Hello everyone, my name is Sadjad Fouladi and I'm
a PhD student at Stanford University.
Today I'm very excited to tell you about Salsify,
a new architecture that we built for real-time
video systems.
So real-time video is a category of internet
video delivery systems, and in this setting we
have a sender that has a camera and it wants to
transmit the video feed over the internet to a
receiver.
What makes real-time video special is its latency
target. We want the video to be delivered as fast
as possible.
So if something happens at the sender side, we
want to see the video of that at the receiver
side almost immediately.
And we care about this latency so much because
usually there is an interaction going on in the
opposite direction that relies on the timely
delivery of that video from the sender to the
receiver.
One good example is called video gaming, where
the game instead of on the local machine is
actually running on a remote server and the
player sees a video stream of that video that is
delivered over the internet.
Another example is the collaboration of robots or
remote surgery, where the surgeon is relying on a
video feed to do the surgery from a remote
location.
And of course, the most prominent application of
real-time video system, video conferencing,
something that has been a big part of our lives
over the past year.
So the problem that we tackled in Solstice was
how these systems react to variations in the
network.
So here I'm going to show you one example. We are
doing a video call with this very nice guy at
NASA using one of these real-time video systems,
WebRTC.
And here you can see the network throughput graph.
That shaded region shows the network capacity and
the line shows the amount of data that the
program tries to send over the network.
In this scenario, the capacity is slowly going to
go down. It reaches zero for a second and then it
will go back up again.
Let's see how WebRTC reacts.
Let's look at the video again. The capacity is
slowly going down and it reaches zero and the
video freezes, understandably.
But now the network is back, but it takes WebRTC
several seconds to recover the video.
So if we look at this part of the graph, you can
see WebRTC actually tried to send more than the
network can handle.
You know, those packets are either going to be
dropped or queued. Both will cause stalls and
glitches.
And then it had to spend several seconds to
recover from the mistake that it made earlier.
You know, a one second outage here caused WebRTC
to spend several seconds recovering the video.
So in Salsophy, we explored a different design.
In Salsophy, we are tightly integrating a
transport protocol and a video codec that allows
the system to respond quickly to the changes that
happen in the network condition.
Okay, before I get to Salsophy's architecture,
let me tell you a little bit about the
conventional design.
So the current systems, they have two important
components. One is a video codec responsible for
compressing the video,
and one is the transport protocol responsible for
sending that video over the network.
And these two components talk to each other
through a very narrow interface.
You know, the transport protocol has some notion
of the network capacity, like a target bit rate,
and it communicates that information to the video
codec occasionally, like every second or every
two seconds.
And the video codec adjusts its internal
parameters, tries to target the bit rate that it
received from the transport protocol by setting
things like quality and frame rate.
And it produces frames that the transport
protocol has to send over the network to the
receiver side.
And you know, there has been decades of research
and development on these components.
We have all these video codecs, you know, that
offer better and better compression ratios.
And we have all these different transport
protocols built for different targets and
different types of networks.
And every time that we want to go and build a
better real-time video system, we go and improve
these individual components.
Let me give you one example. Researchers actually
knew about this problem that I showed you in the
beginning.
They studied Skype and saw that Skype, in the
face of network variability, doesn't really do
well.
Here, Skype tried to send more data than the
network can handle, similar to WebRTC, and it
caused a very huge spike in delay.
So they said, "Okay, it seems that Skype has a
problem predicting the variable network capacity."
So they built a better transport protocol that
accurately can match these variable network
capacities.
Great. But unfortunately, just improving the
network components didn't save the day.
Because we totally forgot about the video codec.
You know, and the video codec has a very big flaw.
And that's the fact that the codec can only
achieve that bitrate on average.
You know, like here, we asked the VPX encoder to
target 2 megabits per second.
And it produced a video that on average is at 2
megabits per second.
But when you look at the size of individual
frames, they are all over the place.
And even a frame like this that is over the
network capacity can still cause packet loss and
congestion in the network.
So the problem is that we have a codec that can
only respond to changes in the target bitrate
over coarse time intervals.
So even if we have a transport protocol that can
have a very accurate estimate at any point in
time,
we don't have a codec that can respond to that
high resolution prediction from the transport.
So in Salsophy, we explored a more tightly
integrated design where the transport protocol
and the video codec work together within the same
control.
So I'm going to start by telling you about the
transport protocol in Salsophy.
The transport protocol in Salsophy only answers
to one question.
What should be the size of the next frame?
So there is no notion of average bitrate or
network capacity here.
The transport tells me if you want to hit your
target latency, your next frame should not be
over a certain size,
like 10 kilobytes or 50 kilobytes, based on the
network conditions.
So now the codec, the video codec, might be able
to use that information.
But as I told you, the codec has trouble hitting
a certain size for a single frame.
It tends to overshoot or undershoot that target.
You know, the transport tells me the network can
handle a frame of maximum 10 kilobytes right now,
but the codec may produce a frame that is larger
or smaller than 10 kilobytes.
So what can we do here? One solution might be
just doing trial and error.
You know, we encode the single frame at different
quality levels and just pick the one that fits
the network.
Sounds like a good idea, but unfortunately with
the current codecs, it doesn't work.
So when we look at the video encoder, you know,
the program responsible for compressing the video,
it receives frames and produces the compressed
bit stream.
When we look inside that black box, it's actually
stateful.
So when you encode a frame, the encoder goes
through a state transition.
And that frame becomes a part of the video stream.
And now it has to be sent over the network and
has to be received by the receiver because the
future frames are going to be dependent on the
state that is produced by that frame.
And when we look at the interface that we get
from the current video codecs, we see that there
is no way to undo an encoded frame.
The state is completely internal to this encode
function and there is no way to save or restore
that state.
So in Salsify, we built a functional video codec
for the VP8 format that makes that state object
explicit.
Now, if you encode a frame and you don't like the
result, like if it's too big, you can just
discard that, go back to the previous state and
encode a new version.
So using this, Salsify can explore different
execution paths without committing to them.
So for every frame in the video, Salsify produces
a version with a slightly higher quality than the
previous one, one with a slightly lower quality
and thus smaller size.
And it also gives the transport the option to
discard the frame.
If the transport doesn't like that frame, the
codec can silently just go back to the previous
state.
So this is how these components come together in
a single control.
The transport has some estimate for what the
network can handle right now.
It says the network can handle 30 kilobytes.
Great.
And the codec gives transports two options, one
with a slightly better quality, one with a
slightly worse quality than the previous frame.
And the transport picks the one that doesn't go
above that estimate.
And it tells the codec, this is the option that I
picked.
And the codec will base the next frame based on
that existing state.
Now the transport repeats the same thing.
And sometimes the situation is that none of the
options fit the network.
Like here, the target is 5 kilobytes and the
options are 70 kilobytes and 50 kilobytes.
So the transport tells the codec, just discard
those frames and base the next frame on that
existing state.
So the frames are discarded and now the codec
moves on to the next frame and the cycle
continues.
So as you can see, there is no notion of frame
rate.
We are not committed to sending like 30 or 40 or
60 frames per second.
We only send the frames over the network when we
know the network can handle them and they can be
received in a timely manner at the receiver side.
OK, let's take a look at the results.
And before I go any further, let me show you a
comparison between Salsify and WebRTC in the
video that I showed you in the beginning of this
talk.
So same situation for both. The capacity is going
to go down and then go back up again.
That's how Salsify reacts. So on the left side,
you see Salsify. On the right side, you see WebRTC.
So the network is out. And now it's back.
And you see Salsify has quickly recovered.
And if you look at the throughput graph in Salsify,
you can see that it gracefully tries to match the
network capacity while WebRTC has trouble keeping
up.
In the second demo, I'm going to show you how Salsify
reacts to occasional network outages.
So at different points in time, the network is
going to be out for one second.
Let's see how WebRTC and Salsify react in this
situation.
Left side, Salsify. Right side, WebRTC.
The network is out. Salsify has recovered.
And now WebRTC.
Again, the network is out. Salsify has recovered.
And now WebRTC.
Network is out. Salsify has recovered.
And again, it takes WebRTC several seconds to
recover from that one second outage.
And when we look at the network throughput graph,
we see that WebRTC basically disregarded whatever
was happening in the network and just kept
sending data at the old rate.
In order to evaluate Salsify, we built a
measurement testbed that is capable of basically
testing any real-time video system as a black box
without requiring any modification to that.
You can read all about this measurement testbed
in our paper.
Just one thing I want to mention is that this
measurement testbed uses a barcoded input video.
Each frame in this input video has a unique barcode,
and using this barcode, we can actually match
frames at the sender and the receiver side, and
we can see what was the change in the frame
quality, and how long did the frame take to reach
from the sender side to the receiver side.
So we are capable of measuring delay and quality
on a per-frame basis here.
Using this measurement testbed, we compare Salsify
performance to Skype, WebRTC, FaceTime, and Hangouts.
Here on the y-axis, you can see the video quality,
and on the x-axis, you can see video delay.
So the x-axis is flipped, better is up and to the
right.
We wanted to see how the ideas that we had in Salsify
actually contributed to the final results.
So we started by implementing a real-time video
system that works very similar to the
conventional design, and it ended up kind of
close to the other systems.
Then, in the conventional design, we went and
just replaced the transport protocol with this
video-aware transport protocol that tries to take
control of the video on a per-frame basis.
We still have a conventional codec, it has to
come up with the right parameters up front, but
now the transport tries to set a size for each
frame.
As you can see, the delay was improved, but the
quality, it actually dropped a little bit.
But now, in the full version of Salsify, when we
put together all these components, you know, the
video-aware transport protocol and the functional
codec, that's when we can achieve better delay
and better quality than the other systems.
So these are the results on Verizon LTE Trace.
Here we have the results on AT&T LTE Trace, where
again, Salsify achieves better delay and better
quality than other systems.
And here we have the T-Mobile UMTS Trace, which
is a very troubled link.
You can see delays up to 18 seconds.
And again, Salsify achieves lower delay and
higher quality compared to other systems.
And of course, there are situations where Salsify
doesn't really offer an advantage over other
systems.
One case is that when you have no variations in
your network.
Here you can see the results when we ran Salsify
on a network with constant capacity and very low
rate of packet loss.
Let's just take a step back and see what happens.
So the individual components that we have in Salsify,
they're not exactly new.
Like the transport protocol is a very dumbed down
version of PacketBear and Sprout.
The video format that we are using is like 13
years old.
And the functional video codec that I told you
about was built for a different purpose, you know,
and because it was built by one person in three
months,
its compression efficiency and speed is way lower
than commercial codecs.
What's new in Salsify is the architecture, the
way that we put these components together,
that allows the system to jointly control the
codec and the transport and respond faster to the
variations in the network.
We believe that in the context of real time video
systems, you know, going and making better video
codecs probably won't give us that much.
But if we go and start making changes to the
overall architecture of these video systems, we
can still yield significant benefits.
So Salsify is a new architecture for real time
internet video.
It tightly integrates a video over transport
protocol with a functional video codec,
allowing the system to respond quickly to the
changes in the network conditions.
Salsify achieves lower delay and higher quality
when compared to FaceTime, Hangouts, Skype, and
WebRTC.
Thank you.