- Hello, welcome everybody.
I'm Ingmar Poese and I'll be talking about
our CoNEXT-19 paper, Steering Hypergiants, Traffic
at Scale.
This is joined work together with Enric Pujol,
Johanna Stavros, George Smaragdakis and Anja Feldman.
And with that, I'm simply going to start.
So when we're talking about hypergiants,
the first thing we need to know
is what actually is a hypergiant.
Hypergiants are, there are multiple definitions
for it,
but the ones that we have chosen is
they are large networks that provide services
mostly to end users.
They run global distributed infrastructures
and they generate enormous amounts of traffic,
which means they are the ones that are dominating
the traffic on the internet.
When we're talking about traffic,
this is a statistic from one of the networks
that we were fortunate enough to have information
about.
The blue line is the traffic share
of the top 10 hypergiants that we have tracked
over the last, now in this case, two years.
You can see that it's sort of steadily between 70
and 80%.
And at the same time, in the background,
the gray area is the traffic increase
normalized to the start of the period.
You can see that over the course of the two years,
we have quite the traffic increase.
Now, the ISP that we're looking at
has about 50 million customers,
generates about 50 petabytes of data daily
and runs more than 10 large pods.
The annual traffic growth is around 30% per year
and the top 10 hypergiants have a traffic share
of more than 75% over this period of time.
But also, we wanna steer traffic.
And here's a very, very simple example
what we mean when it comes to steering traffic.
Let's assume we have our little network here.
We have our hypergiant clusters called HG
clusters.
In this case, there is three interconnection
points.
Whether these clusters are directly connected
or come through a third party
doesn't really matter at this point
since we're focusing on the ISP here itself.
And we're assuming that traffic is being
delivered
by the middle cluster at the moment,
which means we have two bytes per byte
generated from the cluster
or we're running over two links,
which means we need twice the capacity to be
installed
as what the cluster is pushing out.
Now, if we switch over to the cluster on the left,
that would be a bad mapping
since we have even more capacity that we're
burning
in order to get to the sort of smiling users over
there.
And a good mapping would be
if we go to the cluster on the right,
which would only be one internal hop.
And in this extreme scenario,
would be a 50% reduction on the traffic in the
backbone.
Keep in mind, this is just an example,
but that's basically what it's all about,
trying to minimize distance or the hops inside
the network
while also keeping everybody else happy.
Now, this, some of you may say this sounds
familiar
and you've seen this before.
And yes, actually, we have already done this
research
about 10 years ago.
Back then, we introduced the system PADIS.
And today, we're actually talking about the same
thing.
It's, as you can see,
there's also a fair amount of the same people in
this.
So the question is, what's new?
What has changed between 10 years ago and now?
Why are we revisiting this topic?
Well, first of all, the mapping problem
that we talked about 10 years ago is still valid
and it is still an important issue
and it can still cause a lot of headaches
and cause a lot of work.
Second thing is we initially proposed
the PADIS infrastructure.
And we here show the differences
from having a research project,
this called, in this case, PADIS,
go into a fully-fledged production environment
called the flow director
and what needed to be changed to the initial
system,
what we didn't think about,
what needed to be added in order for this to
actually work.
And thirdly, the flow director is actually
operational.
We have more, we have two years,
well, by now, almost three years
of operational experience of steering hypergiant.
So we also wanna report on how this actually
works
in the real world.
First of all, we're going to go into the
user-to-server mapping issue a little bit further.
So what does that actually mean?
This is over the past two years,
we have looked at the hypergiant traffic
and we have looked at the optimal mapping
and how this evolves over time.
This basically takes the traffic
that the hypergiant produce
and checks whether it matches
the current network configuration
to see if the mapping was done optimally
in terms of the mapping function that was chosen.
In this case, the mapping function that we chose
is a combination of hops and distance.
Hops, in order to reduce the load on the ISP,
distance in order to reduce the latency
as shorter links tend to have lower TTL.
So as you can see over time here,
we're steadily around 70 to 60% of optimally
mapped traffic.
So there is a huge potential of actually
optimizing things.
Also keep in mind that this does not tell you
how badly mismatched this is.
This is just a binary decision, is it mismatched
or not?
So the potential for actually reducing or
optimizing traffic
is possibly greater than just this.
And what you can also see,
there is sort of a slow but steady negative trend,
which means as the infrastructures get more
complex,
mapping is getting harder and harder.
Now we're going to go into the individual hypergiants
that we tracked.
Like I said, we tracked 10 hypergiants
over the period of two years.
And we're just going to focus on hypergiant six
here,
which used to run one single interconnect
with the ISP in question that we got the
information from.
They then decided to open a second point
and that reduced the mapping compliance to 50%,
which means this was basically a round robin
and 50% got it right, 50% got it wrong.
So when you actually do the peering at a new
location,
it needs to be communicated how the mapping
actually works.
And if you do not have a mapping system in place
as a hypergiant, you might want to think about
getting one,
otherwise this can happen.
Secondly, there are those hypergiants
that have no direct incentive
to actually do the mapping right.
They just go round robin all the time.
And, or basically they just don't care
or don't have any incentive to do this.
And thirdly, there are those that do try in here,
but they still sort of at the,
none of them ever gets really over the 80%.
So why is it so hard to actually get the mapping
right
if you have more than one ingress point
or more than one interconnection point with an ISP?
It doesn't feel like it,
but this is actually a fairly hard problem.
So let me go into, not detail,
but a few things that have to be taken into
consideration
when doing the mapping decision.
First of all, and I'm gonna start at the top left
here,
we have to think about how many number of POPs
am I actually in.
So how many different mapping,
how many different positions am I ingressing in?
That changes over time, as you can see in the
graph.
More POPs get connected, mapping gets harder,
changes over time.
Next is top right.
The capacity changes.
So just because the configuration was good in a
month
doesn't mean it is good in the next month.
In this case, we're talking about the capacity
of the hypergiant that increases.
So this capacity has to be installed at the right
place
or has to be mapped from the right place.
This is also something that needs to be taken
care of.
Next is the middle left.
This is internal routing changes.
So this basically shows when internal routing
changes occur
or how often internal routing changes
actually affect the mapping decision
or the mapping of hypergiant inside the backbone.
Next is the middle right.
This is the number of events on the internal
backbone
in terms of routing that will give you the,
that will affect the hypergiant
or the number of hypergiants are affected by.
So again, internal routing changes within
that have an effect on the mapping.
Last, I will go into the bottom left,
which is the churn in the IP space.
So reassignment of IP space inside the ISP.
And the same thing as distribution on the top,
on the bottom right,
where you can see what the probabilities are
that within a certain amount of time,
IP address space will be reassigned inside the ISP.
All of these issues that you've seen here
are interesting and hard to get from the outside.
So there needs to be information that is provided
from the end user, from the ISP towards the CDN
or hypergiant in this case.
Of course, there are also unknown factors,
server loads, maintenance, content availability,
cross traffic, and I can only refer you to the
paper
at this point as I am already running out of time
to if you're interested more why mapping is an
hard issue.
But collaboration can clear this lack of
visibility.
So what have we done here?
The flow director is in a nutshell,
it collects data from the internal state of the
ISP.
It builds an inventory and map of the forwarding
and control plane of the internet.
And it regularly does inventory checks.
It computes the best mapping and compiles that
into an easily accessible database.
And it communicates this to the hypergiant
that wants to collaborate in near real time.
And by near real time, I mean, maximum 60 seconds
via multiple protocols.
Alto is one of them, but we also support other
things
like out of band BGP, email, XML, JSON,
and so on and so forth.
A little bit of the history on this.
It took us actually about 10 years in research,
about seven years with the company
to get to the point where we are now
between the first evaluation
and the actually going live in five years.
And as you can see on this timeline,
there is a lot of things in here
that you wouldn't actually be thinking
of needing to implement or needing to do
something
about when you're doing something like this.
All components are adhering to RFCs.
And we also built this horizontally scalable
so we can pretty much take on any network
that we connect to.
Currently, we do run this in a fairly large
environment
just so you get a sort of feeling.
We see over one gigabit per second of net flow
at a low sampling rate.
And we hold about 600 BGP sessions
and we have about a 60 second reaction time.
The architecture itself is split into three parts.
We have the southbound that does all the data
collection
and aggregation.
We have the core engine which does all the model
building
and calculations.
And we have the northbound which does all the
communication
towards the CDN so that the path ranker,
so the mapping decision can actually be
communicated.
Again, due to time restrictions,
I can only refer you to the paper at this point
if you want to know more about this.
Now, let's go into the operational experience
here.
First of all, we have a collaboration
with one hypergiant for this ISP.
So this, at this point, was a one-on-one
relationship.
This hypergiant had more than 10% of the total
traffic
inside the ISP's network.
There were two KPIs that we wanted to adhere to.
For the ISP, we wanted to reduce the long haul
traffic.
For the hypergiant, we wanted to reduce the
latency,
i.e. bring the server closer to the user,
which boiled down into a mapping function
or path ranker function that is a combination
of the path length and the distance.
Also keep in mind that the flow director,
when it gives the path ranking, is a suggestion
and it can be ignored by the hypergiant.
So we do not enforce this.
We put this really on collaboration.
And in the trial run, we actually did this
as a progressive rollout.
So we didn't turn it on all the way,
but we did this in a progressive manner.
Now, first of all, I would like to talk a little
bit
about the benefits for the ISP.
What you see here is over the time
from mid-2017 to mid-2019,
the traffic of the hypergiant
on the backbone, and what you can also see
is the traffic on the long haul links.
There is a steady downward trend.
We've put this into five phases,
denoted by the colors that you can see here.
The most interesting bit is the hold phase in the
middle.
That was a sort of misconfiguration in the system
where the mapping was actually reset to random.
And you can nicely see that when this happens,
that the traffic, especially on the long haul
links,
increases quite drastically.
But then with re-enabling the system, it all went
down.
And in the end, we have quite a significant
reduction,
especially on the long haul links,
which are the expensive ones for this ISP.
At the same time, we can also look at the overall
traffic
produced by this hypergiant inside the ISP's
network.
And also what you can see here
is that we reduce the overall traffic while not,
so this is important,
we do not move away any traffic from the end
users.
So the traffic still increases throughout the
time,
but the traffic on the long haul links gets
reduced.
So which means we localize the traffic more,
we take traffic load off of the long haul links
and use the infrastructure of the hypergiant
more efficiently, mapping the users closer
to the points where the service actually are.
That's the benefits for the ISP.
Next is the benefits for the hypergiants.
And as I said before,
we use the distance as a proxy for latency.
And in terms of distance, we reduced the gap,
so the distance between the server and the client
or the end user by about 40%,
which means we heavily localized traffic.
Again, you can see here that in the whole phase,
this goes up really, really high as the mapping
got broken.
Lastly, we did a what-if analysis
on what could be achieved.
So what you basically see here is the potential
for the 10 hypergiants that we identified
in the ISPs network.
And if all of them would actually go onto perfect
mapping,
so follow the flow director 100%,
there is a huge potential for reducing backbone
traffic
and optimizing the network.
Now, there is one point in here.
We do not assume new server locations.
New server locations can have an even greater
effect.
So there will still always be some long haul
traffic left.
The only thing we assume is that those,
that the infrastructure as it existed
at this in this point of time with the routing
or the network configuration as it existed,
we calculated the potential what would happen
if we remapped according to the flow director.
Across the board, if you take all of this
together,
we see a potential of about 20% overall
long haul traffic reduction for the ISP.
And with that, I am at the end of this talk.
So I just wanna reiterate these points here.
There is an opportunity to operate networks
more efficiently through collaboration.
We've just shown you how this collaboration works.
This is a system that is actually deployed
and it is fully automated.
And we have been running it for multiple years
now.
Nevertheless, there's always a lot of engineering
and a lot of diplomacy involved.
So there's a lot of layer one through nine in
there,
although layer one and two are usually not that
important.
And mostly it works not only for the ISP,
not only for the hypergiant,
but it works for both at the same time.
We're also interested in going into further
directions here.
First of all, we wanna see what happens
if we use different optimization functions,
especially for different types of content
with different optimization functions
from the same hypergiant,
but also optimizing multiple hypergiants
at the same, in the same network.
And secondly, we would like to do a federated
flow director,
which means we do a multi-ISP hypergiant
collaboration.
And with that, I am at the end of this talk.
Thank you for your attention.
And I think I will take questions now,
but I'm not quite sure.