Hi everyone, my name is Niemann Bauer. I'm a
research associate at the Technical University of
Munich
here at the Chair of Network Architectures and
Services.
And today I want to talk about our work on the
evolution of Internet flow characteristics.
As we all know, there's ongoing change on the
Internet regarding new technologies
adapting user behavior or changing Internet
services.
We are all aware of improving network expansions.
We all observed the smartphone boom during the
last decade.
We are aware of the Internet of Things that rises
and of course we know about audio and video
streaming services.
At the same time, previous studies present methodologies
to survey flow characteristics
like flow duration, flow size or flow rates.
But recent insights into flow characteristics in
the Internet are rare.
And therefore our paper pursues the question
how flow characteristics changed during the last
few years.
To answer such question, our paper surveys the
distribution and correlation of flow
characteristics
and applies different taxonomies to assess the
relevance of heavy hitter flows
in the sense of elephant flows but also in the
sense of so-called big fast flows.
Before we start talking about our methodology and
our measurement results
let me briefly introduce a scalable flow analysis
tool implemented in Go
that provides large scalability due to parallelized
packet parsing and flow aggregation.
The tool is published as free and open source
with our paper.
The tool consists of five major components which
are illustrated on the right.
The first component is a reader that reads
packets from a PCAP file.
We have multiple parsers that extract packet
features from such read packets.
Those packet features are then written into a
ring buffer component that is proposed to reorder
packets.
A regular routine then writes such reordered
packet information into pools
where we collect package features per flow until
a flow terminates.
After its termination, flow data is written to
the metric component
where we then calculate our flow characteristics.
Let's talk about flows.
We identify flows based on the IP5 tuple as we
will focus on TCP traffic for our study.
We identify the start of TCP flows with a three-way
handshake.
We terminate a TCP flow when we observe a
connection teardown.
If there is an idle period of a flow for a
certain timeout period
we observe a freshly established three-way handshake
of an IP5 tuple that is already tracked.
For identified flows, we calculate the flow size
in the sense of the sum of layer 4 payload sizes.
We calculate flow duration as the time interval
between the first and the last packet we observe.
We calculate flow rate as the average data rate
based on flow size and flow duration.
For our study, we compose a dataset consisting of
28 traces provided by KIDR.
Such traces have anonymized IP addresses and no
layer 4 payloads.
Each trace provides one hour of traffic that is
captured at a 10 gigabit per second ISP backbone
link.
As illustrated below on the timeline, we select
23 traces taken in Chicago between 2008 and 2016
and 5 traces taken in New York between 2018 and
2020.
As you see, we have three periods without traces
for several months.
On average, we select traces in an interval of
three months.
But there are three larger intervals without
traces because there are simply no traces
available.
Regarding pre-processing of such traffic,
we only consider TCP flows that are longer than
or equal to 100 milliseconds.
This is also done by related work and proposed to
filter out quite short flows
because calculated flow rates are falsified for
short flows in case of single packet flows or if
all packets are sent back-to-back.
Let's talk about selected measurement results.
Regarding the evolution of heavy hitters,
related work proposes a definition of heavy hitter
traffic based on the 99th percentile of a
characteristic.
The 99th percentile of flow duration, flow size,
and flow rate.
We analyzed the evolution of such 99th percentiles
over time.
As you can see plotted here on the right, let me
point out two major findings.
Regarding the 99th percentile of flow duration,
here on top of the plot,
we observe that there is only little increase
during the years 2008 until 2013.
But afterwards, we observe an increase by factor
1.5 between June 2013 and March 2016.
A further major finding of that analysis is the
increase of the 99th percentile of flow rates,
which refers to the bottom plot here.
Here we observe a clear trend towards larger flow
rates in the 99th percentile,
i.e. an increase from around 400 kbps up to 800 kbps
in 2015.
Next, we were interested in the relevance of such
heavy hitter traffic.
Therefore, we calculated the share of transmitted
bytes by such flows within the 99th percentile
for each flow characteristic.
We did not find a specific trend over time, so
here the table shows the average across all
traces taken in Chicago.
In the second column, in the right column, we see
the share of bytes transmitted by different
percentiles.
Especially the flows within the 99th percentile
of flow size represent a large share of totally
transmitted bytes
with nearly 90% of all TCP bytes transmitted by
such 1% of flows.
Further, we had a look at so-called big fast
flows.
Zhang et al. introduced a 2-2 taxonomy based on
two threshold values to group flows regarding
their size and their flow rate.
We had a closer look at the relevance of such big
fast flows, which are represented by only a small
share of flows,
but as we will see, have a large relevance
regarding the share of bytes that they transmit.
We defined three threshold pairs.
The first pair refers to the original threshold
values from Zhang et al., i.e. 100 kbps regarding
size and 10 kbps for flow rate.
Then we increased the thresholds by one magnitude
for pair 2 and pair 3.
Let me highlight the increase of the share of
bytes transmitted by the second threshold pair
illustrated in green.
Here we observe an increase of bytes transmitted
by big fast flows between 20 and 30% for traces
in 2018 until 2010,
up to between 40 and 50% of all bytes transmitted
by TCP.
For more recent traces taken in Chicago, the
values for the New York dataset are smaller,
which can be traced back to a larger share of
small flows for such traces.
To conclude my talk, let me summarize our
findings.
As we have seen here in the talk, we observe a
significant increase of the 99th percentile of
flow duration and rate.
We find a large significance of heavy hitters
regarding the share of transmitted bytes.
We observe an increase regarding the relevance of
big fast flows during the past years.
Further, and not included in the talk, but in our
paper, you will find an analysis of the
distribution of flow characteristics over time
and a study of correlation coefficients between
the flow characteristics,
where we can confirm findings of previous studies,
especially regarding a strong positive
correlation between flow rate and flow size.
That's it from my side. Now I'm happy and looking
forward to answer your questions.