Hello everyone, I am Bano and I am excited to
present our work Scanning the Internet
for Liveness, which was carried out in
collaboration with Philip, Mupin, Srikant, Zakir,
Stephen,
Richard and Wern.
Internet scanning is a key technique to measure
the Internet at scale, and it has diverse
applications such as address-based utilization,
host reachability, topology, service availability,
security vulnerabilities, and application layer
discrimination.
The key concept of IPLiveness is to check whether
a target IP address responds to a
probe packet.
Now this seems pretty straightforward, but
actually it's not.
There are a few questions for which we don't have
a good answer.
For example, what type of probe packets should we
send if we want to maximize the responding
host population?
What type of responses can we expect and which
factors determine such responses?
What degree of consistency can we expect when
probing the same host with different probe
packets?
What we are essentially missing here is a
systematic framework that allows us to understand
IPLiveness and how it manifests in the form of
host replies to active probing.
Now this depends on multiple factors, such as how
the scan was conducted, how different
protocols interact, filtering policies near the
target IP address, and temporal patterns
in IP responsiveness.
Our work makes three key contributions to fill
this gap.
The first is a taxonomy of liveness.
The second is a methodology for systematically
inferring IPLiveness by performing Internet-wide
scans concurrently across a set of different
protocols at various layers, i.e. ICMP, TCP,
and UDP.
And then we perform an analysis of gathered data
to present an in-depth view of liveness.
So in the rest of this presentation, we will dive
deeper into each of these contributions.
We will begin with the taxonomy, focusing on each
layer at a time.
So we begin with the network layer.
When we send a probe packet, we first check if
there is a reply.
If there is no reply, then we say that the target
IP address is dark at the network layer.
So in this example, IP1 is dark at the network
layer because it does not send any response
to our TCP SYN on port 80.
If there is a reply, then we check what is the IP
address from which the reply came.
If the reply came from a different IP address
than the one that we probed, then we flag
the responding IP address as network layer alive
and we label the target IP address as
dark at the network layer.
So in this example, we send TCP SYN on port 80 to
IP1, but we receive an ICMP error from
some other IP address, IP2, which we label as
network layer alive, but the original target
IP address that was probed, because it did not
respond, we label it as network layer
dark.
And if we do receive a reply from the probed IP
address, then we say that it is alive at
the network layer.
So moving on to the transport layer, which is
only relevant to TCP because as we know
UDP is one short communication protocol where the
application layer payload is part of the
original probe and there's no transport layer as
such.
So for TCP, we check if there is a TCP reply, if
the reply is a TCP reset, we say that the
target IP address is active at the transport
layer.
If the response is a TCP SYNAC, we say that the
target IP address is alive at the transport
layer.
Moving on to the application layer, which applies
to both TCP and UDP, we check if there
is an application layer reply.
If there is one, then we say that the target IP
address is active at the application layer.
Next we check if the reply is well formed in the
sense that it complies with the expected
syntax of the protocol that was scanned.
And if it does, then we say that the target IP
address is alive at the application layer.
So this was the taxonomy.
Next we shift our focus to the methodology that
we use to perform our scans.
We carried out eight concurrent scans, one ICMP
echo scan, five TCP SYN scans on ports
corresponding to SSH, Telnet, HTTP, HTTPS, and CWMP.
And this choice was made on the basis of
selecting protocols that represent at least
partially
service level protocols as well as client-side
protocols.
And then we also scanned to UDP-based
applications, that is DNS and NTP.
All of our scans were conducted using ZMap and we
performed the data analysis using Cilk.
We employed a few considerations to ensure the
soundness of our measurements.
The first one of which relates to temporal churn
due to IP addresses legitimately coming
online and offline or because of dynamic IP
assignment practices.
So to mitigate the effects of temporal churn, we
carried out scans for all the eight protocols
that were mentioned on the previous slide
simultaneously.
Next to ensure that we capture a comprehensive
view of liveness, we recorded negative responses
in addition to positive responses.
And finally, because we carried out eight
concurrent scans on shared hardware, the
probability
of packet loss is high, especially close to the
host and also due to transient network
issues.
So to account for that, we sent redundant probes.
However, because network losses usually take
place in bursts, we used an approach which
we call delayed retransmissions, which means that
instead of sending the original probe
and the retransmission to a target IP address
back to back, we instead scanned an entire
block.
So in this example, we have IP block one.
And once we finished with that block, say in 30
minutes, then we sent retransmissions
to the same block before moving on to the next
block, which is IP block two in this
example.
And then we sent retransmissions to IP block two
and so forth.
So we did a block by block scanning, sending
original probes to a block followed by retransmissions
and so forth.
So in summary, we carried out eight concurrent
scans, which finished in about 24 hours,
generating
about 2.3 terabytes of data.
And overall, our scans recorded 487 million
network alive IP addresses out of the 3.6
billion IP addresses that were probed, which we
call IP all.
In the remaining slides, I will present main
results from our study in the context of IP
all.
So finally, presenting the interesting part of
this presentation, that is key results
from characterization of IP liveness.
Let's start with the network layer.
So the first question we ask is that what is the
coverage of different probe types?
So here's the breakdown of IP all covered by
different probe types, that is ICMP, TCP
scans, UDP scans, and then union corresponds to
the union of these three probe types.
We find that ICMP echo probes are most effective
in discovering network active IP addresses,
capturing 79% of IP all, followed by TCP probes.
But interestingly, 16% of IP all can only
exclusively be discovered through TCP, and
2% can only be discovered through UDP probes,
which implies that studies that infer liveness
only by means of ICMP probes will miss out on
about 16% of IP all, which can only be
viewed through TCP and UDP.
Here is the same graph, except instead of looking
at it through the lens of individual
IP addresses, we look at it at the granularity of
/24 blocks.
And as we can see, the variation in liveness
viewed through different probe types is much
less pronounced.
But overall, TCP scans show the highest coverage
discovering about 5 million active /24 blocks,
about 3% more than ICMP echo.
Next we look at the coverage of different probe
responses.
So here we break down our previous graph by
response type.
So here is ICMP scan, and here is the coverage
that we get when we capture different response
types.
So here is echo reply, ICMP error, both, and here
TCP reply, ICMP error, and both,
and for UDP, UDP reply, ICMP error, and both.
We find that 2.3% of IP all are discoverable only
through ICMP error responses, which means
that studies that infer IP liveness through only
successful responses will miss 2.3% of
IP all if they do not capture error responses or
negative responses.
Next we look at key results from our
characterization of IP liveness by transport
layer.
We ask how does the probed port affect the
responsive population.
Now this wouldn't be a relevant question if an IP
address responded to all ports consistently,
but we found that's not the case.
Only 24% of active hosts respond to probe packets
on all 5 ports, which highlights the
potential effect of firewalling or filtering
practices on IP liveness as viewed through
different probes on different ports.
Next we look at what is the coverage by probe
response type.
And we find that 11.5% of all TCP activity can
exclusively be found through CWMP.
In comparison, SSH, HTTP, and HTTPS provide
unique coverage of 3-6% of active IPs.
Now finally let's take a look at a cross-protocol
view of IP liveness.
So this map shows that given that IP is alive or
active for a protocol along the X-axis,
what is the likelihood that it would also be
alive or active for a protocol along the
Y-axis?
So for example, if we focus on this box, what
this means is that if or given that an IP
is active for SSH, the probability that it is
also active for HTTP is 0.68.
So some of the highlights from this map are that
a significant fraction of transport active
hosts, that is 26% on average for TCP services,
and 12% for UDP cannot be discovered through
ICMP.
Another observation is that if a given host is
active for 10-net, then with high probability
it is active for SSH and HTTPS.
And for CWMP, only 56% of active hosts respond to
HTTP probes, indicating an underlying filtering
pattern of the CWMP active population.
So to conclude, this work presented a methodology
to capture a comprehensive and minimally noisy
picture of the state of Internet liveness.
And this taxonomy can serve as a basis for
designing and executing future measurement
studies when it comes to decisions such as what
type of probe packets should be employed,
what type of responses should be captured, how
those responses should be interpreted,
and whether it is appropriate to use the output
of one scan as input for subsequent measurements.
On that note, I conclude, and I look forward to
your questions.