Hi, this is Aqsa. I'm a PhD student at CMU and I'm
going to present my work on analyzing
third-party dependencies in modern web services.
This is a collaborative work with my advisors,
Pyaar Sekar and Yuvraj Agarwal. The study is
motivated by a DDoS attack that happened
in 2016 on a DNS provider named Dyne. There are a
lot of lessons to be learned from this
attack, particularly its root cause, and as I
show in this talk, we can expect more Dynelike
attacks in the future. During this attack in 2016,
a lot of popular websites such as
GitHub, Spotify, and Twitter were inaccessible to
users for many hours. Here is an outage
map of the attack. In total, 178,000 domains went
offline and tens of millions of users
were affected. The list of websites that went
down goes on. It includes almost all the popular
services we know. So how were the attackers able
to bring all of these down? This was
possible because all these websites, such as
Netflix, exclusively relied on Dyne, a
third-party DNS provider that acted as their
authoritative name server. As a result, when
the authoritative name server was down, a lot of
websites were inaccessible to millions
of users. Attacks like these raise many questions
about the resilience of the internet. How
easy is it to bring down the internet?
Particularly, we need to understand how prevalent
these
kind of third-party dependencies are. For example,
how many other websites are at risk
of Dynelike attacks? Are there other hidden
dependencies that we also need to know about?
There might be inter-service dependencies which
indirectly affect a website. And finally,
since Dyne attack had such a huge impact, it is
natural to think that websites would
adapt as a result. How has the world changed in
using third-party services as a consequence
of the Dyne attack? For example, did websites
stop using Dyne or did they start using multiple
providers? To answer these questions, we analyzed
top 100,000 websites, Alexa websites. Alexa
is an external service that ranks websites by
popularity. Alexa rank is calculated from
a combination of daily visitors and page views on
a website. To see how the world changed
in response to the Dyne attack, we look at the
dependencies of top 100,000 websites in
2016 and 2020. In this talk, I'll first describe
our measurement methodology to measure the
third-party dependencies of websites and then
explain our main findings. Then, after the
findings, I will also present some
recommendations in light of our findings and
finally, some
limitations of our work. To start with the
measurement methodology, we first need to
identify the services which are crucial for a
user's access to the website. To do this,
we look at the lifecycle of a web request. When a
user makes a request to a website,
it first goes to the authoritative DNS server,
which provides the IP resolution of the website.
After the IP resolution of the website, user
starts a TCP handshake followed by an SSL
handshake if the website supports HTTPS. If yes,
then the website provides a certificate,
which is verified for revocation by the user by
contacting the OCSP server or CRL distribution
points set up by the certifying authority.
Finally, if the certificate is valid, the
user requests the content of the website page. If
the website is using a CDN, then CDN will
serve this content. Based on this, we identify
three services. The first one is DNS, the
second one is certificate validation by
certificate authority, and the third one is CDN.
Note
that these by no means are the only services that
a web request encounters, but we just
identify these three services and define the
scope of our work. Given these services and
websites, we need to identify what exactly we
want to measure. So we are interested in
third party dependency of a website. For example,
here, the website Techradar uses another entity,
Let's Encrypt, as its certificate authority. Next,
there might be indirect dependency between
websites and service providers. For example, here,
Let's Encrypt uses Cloudflare as its
authoritative name server, and hence Techradar
has an indirect dependency on Cloudflare.
We are also interested in critical dependencies
which exist if a service provider is integral
for a given service. For example, here, if Cloudflare
is down, OCSP servers of Let's
Encrypt will not be accessible. Hence, Let's Encrypt
has a critical dependency on Cloudflare.
In case of DNS and CDN, we compute critical
dependency by measuring if the website is
redundantly provisioned, which means that the
website is using multiple DNS or CDN providers.
And in case of certificate authorities, we
compute it by measuring if the website has
enabled OCSP stapling. Here, Techradar does not
support OCSP stapling, and hence Techradar
has a critical dependency on Let's Encrypt as
well as Cloudflare. If OCSP stapling is
enabled, the user doesn't have to contact the OCSP
server for certificate validation,
so Let's Encrypt will not be contacted here. But
if OCSP stapling is not enabled, then
users of Techradar will have to contact the Let's
Encrypt OCSP servers to validate the
certificate.
Now, given this, we first describe our
measurement methodology for DNS. What we
essentially want
is to find the authoritative name server of each
website. This can easily be done by looking
at the name server's resource record of a website.
Then we want to identify if this
name server is private or a third party. I would
go into details for this in the next
slide that how do we actually identify if it's a
private or a third party. And finally,
for a website and its name server, we want to
identify which of these name servers belong
to the same entity. For instance, here, Azure DNS
and O365 filtering both belong to Microsoft.
So since live.com is using these two which belong
to the same entity, it is actually
not redundantly provisioned.
To identify third party name servers, prior work
looks at second level and top level domains
of the name server and the website. If they match,
then the name server is private, otherwise
third party. For example, here in case of Google,
its name server also has the same
second and top level domain, google.com. Hence,
it gets correctly classified as private. However,
this fails in case of YouTube, where the name
server gets classified as third party when
in fact it is not.
Other approaches look at the start of authority
DNS records, which contain administrative
information about the DNS zone. If the start of
authority record of the website and name
server are the same, then it's private, otherwise
third party. This works in case of YouTube,
but it fails for instances in case of Twitter,
where it is using a third party name server
and its start of authority is also that third
party. For instance, here Twitter is using
dynic.net, which actually belongs to Dyn.
To cope with these issues, we add a few more heuristics.
So for all website and name server
peers, we classify the name server as private if
the second level domain and top level domain
match for the website and the name server, or if
the name server is present in the subject
alternate name list of that website. SAN list is
present on the HTTPS certificate of a website,
which tells that this certificate will also work
for the domains present in the SAN list.
Hence, it's a good way to know if a given domain
is private or not.
If the start of authority do not match, we
classify it as third. And finally, if the
name server serves more than 50 websites, then we
say that it's more likely third party.
In general, we identify approximately 10,000
third party DNS providers in our data. For
measuring CDNs, we fetch the web page of the
website that we want to measure the CDN of,
and then we extract all the resources in the web
page and identify the resources which
are internal. For example, for reddit.com, this
resource that comes from redditmedia.com
is an internal resource. We use TLD matching,
start of authority, and subject alternate
name lists here as well to identify internal and
external resources. We do this because
internal resources, if they come from a CDN, then
it means that the website is actually
using a CDN. For all the internal resources, we
fetch their CNAME records. If the website
is using a CDN, then these CNAMEs will point to
the CDN. For instance, here, the CNAME
actually points to Fastly. We input this CNAME to
our CNAME to CDN map, which will
output Fastly here. We manually build this map by
taking help from available CNAMEs on
the internet, on the CDNs' websites, and also in
public suffix lists. After we have
identified the CDN being used by a website, we
also need to see if the CDN is private
or a third party. To do that, we again use TLD,
start of authority, and sans list to
see if the CDN is private or third party. In
general, we identify a total of 86 third
party CDNs in our data. For certificate authority
dependency, we just fetch the certificate
of a website, and from that, extract the links
for the OCSP servers and the CRL distribution
points. And we use the same techniques, which are
TLD matching, start of authority, non-matching,
and sans list to identify if these are third
party certificate authorities or not. In our
data, we identify a total of 59 third party
certificate authorities. Now we present our
main findings. I will start by answering the
question, how prevalent third party dependencies
are and how concentrated different third party
providers are. Overall, we find that 89% of
the top 100,000 websites use third party DNS, 32%
use a third party CDN, and 76.8% use a
third party certificate authority. Moreover, 85%
critically depend on a third party DNS
certificate, 28% critically depend on a CDN
provider, and 59.5% critically depend on a
certificate authority. For DNS and CDN, it means
that they're using just a single DNS
or CDN provider, and for C, it means that they do
not support OCSP stapling. All in
all, we found that 89% of the top 100,000
websites critically depend on a DNS, CDN,
or CA provider. In addition to this, we also
found that critical dependencies are higher
for less popular websites. For example, we
observed that 49% use a third party DNS provider
in the top 100 websites, and 89% in the top 1,000
websites. Same is the case for critical
dependencies. For instance, 27.9% critically
depend in the top 100, while 84.8% in the
top 100,000. This means that more popular
websites care more about availability. Now,
since third party dependencies are very prevalent,
it raises the question of how concentrated
various providers are. The greater the
concentration, greater will be the single points
of failure.
This is a dependency graph of website to DNS
dependency, where each edge shows the dependency
of a website on a third party DNS provider. The
size of each node here is proportional
to its n-degree, which in the case of DNS
provider is the number of websites using it.
We observed that the top most DNS provider, Cloudflare,
alone serves 23% of the top 100,000
websites. In general, we observed that only top 3
serve almost 40% of the websites, when
the total DNS providers we observed are almost 10,000.
This indicates that DNS space is highly,
highly concentrated. We see similar trends in CDN,
where the top most CDN provider, Amazon
Cloudfront, alone serves 26% of the websites that
use CDN. Particularly, we observed that
only 3 CDNs serve almost 60% of the websites that
use CDN, which are Cloudfront, Akamai,
and Cloudflare. The situation for certificate
validation is
also similar. DigiCert alone serves 28% of the
websites that support HTTPS. Particularly,
only 3 certificate authorities, DigiCert, Let's
Encrypt, and Sectigo, serve 60% of the
HTTPS supporting websites, out of the total 59
certificate authorities we observed.
To summarize, our main findings were that third
party critical dependencies are highly
prevalent and third party services are very
concentrated. A direct implication of this
is that 89% of the websites are vulnerable to Dyn-like
incidents. A single provider can
affect as much as 25% of the top 100,000 websites.
Now the question is, if this is the complete
picture or do we need to take indirect
dependencies into account as well?
And this brings us to our second question which
is, are there any indirect dependencies
between websites and their third party providers?
To do this, we first need to analyze inter-service
dependencies between DNS, CDN, and CA providers.
In case of CA to DNS dependency, we found
that out of the 59 CAs that we observed, 48% of
them use a third party DNS provider. In
case of CA to CDN dependency, 36% used a third
party CDN provider. Moreover, out of the 86
CDNs that we observed, we found that 36% use a
third party DNS provider. Hence, third party
dependencies are also prevalent among service
providers.
In case of critical dependencies, we found that
31% of certificate authorities relied
on a single third party DNS provider, 36% relied
on a third party CDN provider, and
17% CDNs relied on a single third party DNS
provider. These critical dependencies between
service providers give rise to indirect
dependency. But why should we care about indirect
dependencies?
I mean, okay, sure, they exist, but what's the
harm? Turns out that as a result of indirect
dependencies, we see huge amplification in the
impact of a given provider. For example,
in case of CA to DNS dependency, Cloudflare now
critically serves 37% of the top 100,000
certificates as compared to 24% before. And in
case of DNS Made Easy, the impact goes
from 1% to 23%. Hence, because of these indirect
dependencies, we see amplification of provider
concentration. To summarize, critical
dependencies are also
widespread among service providers. And these
dependencies can amplify the concentration
of service providers. Which means that the effect
of single points of failures in the
internet is amplified. And a single provider can
now impact 37% of the top 100,000 websites.
This is what we observed for our data in 2020.
Now we see how the world changed after the
Dyn incident by comparing 2016 dependency data
with 2020 data. Which brings us to our
third question, which is, how did the world
change after the Dyn incident? Did websites
try to reduce their third party dependency on
service providers? Did the concentration
among providers decrease? We observed that
critical dependency in DNS increased. With
respect to CDN and CA, we see no significant
change in the critical dependency.
Similarly, in case of inter-service dependencies,
we observe an 8% decrease in critical dependency
in CA to DNS dependency, and a 4% decrease in CDN
to DNS dependency. So overall, in terms
of provider to DNS dependency, we see a
decreasing trend, which is encouraging.
Now let us look at the change in concentration of
service providers. According to this figure,
in 2016, 2700 DNS providers served 80% of the
websites, while in 2020, only 54 providers
served 80% of the websites. This means that we
are in general moving towards more concentration.
This is not an encouraging result because it
means that single points of failure actually
got bigger. And we see similar trends in CA
providers while concentration decreased in
CDNs.
To summarize, we see no significant change in the
third party dependency trends among
websites. We do observe a decline in third party
dependency among service providers.
We also observed that services became more
concentrated in 2020 as compared to 2016.
All in all, we do not see an increasing trend in
redundancy. We did observe that websites
using Dyn were more redundantly provisioned in
2020 as compared to 2016. But in general,
there was no significant trend towards redundancy.
And because the concentration of providers
increased in 2020, single points of failure
actually got bigger.
Now, based on these findings, we make some
recommendations.
For websites, they should consider redundancy
while using a third party provider. Moreover,
they should try to understand their indirect
dependency arising from the inter-service
dependencies of the providers that they are using.
Similarly, for service providers, they
should make it easy for websites to be redundantly
provisioned. As we observed that many DNS
providers do not support redundancy. And also,
service providers should try for redundancy
in their inter-service dependencies. Finally,
service providers should also try to be more
transparent about attacks on their infrastructure.
To highlight some of the limitations of our work,
our measurements are from a single vantage
point only. And hence, we may miss region-specific
dependencies. We analyze dependencies only
on the landing page of a website. Hence, we may
miss dependencies that manifest deeper
in the content hierarchy. A direct consequence of
this could be that we may miss certain
CDNs used by a website because their landing page
didn't access any resource from a CDN.
Finally, we also did not look at the physical and
networking infrastructure dependencies
such as routing and hosting, mainly because it's
not the scope of this work.
To summarize, the DDoS attack on Dyn in 2016
raised some questions about the resilience
of the internet. In our paper, we try to answer
some of these questions, particularly about
the prevalence of third-party dependencies and
their impact. To do so, we analyzed top
100,000 popular websites. We also look at their
inter-service dependencies to get a
holistic view. And we identify the potential
single points of failure in the internet.
We see that third-party dependencies are highly
prevalent. Indirect dependencies amplify
provider concentration. And we do not see a move
away from third-party dependency after
Dyn incident. The code and data used in this
paper is actually also available online. Thank
you so much.