Hello everyone, I'm Chaoyi Lu from Tsinghua
University, and thanks for joining us.
In this talk, we will discuss encrypted DNS
protocols, or what we call here DNS over
encryption.
We present a large-scale measurement study on the
operational status of the protocols,
from angles of servers, DNS clients, and passive
traffic, and give an early view into this
ecosystem.
The major results in the paper were obtained in
2019, and in this talk, we will also present
some more recent updates.
So today, almost all internet activities start
with DNS.
I want to attend this year's IRTF Open Meeting,
so I need to resolve its website at irtf.org.
So my DNS queries actually can say something
about my internet behaviors.
But there is no confidentiality or authentication
in DNS, so my queries are susceptible to both
passive and active attacks.
For all these people, they can see that I query
the IRTF domain often, and learn that
I may be interested in the activities.
So that was just a juvenile example about DNS
privacy, but massive DNS surveillance
and manipulation is not in theory only.
For example, release documents show that the NSA
has been monitoring DNS data under secret
projects.
And also in the literature, works show that DNS
data can be used for privacy-threatening
activities, such as device fingerprinting, user
tracking, and user behavior analysis.
So to mitigate DNS privacy issues, one of the
mainstream solutions is to encrypt DNS
communications, or what we call here, design DNS
over encryption protocols.
Starting from 2009, we see some early protocols
from the industry, like DNS curve and DNS
Crypt.
From around 2014, DNS privacy problems have drawn
attention from the IETF, leading to
some informational RFCs.
And since then, three working groups have been
established to work on solutions.
And as a result, we have seen three standardized
DNS over encryption protocols.
And of course, there are still more protocols and
tests coming to solve the problem, but
we do not discuss them today.
In this study, we focus on two standardized and
extensively supported protocols, DNS over
TLS, or DOT, and DNS over HTTPS, or DOE.
The idea of DOT is very straightforward.
Before DNS queries, I negotiate a TLS session
with my resolver and use it to encrypt wire-format
DNS packets.
DOT runs over a dedicated port 853, and to use it,
some client software needs to be updated,
like the stub resolver of my operating system.
DOE, on the other hand, embeds DNS packets into
HTTPS messages.
It shares port 443, and is especially convenient
for user-space applications, such as web browsers.
And currently, both protocols work between stub
and recursive resolvers.
And here are two examples on how to use these
protocols.
DOT queries can be issued with tools like KDig,
which is the stub resolver of NotDNS.
DOE queries can be issued, for example, from a
browser, using a URI template like this.
The query names or DNS packets can be embedded
into the URL or HTTP message body.
Actually, the standard DOE uses wire-format DNS
packets in the HTTP message body.
The format displayed here is the JSON format
adopted by some public resolvers, and I use
it here because it's readable.
The history of DNS over encryption protocols is
not long, but they have been rapidly developing
recently.
From the industry, several large public DNS resolvers,
DNS software, operating systems,
and web browsers have implemented DOT or DOE.
And some more recent updates that will further
push them forward.
Web browsers and operating systems have been
rolling out DNS over encryption, and these
are just some examples from large vendors.
This year, Firefox has even enabled DOE by
default for users in the United States.
So after the efforts from the community and the
industry, we aim to understand the current
DNS over encryption ecosystem from Internet users'
perspective.
We seek answers to these questions.
How many DNS over encryption servers are there in
the wild, and how are they currently operated?
How are the performances of DNS over encryption
services, and are there any issues preventing
global users from accessing them?
And what does the real-world usage of the
protocols look like?
So first up, how many servers are offering DNS
over encryption services?
Discovering open DOT servers is simple.
It runs over a dedicated port, so an Internet-wide
scan will suffice.
DOE servers, however, are different.
They use URI templates and a shared port 443, so
we cannot simply use port scans to discover
them.
But the RFC documents and large providers have
used some common URI templates, such
as /dns-query, which inspires us to look for them
in large-scale URL datasets.
And in more detail, to find open DOT resolvers,
we first launch Internet-wide scans of port
853 using ZMAP.
For hosts with this port open, we send them DOT
queries using the GET DNS API, and only
those correctly answering our queries are counted
as DOT resolvers.
And finally, we fetch and verify their
certificate chains on port 853 with OpenSSL.
And in this study, we only consider resolvers in
the IPv4 address space.
So during February and May 2019, we repeated our
scan every 10 days and found around 2000
DOT resolver addresses.
We further identified them by the subject of
their certificates, and found several big
players actually dominate in the count of servers.
For example, clean browsing DNS alone accounts
for over 1000 resolvers in almost each of
our scan.
And here are some more recent updates.
In our latest scan on July 1st 2020, we find that
there has been a significant increase
in the number of open DOT resolvers.
Compared to a year ago, which is July 1st 2019,
the number of open DOT resolvers has
risen to over 7.8 thousand, almost quadrupled.
And our results also show that there have been
more providers establishing their DOT
resolvers.
And meanwhile, from their certificates, we spot
several big providers, which maintain
many of them.
And this time, Cloudflare tops the list with
nearly 2000 resolver addresses, followed by
Fortinet Firewall devices, clean browsing, and
NextDNS.io.
DNS over TLS uses certificates for authentication.
The DOT usage profile suggests that under strict
privacy, DOT clients should validate
the full path of resolver certificates, so we
check all DOT resolvers in our list with
OpenSSL.
During our regular scans from February to July
2019, we find that around 25% of all
DOT providers have at least one resolver which
use invalid certificates on port 853.
A year later, we find that invalid certificates
still poses as a problem.
In our latest scan this July, the ratio of open
DOT resolvers with invalid certificates
rises to 28%, but they belong to a smaller
portion of DOT providers.
We take a look into the error codes that OpenSSL
shows, and find that the invalid certificates
belong to three categories.
Around 70% are self-signed certificates, and most
of them are installed on firewalls and
TLS inspection devices.
Under 15% of the invalid certificates are expired,
and one third of them have expired
before 2020, that is, more than half a year,
which could have already been out of maintenance.
The rest has problems in their certificate chains
which cannot be correctly validated.
Because the authentication of DOT relies on the
certificates, we suggest that the providers
keep their servers under regular maintenance.
To discover OpenDNS over HTTPS resolvers, we
search for the common URI templates in
a large-scale URL dataset provided by our
industrial partner.
The dataset consists of URLs collected from their
web crawlers, sandboxes, and some external
data feeds.
We find that the number of DOT resolvers is
modest compared to that of DOT.
In May 2019, only 17 DOT providers are discovered
by our method.
And most of them have already been included in
previous lists like this one, maintained
by the Curl project, and we also find two resolvers
beyond the list.
And more recently, in this July, the number of
OpenDNS resolvers also rises.
We discover over 50 valid URIs which are operated
by 37 providers.
And our list also contains several resolvers out
of no lists, and here we give some examples.
And in particular, we find that the lists also
include some resolvers which are established
by large ISPs.
Our second question.
We know that clear-text DNS are manipulated under
various policies.
So are DNS over encryption services reachable by
global clients?
And are there any issues preventing their access?
We use SOCKS5 proxy network platforms that allow
large-scale client-side measurements.
To explain, clear-text and encrypted DNS queries
are issued by our measurement client, passing
a superproxy, forwarded to global exit nodes, and
eventually reach the target resolvers.
By checking its response at the measurement
client, we can tell whether a DNS resolution
from the exit node is successful.
To perform the tests, we use two proxy networks,
one global, which is ProxyRack, and one in
China only, to see the behaviors of a typical
censored network.
In total, we collect 29,000 global vantage points
from 166 countries and 85,000 Chinese
vantage points.
And for each exit node, we first issue clear-text
DNS, DOT, and DOE queries of our control domain
to three popular DNS services, Cloudflare, Google,
and QWAP9, to see whether they are
successful.
From the nodes which fail the queries, we fetch
the TLS certificates, probe common ports,
and fetch webpages on the resolver addresses to
check whether the failed nodes are actually
connecting to the real services.
And here are our test results obtained in 2019.
Globally, we find that the failure rate of DNS
over encryption is lower than that of
clear-text DNS.
We suppose it is because interception policies
are currently not considering port 853 or
443.
And as a result, clients may switch to DOT or DOE
when their traditional DNS has problems.
However, global clients see more query failures
when using Cloudflare DOT, and we find that
the address quad1 can be hijacked in these
networks, which actually has been an old problem.
And here are some examples.
These ports are not open on the Cloudflare quad1
address, but from the failed clients
when we probed these ports on quad1, they are
found open.
So these clients are not able to use Cloudflare
DOT because this address is already taken
by something else, like network devices.
And back to the results, we also notice a high
failure rate for quad9 DOE.
We find that quad9 forwards DOE queries to itself
on port 53 and sets a small timeout
for this, which causes unnecessary server
failures.
We have reported this issue and they are
considering losing the timeout.
Another finding is that nearly all clients in
China fail to use Google DOE.
And yes, it's because of censorship.
At the time of experiment, Google DOE was not
running on its Anycast address.
But since August 2019, Google DOE has migrated to
Anycast, and hopefully this issue should
be resolved.
Our next question, encryption can bring some
performance downgrade.
So is the query latency tolerable for DNS over
encryption services?
We aim to measure the relative query time of
clear text DNS and DNS over encryption.
And more importantly, we measure their query time
when connection is reused, as required
by their standards.
Connection reuse is also supported as the default
option of many stub and recursive
resolvers.
And again, we use ProxyRack to recruit 8000
global clients for the test.
On each client, we issue clear text DNS, DOT and
DOE queries to the popular resolvers and
compare their differences in query latency.
Because we focus on connection reuse, we only
record their DNS transaction time, which is
after the connection setup.
Our results here show that when connection is reused,
the extra query latency of DNS
over encryption is tolerable.
Take the Cloudflare resolver as an example.
When using its DNS over encryption services, our
global vantage points see extra query
latency on the order of several milliseconds
compared to using its clear text DNS.
And of course, we also see that for individual
clients or vantages in different countries,
their performances can fluctuate, but on average,
the overhead is modest.
And finally, after all these efforts, how are the
DNS over encryption protocols now
used?
And what does their traffic scale look like?
We tried to answer this question from passive
traffic datasets.
Observing DOT traffic is easy because it uses a
dedicated port.
We can find port 853 traffic to DOT resolver
addresses from ISP NetFlow datasets.
For DOE, it is again different.
But recall that DOE resolvers use URI templates,
often with a resolver domain name in it, like
dns.google or cloudflaredns.com.
And these domain names need to be resolved before
using DOE, and that inspires us to
observe the query volume of these domain names in
passive DNS datasets to reflect their usage.
To begin with, we acquire 18-month sampled NetFlow
datasets from a large Chinese ISP,
which is locked by its backbone routers.
If a TCP flow is sent over port 853 to a DOT
resolver address, we record it as DOT traffic.
We find that the traffic scale of DOT is small
before 2019, around 2-3 orders of magnitude
less than traditional DNS over port 53.
That being said, we also notice a growth of DOT
traffic over the months.
In compliance of this ISP, traffic to cloudflare.rises
by over 50% from July 2018 to December 2018.
If we zoom into the client subnets of DOT traffic,
we find some giant netblocks that
account for more than half of the traffic.
We suppose that they could be associated with
proxy services or not.
But on the other hand, in the beginning of 2019,
we also find a lot of temporary DOT
users, which were only active for less than one
week.
For DOE, we inspect large passive DNS datasets
for the query volume of the resolver domain
names.
Not surprisingly, DOE queries mostly flow to
large providers and can rise due to events
like experiments from the industry.
And also, until very recently, there has been a
growing trend on their query volume, which
suggests growing usage.
Actually, public DNS service providers have a
better angle to look into DNS over encryption
traffic.
And from their data, we see a more promising
growth of usage.
For example, cloudflare reported in 2019 that up
to 8% of its queries have been encrypted.
And qihu360 reports that its DOE service has
gained over 1 million active users so far.
And some takeaways and recommendation.
For protocol designers, we think one of the
reasons that DOT and DOE got strong support
is that they are built on well-developed
protocols.
For service providers, we recommend they correct
misconfigurations and keep their servers under
regular maintenance.
For internet users, we believe education is
necessary to let them know the benefits of
DNS over encryption.
We also publish our scanning results and source
code on our project website for public use.
And finally, a quick summary on our key
observations.
First, we discovered thousands of active DNS over
encryption resolvers in the wild, and
a lot of small providers which are less known to
public.
But their implementations are not always secure.
A quarter of them use invalid TLS certificates.
For global clients, DNS over encryption services
are less interrupted by empath policies, and
their performance is tolerable with reused
connections.
And finally, from real-world traffic, we find the
usage of DNS over encryption is still
less than traditional DNS, but there has been a
growing trend.
So that wraps up my talk today.
Thank you for your time, and I'm happy to take
questions.