Hey everyone, my name is Audrey Randall.
I am a PhD student at the University of
California, San Diego.
I'd like to talk to you today about our paper Truffle
Hunter,
cash snooping rare domains at large public DNS
resolvers.
When you think about abusive behavior on the
Internet today,
you might first think about the more common types
such as spam emails,
or botnets, or malware,
all of that stuff is everywhere.
The thing about it is it's easy to find in the
wild simply because it is so common.
But there's another type of abuse which is much
harder to find because of its rarity,
and that includes things like typo squatting,
hack for hire services,
stock aware services, where we really don't know
how many people are
affected by it because it's hard to find in the
wild for whatever reason.
That's the type of abuse that we are really
interested in studying.
We've made the observation that all of these
types of
rare harmful behavior have something in common,
which is that they all need to make
DNS requests to the servers they require to
function.
If you could observe enough DNS requests,
you could presumably study these types of harm in
more detail.
You could figure out how prevalent they are,
where they occur, how frequently they occur.
But to do that,
you're going to need to observe a lot of
DNS requests because you're looking for the
needle in the haystack.
You're looking for a very small amount of signal
and a large amount of noise.
Fortunately, we're entering this new era in DNS
right now,
and that's because these public DNS resolvers are
starting to gain more popularity.
It used to be that it was only power users and
people who were really tech savvy,
who would be using public resolvers,
but we're starting to see them get hard-coded by
default.
For example, Google Home routers all use Google's
Quad 8 service by default,
and Firefox routes all their DNS queries to Cloudflare.
We even see that New York City's entire public Wi-Fi
network now uses Quad 9.
Of course, we are not Google,
we are not Quad 9, we are not Cloudflare.
But can we, as third-party observers,
still use these services to observe the kind of
rare behavior we want to study?
Well, of course, the answer is yes.
There is a well-known technique that's been
around since at least 2004 called DNS cache snooping.
But in the past,
it's been presented as an attack and it's
considered a privacy threat and for good reason.
Most of the time, what researchers were doing
when they did cache snooping,
was they would scan the whole Internet and they
would try and
see which devices would answer a DNS request.
The problem was, most of those devices were
misconfigured home routers,
so they've only got a few users behind them.
If you find some domain on them that might be
invasive of privacy,
it's not too difficult to figure out which actual
user put it there.
Public DNS resolvers, on the other hand,
allow you to preserve privacy because so many
people use them,
it's almost impossible to de-anonymize them.
It's almost impossible to figure out just based
on the fact that
a domain is in cache, who put it there.
That's great. We can use cache snooping as
a measurement technique instead of a privacy
threat on public resolvers.
But public resolvers are also more
challenging because they've got complicated caching
strategies,
and that's why they're of interest to the IETF,
because this does lead to some protocol non-compliance.
For the remainder of this talk,
I'm first going to go over some brief background
on
cache snooping for anyone who hasn't seen the
details in a while.
Then I'm going to talk about how to do it on
public resolvers.
To do that, you need to understand their caching
strategies.
We as researchers had to reverse engineer
the caching strategies of four large public resolvers.
Next, I'm going to talk about our tool,
which is called Truffle Hunter,
and how we used it to measure certain case
studies,
and learn a little bit more about
these types of rare abuse that we're interested
in measuring.
So I'll get started on background of cache snooping.
If somebody were to make a request for example.com
to their local DNS resolver,
that resolver would have to look at its cache and
say,
"All right, is example.com present?"
If so, they can return the response quickly to
the user.
If not, they have to go to the authoritative name
server.
What you can do if you are a snooper is make a
request for example.com,
but set a flag that tells the resolver it is not
allowed to check the authoritative name server.
That way, if you get a valid response back with a
valid IP and a valid TTL,
or time to live value,
then you know that the domain was cached.
I should note that all of the resolvers we've
been using this on do
respect the recursion desired flag with one
slight exception,
which I'll get into later.
But the thing about cache snooping as a
measurement technique is that it only
provides a lower bound on the number of users
that are accessing a domain.
If multiple users have hit the same cache for
the same domain before that TTL expires and the
record is removed from cache,
you won't be able to observe them.
You can observe a maximum of one user per cache
per TTL.
But that's okay for our purposes because we're
looking for
the types of phenomena where nobody knows how
frequent they are,
how frequently they occur in the wild,
and it's valuable even just to get that lower
bound
because seeing any of them is actually really bad
news.
But cache snooping on a single resolver is
actually reasonably straightforward.
In order to do it on a public resolver,
things get significantly more complicated.
Let me talk next about how public resolvers work
in broad terms,
and then how they work in more specific terms.
When a user wants to send a request to a public
resolver,
that query is first routed using IP anycast to
the first available or the closest point of
presence or POP.
Once there, it can be routed to one of any number
of front-end caches.
There are a lot of these.
If it misses in those front-end caches,
it will be sent to one of usually several back-end
resolvers.
That's good for us in some ways because each of
those caches represents another possibility of
observing a user in the wild.
But of course, these complicated caching
techniques make our job a little bit
harder because we need to be able to count how
many caches our queries have hit,
and we need to be able to differentiate between
them
so that we don't double-count full caches.
This challenge is exacerbated by the fact that
each of
these public resolvers implements their caching
differently.
We have found that that inconsistency could cause
potential problems.
While some of the resolvers we've looked at
always do seem to serve
records with correct TTLs,
we have found some that can either serve records
with
incorrect TTLs or will serve records after the TTL
should have expired.
In a few slides, I'll go into more details on
that.
I've mentioned that in order to do our
measurement study,
we need to count how many caches have been filled
with the domain,
and that means we need to be able to identify
which caches our queries have hit.
We had to understand the caching strategy of each
of these four public resolvers.
To do that, we only have access to the TTL and
the timestamp of the DNS queries that we
were sending which made reverse engineering these
things challenging.
I'll go over now how we did that.
We ran an experiment where from a single central
location,
we would repeatedly query a resolver and try and
fill its caches.
Then we looked at the query responses that came
back and we looked at
their TTLs and their timestamps to try and figure
out how the caches worked.
At this point, I have to introduce the concept of
a TTL line,
which is just our model of how a TTL decreases in
a cache.
A TTL in a cache ought to decrease by about one
second per second.
If you plot a bunch of measurements that have all
hit the same cache,
you ought to see if you plot their timestamp
against their TTLs,
that that decreases by one second per second.
That's this green line in the figure here.
That's our model of what the TTL should be doing
in the cache.
When we do that against real resolvers, in this
case,
OpenDNS and Quad9, we get something like this.
It looks like about what you would expect.
Now, note that this plot is zoomed in on the
first 50 seconds just to show detail.
The TTL lines aren't shown going all the way to
zero,
but of course, we expect them to do that
eventually.
The first thing that you might notice is this row
of measurements across the top,
which came back with the maximum TTL value for
the domain we used.
They're circled in red.
We assume that each of these measurements filled
a new cache
because they came back with the maximum TTL value.
And we were able to confirm this
because we controlled the authoritative name
server
for the domain that we were querying.
So we confirmed that every time our authoritative
name server got a new request,
we got a measurement back at our measurement
source
with a maximum TTL value.
You'll also notice that all of the measurements
that are not one of the top row of circled dots
lie on one of the TTL lines.
So they look like they came from one of the caches
that we observed to be filled.
So that's great.
That means that OpenDNS and Quad9's caching
architecture
is reasonably straightforward.
Requests can hit any of several independent front-end
caches.
And if they miss, they're sent to a group of
independent back-end caches.
And we can just count the TTL lines that we see
and assume that that's the number of filled caches.
When we ran this experiment on Cloudflare,
we got a very different looking graph.
So we do get a first measurement,
which came back with the maximum TTL.
But all of the measurements after that
look like they came from the same cache,
which we would have thought would be unusual
in a resolver of Cloudflare's size.
You can also see that for a while,
the measurements look like they're exactly on the
TTL line,
but then they start to drift over time.
And we noticed that they would always drift
upward.
So what we think is happening
is that Cloudflare has a shared front-end cache,
shared and distributed.
As soon as a measurement arrives in one cache,
it is shared with all of the others.
So that's a little bit disappointing for our
purposes,
because at the whole POP,
we can only see one cache get filled.
So we can only measure one user per POP on Cloudflare.
There is a question with Cloudflare's strategy
of whether or not it is completely compliant
with the DNS RFC for TTLs.
The maximum drift that we saw
away from the true TTL value was about 80 seconds.
So we were using a domain at the time
with a TTL of about three hours,
and we saw that there were still measurements in
cache
whose TTLs hadn't yet expired
for about 80 seconds after they should have
expired.
Now, it's important to note
that the drift scales with the maximum TTL.
So probably, even if you have a 60-second TTL,
you're only going to have a drift of a few
seconds,
and that's probably not going to be an issue for
you,
even if you have such a short TTL.
And if you have a long TTL,
you're probably tolerant of more drift.
So we concluded that the actual problems here
are likely to be very small.
And then finally, we looked at Google DNS,
which is the resolver where the caches fill
themselves.
This actually isn't just our observation.
There's been prior work that has observed this as
well.
Schaumbet et al found that they could make
requests
and get a accurate TTL back on the original
requests,
but then they would keep making requests
and they would find subsequent TTLs to be wrong
because it looked like those TTLs were coming
from caches
that had never been filled.
And then Rapimardo et al noticed the same effect
and called these mystery caches ghost caches,
which we thought was a great name for them.
So why on earth are these caches getting filled
without being queried?
I'll show you what we mean here.
If you look at all of these blue lines,
you will notice that there is no measurement
at the start of these blue TTL lines, these
mystery caches.
According to our theory,
every filled cache should correspond to two
things.
A, a measurement that we made with a maximum TTL
at its start,
and B, a request to our authoritative name server.
And we didn't see either of those here.
What we eventually noticed is that every light
blue cache
does appear to get filled at the same time
as one of our measurements was made.
You can see that the dotted lines
that descend from the start of each cache line
each pass through a measurement.
So what we think is happening is this.
Google is using what we call a dynamic caching
strategy.
When a request comes into Google
and it misses in a front-end cache,
that's this light blue cache here,
then it's going to get forwarded to a backend
cache.
And let's assume that that backend cache is
already full
and it has a TTL less than the maximum value.
The first time that TTL, less than the maximum
value,
let's say 550 seconds,
is going to get sent back to the user.
But at the same time,
the front-end cache is going to store the record
and it's going to store the maximum TTL,
which in this case, let's say it's 600 seconds.
So you can think of it like this.
Every request that comes into Google DNS
has a chance to spawn a new cache
that is visible to cache snooping.
So that's great news for us
as researchers running a measurement study
because we will see a much greater percentage
of unique queries on Google than we will anywhere
else.
But you do have the question of whether this
strategy
is going to lead to inaccurate TTLs.
Now, we did observe that if we queried a domain
that we had placed in cache
and we queried it all the way until it's TTL
expired,
these ghost caches or these front-end caches
that we filled ourselves,
they did expire even if the TTL had not reached
zero yet
when the original back-end cache expired.
So that's good.
But we noticed that a user could make a request
just before the TTL of the back-end cache expires
and get a cache that had just been filled.
And that could lead to extending the TTL
to twice as long as it should be.
So the maximum drift is just twice as long
as whatever the maximum TTL is.
Whether or not this is actually a problem
is not up for us to decide.
We couldn't think of a use case
where it would be super problematic.
But we do have a question of
why would this be a useful strategy?
Why store the maximum TTL in the front-end caches
rather than just copying the TTL from the back-end
caches?
Now, it's great that Google did this from our
point of view
because it really enabled our measurement study,
but we couldn't come up with a reason
why it would be more efficient or more performant
to do that.
So if anyone is here from Google
or if anybody wants to weigh in,
I would love to get somebody's thoughts on that
when I'm done with this talk.
So to summarize, OpenDNS and Quad9
appear to have a pretty straightforward caching
strategy.
And we don't think that that caching strategy
ever manipulates the TTLs of the responses at all.
Cloudflare has this shared and distributed front-end
cache.
And we do notice that the TTLs are affected
slightly by it,
but we don't think that's likely to be too much
of an issue
because the drift is so small
compared to the length of the maximum TTL.
And Google has what we call a dynamic caching
strategy.
And that can result in a TTL received by the
client
being about twice as long as it should be
because you could receive a maximum TTL
right before the back-end caches TTL was set to
expire.
So you should have been receiving a very small TTL
and you receive one that's closer to the maximum.
Now that we've talked about how to use cache snooping
on public resolvers by counting the caches
that your queries hit, let's talk about our tool,
which we have nicknamed Truffle Hunter.
It's our distributed measurement tool,
which we've deployed on KEDA's Archipelago
project.
That means it's on 46 different measurement nodes
scattered across the United States.
All that it does is send continuous DNS queries
across the US for the domains that we're
interested in.
When it gets the responses back,
it interprets them according to our models
to try and figure out how many caches were filled.
And we go from there to estimating counts of
users
in some cases.
We have three months of data
or data at the time we wrote this paper
from March to May in 2020.
Truffle Hunter is of course not perfect.
And the first big question that we had when we
deployed it
was how accurate it was at estimating the number
of caches
that were filled.
Because we know that our models of cache
architecture
might not be 100% accurate.
So we ran an experiment
where from 900 different ripe Atlas probes,
we placed a domain we controlled
into the caches of public resolvers.
The idea was just to put it there
as if people across the US had done it naturally.
Then we use Truffle Hunter to try and observe it
in those public resolver caches.
Because it's a domain we control,
we could conclude that the number of requests
that came into our authoritative name server
should be the true number of filled caches,
except in the case of Google,
which of course does its own thing.
We found that we performed best on OpenDNS and
CloudFlare,
except in the case of one particular CloudFlare
pop
where we think there was some routing going on
that we didn't account for during our experiment.
On GPDNS, it turned out to be difficult
to accurately remove all of the front end caches
that have been filled by our own probes.
So we conservatively removed more
than we had actually created
in order to ensure we never over counted.
That's consistent with our goal
of always providing an underestimate
rather than an overestimate of the number of caches
that have been filled
and therefore the number of people that are
filling them.
And then Quad9 was very interesting.
It had the same architecture as OpenDNS,
so you would expect it to be just as easy to snoop.
But as it turns out,
Quad9 runs two types of software at each of their
backends.
And one of them refuses to respond to non-recursive
queries,
that's Unbound.
So if it turned out to be that Unbound
was the software that had cached our domain,
that record essentially became invisible to us.
So it does mean we can't observe
about half of the filled caches at any Quad9 pop.
The takeaways here are,
first, we were able to tune our algorithm
so that we almost always underestimate,
which is good because our goal
is to provide these lower bounded estimates of
prevalence.
And second, even on the resolvers where we have
high error,
we do see at least half of the filled caches.
The next limitation that TroubleHunter has
is inherent to cache snooping as a measurement
technique.
And that is that you can only observe one user
per cache.
So we looked at how many caches were visible
on each of these resolvers at any one time.
We found that Google had a great many of them
that were visible to us
because of their dynamic caching strategy,
about 100 times more than either OpenDNS or Quad9.
Cloudflare, of course, only has one visible cache
per pop,
so its graph is omitted here.
OpenDNS and Quad9, you might notice
that some of the larger pops like NYC and IAD
have a lot more visible caches than the smaller
ones.
So that's TroubleHunter in a nutshell.
Now, the interesting bit
is what we actually use TroubleHunter to study,
and that's our case studies, which I'll talk
about next.
We ran four of these.
I'm only going to talk about three of them here.
And these were stalkerware, contract cheating,
and very old typosquatting domains.
We didn't expect to see very much of any of this
stuff
for various reasons.
And previously, all of these things
were kind of difficult to measure,
maybe typosquatting not so much,
but certainly contract cheating and stalkerware
have been very difficult to observe in the wild.
And nobody really had any data about how
prevalent they were.
First of all, let's talk about stalkerware.
If you haven't heard the term before,
it's kind of this emerging spyware thread.
It's this software that can be installed
on a target's device, either a phone,
usually a phone or a desktop computer, and it
tracks them.
It can record location.
It often has key loggers to record texts,
social media, browsing history, things like that.
And oftentimes it can record ambient sound
and video of the device as well.
And it can hide its presence on the device.
So even if you are downloading it on an Android
phone,
which ought to tell you how many apps
and what types are installed on it,
it can hide its icon so that you can't tell it's
there.
So we downloaded and profiled 24 of these apps
to try and get their network signatures
so that we could figure out
what DNS requests they were making.
Six of these apps were what previous work calls
dual use.
Those are apps that have a legitimate purpose,
which is separate from their ability to stalk
somebody,
but they can be repurposed to act as stalkerware.
The line between dual use and overt apps
is a little bit blurry,
but dual use apps tend to be marketed
for parental control or employee surveillance
or things like finding a lost phone or backing up
your data
rather than for spying on a spouse or a
girlfriend.
Overt apps, on the other hand, are marketed as
undetectable
and marketed specifically for,
or at least often marketed specifically for spying
on a spouse
or a girlfriend or a husband or what have you.
So they tend to be the more dangerous ones.
The prevalence of overt stalkerware
is hard to estimate by any other means
because it's very difficult to observe it in the
wild.
Prior work in this space
has mostly been conducted in clinical settings.
So researchers will conduct
individual one-on-one interviews with targets.
And unfortunately that gives them a low sample
size.
During these interviews,
they found few to zero of these overt apps in the
wild,
but a simple Google query will turn up dozens of
them
and there are ads all over the place.
So it really does beg the question
of how much of this overt stalkerware is out
there.
Additionally, by the time a target
has come in to talk to a professional,
they have often already reset their devices.
So it's difficult for a clinic to tell
which apps were on there before the reset.
And finally, clinics often lack technical
expertise.
So if they aren't working with someone
who does have expertise in this space,
it can be very difficult for them to tell
if a device has stalkerware installed.
Up until this point,
we've only been talking about counting filled caches,
not counting the number of devices
that have made the requests.
And we haven't made any attempt to differentiate
between a single device that is making multiple
requests
and multiple devices that are each making a
single request.
Stalkerware is a really good app to use,
really good use case to use for figuring this out
because it's supposed to be often installed
without the user's knowledge.
So it has to make its DNS requests automatically
without any user interaction.
And that means it often makes them
at regular predictable intervals.
So if you want to know how many devices
have stalkerware installed,
all you have to do is measure that request rate
and then divide the number of filled caches you
see
by the request rate of the app.
That's the technique that we use
to come up with this figure here.
This graph shows the maximum targets
that we ever observed with stalkerware installed
at any one time across the United States.
We found that nearly 6,000 people are being
targeted
by overt stalkerware in the US today.
And recall that that's a strict lower bound.
The number may very well be higher,
but we suspect it's not lower.
The two most interesting apps that we found
were called Mobile Tracker Free and Spy to Mobile.
Those are the two most frequent
that you can see at the top of this chart.
Mobile Tracker Free, we suspect is so popular
because out of all the overt apps we studied,
it was the only one that wasn't subscription-based.
It was free to install, free to use.
And Spy to Mobile, while not being free,
was the only one of the overt apps
that was available on the Google Play Store
because it has a habit of changing its developer
name
and its name and re-uploading itself multiple
times
to evade Google's rules.
We also looked at the dashboards for stalkerware.
And when I say a dashboard,
I mean the website that an attacker will go to
when they want to view the data
that has been collected by the app.
We again see that Mobile Tracker Free
is the one that was visited most frequently,
but Spy to Mobile has fallen down
in the rankings a little bit.
So clearly it is the case that the popularity of
the app
does not necessarily correspond
to how many times somebody is checking the
dashboard.
We theorized that might be
because of differing app capabilities.
Mobile Tracker Free has a lot more features
than Spy to Mobile.
Spy to Mobile is mostly good at tracking location.
And the next case study that we looked at
was contract cheating,
which is the new form of plagiarism
that the kids are using these days.
So in case you haven't heard of this,
these are services that you can buy as a student
to complete your homework or your projects
or even entire classes for you.
I've even seen a few that are offering
to get entire online degrees.
It's pretty hard to detect when a student is
using this
because they're not actually plagiarizing.
They're not copying work that exists already.
They're hiring someone to create original content
for them.
Of course, your mileage may vary.
Some of these services are better than others,
but there are a few that are good enough to get A's
in most cases, even including college
and sometimes graduate classes.
So it's of course hard to observe in the wild
because students aren't going to just admit
that they have done cheating, even on anonymous
surveys,
which is how a lot of this work has been done in
the past.
So we had Trouble Hunter look for it
and we observed that yes,
you see a lot of requests per day
to these contract cheating websites.
Now, of course, a request made for the website
doesn't necessarily mean that a student bought
anything,
but it's still an interesting number to observe.
We saw that some of these services,
which we measured over the last couple of weeks
of May,
were decreasing over time,
which we thought was interesting
and might indicate that schools are letting out
for summer break, so demand for cheating is going
down.
And then finally, because we had some of these
domains,
we looked for typosquatting.
These domains are pretty old.
We don't expect that they are being used
to phish anybody anymore
because received wisdom and prior work
says that phishing domains and typosquatting
domains
usually roll over very quickly.
They get blacklisted
and then the miscreants move on to other domains.
So you wouldn't expect that any of these
would still be receiving DNS requests,
but we saw that yes,
they are still getting a few requests per day,
which is interesting.
So the takeaway from our point of view
is that cash snooping on public resolvers
shouldn't actually be gotten rid of yet.
We argue that there are minimal privacy concerns
when you're cash snooping on public resolvers
because there are too many users
to figure out which user put a domain into cash.
And if you allow cash snooping on these resolvers,
then you can measure types of harm
that are otherwise very difficult to study.
In particular, for stalkerware,
it's very difficult to figure out
how much of this stuff exists in the wild
and each instance of stalkerware
represents a significant amount of harm being
done.
Furthermore, contract cheating is difficult to
study
because students are just not honest
about whether or not they've bought cheating
software.
And then there's other phenomena,
which we didn't get a chance to measure very well,
which we would like to look into more in the
future,
such as these new hack for hire services
and phishing, which by all accounts is quite
common,
but we would like to see how much of it is
happening
in various places around the world.
To conclude, we found that public DNS resolvers
enable us to use cash snooping
as a privacy preserving measurement technique
rather than an attack.
We think this is a valuable measurement technique
that should not be disabled
on specifically public DNS resolvers.
We also found that to use cash snooping
on public resolvers,
you have to understand their cash architecture,
which is quite complex.
So we reverse engineered four of these resolver
strategies
and we did find that Cloudflare and Google
caused some minor TTL noncompliance.
Whether that's an actual issue or not
is not up for us to decide,
but we suspect it's not that much of a problem.
And then finally, we used our tool
to find non-trivial lower bounds
of the prevalence of internet phenomena
that were previously very difficult to study.
So Travel Hunter is open source.
If you want to try it out for yourself
from a single location,
you can go to this GitHub link at the bottom of
the slide.
And I'd like to thank you for your attention
and take questions at this time.