Hi everyone, my name is Justin from the
University of Liège in Belgium.
Today in this talk I'm going to introduce you to
something that we have called cross-layer telemetry.
So basically we try to find a name as explicit as
possible and I will go much into detail about
this.
But first let me remind you some basics.
So we moved from a kind of monolithic
architecture to microservices.
So you can see microservices everywhere now.
And there are a lot of reasons for that.
Mainly that it's easier and faster to deploy, to
maintain, but there are also other reasons.
So if you look at this kind of architecture and
let's assume there is a problem somewhere
and I ask you to debug and to find the problem
here, you would tell me "okay, easy game".
But what about this one? So it gets a little bit
more complicated.
So hopefully for that kind of spaghetti microservice
architecture,
you have application management performance,
which is actually application performance
management,
and in this case, more specifically, you have
distributed tracing tools.
In this talk I will take Jaeger as an example,
but you have to know that there are a lot of
other alternatives.
Jaeger is just a famous one in all of them.
So such tool is very useful when we are dealing
with microservices.
And they all have something in common.
They all have the same notion, the same concept
of trace and spans.
So you can see a trace as a hierarchy of trace
and spans.
So a trace can contain subtraces and also spans.
And you can see a span as a unit of work, of code.
So this is just a part of your code that you want
to monitor.
And here at the bottom of the slide, you can see
a screenshot of the Jaeger visualization.
So you have basically a main trace, which
contains two subtraces,
and each trace has some child span.
So here a span, and then a span, a child span,
etc.
So you can see on the left that you have a clear
visualization of the flow of the code path.
And on the right, you have the time taken by each
part.
So it's very useful for those two reasons.
You can see, as I said, the path taken by your
code.
So if you execute this one, what is going to be
triggered, etc., etc.
And you can see the time that it took.
But such tool is limited to layer 5 and above.
So if you have a problem, a low-level issue, so a
network issue, for instance,
well, that's when you're facing a problem.
And just to cite an example, let's assume that my
database lookup is slow.
So I can see in Jaeger that my database lookup is
slow because I traced it.
In that case, maybe should I just blame the app,
or should I put the blame on the server, or even
the database?
Or maybe this is a network issue, or actually it
can be anything else.
So it's hard to know exactly what it is.
So you don't know why, and you don't even know
where exactly the problem is.
So the only thing that you know is what Jaeger
tells you, that it's slow. That's all.
So let's just see a basic, simple topology here.
So you have the app, the database, and this guy
in the middle of the path with a congestion of
one of its interfaces.
And so Jaeger will just report a slow execution
time.
So when you see this, you will investigate on the
app, you will investigate on the database,
and actually you wouldn't find anything.
So you would be left scratching your head and
wondering why it takes so long.
And that's a big problem because root cause
analysis, it's hard in this case.
So the goal here would be to also include layer 3
and layer 4 in such tools.
And actually this is exactly what we provide with
cross-layer telemetry.
And to reach such a goal, we first have to answer
two key questions.
The first one is how do we find a way to
correlate the traces from the APM with the
corresponding network traffic?
So back to my example, if you want to trace
exactly your database lookup in your code,
you want to match the trace generated by the APM
with the network traffic, which is the DB lookup
on the link.
And for that, we will use IOM.
So IOM is just, it's in-situ operation
administration and maintenance.
It's actually used to carry some useful data in
packets.
We have developed it in the kernel and it should
be available soon.
Why do we use IOM?
Well, we want to kill two birds with one stone.
So as I said, it can carry a lot of useful low-level
information.
So for instance, you have the queue size, the
buffer location,
IDs of nodes and interfaces from where the packet
is coming from and where it's going to,
and so on and so on.
And on the other side, we just enhanced the IOM
header to carry both the trace ID and the span ID.
So remember what I told you about the common
point of those tools.
They all have the same notion of trace and span.
So a trace ID and a span ID both represent a
unique ID of a span.
So that's why we just carry the two.
So the first important question is answered.
Let's face the second one.
When and how should we inject these IDs?
So when?
Well, we have two possibilities, either at socket
creation or when sending data.
So if you think a little bit about it, at socket
creation, it wouldn't be enough.
Why?
Well, because an operator could use the same
socket for all connections all along,
or you could have for the same connection,
multiple trace.
See, if you want to monitor different part of
your code, you would have multiple trace on the
same socket.
So that's not an option to inject those IDs at
socket creation.
And moreover, you don't want to modify the C
library, because you would have to modify also
high level languages.
That's not an option.
We want to provide a tool, an improvement to
those tools, which is not a burden.
So you want to integrate it easily without
changing everything.
So we will just select the option of injecting
those IDs when we send the data.
But now how?
So again, we have several possibilities.
The first one that comes in mind is to modify the
send functions offered by the C library.
But again, you don't want, as I said, you don't
want to modify everything, because if you modify
this,
you would have to modify also high level
languages.
So not an option.
So we are left with two other possibilities,
which is to add a new syscall or use a netlink
call.
Again, if you add a new syscall, syscalls are not
always portable.
And the preferred way of doing this is usually to
use a netlink call.
So from a kernel perspective, it's always the
best option.
So we just select the netlink call as an option.
So let me explain you a bit this architecture of
cross-layer telemetry.
So first of all, we have a client, which is a Jaeger
client in this case,
which is used to add tracing code to your
application.
And actually, when a trace is available, it will
be sent to the agent, which will forward to the
collector,
which will apply some action on it and then store
it in the database.
Now we had a CLT library, which is also a client
library.
We had the netlink call. And we add an IOM agent
and an IOM collector.
So the CLT library is just an abstraction of the
netlink call.
So the application will just pass those IDs to
the CLT library.
So those IDs will be injected through a netlink
call and they will be injected where?
Well, to the socket.
So again, someone will tell, hey, this is a per-socket
solution.
So what if you have some congestion in your queue?
Maybe that one packet will be marked with those
IDs and it shouldn't be.
Well, you're correct. And we are currently
working on a perfect solution, which will be per
packet.
So it comes with a lot of other challenges.
But I think we are pretty close to a perfect
solution.
But right now, this one is working pretty well,
except in the corner cases I mentioned.
But again, the main goal of this one is to have
some useful info to debug.
So from layer 3, layer 4, as long as it's low
level.
It's not that important that you have a perfect
match between the connection and traces.
If you want a perfect match, then stay tuned and
we will post the next version, which will be a
lot more precise.
So back to our architecture.
When there is a request, so here, HTTPS request
or it could be whatever it is, as long as it's
network traffic.
So when the packet goes out of the interface, it
will be marked with the IDs that were injected
here.
And so actually, the application or it's not
important if it's an application, as long as it's
a node on the other side.
And you have an IOM agent that will run on this
part.
And this one is responsible for gathering IOM
data.
And it will forward those to the IOM collector.
And this guy will just correlate IOM data and any
data that you want from layer 3 and layer 4 with
the IDs that you have passed.
And so it will send this to the IOM collector.
So this is a correlation request.
And so the IOM collector will be responsible for
just storing the correlation inside the database.
And actually, as a result, you can see directly
in the Jaeger visualization IOM data or layer 3
or 4 data as you want, as I said.
So back to our example, you have the application
database.
We introduced a simulated congestion here on this
guy on its interface.
And as you can see, so this is the first node.
So this is the app.
This is the second node and third node.
So the second node is this one.
And you can see that the egress queue is
increasing.
So if you are the operator, now that you see this,
well, you directly find root cause analysis.
So you can see, OK, there is a problem in the
queue.
Maybe I could rebalance it.
Or you are now capable of applying some actions
to solve the problem.
But again, without cross-layer geometry, you
wouldn't have those data.
So the only thing you would know is that it's
slow, once again.
OK.
So let me conclude on this talk.
I definitely think that it's a hot topic in the
industry.
So I've heard that there is a lot, a lot of
interest on this one.
And I do believe that CLT solves a lot of
challenges in the tracing world.
We are still working on some part to improve it.
So I mentioned earlier another version which will
be per packet.
So to have a perfect correlation and match
solution.
And there is -- there are also some other things
to improve, but not that important.
So I insist on the fact that this solution is
working.
So this is something that you could use right now,
if you want.
And there is a link to the GitHub repo.
So feel free to have a look at it.
There is also a video to demonstrate how it works.
Everything is explained here, there.
So feel free to visit it.
And that's it for me.
Thank you very much for your attention.
Feel free to contact me on my e-mail address or
you can ask me a question right after this.
Thanks again.