Hi, I'm Tushar Swamy, and at the IRTF Open
Meeting on Monday, July 25th, I'll be presenting
our work on TORUS, a data plane architecture for
per packet machine learning.
So, this here is a quote from a 2015 Google blog,
and at that time Google was already
dealing with more than one petabit per second of
bisection bandwidth, and their networks
have only continued to grow since.
As a result, they're getting harder and harder to
manage, and these networks require more
and more complex management, but without
sacrificing performance.
So the time is ripe to look for new solutions
here.
And one potential solution is machine learning.
Machine learning actually serves as a good system
to scale complex management in networks.
An ML model can take in data and train a model to
make progressively more and more accurate
decisions, and they can approximate network
functions where the analytical solution is
too complex or slow.
And furthermore, machine learning will customize
its function to the data it sees, and in turn
customize itself to the network in which it
operates.
So both of these things are already happening
with network functions to some extent in the
form of hand-tuned heuristics, like hashing and
load balancing, and TCP congestion control
algorithms with operator-tuned parameters.
So why not scale the process by automating it
with machine learning?
So if we're committing to machine learning, then
the natural next question is, where and
how does machine learning happen?
And the core tenet of TORUS is that machine
learning inference must happen on a per-packet
scale in the network data plane.
Per packet because we want to have the finest
granularity of operation, and the network
data plane is simply where the packets are.
So in this diagram here, I have on the left a
picture of a typical software-defined network
with a split control plane and data plane, and on
the right, I have a software-defined
network modified to work with TORUS.
And TORUS is all about introducing ML to network
for management.
In a TORUS network, the control plane at the top
here is going to create policies by training
machine learning models in addition to flow rules,
and then it will install the model
weights along with flow rules into the data plane.
And in the data plane, we're actually performing
machine learning inference, which is the decision-making
process.
And so this split is not arbitrary.
Machine learning training is left in the control
plane because it's not a critical path operation.
It's not going to be done on a per-packet basis,
and it's not going to happen in the
middle of the network as traffic is moving
through it.
Instead, we leave it in the control plane where
we can use the latest and greatest ML
accelerators to essentially optimize that process.
And in the data plane, we're going to be
executing just inference to enact to learn
decisions.
And the next question here is, how exactly do we
enable a data plane to do this?
We can use accelerators and typical servers for
the control plane training, but there
isn't a good platform to do machine learning
inference in the data plane.
So this is exactly what TORUS is built to do.
TORUS is a swish-level architecture for per-packet
machine learning inference in the data plane.
So once we decided we wanted to tackle this
problem of an ML-capable data plane, we started
with a discussion of what kind of abstraction
would be appropriate, because this would in
turn dictate what kind of hardware we have to add
into a switch pipeline.
And in TORUS, we use the map and reduce patterns.
Map reduces is a commonly used abstraction for ML
because it supports common linear algebra
operations found in a variety of ML algorithms,
such as neural networks, support vector machines,
and k-means.
And in addition, it's a SIMD pattern, so it
allows for a lot of parallelism with minimal
logic.
And in the picture here, we show how a single
neuron in a deep neural network can be
implemented.
Map can be used to perform element-wise
multiplication between inputs and weights, while
reduce combines
elements of the resulting vector along with the
bias into a scalar value through addition
before applying an activation function at the end.
And so you can stack this pattern ad nauseum for
multiple neurons in a layer and then multiple
layers.
And in that way, you can build an entire DNN, for
example, using just map and reduce.
So now that we decided we want to use map reduce,
we went ahead and added map reduce
hardware into the programmable switch pipeline.
So we actually keep the majority of a typical
switch pipeline with packet parsers, match
action tables, and traffic managers.
But we split the match action tables into two
blocks and insert this map reduce unit
in the middle.
And so the match action tables are split into two
blocks for pre- and post-processing of
ML inputs and outputs, while the packet parser is
used for ML feature extraction.
And we wanted to design our map reduce units so
that it's reconfigurable, so you can program
it, it meets line rate, and it has minimal area
and power overheads.
And so you can see the full evaluation of these
in the talk and also in more detail
in the actual paper.
In addition to an ASIC evaluation, we also wanted
to provide an open source FPGA testbed.
We don't want people to wait for some sort of Taurus
ASIC to be mass produced before
writing their own ML data plane applications.
And so we've made this testbed to allow people to
try their own ML for networking apps.
We use an FPGA here to emulate the map reduce
block, while a Tofino-based switch serves
as the programmable switch elements of the
pipeline.
And in our repository, we also have code for
applications like anomaly detection to help
people get started.
So we hope people can make use of this.
So please come see the talk at the IRTF open
meeting if you have any interest in this.
I'll be going over the ASIC evaluations and FPGA
prototype evaluations in more detail,
along with discussion of some applications.
And of course, I have the link here for the full
paper if you want all of the details.
And finally, there's a link here to our GitLab
repository if you'd like to actually try out
ML for networking apps using a Taurus
architecture.
So I look forward to seeing you there.