Writing a Kubernetes Network Plugin (CNI) in Rust

You can check the plugin yourself here, or fork it and use it as a boilerplate to start your own rust CNI plugin.

During writing my rust operator, I was intrigued about how the underlying networking in Kubernetes really worked, not just on the service and pod abstraction levels. And what a better way to learn about something than to build one.

The Journey

First I want to give credit to this article here that helped me the most through this project.

The CNI specs need you to handle 4 commands in order to be able to communicate with kubelet of any other implementation, namely: ADD, DEL, CHECK, VERSION

I decided to follow the foot steps of the bridge CNI plugin, and try to replicate that in rust, I followed the official plugin repo reference implementation. Believe me it is not an easy task to do. First most of the sugar coating that is available in the official Golang framework wasn’t available, fortunately, I stumbled upon this Rtnetlink implementation for rust that made it easier to implement the needed network interface changes.

The Steps

• First I delegated the IP management to the ipam plugin host-local which had its own limitations, forcing me to use different subnets for each node and then needing to add routing for each subnet to each not routing stack.
• Create a bridge interface and assign the gateway IP to it.
• Add the post routing and NAT rules to the iptables to route traffic from this new subnet to the internet and to through the bridge connection.
• Create virtual network interface pair one for the host and one for the container and move each pair in the correct network namespace.
• Connecting the host virtual network interface to the bridge connection.
• Assign IP and gateway for the container network interface and bring it up.

The Testing Environment

To test the plugin I used vagrant to quickly spin up and provision a two workers and a master Kubernetes 1.26 cluster to test the plugin.

The testing was bit annoying because you had to build the plugin on your local machine push to the nodes and the master and clean the old interfaces and iptables before you could test your latest build. A tool that helped me with that is cnitool which allows you to simulate a CNI request which as useful to quickly test changes in between real cluster testing rounds.

The Conclusion

Writing a CNI plugin is a complex task not because of the coding itself but because you have to really good at a lot of things the Linux networking stack, iptables, CNI and Kubernetes.

I’ve cut a lot of corners and I know the code is far from perfect, some of the edge cases I’ve willingly ignored because it would have multiply the code by a factor of 2 or 3 but you cannot ignore them in a production plugin. For example managing the subnets and automatically adding the required routes to each node which I’ve created manual scripts for.

The rust ecosystem is maturing and becoming a place which provides the best of both worlds, you can get a lot of crates to speed up your production releases while enjoying the performance and the elegance of writing rust.