At Joyent I’ve been spending my time designing and building support for network virtualization in the form of protocols like VXLAN. I’ve gotten far enough along that I’m happy to announce the first SmartOS developmental preview of this work. The goal of this is to just give something for folks to play around with and start getting a sense of what this looks like. If you have any feedback, please send it my way!
All the development of this is going on in its own branch of illumos-joyent: dev-overlay. You can see all of the developments, including a README that gives a bit of an introduction and background, on that branch here.
The development preview below is a debug build of illumos. This is not suitable for production use. There are bugs. Expect panics.
What’s in this release
This release adds the foundation for overlay devices and their management in user land. With this you can create and list point-to-point VXLAN tunnels and create vnics on top of them. This is all done through dladm. This release also includes the preliminary version of the varpd daemon which manages user land lookups and will be used for custom lookup mechanisms in the future.
However, there are known things that don’t work:
- All overlay devices are temporary — not persisted with dlmgmtd
- Overlay device deletion isn’t properly wired up with varpd
- Overlay devices only work in the global zone
Getting Started
This development release comes in the standard SmartOS flavors:
Once you boot this version of the platform, you’ll find that most things look the same. You’ll find a new service has been created and should be online — varpd. You can verify this with the svcs command. Next, I’ll walk through an example of starting everything up, creating an overlay device, and a VNIC on top of that.
[root@00-0c-29-ca-c7-23 ~]# svcs varpd
STATE STIME FMRI
online 21:43:00 svc:/network/varpd:default
[root@00-0c-29-ca-c7-23 ~]# dladm create-overlay -e vxlan -s direct \
-p vxlan/listen_ip=10.88.88.69 -p direct/dest_ip=10.88.88.70 \
-p direct/dest_port=4789 -v 23 demo0
[root@00-0c-29-ca-c7-23 ~]# dladm show-overlay
LINK PROPERTY PERM REQ VALUE DEFAULT POSSIBLE
demo0 mtu rw - 0 -- --
demo0 vnetid rw - 23 -- --
demo0 encap r- - vxlan -- vxlan
demo0 varpd/id r- - 1 -- --
demo0 vxlan/listen_ip rw y 10.88.88.69 -- --
demo0 vxlan/listen_port rw y 4789 4789 1-65535
demo0 direct/dest_ip rw y 10.88.88.70 -- --
demo0 direct/dest_port rw y 4789 -- 1-65535
[root@00-0c-29-ca-c7-23 ~]# dladm create-vnic -l demo0 foo0
[root@00-0c-29-ca-c7-23 ~]# ifconfig foo0 plumb up 10.55.55.2/24
Let’s take this apart. The first thing that we did is create an overlay device. The -e vxlan option tells us that we should use vxlan for encapsulation. Currently only VXLAN is supported. The -s direct specifies that we should use the direct or point-to-point module for determining where packets flow. In other words, there’s only a single destination.
Following this we set three required properties. The vxlan/listen_ip which tells us what IP addresses to listen on. The direct/dest_ip which tells us which IP to send the results to, and finally, direct/dest_port which says what port to use. We didn’t end up setting the property vxlan/listen_port because VXLAN specifies a default port which is 4789.
Finally, we specified a virtual network id with -v, in this case 23. And then we ended it all with a name.
After that, it became visible in the dladm show-overlay which displayed everything that we wanted. You’ll want to take similar steps on another machine, just make sure to swap the IP addresses around.
Concluding
This is just the tip of the iceberg here. There’s going to be a lot more functionality and a lot more improvements down the road. I’ll be doing additional development previews along the way.
If you have any feedback, suggestions, or anything else, please let me know. You can find me on IRC (rmustacc in #smartos and #illumos on irc.freenode.net) or on the smartos-discuss mailing list. If you’d like to work on support for other encapsulation methods such as NVGRE or want to see how implementing a dynamic mapping service might be, reach out to me.