Host based routing - EVPN-VXLAN to the host - Single VXLAN Device (SVD)

[joolli]

Hey guys,

I want to try out host based routing with Cloudstack and I'm in the process of setting it up... One thing I notice is that the EVPN-VXLAN setup uses the "Traditional mode" of VXLAN setup, where there is a bridge and an associated vxlan device. The new way of doing it is the SVD (Single VXLAN Device) where there is a vlan-aware bridge and a vlan-aware VXLAN device and there is no need for additional devices, just add a VLAN to both of those devices and you have another VNI and add VMs to the vlan aware bridge and associate them with the VLAN, just like a switch. Scales better and less clutter in "ip addr show".

Has anyone tried this method by adjusting the modifyvxlan.sh script to accommodate that type of setup?

[wido]

Interesting! I have not tried it yet. Do you have some example commands of "ip" and "bridge" on how to do this?

I know that vlan-aware exists, but I wasn't aware you can do this with VXLAN as well.

[joolli]

Sure... Here are the iproute2 commands to create that type of setup with one namespace (to imitate a VM) on VNI 10330:

Create the vlan-aware bridge, add a VLAN (330) to it and MAP a VNI (10330) to the VLAN.

ip link add dev br0 type bridge
ip link set dev br0 type bridge vlan_filtering 1
bridge vlan add vid 330 dev br0 self
ip link add dev vxlan0 type vxlan external local [LOOPBACK ADDR] dstport 4789 nolearning
ip link set dev vxlan0 master br0
bridge link set dev vxlan0 vlan_tunnel on
bridge vlan add dev vxlan0 vid 330
bridge vlan add dev vxlan0 vid 330 tunnel_info id 10330

• Add an SVI if you need one:
  ip link add link br0 name vlan330 type vlan id 330 protocol 802.1q
• Then you would add an SVI IP, otherwise you probably wouldn't need the SVI.
• And since you wanted an SVI, you might want an anycast gateway, using macvlan interface, in tandem to the SVI.

Create a "cable" (veth) to plug into the switch and the namespace

ip link add veth-a type veth peer name veth-b
ip link set dev veth-a addrgenmode none # not necessary but prettier for ip addr show
ip link set dev veth-a up

Plug the A side of the "cable" into the "switch" (vlan-aware bridge).

ip link set dev veth-a master br0
bridge vlan add vid 330 dev veth-a pvid untagged
bridge vlan del vid 1 dev veth-a

Create the namespace and plug the B side of the "cable" into the namespace:

ip netns add fakevm
ip link set veth-b netns fakevm
ip netns exec fakevm ip link set dev lo up
ip netns exec fakevm ip link set dev veth-b up
ip netns exec fakevm ip addr add 10.1.1.2/24 dev veth-b
ip netns exec fakevm ip route add default via 10.1.1.1

[wido]

Interesting. So you add a virtual switch (veth) to the bridge and you also tell how to map the VLAN to a VNI. In your case you use 330 and 10330, but in the CS case these would be identical.

How would this work with libvirt? Because right now we use an XML definition where we just plug the VM into a bridge.

In this it seems we also need to know the VNI when creating the VM, this means you would need to make more adjustments to CloudStack.

And my last question: Super cool, but what is the true benefit? Performance? Scale?

[joolli]

That bridge (br1) is created before the VM, the same would apply to the VLAN to VNI mapping. So all you need is the VLAN information, which would need to be included in the XML. You can think of the creation of the bridge and vxlan in the traditional mode as the following, in the Single-VXLAN-Device method:

bridge vlan add dev vxlan0 vid 330
bridge vlan add dev vxlan0 vid 330 tunnel_info id 10330

All the other commands above those are only done once.

[joolli]

So these are the commands that would be needed to be executed for the VM to be a part of the VNI:

Created on host startup

ip link add dev br0 type bridge
ip link set dev br0 type bridge vlan_filtering 1
ip link add dev vxlan0 type vxlan external local [LOOPBACK ADDR] dstport 4789 nolearning
ip link set dev vxlan0 master br0
bridge link set dev vxlan0 vlan_tunnel on

Created by CloudStack on new network creation (I assume, haven't accessed the GUI yet)

bridge vlan add dev vxlan0 vid 330
bridge vlan add dev vxlan0 vid 330 tunnel_info id 10330

• This would be taken care of by the modifyvxlan.sh script.

To make the VM a part of the VLAN (thus the VNI)

ip link set dev vnet7 master br0
bridge vlan add vid 330 dev vnet7 pvid untagged
bridge vlan del vid 1 dev vnet7

[wido]

Got it. But the last part is tricky, as we don't know the vnetX device of the VM while we execute modifyvxlan.sh.

[joolli]

The last part is taken care of by a libvirt hook script. So what Cloudstack needs to do is add a few lines to the xml.

  <metadata xmlns:vlc="https://github.com/modzilla99/libvirt-vlans">
    <vlc:vlanconfig>
      <vlc:iface pvid="330">
        <vlc:vlan untagged="yes">330</vlc:vlan>
      </vlc:iface>
    </vlc:vlanconfig>
  </metadata>

Hook script:
https://github.com/modzilla99/libvirt-vlans

The metadata is processed top down and the interfaces as well. So the first interface will get the first vlan definition.

[joolli]

One benefit is that you have the possibility to add multiple vlans to an interface... Maybe not a big one, as not many are likely to use it but, it's there.

[joolli]

I'll leave this here...

https://docs.nvidia.com/networking-ethernet-software/knowledge-base/Configuration-and-Usage/Network-Configuration/Calculating-the-Limitation-of-a-Linux-Bridge-in-Traditional-Mode/

[wido]

Yes, good! So I'm not against this change, but somebody needs to implement it :-)

[joolli]

Didn't take it as you're against the change. They were good questions that helped me describe the setup so others will hopefully understand how it's different from traditional mode. So thank you.

I will add to modifyvxlan.sh for it to accommodate SVD mode... Can someone point me to the XML generation code?

[DaanHoogland]

Do you mean https://github.com/apache/cloudstack/blob/2df1ac51068220aeac75c9d2365af6dc382d59ab/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java , @joolli ?

[joolli]

Thank you! That's it...

I checked if vlan support for Linux bridge has been added in libvirt, recently, and it has. So the hook scripts and the metadata XML clause won't be needed, only the support for <vlan> under <network>.
https://libvirt.org/news.html#v11-0-0-2025-01-15

[joolli]

@wido I have added to your script the possibility of using SVD mode. It's not intended as a final solution but it's comparable with the current solution, which is designed for a libvirt that doesn't support VLANS for Linux Bridges. I don't have a lab using the traditional mode, so I haven't tested to see if my changes break that type of setup... It shouldn't, as I didn't change those functions, but better test it. Would you be willing to try it out?

https://github.com/joolli/cloudstack/blob/evpn-vxlan-svd-mode/scripts/vm/network/vnet/modifyvxlan-evpn.sh

[joolli]

@bhouse-nexthop Oops, forgot to push it. Sorry about that. Sorted now.

[bradh352]

wow, that's some awk there. I see you're using bridge -j, think about using 'jq' instead?

Anyhow other notes:

• When generating the vlan to vni map, shouldn't you restrict it to only the bridge you're interested in? Couldn't there be other bridges on the system that should be ignored?
• If the VNI already exists I see you pretty much just return. But shouldn't you instead extract the VLAN its mapped to and call bridge vlan add vid ${Vlan} dev ${VETHPEER} ? I'd think you could have dozens of virtual machines on a host attached to the same VNI so that needs to be supported.
• Similar comment on delete, can only delete the VNI/VLAN once the count of VETH associated with it hits zero.

[joolli]

Yeah, jq was much easier but it is not installed with cloudstack and I didn't want to create another dependency.

There can be only one bridge with the SVD according to Cumulus... I ended up doing it this ways, thinking it would be redundant to include the interface... In any case... I don't see a case where you would want to use the same VNI on a different vxlan dev.

No, the tagged veth device is a connection between the vlan-aware-bridge and the vlan-unaware-bridge where the VM devices are placed, so the bridge (VUB) and the tagged veth is only created once and then deleted after the last VM is removed from the bridge.
Do I understand your concern?

[bradh352]

oh really? I guess I made some incorrect assumptions without actually looking under the covers of what is happening.

How then with that model can you send multiple vxlans to the vm over a single interface? Then potentially a different VM wants to use an overlapping subset of vxlans but not the entire set. I thought that was the ultimate goal of this change.

[joolli]

This change is to accommodate those that use the SVD type of setup on their hosts. It does not support multiple vlans on a single VM interface... You would need another interface for the VM. However the SVD type of setup does support it if libvirt would place the VM's interface on the vlan-aware-bridge and Libvirt recently got support for that. The cloudstack xml code doesn't have support for that yet.

I don't expect this to be used for long in it's current image, as cloudstack might get support for vlan-aware-bridges and then the script will not need the "addVlan" and "deleteVlan" functions, as libvirt will place the VM's directly on the VA-bridge, but the script is very useful in it's current form to transition from a traditional mode setup to SVD or the other way around because you need a bridge with the same name to be able to migrate the VM's.

[bradh352]

I'm back with playing with this. I've switched to using systemd-networkd instead of netplan, but I haven't had any luck so far getting this online. Using your manual ip/bridge commands does indeed work, however, on Ubuntu 24.04. Interestingly one difference I noticed was ip neigh doesn't any vxlan evpn learned neighbors at all, I'm used to seeing them as extern_learn zebra entries, so at first I thought it wasn't working.

I'm still researching though, I'll share if I get a working networkd setup.

[bradh352]

So it looks like SystemD 258 (not yet released) has added some SVD support, it doesn't have all of it. See thread on systemd/systemd#36793 (comment)

That's unfortunate, as it means not even the upcoming fall distro releases will support this.

[bradh352]

opened a feature request as they said: systemd/systemd#38581

[joolli]

You should see the zebra extern_learn entries just as with traditional bridge mode.

For cloudstack, you don't need systemd to map a vlan to a VNI unless you want to have some vlans defined outside of cloudstack. The modifyvxlan script should take care of creating the vlan and the mapping between that vlan and a vni.

[bradh352]

I didn't neighbors at all for vxlan, but also in your example that I tested, I didn't see vnifilter next to external when adding vxlan0, which from what I'm reading I think should be there. I haven't tested it. Traffic worked, so the ip neigh thing was curious. I'm might tear down the network on a node and retest. I was really hoping I could do everything with systemd-networkd though.

Regarding vlans outside of cloudstack, yes I'm only L3 and use bgp unnumbered for peering, so for the host to be reachable, I have to break out vxlans and assign addresses for which the host needs to reside (public, management, and storage networks are what I break out).

[joolli]

I missed your answer @brad352, sorry. What do you mean by break out VXLANS? Can you show me the configuration (maybe with some comments so I can understand better)?

I also use BGP-unnumbered and do L2 over L3. Public, storage and guest traffic are all on their own VNI's outside of the guest VNI range.

vnifilter is not needed. vlan_filtering (on the bridge), vlan_tunnel (on the vxlan dev) and no vxlan id specified (vxlan id 0 on the vxlan dev) are the settings for SVD mode.

[Ilyaemma]

Is it alive?

[DaanHoogland]

I heard some one talking about it recently, but no concrete plans. (cc @rajujith ?)

[rajujith]

No plans for SVD yet, unless I missed something.