How much slower is Linux SRv6 H.Encaps without tunsrc?

· srv6, linux, network, frr

I have been implementing the SRv6 Mobile User Plane (RFC 9433) behaviors in the Linux kernel, and I am measuring them with SRPerf to see whether they keep up with the existing SRv6 behaviors. SRPerf is the measurement framework from the paper SRPerf: A Performance Evaluation Framework for IPv6 Segment Routing, which benchmarked the SRv6 behaviors of Linux and VPP.

The paper says that H.Encaps gets slower if the tunnel source address (tunsrc) is not configured. It does not say how much slower, so I measured it.

What the SRPerf paper says about TUNSRC

The description of the experimental setup in the paper reads:

We configured the TUNSRC for the SRv6 policy headend behaviors doing encapsulation. The latter allows to configure the IPv6 source address of the IPv6 outer header. The TUNSRC has to be configured otherwise the Linux kernel will try to get the address from the interface which will cause a performance drop in the performance of the encaps behavior.

The authors measured with tunsrc configured, but the paper gives no number for the drop. Note that the published SRPerf forwarding-behaviour.cfg does not configure tunsrc, so you need to add it yourself to measure under the same conditions as the paper.

What happens without tunsrc

The source address of the outer IPv6 header is chosen by set_tun_src() in net/ipv6/seg6_iptunnel.c (v7.1 source).

	if (route_tunsrc && !ipv6_addr_any(route_tunsrc)) {
		memcpy(saddr, route_tunsrc, sizeof(struct in6_addr));
	} else {
		rcu_read_lock();
		tun_src = rcu_dereference(sdata->tun_src);

		if (!ipv6_addr_any(tun_src)) {
			memcpy(saddr, tun_src, sizeof(struct in6_addr));
		} else {
			ipv6_dev_get_saddr(net, dev, daddr,
					   IPV6_PREFER_SRC_PUBLIC, saddr);
		}

The kernel looks at the per-route tunsrc (since Linux 7.1) and then the per-network-namespace tunsrc (the one set with ip sr tunsrc set). If both are ::, it calls ipv6_dev_get_saddr(). tunsrc defaults to ::, so unless you configure it, the kernel compares the addresses on the device against the RFC 6724 rules for every packet. This is what the paper means by “the Linux kernel will try to get the address from the interface”.

Measuring with and without tunsrc

I followed the paper’s method: the SUT (the machine under test) forwards on a single core, and I measured PDR (Partial Drop Rate, the highest offered rate with at most 0.5% packet loss) and MRR (Maximum Receive Rate, the receive rate when sending at line rate), 10 runs of 10 seconds each. The SUT has a Xeon E5-2650 v3 (2.30 GHz) and is directly connected to the tester running T-Rex with Intel 82599ES (10GbE) NICs. The IOMMU is in passthrough mode (iommu=pt).

With the same kernel and the same configuration, I only switched tunsrc between :: and 12:2::2. 12:2::2 is the address the kernel picks when tunsrc is ::, so the only difference between the two runs is how the source address is chosen. For comparison, I also measured plain IPv6 and H.Insert (inline), which do not build an outer IPv6 header and therefore never reach set_tun_src().

without tunsrcwith tunsrcchange
plain IPv61196.01192.2-0.3%
H.Insert998.31001.3+0.3%
H.Encaps.V6569.8967.2+69.7%
H.Encaps.L2445.2656.2+47.4%

(MRR, kpps)

plain IPv6 and H.Insert barely moved, and only the two encap behaviors improved. Without tunsrc, H.Encaps.V6 was 41% slower and H.Encaps.L2 was 32% slower. H.Encaps.L2 also has a separate slowdown unrelated to tunsrc, and these numbers were taken on a kernel without that fix.

Converting PDR to cycles per packet (2.30 GHz / PDR), the difference is 1660 to 1670 cycles for both H.Encaps.V6 and H.Encaps.L2, which is the cost of the source address selection alone. This cost may depend on how many addresses the device has, which I did not measure.

With tunsrc configured, H.Encaps.V6 reached a PDR of 976 kpps, almost the same as the 978 kpps in the paper.

How to configure tunsrc

With iproute2:

# per network namespace
ip sr tunsrc set 2001:db8::1
ip sr tunsrc show

# per route (Linux 7.1 and iproute2 7.1 or later)
ip -6 route add 2001:db8:100::/64 encap seg6 mode encap tunsrc 2001:db8::1 segs fc00::1 dev eth0

With FRR, put the following in the configuration (frr.conf):

segment-routing
 srv6
  encapsulation
   source-address 2001:db8::1

FRR does not set tunsrc unless source-address is configured. I checked this by starting FRR: with only an SRv6 locator configured, ip sr tunsrc show stayed at ::, and after adding source-address it showed that address. bgpd’s SRv6 L3VPN does not set a per-route source address either, so without source-address you get the slower path.

References

Assisted-by: LLM