Baby Jumbo Frames (RFC 4638) with igb-based NICs on pfSense
Baby Jumbo Frames (as defined in RFC 4638) are a pretty useful feature. In short, they allow you to have a standard 1500 byte MTU on PPPoE connections. Normally, due to the 8 byte PPPoE header the connection is reduced to a MTU of 1492 bytes. Baby Jumbo Frames allows for the MTU of the underlying interface to be increased to 1508 bytes, and thus after the PPPoE overhead you’re left with a standard 1500 byte MTU connection. Admittedly, the performance gains are minimal but that’s no fun!
In order to support Baby Jumbo Frames, both devices must support them. I’m using a pfSense based router and the interfaces support Jumbo Frames. In theory, by manually setting the PPPoE connection’s MTU to 1500 bytes in pfSense, I should be able to use Baby Jumbo Frames but inexplicably my connection was always negotiating an MTU of 1492 bytes. I knew that my ISP (Zen) and modem (ECI B-FOCus) supported Baby Jumbo Frames and I had successfully used them in the past with an OpenWRT based router, so why couldn’t I get them to work now?
My WAN NIC is the Intel i211, and thus the Intel igb driver is used. Reading over Intel’s documentation for their igb driver however, I noticed an interesting line;
– Using Jumbo frames at 10 or 100 Mbps is not supported and may result in poor performance or loss of link.
Both of the commonly supplied VDSL modems in the UK (ECI B-FOCus and Huawei HG612) only have a 10/100mbit connection – you cannot negotiate a gigabit connection. So it seems there’s two conflicting things here;
- NICs using the igb driver do not support Jumbo Frames (including Baby Jumbo Frames) at 10/100mbit speeds.
- Common VSDL modems won’t support speeds other than 10/100mbit.
There’s an easy solution though. I put a spare unmanaged gigabit switch between my pfSense router and VDSL modem. The igb NIC in my pfSense router now negotiates a gigabit connection to the switch, and from the switch to the VSDL modem a 100mbit connection is negotiated. As the igb NIC is now working at gigabit speeds, it can use Baby Jumbo Frames and the PPPoE connection now correctly negotiates the standard 1500 byte MTU size as confirmed by ifconfig . Success!
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
Of course, we can’t declare victory just yet until we’ve verified that we can successfully send 1500 byte frames without fragmentation. This is easily checked with a simple ping though;
ping -f -l 1472 github.com Pinging github.com [18.104.22.168] with 1472 bytes of data: Reply from 22.214.171.124: bytes=1472 time=90ms TTL=51 Reply from 126.96.36.199: bytes=1472 time=90ms TTL=51 Reply from 188.8.131.52: bytes=1472 time=93ms TTL=51 Reply from 184.108.40.206: bytes=1472 time=91ms TTL=51 Ping statistics for 220.127.116.11: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 90ms, Maximum = 93ms, Average = 91ms
The size was set to 1472 bytes as the ICMP and Ethernet headers combine to form an overall Ethernet frame size of 1500 bytes. The -f option forces Window’s ping command to not fragment packets (Linux uses -M do and FreeBSD/pfSense -D for their ping commands).
Thanks for this, I’ve been battling with getting baby jumbos working on my current ISP (AAISP) on pfsense. (see here: https://forum.pfsense.org/index.php?topic=126011.0 )
Unfortunately it still isn’t working correctly (despite a dumb 1gb switch between the openreach modem and pfsense) It worked fine for a few hours then seemed to revert to 1492.
Anyway, the main reason for my post – did you have to do anything else with zen to get baby jumbos to work? I am moving house and considering moving to zen as my isp. I contacted them on the subject of baby jumbo frames, but they said that it wasn’t supported and wouldn’t work – saying that the maximum MTU will always be 1492.
Is this just a case of the person I’m speaking to not really knowing, or do you have to have a special service for it to work?
I didn’t need to do anything beyond what I mentioned in this post – Zen didn’t need to enable anything on their end. I think that this works on any FTTC ISP here – the support for it is in the Openreach network rather than at the ISP themselves but I might be wrong. Other Zen users (including the original author of RFC 4638 support in pfSense) have been able to get it to work ‘as is’.
Even with the solution in this post, I did see my connection negotiate down to 1492 briefly but after a resync it returned to 1500 again and has remained there since. It might be worth trying to leave the connection alone for a few days, then restarting your pfSense box to see if it negotiates a 1500 byte MTU? Fingers crossed!
Thanks, I’ll give it another reboot and see if that spurs it back into life.
Still doing the same thing unfortunately. Do you know which openreach modem you have?I wonder if that has anything to do with it?
I’m actually wondering now if my “dumb” netgear gs605 v1 switch is a bit too dumb, as it doesn’t say it supports jumbo frames. Will see if I can try something that does say it is supported.
Longer term I might get a Draytek 130 to use as my modem, which has a gig port anyway.
For ref, I have a DrayTek 130, Zen ADSL and pfSense. I haven’t managed to get baby jumbo frames to work either, pfSense tries to negotiate and fails and sits at 1492. I’ve not tried putting a switch in between pfSense and the Draytek because as you’ve identified its got a Gigabit port, however when KingJ and I discussed it we did wonder if it’s down to the fact I’m a standard ADSL user with Zen rather than FTTC. I’ve not investigated this too heavily but thought might be worth mentioning. I might contact Zen and get their thoughts as their customer service is pretty great.