Maximum packet sizes on the internet (posted 2014-12-17)

After some heated discussions about packet sizes on the mailinglist of the IETF v6ops working group, I decided to do some measurements to find out what maximum packet sizes are supported on today's internet. I did this by capturing two types of packets: the ICMP "too big" messages that routers send to tell a computer to send smaller packets, and the first packet of a TCP session, which contains the MSS option. The maximum segment size (MSS) option is used in TCP sessions to tell the other side what the maximum packet size is that we can receive. This depends on the maximum transfer unit (MTU) of the hardware, which may be further reduced by system administrators.

The Ethernet standard uses an MTU of 1500 bytes, although a lot of Ethernet hardware can support more, such as 9000-byte "jumboframes". Wi-Fi also uses 1500 bytes to be compatible with Ethernet. However, sometimes one protocol needs to be tunneled over another protocol, such as IPv6 over IPv4 (over Ethernet) or PPP over Ethernet, which reduces the supported packet size to 1480 or 1492, respectively. The IPv6 specifications require that a minimum MTU of 1280 bytes is supported. IPv4 has no minimum MTU. Note that all of this is about the maximum packet size, it is of course perfectly fine to send smaller packets.

TCP (and UDP) use segments which are put inside IP packets that are then transmitted inside Ethernet frames. A 1500-byte IPv4 packet supports 1460-byte TCP frames (1500 bytes minus the 20-byte IPv4 header and the 20-byte TCP header). This 1500-byte IP packet is transmitted as a 1518-byte Ethernet frame, although some people only count 14 bytes for the Ethernet header, ignoring the 4-byte checksum that's at the end of the Ethernet frame. Because the IPv6 header is 40 bytes, a 1500-byte IPv6 packet can hold a 1440-byte TCP segment, while a 1500-byte IPv4 packet can hold a 1460-byte TCP segment. I'll be talking about IP MTU sizes rather than segment/MSS sizes to make it easier to compare IPv4 and IPv6 results.

Over the better part of a week, my server received 41753 incoming TCP SYN packets with an MSS option on port 80. Another 140 packets didn't have the MSS option, and looked like they were mostly TCP-based traceroute packets. 24246 packets were IPv4 packets, coming from 4164 unique IP addresses. 17507 were IPv6 packets, which I reduced down to 227 unique IP addresses. Turns out, most of the IPv6 traffic on my server is from bots that check if I've added any new content to the site. Some of them use the same address each time, but others keep using different addresses, but, strangely, the same source port number (12000 - 12006). I removed these addresses to keep them from drowning out the real data.

The data showed no fewer than 72 different MTU sizes for IPv4, ranging from 576 to 9198 bytes. However, both of these extremes only showed up once, and other values below 1280 and above 1500 are also quite rare:

• 576: 1 (0.02%)
• 1200: 2 (0.05%)
• 1240: 4 (0.10%)
• 1242: 1 (0.02%)
• 1252: 4 (0.10%)
• 1272: 1 (0.02%)
• [...]
• 1504: 4 (0.10%)
• 3000: 1 (0.02%)
• 9000: 5 (0.12%)
• 9001: 47 (1.13%)
• 9198: 1 (0.02%)

I found the 9001 value quite curious; computers really like to work on nice round multiples of 2, 4 or 8 bytes. 9001, on the other hand, is a prime number. Turns out that 9001 bytes is used Amazon's datacenters, where some of the bots that index my website reside. These are the more common MTU sizes advertised in the TCP MSS option:

• 1300: 46 (1.10%)
• 1400: 78 (1.87%)
• 1420: 152 (3.65%)
• 1440: 99 (2.38%)
• 1460: 96 (2.31%)
• 1470: 233 (5.60%)
• 1492: 370 (8.89%)
• 1500: 2672 (64.17%)

1300 and 1400 look like someone set them manually; 1300 is also a common VPN MTU. 1440 bytes seems to be hardcoded in some home routers. 1460 could indicate IPv4-over-IPv6 tunneling. 1470 seems to be used by a number of broadband ISPs and 1492 results from PPP over Ethernet (PPPoE). Last but not least, just under two thirds of IPv4 visitors support the Ethernet MTU of 1500.

These are the results for IPv6 with the < 1% values removed (there were no values below 1280 and above 1500):

• 1280: 13 (5.73%)
• 1426: 3 (1.32%)
• 1428: 9 (3.96%)
• 1446: 14 (6.17%)
• 1472: 19 (8.37%)
• 1480: 5 (2.20%)
• 1492: 14 (6.17%)
• 1500: 142 (62.56%)

1280 and 1480 are probably IPv6-in-IPv4 tunnels and 1428 AYIYA tunnels. 1472 could be IPv6-in-UDP-in-IPv4 tunnels or IPv6-in-IPv4-over-PPPoE. The image below shows the cumulative frequency of MTU sizes for both IPv4 (red) and IPv6 (blue), where the line shows how many systems support a given MTU value, starting at 99.98% for 1200 and ending at 65.56% for 1500 (for IPv4).

The 90th percentile MTU size is 1428 for IPv6 and 1440 for IPv4. Obviously 100% of IPv6 systems support 1280, but 99.7% of IPv4 systems also support this size.

The MSS reflects the maximum size that the systems at both ends of a connection think they can use. However, there may be a bottleneck somewhere along the path. In that case, routers send back an ICMP Packet Too Big message. Tomorrow, I'll look at those.