ccie blog

MTU Values And Ping Sizing On Cisco Products

This is an deep dive into understanding the layer 2 and layer 3 MTU settings, as well as understanding what MTU value is actually being used when you are sending pings with different sizes.  There’s more to it than actually meets the eye on the CLI.

I’ve strung a couple of switches together to setup test environment.  The 10 gig interfaces will be set to use an MTU of 9216, then I’ll discuss what this means at layer 2 and 3.

6500-to-6880

Configuration

6500
interface TenGigabitEthernet1/1
no switchport
mtu 9216
ip address 1.1.1.1 255.255.255.0
6880
interface TenGigabitEthernet1/1/1
no switchport
mtu 9216
ip address 1.1.1.2 255.255.255.0

As you can see, both sides have their layer 2 MTU value set to 9216.  This value is actually the maximum allowed size of the Ethernet frame before the layer 2 headers and FCS value (i.e. the CRC check) are are added.

A traditional 802.3 datagram is 1518 bytes large, which would break down into:

  • Destination address = 6 bytes
  • Source address = 6 bytes
  • Type/Length = 2 bytes
  • FCS = 4 bytes
  • Data/Payload = 1500 Bytes

However additional fields can be added to expand the size of the layer 2 frame above the traditional 1518 bytes, such as: VLAN tagging (4 bytes), QinQ (4bytes), ISL trunks (30 bytes) etc.  It just appears that Cisco have taken these values out of the equation, making it much more simple to configure the MTU value.  Moral of the story is: When setting the MTU at layer 2, it’s exclusive of the layer 2 headers and CRC.

What’s interesting on Cisco devices is that configuring the layer 2 MTU, it also automatically adjusts the layer 3 IP MTU value; as shown in the output below.

6500#sh ip int te1/1 | i MTU
  MTU is 9216 bytes
6880#sh ip int te1/1/1 | i MTU
  MTU is 9216 bytes

That’s a bit confusing right? Considering that the the IP header is 20 bytes, you would have thought that the IP MTU value would actually be 9196.  However what Cisco devices do, is add the 20 byte IP header and 9196 byte payload into the total configured layer 3 PDU size. So when you see the IP MTU is set to 9216 byes, this is the total layer 3 packet size (i.e. the payload + the IP header, as a combined value).

Cisco Pings

This methodology is also carried over to their ping requests.  So when I send an icmp echo between the devices with a size of 9216 bytes, this is the total layer 3 PDU size as it is encapsulated into a layer 2 frame. In the ping below, I’m sending a total layer 3 sized PDU (inclusive of the IP and ICMP header).

#ping 1.1.1.2 size 9216 df-bit
Type escape sequence to abort.
Sending 5, 9216-byte ICMP Echos to 1.1.1.2, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

So let me elaborate on what I mean by breaking down the packet structure for this ping:

  • Ethernet Frame size (exclusive of header and CRC) = 9216 bytes
  • IP MTU size = 20 bytes
  • ICMP header = 8 bytes
  • ICMP payload = 9188 bytes

So actually the ping size is 9188 bytes.  But when you issue the ping command, and set the size to 9216, this actually defines the total ping size inclusive of ICMP and IP headers.  You absolutely need to be aware of this for Cisco products. If you were to send a ping with an adjusted size in operating systems such as Linux, Microsoft Windows, and Juniper, they all actually refer to the to the ICMP payload size.  For example.  I am on a Windows laptop at the moment, connected hard wired into a switch that has the interface IP MTU set to 1500 bytes. If I ping the switch with the df-bit set, at size 1472 it works, but at 1473 it fails:

C:\>ping -l 1472  10.15.0.254 -f -n 1

Pinging 10.15.0.254 with 1472 bytes of data:
Reply from 10.15.0.254: bytes=1472 time=3ms TTL=255

Ping statistics for 10.15.0.254:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 3ms, Average = 3ms
C:\>ping -l 1473 10.15.0.254 -f -n 1

Pinging 10.15.0.254 with 1473 bytes of data:
Packet needs to be fragmented but DF set.

Ping statistics for 10.15.0.254:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss)

So this would be broken down into:

  • Ethernet Frame size (exclusive of header and CRC) = 1500 bytes
  • IP MTU size = 20 bytes
  • ICMP header = 8 bytes
  • ICMP payload = 1472 bytes

 

Summary:

The #mtu 9216 value sets the maximum possible layer 2 payload (i.e. exclusive of layer 2 headers and CRC check).

The #ip mtu 9216 value sets the maximum possible layer 3 PDU (i.e. inclusive of the IP header).

In my topology these are set to the same value.  However, there will be times where the IP MTU will need to be lowered to accomodate for things like GRE or IPSec headers.  The layer 2 MTU will never change, but the IP MTU value certainly will need reducing for things like tunnels.

Leave a comment

Your comment