Blog Archives

FHRP (First Hop redundancy Protocol) best practices in Nexus 7K

vPC (virtual port channel) allows two Nexus switches to share port-channel Attached devices believes that they are connected to a single device via ether-channel bundle so that STP treat them as a single link. To achieve this, the paired Nexus switches uses to different communication channel. The first link is called Peer-Keepalive link which is a continuous L3 UDP ping going between them and use to detect the presence of peer.


Second link is called Peer-link, it is a L2 connectivity and mainly use for control traffic between switches.

When running HSRP between Nexus, by default Nexus switches will work in active/active mode despite of its configured role i.e. if a frame received on standby switch it will not forward it to Active HSRP switch but forward itself. This behavior of HSRP is tweaked specially for vPC optimization.

However till this point everything is good but then some storage and data center equipment manufacturers like NetApp, EMC, F5 load balancers etc thought it would be good idea to optimize their handling of Ethernet Frames. Some NetApp and EMC equipment ignores ARP reply given by HSRP primary and instead forward Ethernet frames to whichever MAC address it receives frames from. NetApp called this “Fast Path Source Mac address caching”. It is a nonstandard behavior.

So what is wrong with this vendor optimization. According to Cisco, “Packets reaching a vPC device for the non-local router MAC address are sent across the peer-link and could be dropped by the built in vPC loop avoidance mechanism if the final destination is behind another vPC.”. Because of this at the application level we saw very poor performance due to these dropped packets.  Enough of the packets got through to allow access to the storage device, but file load times were measured in the tens of seconds, rather than milliseconds.

So How Peer –Gateway help :

Configuring peer-gateway will allow the nexus switches to route frames which are destined to the mac address of their peer device.  Only exception is if a packet is destined to both the physical mac of the peer and the physical ip address.  Under that circumstance the packet will be tunneled across the peer link.

Configuring Peer-Gateway :

Configuring the peer-gateway feature needs to be done on both primary and secondary vPC peers and is non-disruptive to the operations of the device or to the vPC traffic. The vPC peer-gateway feature can be configured globally under the vPC domain submode.

When enabling this feature it is also required to disable IP redirects on all interface VLANs mapped over a vPC VLAN to avoid generation of IP redirect messages for packets switched through the peer gateway router. When the feature is enabled in the vPC domain, the user is notified of such a requirement through an appropriate message.

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

Therefore Peer-Gateway should be enabled when dealing with nonstandard behavior of date center devices available. To enable this feature configure on both pair of Nexus switches as following  :

switch# config t

switch(config)# VPC domain <domain-id>

switch(config-vpc-domain)# peer-gateway