Some Important Things to Know about STP Loops

Cisco’s troubleshooting guide for STP loop detection is worth a read:

• Some links sustain a loop, some links simply carry the loop traffic. Shutting down the non-sustaining links will not break the loop.
• The main reason why loops may occur is because BPDUs are not received properly. (The document mentions two reasons, but it, IMHO, boils down to one reason).
• Typical symptoms of a loop are – high link utilization on boxes, protocol session flaps, high CPU utilization etc.
• Finding the loop and identifying the source of the loop involves – find the highly utilized links, understanding the topology, finding redundant paths in the network, making sure some of the paths are blocked (that is finding whether the switch knows the correct STP root – knowledge may not be consistent across the system because of lack of propagation of information),

What’s more, when a forwarding loop happens, its not just the network, hypervisor hosts too get hogged. As Ivan points out, hypervisors keep their NICs in promiscuous mode, so they end up punting every packet to the CPU! Hypervisor hosts typically have their NICs in trunk mode and BUM traffic from every VLAN would come hit the hypervisor.

Just by going through the troubleshooting guide, it gets pretty clear that STP is a mess and is at best avoided. No wonder, the network world wants to move to L3 fabrics.

VxLAN Use Cases

This IT 2.0 blog post on Typical VxLAN Usecases makes some interesting points about VxLAN:

• That it is not Layer-2 that has a problem getting extended across the data center, but Layer 3 routing and its architecture which limits the layer 2 broadcast domain under its head. Extending layer 2 across layer 3 domains will confuse layer 3 architecture. Moreover host discovery through ARP will not be possible.
• The basic difference between GRE (and likes) and VxLAN is that GRE requires configuration across all the network nodes to run tunnels, while VxLAN needs only software configuration that to inside a hypervisor. Without the network admins knowing it, layer 2 domains will get extended across VMWare nodes. This is significant.

Implications of Virtual Switches in the Data Center

Where do virtual switches reside? Inside the hypervisor. It interconnects all the VMs together. In case, a hypervisor is hosting VMs from different security zone together, then they may have to be segmented with VLANs inside the virtual switch.

A small note about this virtual switch – it does not participate in the STP protocol with outside world. It distinguishes between VM facing ports and outside facing ports (uplink ports – that is those that are connected to the NICs). It will never allow a loop to form by making sure that – traffic from uplink ports are not redirected again to uplink ports. It will also not allow incoming traffic from uplink ports with a source MAC of that of one of the residing VMs. VMWare’s vSwitch is stringent about this.

Once we get VLANs into picture, then the NICs on the physical server will have to have trunked ports. Which means VLAN trunking configuration is not clearly delineated at the border network switch of the data center, but it comes into the hypervisor as well. The vSwitch needs to be configured with all the VLANs that it may carry traffic for. This entails server and network admins to talk to each other and get to an agreement about which VLANs to use.

Apparently, Nexus 1000V virtual switch is released by Cisco to cater to this need. It sits inside the hypervisor and replaces the VMWare’s vSwitch (?). So, the network admin who is already familiar with the Cisco CLI will have no learning curve to configure and manage the vSwitch inside the hypervisor.

A nice article about the same.

Some Subtleties About vMotion

Ivan has a nice article about vMotion:

• Virtual Machines are in the order of GBs and moving them is not a simple operation. Across WAN this is significant traffic
• The requirement of retaining the sessions mandates the VM to retain the IP address even after movement. The traditional way to handle that is to still keep it in the same VLAN domain “under a router”.
• All switches under the router will have to be part of all VLANs
• The VLAN configuration have to be synchronized on all the servers and switches. How much of this config problem is solved by overlays
• VM’s communication with other services will not change – it will continue to flow and now the paths may be longer.