Right – this might be a bit long.. I haven’t yet worked out how to make a ‘short but sucint’ blog post..

I had one of those ‘tada‘ or ‘eurika‘ or ‘bloody hell, why didn’t that occur to me earlier’ moments this afternoon (it was the latter).  You know what it’s like:- you’re at the end of your tether trying to get something to work, you’ve been fumbling around for hours, and out of the corner of your brain a little flicker resembling a thought process occurs.  You give it a moment to surface, chew on it and then ‘oh hell, I’m an idiot’ as you prove you’ve fixed your own mess of a problem. This happened while staring at a multicast convergence problem today – and it was all due to bad CoPP.

CoPP – or Control Plane Policing – is regarded by Cisco as a security feature/mechanism. It’s designed to protect the switch’s CPU from being overwhelmed by control-plane traffic (whether that traffic is legitimate, accidental or the likes of a DoS attack).  The Catalyst 6500 had it – but no-one ever seemed to configured it. In the Nexus 7000 and all new NX-OS based switches, it’s a default configuration (unless you’re a monkey and choose ‘none’ during the startup script).

CoPP is configured using MQC and allows you define classes of traffic that might head to the CPU and apply a policing policy to it..  Usually, the default policies (lenient, moderate or strict) are fine for most network deployments, say 80% of them.  For the other 20% you have to tweak the policy a little.

Case 1 Bad CoPP – A common [and in my opinion, stupid] method for a server detecting the loss of a gateway is by using ARP. In the old days, this wasn’t a problem – the 6500 would just churn through the ARPs and spit out responses, perhaps missing one or two. When a customer decided to test the Nexus 7000, they found that their servers kept seeing gateway losses.  Turns out, the default policy was being exceeded by the sheer number of servers sending out ARPs. The customer moaned. Of course they would, they think the switch is bad and broken and.. anyways. So to get over the [stupid stupid idea] the CoPP is tuned to allow more ARP up to the CPU.  It’s a solution to a problem that’s easy to implement, rather than fixing the fact that your servers don’t need to ARP for the gateway as you have HSRP (but hey, who am I to argue with the customer?).

Case 2 Good CoPP – A customer has a setup where they need a fast multicast-convergence time but are also receiving the same (S,G) streams on two different interfaces.  Fast multicast-convergence means we need to register the multicast frames with the RP as quickly as possible, so the RP can learn and then (as we happen to be using PIM Anycast-RP) relay the PIM registers to other RPs.  For this, we can increase the policing of PIM protocol messages (the default was 200pps, so we upped it to 600pps).  This is fine, we’re just allowing the policy to scale upwards.

The trouble of balancing the CoPP came with the (S,G)s being received on two interfaces.  In multicast we can only have one interface being the incoming interface – this incoming interface is determined by the RPF check (reverse path forwarding) and is programmed into hardware.  Once programmed, all matching multicast on that interface is forwarded in hardware.  The (S,G)s being received at the non-RPF interface would not match a hardware (FIB) entry – and thus would be forwarded up to CPU for software processing.  The problem with this is we’re punting useless traffic up to the CPU, wasting CPU resources and preventing that inband bandwidth for being used for other things (such as that fast multicast convergence).  When testing this initially, I identified IPMCMISS as the class which this useless, RPF-failing, traffic was hitting in CoPP and trimmed it right back to 10pps.  When I went to do another convergence test, I found that convergence was super-slow, even though I had tuned the PIMREG upwards.

What I didn’t realised was that IPMCMISS doesn’t actually [just] match RPF-failing traffic – it actually matches any multicast traffic that triggers a ‘FIB-miss’ – this was the ‘bloody hell’ moment.  Whenever we receive multicast traffic into hardware, and there’s no hardware-programmed FIB entry, it’s a FIB-miss – and this is punted to CPU for processing or software switching.  FIB-miss would be triggered the first time we see an (S,G), which is how we get into the process of punting to CPU, PIM learning, inserting into MRIB, generating a PIM Register and programming the hardware. So by cutting away the bandwidth available to IPMCMISS, I was also reducing the chance of new (S,G) frames making it to the CPU for learning.

So to summarise – I now I have to work out how to balance policing of the useless traffic and wasting CPU bandwidth against the need to learn new (S,G)s.  I would never condone opening CoPP up for something like ARP, it sounds silly to me.

The end.

PS – [Just an updated thought] – It’s worth noting that on the N7000 you can define different class-maps for IPMCMISS and RPF-failing traffic, you can’t yet do this on the N3000, and I have yet to check the N5x00.
class-map type control-plane match-any mc-rpf-fail
match exception multicast rpf-failure