Puppet Apprentice

Puppeteer

As I move my career towards the world of DevOps and automation / orchestration, I feel I need to learn the latest tools of trade for these fields. So, to that end, I'm installing the free version of Puppet Enterprise as a proof of concept. I've heard many people tell me how fantastic a tool Puppet is but haven't actually played with it.

So here I go.

For those wondering, there isn't technically any difference between the enterprise version and the normal puppet version but there are plenty of add ons that require the enterprise version (so I believe). The enterprise version will allow control of up to 10 nodes for free which is more than enough for a proof of concept.

I'm also hoping to build Puppet into a larger story about automation and orchestration.

Just to clarify, if you don't know the difference between automation and orchestration or think they mean the same thing here's the useful definition:
Automation is like musicians in an orchestra, looking the music and following the instructions, the orchestrator is the person up the front conducting, making sure everyone is on the same page and working together as a team.

In other words, the musicians are like the scripts or tools we use in lots of place to do useful work, like puppet agents or like simple shell scripts whilst the orchestrator would be something that can connect lots of systems and coordinate their efforts into something more useful than individual automations.

Hopefully that makes sense. There's a definite distinction between these two terms and it's useful to have an understanding of both for forming a bigger picture.

I'll post again when I've had time to play with my new puppet installation and have managed to make it do useful work.

Cheers.

Tail in Windows

One of the most useful tools to solve a computer problem is log files. Log files can tell you lots of information about what is happening when a problem occurs. The problem with log files though is that they can often be enormous in size and this creates its own needle in the haystack problem.

The solution as any seasoned *nix admin will tell you is to use tail -f to monitor the log file in real time then attempt to trigger the problem again and see what pops up. If you are lucky the last few lines will relate to your problem and you'll be much wiser about how to solve it.

Now move that process to windows. I've been doing a lot of automation work in windows of late using powershell. Having to solve various problems under windows and lamented the lack of a useful tool like tail. Sure there are stand alone utilities that can perform that function but none seemed to be as simple as tail.

That is until I discovered powershell. Specifically Powershell version 3. Perhaps to appease people like me they have added a '-tail' option to the get-content commandlet. This command simple gets the content of a file. When you combine the tail option with the wait option, you can see anything that's added to the file in real time:

C:> get-content -path "somelogfile.txt" -wait -tail 10

The wait parameter tells get-content to keep waiting for new data to be added whilst the tail parameter says to show the last 10 lines of the file.

In order to get powershell version 3 onto you machine you will need to download an update.

Hacking Active Directory

So here's the deal. I've been given the task of automating creating objects in active directory from outside the domain.

Being someone with a programming background I thought perhaps power-shell was the way to go. As it turns out, the ActiveDirectory powershell module is included when you made a Windows 2008 server into a domain controller by adding the feature 'Domain Services'.

Ok, so thats ok if you are running a script on the machine itself but that doesn't solve the problem of running from another machine.

Further research discovered Active Directory Web Services (ADWS) which export that powershell functionality over the network. These web services, whilst adhering to normal web services standards, are not meant to be consumed by a web application using WSDLs but by the ActiveDirectory powershell module.

This web service is installed by default in the same way the power shell module is and it listens on the port 9389. This gave me a nice simple test to see if it was running on a given domain controller.

A simple command you can run with the powershell command-lets is:

PS > Get-ADOrganizationalUnit -filter 'name -like "*"'

If you do not specify anything else, it will try to communicate with the ADWS running on the same domain as the machine that is executing the powershell. For a lot of use cases this is what you want and is quite the expected behaviour.

For me, I wanted to run this from a more distant machine. You can specify which machine to query using the server parameter. At this point you will also have to supply credentials for this new domain using the -credentials parameter.

How to create credentials for non-interactive scripts can be found here.

PS > Get-ADOrganizationalUnit -filter 'name -like "*"' -Credential $mycreds -server <servername>

Using this simple set of tools I can now manipulate active directory from outside the domain (given suitable credentials) and automate regular tasks like adding or deleting users or OU.

Hopefully this has been helpful to some.

CCIP It Isn't

If you are still listening, then perhaps you've heard. Cisco is retiring the CCIP certification. This both good and bad for me.

Firstly, it's bad as now, instead of studying only two more exams to get CCIP, I now have to study 5 or 6 more (more on that later) to get there.

On the other than, it's much closer to the real world in terms of the type of routing you need to know (e.g. IS-IS is back in) which I think is a good thing.

But what this means now is that instead of going for a CCIP certification, I will instead be going for a CCNP SP (SP = Service Provider). In line with other Professional level certifications, there is now a corresponding Associate level certification (CCNA SP) that is a prerequisite for this.

So going back to Associate level, means my certification path is now:
CCNA SP => CCNP SP

The CCNA SP is now two exams and the CCNP SP is four exams for a total of 6 exams but there may be some overlap with the CCNP and thus a possible exam or two that I won't need to do.

There are of course now new topics like IS-IS to learn that weren't in CCIP (though they used to be under the old routing exam BSCI) if you took the ROUTE exam.

I'm actually looking forward to this as it seems much more aligned to work I'm doing now and we all know that's a huge help when trying to learn new things.

Here's to more certificates and the joy of learning.

CCIP It is.

The treadmill continues. I'm embarking upon another Cisco certification. After my failed CCDP attempt  due to lack of study materials, I'm aiming instead for a CCIP which seems much more interesting to me considering it's application for cloud (tm).

First up is QoS which I hope to finish by end of March this year. I'll update you as to how I go.

The IPv6 Conundrum

I have been making use of IPv6 for over 10 years now and only recently has support for it become good enough that I can think about treating IPv6 like IPv4. For a long time IPv6 was the poorer cousin of IPv4, less features, less support, more bugs.

Nowadays, I think it's reasonable that most people would expect IPv6 to be production ready. I certainly do. I would expect IPv6 to be able to support all my network needs and in theory replace IPv4.

Yet even today I've managed to find an issue that has broken down this assumption. I'll walk you through it but it's reasonably obscure but still annoying nonetheless.

First a little background. Here at work recently upgraded one of our switch blocks to support gigabit speeds to the desktop PCs. We used Cisco 2960S series for the access layer and 3560X series for the distribution layer.

Something interesting about these models of switches is that they all have a dedicated management port. This is a Good Thing &tm; as it allows you to completely separate your management traffic from your production traffic. For those vague on security, this means it's that much hard to take control of the network. Done properly, and your management IP addresses don't need to be routable on your production network at all.

Of course, this sounded great to me. Given, that this post is about IPv6, you might be able to see where this is going, but stick with me.

The configuration for the 2960S is easy enough. The management port is the only port configured for routed mode and given that they aren't normally L3 switches, this makes the management port the only physical port with an IP address. Simply string all the management ports together onto a management switch and you're away.

So far so good, I have a management port on each access switch, assigned both a IPv4 and an IPv6 address. Each port hooks back to a management switch which is connected to the rest of the network via a router (ok, it should be a firewall I'll admit).

Then we come to the 3560X switches. Since these switches are operating as distribution devices they do routing. Now here's the complicated bit, in order to provide separation for the management port, it needs to be placed into a vrf instance (virtual routing and forwarding) else the management traffic will be routed as per normal along with the production traffic.

Not so hard you say? VRF Lite has been supported on these devices for a while now that's true. VRF lite works a treat, for IPv4. When you start talking about IPv6 inside a vrf, you need to move it up a whole notch. Currently that technology only exists for complicated solutions like 6VPE over an MPLS network.

Thus, whilst I can separate my management traffic for IPv4, IPv6 still isn't supported to the same level and thus I cannot have separated management traffic for these devices if I wish to use IPv6 for management.

Yes, I can hear you say, why use IPv6 for management? You know, the simple answer is because I should be able to. After all this time, there shouldn't be any operational differences between these two protocols.Yet there still are which created issues if one is to try to head down the IPv6 only path.

As a result, I'm still forced to treat IPv6 (as much as I love it) as a lessor protocol compared to IPv4. That's just the way it is.

CCDP ARCH Attempt 2

Yep, It's treadmill time again. Sometimes I wonder, when it feels like this that I bother putting myself through it. Then I remember I really enjoy the feeling after getting that pass mark. But I don't' want to get ahead of myself so I'm just putting it out there that hopefully by lunchtime tomorrow (15hrs time) I'll know if I've passed my 2nd attempt at the ARCH exam. Missed by 4% last time (that's not giving anything away I hope) so I'm hoping the 3months since and the boat load of study have been enough to get me over the line.

ASA and OSPF

I have recently updated my ASA cluster to 8.4.X in the hope that this might introduce some stability in my IPv6 support. Little did I know it would introduce other issues for IPv4 that I have only now resolved.

Firstly a little bit of background. I run a pair of Cisco ASA 5510 in active/standby mode. They run OSPF to learn about internal routes whilst having a default route pointing out to my policy routers.

The problem I had encountered since the upgrade (or perhaps shortly after) was that the ASA cluster was no longer sending messages to syslog. Upon investigating I could see an obvious route in the routing table, learned via OSPF yet the firewall would state that there was no route.

This was unusual behaviour and whilst I wanted to resolve it , I also needed syslog working. So I created a static host route for the syslog server so that I could get that working. In order to complete the process I had to disable redistribution of static routes within OSPF else all other syslog traffic would end up at the firewalls.

This of course then broke remote access VPN as an IPSec VPN would generate a static route on the ASA which would then need to be redistributed. So I lived with that for a while whilst I continued to investigate.

Today I needed to get the VPN working again so I renabled the static route redistribution which then caused the syslog routes to dissapear. I finally solved it when I looked at the configuration in CLI mode and discovered that OSPF had three network statements! Now normally this wouldn't be an issue but when I looked closely at the routing table I discovered something unusual. For each internal destination there were multiple routes in the routing table, only one of which had an interface associated with it.

I theorized that a bug in ASDM prevented any of the network statements from showing up and thus caused me to add superfluous network statements. In turn, these superfluous network statements caused invalid routes to be added to the routing table and prevent normal traffic from working.

I believe this is correct as upon removing the extra network statements and bouncing the OSPF process, the interface-less routes dissapeared and normal traffic resumed. I was able to remove the static routes and everything functioned as expected.

For reference, I'm running 8.4.2. I can only assume this is a bug and will be resolve in a future update. I'm just glad I've resolved it.

Whole Lotta IPv6

Wow, I've just had the pleasure of receiving my allocation from APNIC of a /32 prefix of IPv6 address space. That's a lot of address space.

It's a hard thing to think about how big IPv6 address space is, so to put it into perspective, I'll equate it to the size of a single LAN. Now in IPv4 world, a single LAN is usually a /24 which gives 254 hosts on a network. That's a good number for most applications. When we move to IPv6, to support auto configuration, the recommended number of bits was set at 64 bits. Already that's mind boggling-ly huge but ignoring that for now, we can use a /64 as the basis for measuring the size of various IPv6 prefixes.

So a single IPv6 LAN is 1 /64 prefix. The recommended allocation for enterprise networks is a /48 which adds 16 more bits. So this means we have potentially 65,436 IPv6 LANs in a /48 prefix.

Can you see where I'm going with this?

My allocation is a /32 because we are a data center and we allocate address space to clients. So this is 16 more bits again from  a /48. One way of looking at it is that there are 65,536 /48s in a /32 or 4,294,967,296 IPv6 LANs in a /32.

If you recognize that big number, you're doing well, it's the total number of IPv4 addresses possible. So my /32 IPv6 allocation has as many LANs as there are IPv4 addresses possible.

Wow. Each one of these IPv6 LANs is also stupendously big but lets not think about that!

To top it all off, this is just a very small piece of the IPv6 global pool. Is it sinking in how big it all is now? There's just so much more room in IPv6 land to spread your stuff out without worrying about conserving every last bit.

Now I just need to renumber everything!

ASA, Failover and IPv6 , Part 2

After posting the previous post about my ASA cluster and IPv6 I began to have problems.

Initially the solution described (setting the next IPv6 hop to an anycast address) worked as expected and I could get IPv6 traffic through the firewall without trouble. But after a while, for some reason, the traffic would stop working.

My standard investigation process eventually led me to log onto the ASA and try to ping the next hop address which in this situation is an anycast address shared by both next hop routers.

Usually the ping to the anycast would fail whilst the ping to the individual IPv6 addresses would succeed after which point the anycast address would start working again.

This was annoying to say the least and I started to have doubts about the design using an anycast address.

I did know that I wasn't running the latest firmware for my ASA (only 8.2.X) but to upgrade required a memory increase for the new firmware. I simply had to live with it for the mean time until I could get an memory upgrade through.

Fast forward to the (almost) present. Memory upgrade has been completed and I've now got the latest (8.4.X) firmware on the ASA cluster. Before you ask, yes, my problem has now been resolved with the IPv6 routing through the firewall working consistingly for several weeks now. I haven't changed the design, the next hop out of the ASAs is still an anycast IPv6 address. Similarly, the internal next hop address is also an anycast address. Both directions work a treat and have been since the upgrade.

I can take away from this the fact that it is a valid design to use an anycast address as a next hop. Sure it's not quiet the same as a redundancy protocol but it works and that's all I care about. I presume there was a bug or issue with the older ASA firmware that prevented this from working properly.

Now I can move forward to World IPv6 Day testing.

ASA, Failover and IPv6

Today I finally solved a long standing issue with IPv6 that has been bugging me.

Firstly a bit of background. I have long been an IPv6 advocate and when possible I have been enabling IPv6 in my infrastructure in the hope of one day having a production ready IPv6 network. The last piece of the puzzle was my firewall infrastructure. Last year I upgraded from a single PIX firewall to a pair of ASA firewalls in an active/standby failover cluster.

So far so good. I had already had IPv6 working under the PIX firewalls and soon had it working under the ASA. Until, that is, I enabled failover.

The zeroth problem that has yet to be resolved is the lack of dynamic routing support for IPv6 on the ASAs. There are ways around this but it means that the solution used for IPv4 cannot be used for IPv6 (e.g. the advertising of VPN /32 routes via OSPF) and I believe any differences in operation like this will make transition all the more difficult.

The first actual problem was that the code at the time (8.0 ) did not support failing over of the IPv6 addresses. What this meant was that when the firewalls failed over (often for apparently no reason) the IPv6 address configured on an interface would be come unavailable to the rest of the network. Since there was no dynamic routing this meant that static routes pointing at the ASA for IPv6 needed to be updated both inside and outside every time a fail over occured. This wasn't ideal but I lived with it whilst the fail overs were few and far between.

A subtle issue here is related to IPv6 auto-configuration. Initially I used EUI-64 on both internal and outside interfaces which meant that a fail over would actually change the address on the interface due to the different MAC addresses for each physical NIC. Sure, the obvious solution would to be configure a manual address on each interface. The problem was since the code didn't do correct fail over of the IPv6 addresses, each firewall would hear the others address and complain about duplicate IPv6 addresses and shutdown IPv6 on each interface. Totally useful!

Fast forward a bit and a new release of code now supports IPv6 fail over (8.2.4). This means the internal and external IPv6 addresses get moved across when there's a fail over event. This is great but for reasons unknown to me didn't actually solve my problem. My ASA cluster has two routers inside and two routers outside. After each fail over event, the active ASA could only contact one of the external routers which meant that the static IPv6 routing on the ASA (remember no dynamic routing) needed to be updated each and every time there was a fail over event. So despite supporting IPv6 fail over properly (i.e. each firewall no longer complained about seeing each other) my IPv6 solution was no better off.

Now, I no longer needed to change the static routes on the internal and external routers, but I did have to change the routes on the ASA itself.

I was getting frustrated with this state of IPv6 support given the imperative to embrace IPv6 in the Asia Pacific region. I imagined that it shouldn't really be this hard.

Fast forward to today.

From some dusty corner of my mind I recalled today, whilst trying to debug this issue, the anycast type of IPv6 address. I had it in my head that my networking kit didn't support anycast but it was worth a try.

Checking on the external router, I was able to enter:

ipv6 address xxxx:xxxx:xxxx:ffcb:ffff::1/64 anycast

on one of the external routers. Excellent. I repeated the same on the second router then checked for connectivity on the ASA. It worked! So I updated the routing on the ASA to point its static IPv6 default route to the anycast address now instead of the router specific one.

The next step was to repeat this for the internal routers. My internal 'routers' are L3 switches and I was expecting less complete IPv6 support from them but to my surprise I was able to repeat the above command on the internal L3 switches.

Following this, I repeated the ping test on the ASA and was thankful to get a response. So again, I updated the static route pointing to the internal IPv6 /48 network to go via the anycast IPv6 address.

Now, after all this wrangling, I have a stable IPv6 infrastructure. I can ssh to an IPv6 address on my external routers and presumable, it will all work after a fail over event. I can't see why it wouldn't. I'm now starting to see the usefulness of the anycast addreess type. For a long time (i.e. up until today) I had the impression that they were only good for application level stuff, e.g. dns queries.

I'm now keen to make more use of anycast within my network for say dns or ntp servers instead of using mechanisms such as multicast routing (which isn't yet supported on L3 switches for IPv6!).

Happy routing!

Passed CCNP

Well finally after redoing my SWITCH exam then finally sitting TSHOOT on the 14th of Dec last year, I have completed all the requirements for my CCNP and am now certified.

I even have a nice little logo that I can include in signatures which is a little bit of icing.

Of course there is no rest and I've already started in on my next certification, CCDA though I'm going to be studying (I think) for the new 2.1 curriculum that has recently been released. This of course makes things harder as the reference material is still being written for this certification but hey, nothing wrong with a challenge!

After CCDA, on to CCDP which is also being updated.

After I get my CCDP its on to the CCIE.

Wish me luck!

The end of the global IPv4 address pool

Yes, the end is nigh, rumor has it that 2nd of Feb 2011 will be the official announcement day for the end of the global IPv4 pool. Technical types will no doubt understand the implications of this.

What irritates me though, is the media who choose to display their complete ignorance in order to drum up business (I can only assume).

Here are a few pointers for any media types out there:

"IT'S the end of the web as we know it.
Since its inception, the internet..."

Read more: http://www.news.com.au/technology/the-internet-has-run-out-of-ip-addresses-and-what-happens-after-that-is-anyones-guess/story-e6frfro0-1225995086627#ixzz1CCKnzW6J

The 'web' is not the same thing as 'the Internet', you cannot use these terms interchangeably.

 
"Web developers have compensated for it by creating IPv6"

Read more: http://www.news.com.au/technology/the-internet-has-run-out-of-ip-addresses-and-what-happens-after-that-is-anyones-guess/story-e6frfro0-1225995086627#ixzz1CCKVmEgk

Web developers did not invent IPv6, the IETF did many years ago.


"At best, their user experience will be clunky and slow."

Read more: http://www.news.com.au/technology/the-internet-has-run-out-of-ip-addresses-and-what-happens-after-that-is-anyones-guess/story-e6frfro0-1225995086627#ixzz1CCL2jad9

IPv6 will not break things or make your Internet experience worse. Buggy software will but that's harder to blame in a news story.

"The current generation of iPhones, for example, won't display anything with an IPv6 address correctly."

Read more: http://www.news.com.au/technology/the-internet-has-run-out-of-ip-addresses-and-what-happens-after-that-is-anyones-guess/story-e6frfro0-1225995086627#ixzz1CCLTJSHd

Apple iPhones can do IPv6 just fine (those that run iOS 4.X) and will have no issues accessing the IPv6 Internet when it becomes more available. The issue is carrier support for IPv6 which is still absent in most markets.


As a matter of fact, most recent devices will support IPv6, the issue has been the infrastructure. Sure there's no content  and what there is has restrictive terms (e.g. Google over IPv6) that see only a few make use of it. Yet the biggest issue is the infrastructure upon which the Internet exists. There's no great incentive to move over to a new protocol considering the costs that could be incurred.

That being said, as part of a companies normal upgrade cycle, IPv6 will be included on newer kit purchased. Some companies will choose to take the initiative and enable IPv6, perhaps gradually; other companies will choose to pretend it is a security risk and disable it everywhere lest someone hacks them via IPv6.

My only hope is the huge media beat up about the exhaustion of IPv4 will get more average businesses aware of IPv6 so that will start asking their ISPs about it and thus create some level of demand!

Will you?

2% of IPv4 address space left!

Yes, you heard me. IANA has only 2% of all IPv4 addresses left to hand out. In terms of /8 this is 7. Of these 7, five will be automatically handed out to the 5 regional registries meaning there are really only 2 /8s left up for grabs and rumor has it these will go early next year making and end to it all.

I just wonder how much media beat up it will get?

And the other burning question, are you IPv6 ready will soon be on everyones lips. I at least know my work is ready but still I haven't got native IPv6 yet and I've no idea about my home or mobile operators as to wether they will have IPv6 available anytime soon.

Even major content is still shy about IPv6, come on Google, it's time to get out from behind your Google over IPv6 program and offer native IPv6 to anyone who asks without some prior arrangement. It's not helping.

If you're worried about the .01% of people who might have an issue, make sure you make available lots of resources on how to fix their connectivity!!

Anyway, that's my IPv6 rant for today.

Hope you enjoyed!

In the 1/8 block.

Some while ago I noticed that APNIC had been allocated the 1.0.0.0/8 block. This was a cause for concern as it has been one of the more well know bogon prefixes. These are address blocks that may be used within organizations instead of the rfc1918 address space, e.g. 10.0.0.0/8. I've spoken people who have done this and the usual reason is to avoid issues with others who might be using rfc1918 address space.

iPhone screenshot.

The result, many ISPs would filter these bogon routes lest they leak out onto the Internet.

Now, to cut a long story short, I've noticed my iPhone now has an IP from this range. Time will tell if there are any connectivity issues.

TSHOOT Books

Yay, today I finally got my foundation learning guide for TSHOOT. I discovered recently that you can get two types of Ciscopress books, Exam certification guides which are all about passing the exam, and the Foundation Learning guides which are more like a course and give you much more depth.

I already had the certification guide so I thought I should really have the foundation learning guide (after failing SWITCH) so I ordered it from Amazon but as usual when you are waiting for something it takes forever to turn up.

So today, some 19 days later it finally arrived on my desk so now for some hard core reading as my exam is less than 1 month away!!

Wish me luck!

CCNP SWITCH Passed

Yes, finally I managed to pass CCNP SWITCH exam.

Lesson to be learned, make sure you read all those tiny sections at the back of the book that mention exam updates.

It turns out that between publishing of the official certification guide and my sitting the exam the first time around, Cisco had updated the exam curriculum. As a result, the book author had written extra content as a PDF to be downloaded from the publishers website.

If I had know that before my first attempt, perhaps I would have passed, who knows. As it is I took advantage of several other aids to study for my second attempt and I guess it all worked as I passed.

Only one more exam for CCNP, that's TSHOOT booked for Dec 14th!

Wish me luck.

Fixed my Internet

Last night I finally fixed my Internet! I have a Cisco 857W router for my ADSL connection (thank you eBay) but had problems with it. Every so often, it would simply stop passing traffic. The router wouldn't report any particular issues but wouldn't pass any traffic down the ADSL line. Everything else on the router appeared to be normal.

So, being the network engineer, I downloaded the latest IOS and updated my router. Not only did this not fix the problem but it introduced another one. It would appear that every new connection would loose the first few packets. A ping would loose the first packet, telnetting to port 80 on a website, would delay for several seconds before responding.

It was enough of a problem to prevent the ABC's iView on the PS3 to stop working which was annoying.

Normal web surfing would still work but felt a tad slower. Speed tests would show the expected bandwidth but have a horrendous ping response (> 1000ms).

So last night I downgraded slightly but went from an ED release (Early Deployment in Cisco speak) to an MD release (maintenance release) and found that all these problems have gone away.

I guess this says that there are problems with the latest ED release for my router. It remains to be seen if this MD release fixes the original problem that I was trying to fix.

Wish me luck!

CCNP Switch Fail

I recently failed my CCNP SWITCH exam. Only by a few marks ( < 5%) but enough nonetheless. At first I was upset but now I've come to realize that I wasn't thinking the right way about a lot of the questions, particularly the simulations.

Previously, I had seen sims in Cisco exams which clearly stated what the end goal was in technical terms but this exam was more about giving a set of requirements and letting you figure out what was required. During the exam I wasn't thinking through the requirements at all, just my own set of technical goals, e.g. must get this ether channel up.

Having had time to think about, I now realize that this comes down to my lack of experience (despite doing Cisco networking for 8 years) of doing design work. I have never sat down at a planning meeting with a client to determine their requirements, not have I been involved in a peer review process in an engineering team.

My networking role is just me looking after a large network (> 800 virtual users) all by myself, with no other network engineers and no 'client'.

The end result is I know my network very well but I have never had to plan or design from scratch which showed up in the exam.

I want to be a good engineer but planning and design are hard to study for unless you've had the exposure.

Never fear, I will keep at it and I hope now I at least better understand what is required of these types of questions in the exam.

Enabling rapid-pvst

This post is about my attempts to enable rapid-pvst (802.1w) on one of my switch blocks at work. My previous attempt had resulted in lots of loopback errors disabling up-links on the access switches.

My only thought at the time was that perhaps I had way too many end-to-end VLANs that took too long to converge. Now, having read up on rapid-pvst , I now believe it was probably the fact that during the process of enabling rapid-pvst, the default pvst has to be switched off and for a small period you are running without any spanning tree. If you have enough traffic then the probability is high that you could get a loop during this interval. It could still be a combination of effects though and since my original failure, I have been agressive with switchport trunk allow vlans ... to restrict the number of end-to-end VLANs and have been rewarded with success.

I now have managed to get rapid-pvst working on one of the switch blocks to which I originally had problems with. This time though, knowing more I took a cautious approach.

• Step 1, Enable loopback errdisable recovery so that if for some reason the original problem reoccurred I wouldn't have to get console access to my access switches or reboot them. This can be done with errdisable recovery cause loopback
  
• Step 2, Turn on consoles message so I can see any errors that might occur. (terminal monitor)
• Step 3, Enabled rapid-pvst and wait spanning-tree mode rapid-pvst
• Step 4, Save your work once switch is stable.

I noticed on each access layer switch, the management interface went down and came back up. Once I had completed all the access layer switches, I then did the core switch (only one) and once finished I had a stable switchblock running rapid-pvst. Here is the output of show span vlan 188:

VLAN0188
  Spanning tree enabled protocol rstp
  Root ID    Priority    24764
             Address     001b.8f97.2180
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24764  (priority 24576 sys-id-ext 188)
             Address     001b.8f97.2180
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/1               Desg FWD 4         128.1    P2p
Gi0/2               Desg FWD 4         128.2    P2p
Gi0/3               Desg FWD 4         128.3    P2p
Gi0/4               Desg FWD 4         128.4    P2p
Gi0/5               Desg FWD 4         128.5    P2p
Gi0/6               Desg FWD 4         128.6    P2p
Gi0/7               Desg FWD 4         128.7    P2p
Gi0/8               Desg FWD 4         128.8    P2p
Gi0/9               Desg FWD 4         128.9    P2p
Gi0/10              Desg FWD 4         128.10   P2p
Gi0/11              Desg FWD 4         128.11   P2p
Gi0/12              Desg FWD 4         128.12   P2p
Gi0/17              Desg FWD 19        128.17   P2p Edge
Gi0/27              Desg FWD 4         128.27   P2p Peer(STP)
Gi0/28              Desg FWD 4         128.28   P2p Peer(STP)

This switch block is one of the ones that I had the loopback issue with so I am happy to have gotten it working here. I will now rinse and repeat and see if I can get it working everywhere!