An Open-Source, CCIE Lab Rack Automation Control System

The lab portion is the most challenging part of the Cisco Certified Internetwork Expert (CCIE) exam. To pass, you need hundreds of hours of experience with the equipment, so you will need access to a practice lab.

There are generally two ways to go about this. The first is to rent rack tokens from your CCIE training provider. These racks are nice because they match the workbooks they sell and make it easy to lab. More importantly, they have automation in the racks to reset configs to get the to a proper starting point for your lab. This makes good use of your time.

The second way is to build a home lab. After earning my CCIE in 2015, I found having my own lab was extremely important because you can’t get everything done in an 8 hour session or a day on a rented lab rack. On the flip side, a home lab takes a long time to reset to starting point zero.

I recently decided to pursue a second CCIE in wireless and I built a home rack. I wanted some automation to make studying more efficient.

That’s where our solution comes in. If you’re taking your CCIE soon, I hope you will benefit from some automated rack control to help you manage your configs and labs.

You can find the application and repository here.

How does it work?

What I’ve built is a small web application (on Ruby on Rails) that has some hooks into SSH and VMware API’s to reset and restore configurations in an easily configurable manner. This application has a few concepts in it:

  • Lab Rack: This is your pod of equipment as a whole. I have 2 racks in my system, one for CCIE Wireless and one for the new CCIE Collaboration v2. Almost all of you will only have one rack out there, but it supports multiple racks if a partners wanted to set up training pods.
  • IOS Devices: These are your traditional Cisco hardware boxes. We basically Secure Shell (SSH) in and have the ability to restore configuration files that are stored currently in the device’s flash.
  • VMware devices: These are your virtual machine’s. We have built-in automation to restore to a previous snapshot.
  • Lab Scenarios: These live under lab racks and they associate a Lab Scenario to a set of devices and a set of configs for those devices. The reason for this is you may have a different set of configs; for example, if you working on Layer 2 switching or Layer 3 routing. This give you ultimate control over how many different configuration iterations you have. You define these labs yourself in the application, so its completely extensible.

I’m using this to successfully power my wireless lab at this point. If you find something it can’t do, just let me know. I’ll try and add it!

Prerequisites

In no particular order:

  • Ruby 2.5.3
  • Postgres
  • VMware vSphere or vCenter with your machines already built and running.
  • Cisco IOS Device with SSH access configured and an enable password set! I haven’t tested telnet yet (thought I did take a shot at the code) and it won’t work without an enable password.
  • All config files for Cisco IOS devices uploaded to their flash and uniquely named.

Installing the application

The application is written in Ruby on Rails so you’ll need Ruby 2.5.3, Postgres database and git installed. Installation will be easier on a Mac or Linux/Unix box. If you’ve never done this before, I suggest following a blog like this.

Note: you can also get this running on Windows with some work. There is a Ruby engine and Postgres for windows.

First, we need to clone our repository. Here is a screenshot where you can find the URL.

You can clone it into your home directory with the command:

git clone https://github.com/Cloverhound/RackControl.git

If you look in the configuration file for my application’s database connection, you will see I’m using environmental variables for myself. Replace these with your username and password for Postgres. You can find the db config file in rails at config/database.yml.

You'll want to replace <%= ENV['DBUNAME'] %> and <%= ENV['DBPASS'] %> with your information.

These are also called environment vars and on nix and OSX they are located in the ~/.bash_profile. You can set DBUNAME and DBPASS in this file. Here is more info on environmental variables and how to use them.

Now we are ready to install the database. We are going to move into the project directory and run

rake db:create

to make the databases and 

rake db:migrate

to build all the tables.

It gave me an error message for running version 2.5.4 so I used RVM (Ruby Version Manager) to change versions. If you use Ruby for more than one project and are on Nix/OS X, you’ll want this.

Once everything is installed, you can start it with 

rails s

in the project directory. You’ll be able to access the project on you local machine at

http://localhost:3000

The hard part is done.

Setting up your rack

Once we have our webpage up we will see this:

Click New Lab Rack and add a name and description for the rack.

Click Create Lab Rack, and we will be ready to add devices.

Next we are going to show you how to setup VMWare device under your rack.

Configuring VMware Devices

The first thing we are going to do is access vSphere or vCenter, find a few pieces of information and take at least one snapshot. You can access vSphere by going to its IP address 

https://<IP ADDRESS>

and logging in with the correct credentials. You will need these same credentials and IP for our Rack control application.

Once you’re in, prepare to take some notes. Browse the list of your VMs. We need to know where the Datacenter is located; in this screenshot, it is located under Cloverhound. It’s right under vCenter on the left.

Find your VM in vCenter, right click on it and Take a Snapshot. This Snapshot is the state your machine will be reverted to from the rack control application. If this was a Communications Manager, for example, this would probably be right after install, with no configuration on it other than an installed subscriber. You’ll want this to be the starting point for your configuration.

You’ll need the name of the Virtual Machine and the name of the Snapshot as the remaining pieces of configuration needed for our application to automate. You’ll make a name when you take it, but you can also find it by right clicking on the VM and choosing Manage Snapshots.

We finally have everything we need from VMWare. You’ll need to repeat this process for every machine you want to control in the rack(s).

Configuring Rack Applications for VMWare Devices

Now we need to go back to our rack application. Click on our lab rack and click on Configure VMWare Devices.

Next, click on New VMware Device.

You can see that it’s asking for all of the information we just found or gathered in vSphere. Fill that in and click Create VMWare Device. This will add the VMWare device to our rack.

Note: We use the snapshot name in a later process that is associated to an individual lab so save that information for later.

From here our device will be added and we’ll go back to the VMWare Device page. Let’s click Back to Lab Rack and learn how to add a Cisco IOS Device to our rack control.

Configuring Cisco IOS Devices

I made some assumptions here:

  • Most racks you will find out there are going to work off console cables. Ours currently works off of SSH.
  • I have a management VLAN that I’m keeping up no matter what in my configurations so all of my configurations have at least one reachable interface. When I do layer 2 things, I just have to configure them back to at least reachibility, not the worst thing.

You could easily modify this application to run on a PC with a bunch of console cables and I probably will do that in the coming weeks seeing as some folks at Cloverhound want to make a R/S rack.

Your IOS Devices need to be configured for SSH with a username and password and must have an enable password set.

Another important note is all of the configuration files you are going to restore should be located on the flash of the Cisco device. If you have 12 lab scenarios then you might name the configurations lab1.txt through lab12.txt. You can use any name you like. You will configure it in the app later.

You can confirm they are on the router correctly and their names by typing 

sh flash.

Configuring Rack Application for Cisco Devices

So from here we are going to go back to our Lab Rack within the Application and click on Configure Cisco IOS Devices.


Then New Cisco IOS Device.

It’s going to ask us for some information you should be able to fill in just fine. The one key here is you MUST type in ssh as the transport type. I plan to add telnet support shortly

Alright, click Create Device and we are good to go. You’ll want to add all of your Cisco Devices. Now we are ready for the fun stuff.

You’ll be back at the Cisco IOS Devices page, click Back to Lab Rack.


Creating Lab Scenarios

Here is where it all comes together. Your going to create a Lab Scenario and then associate the devices you want automated under that scenario. Finally, you’ll associate a config file name for Cisco IOS Devices and a Snapshot name for VMWare Devices.

You’ll be back at the Lab Rack page now. Click Configure Lab Scenarios.

From here click New Lab Scenario.

It’s going to ask you for a name and description.

Click Create Lab and then you will be taken to the Lab Configuration screen.

The final step here is to add your VMware devices and your Cisco IOS Devices to your lab.

Click on Add VMWare Device. You’ll be asked to select the Virtual Machine you would like to associate to the Lab Scenario and the name of the VM Snapshot, which we made above.

Click Create VMware Device Lab. You’ll be taken back to the home screen of the lab now. Finally we can test all our hard work by pressing the restore button next the Virtual Machine we added to the lab.

After we click restore, next to our VMware device. We will see a notice saying that it’s restoring, but let’s check VMWare just to be sure it all worked fine. However, we would see an error if it failed.

If we pull up our Virtual Machine in vCenter and choose Monitor -> Tasks we can see that the Virtual Machine just reverted a snapshot. Success! All that’s left is Cisco IOS Devices.

It’s going to be the same procedure for Cisco devices, let’s click on Add Cisco IOS Device. Just select you Cisco Device and type in the name of the config file you put on the flash that you want restored. In my case lab.txt.

Click Create Device Lab and that’s it, you’ll be taken back to the Lab Scenario home screen. Once again we can test it by clicking restore.

When we click Restore we can see it will tell use the device is restoring.

We can confirm this by trying to ping the device. You’ll see it’s offline because it’s reloading.

Lastly if you try and issue another restore command before the device is back online and ready, it will tell you it’s offline.

The final piece of this magic is now if we click Back. You’ll be taken back to the Lab Rack home page. Now you’ll see your Lab Scenarios listed and next to it, you’ll see a Restore button. Clicking this button will restore all the devices at once!

This is a single threaded app so if you have 10 or 20 devices it may very well take 2-3 minutes to complete. Just give it time.

You can see when I click restore now, it will do both at once!

Conclusion

I built this because when I was doing my CCIE Collaboration I found managing my home rack to be a very tedious process. I also thought it was a great way to test out some automation on VMWare that I had been meaning to try for a few years.

You can find most of the meat for the interaction with vmware and cisco in 2 files. app/models/cisco.rb and app/models/vmware.rb.

I hope future CCIEs find this and use it in their studies. Feel free to use this or hack this in any way you like as long as it’s for personal use or internal training. What I do not approve of is commercial use or any CCIE training programs using this IP for rack control.

I also think new coders can derive some value of how to interact with other devices on the network. Always keep learning.

-Chad