Category Archives: Cisco

DHCP Scope Configuration – Oops

So for the last couple of days, I’ve been in Nevada at one of our remote sites. (On a side note, the “middle of nowhere” pretty much describes all of Nevada!!) I was setting up a wireless bridge to connect separate parts of a large aggregate plant…it was a very busy few days. We kept running into problems, which took up a lot of time to resolve, but eventually we got things working. As I was testing the new subnet hanging off the bridge, I noticed that DHCP was not working…hmmm, very strange. I’ve configured DHCP many times over the years, and it just works. Time to troubleshoot…

First test was easy…I configured a static IP on my laptop and everything worked great. Next I drove over to the other end which housed the main switch and router, and plugged into a port configured for the new VLAN….and no DHCP. Say what? Hmmm…I must have made a mistake on my configuration…but the DHCP pool looks good…

Config for the DHCP pool

Config for the DHCP pool

And the subinterface configuration looks good too…

Sub-interface config looks good too

Sub-interface config looks good too

Very interesting…the only thing left was the DCHP excluded-address config, but that’s so easy, I know that’s not the problem.  But I checked it out anyways…

DHCP excluded-address config....oops

DHCP excluded-address config….oops

Say what?? How could I have messed that up? But I have to say, the configuration was doing exactly what I asked it to do…basically not handing out any IP’s!! So after a quick edit, everything was working properly…

The proper excluded-address configuration

The proper excluded-address configuration

So remember, most of the time, it will be the simple things that get you.

Cisco VWIC3-1MFT-T1/E1 Controller PRI Issue

Hats off to Dan, my co-worker….he’s been chasing a PRI issue at one of our new sites. He has a Cisco 2951 router with a VWIC3-1MFT-T1/E1 card installed, and when the PRI circuit is un-plugged and plugged back in, the circuit refuses to come up. However, if you reboot the router, the circuit will come up. (Rebooting the router is not a good fix, just in case you are wondering.)

As part of his troubleshooting, he replaced the VWIC3 card with a VWIC2 card and it works just fine…no issues. After working with Cisco TAC, he found out he was hitting a known bug with this VWIC3 card and IOS software (15.3(3)M6). At least the fix was easy…he simply had to add the command “hwic_t1e1 equalize” under the controller T1 interface…like this…

Adding in the hidden controller T1 command

Adding in the hidden controller T1 command

What’s interesting is it’s a hidden command…if you list the available commands under controller T1 0/0/0, you won’t see this command…

A hidden command...interesting!

A hidden command…interesting!

And I just bet there are a bunch more hidden commands that we don’t know about!!

Cisco Security Alert – ROMMON Firmware Hack

Cisco Security Alert

Cisco Security Alert

Well, it looks like the hackers are at it again. (BTW…I use the term “hackers” as my preferred term “slimy dog-poop scum” is too wordy…but either one works just as well.) Cisco just released a security alert concerning a hack which replaces the ROMMON firmware (the boot firmware) with malicious ROMMON code. This code does work, in terms of booting the router/switch properly, but it also contains malicious code. Fortunately, you do need either privileged access or physical access to the device. Note the credibility level…”Confirmed”.

Check out Cisco’s security alert here.

Securing SNMP Access on Cisco Switches

Here is a quick and easy one…

I installed and configured Solarwinds Network Engineer’s Toolkit on a new server today, and did a quick SNMP (Simple Network Management Protocol) test to my core switch. Well, it didn’t work…which actually is good. It meant that I did configure access restrictions via SNMP. And you should too…if you don’t, then ANYONE can install an SNMP utility and try and gain access to your switches, or other network devices.

Here is my SNMP config on my switch…

SNMP Configuration

SNMP Configuration

The “2” at the end of the line references access control list number 2…

ACL #2 - Restricting SNMP access

ACL #2 – Restricting SNMP access

As you can see, I have configured SNMP access from two separate servers, which did not include the new server I was using today. (If no ACL was referenced, then anyone can access the switch via SNMP). I then added that server into ACL 2…

Adding another server to ACL 2

Adding another server to ACL 2

And everything worked just fine! So, moral of the story is to make sure and secure your SNMP access…and test it every now and then to make sure it’s working properly.

CiscoLive and a Leap Second

Greetings everyone…

Yes, I’ve been way too busy…sorry for the lack of posts. I am going to try and get back in the swing of things!!

First, a bit of fun news…I’ll be out of town next week, down in San Diego, attending CiscoLive 2015!! And even better, my wife is going with me! I bought her a Social Events Pass…this allows her to attend all of the Keynote speeches, the evening vendor meets, and the closing night concert with Aerosmith at Petco Park (the Padre’s stadium)!! We are both very excited!! I will try to post some notes next week from CiscoLive…if I can find the time.

Next…since we are talking about time….on Tuesday, June 30th, there is going to be a leap second event. (I’m not kidding folks…this is actually very cool stuff!!) Over time, the Earth is gradually slowing down…and to keep the “real” time sync’d with our perception of time (sundown, etc.) they have to add a second to the day. The last time this happened was back in 2012. Is this something you need to worry about? Probably not, but you should be aware of it, and keep an eye on things that might be acting strange (frozen, pegged CPU, etc). Here is Cisco’s take on this years leap second.

Details:  The leap second is going to occur at 23:59:59 UTC on June 30th, 2015. In other words, one second before midnight UTC time. For me, here on the west coast with daylight savings time, it will be 4:59:59 PM in the afternoon (PDT). That final minute will actually last 61 seconds. This will be orchestrated by all of the NTP servers on the Internet…so if all of your network equipment is time sync’d to NTP servers, you should be fine. If you equipment is not time sync’d, well…I guess you don’t really care about all of this anyway. (And you are not being a good network engineer either…get your network time sync’d!!)

Linux and UNIX systems should handle this well…NTP will announce to any system running true NTP that a leap second will occur (there is a leap second flag that gets set). Windows systems do not know how to handle the NTP announcement, so they will just end up a second off. However, within 20 minutes they will be re-sync’d properly. What to watch for? Anything acting strange that relies on exact time…applications that use GPS would be key to watch.

So…you have been warned. Hopefully the addition of a leap second will not trigger Armageddon.

CiscoPress – eBook Deals, a Good Value

I love to keep up on the latest technologies and I love to read, which tends to be a good combination. I also like a good deal, so every Monday morning I check the CiscoPress website and see what the current eBook special is. You can get an eBook version of a Cisco book at a very good price, typically 50% off. Of course, if you are not interested in the weekly deal, then just pass on it…it changes every week. I’ve bought a number of eBooks over the last year or two…books on Data Center technologies, IPv6, Cisco Unity and the updated CCIE Routing (v2)…all at a great price.  I then upload them on my iPad. Look on the right side of the page, about a third of the way down…

CiscoPress eBook Deal of the Week

CiscoPress eBook Deal of the Week

There is also a Video deal of the week, if you are into that…I’m not.

A Day of Mourning…Missing My C6509-V-E

A while back, my company outsourced our Data Center to a third party hosting firm. For the most part, it has worked out well, and from a business point of view, it made lots of sense. From a personal point of view, however….well, it did hurt a bit. I mean, if you are a Sr. Network Engineer, without a Data Center, are you really still a Sr. Network Engineer?

Oh well, it is what it is. After the move, our Data Center was rather empty, except for our core 6509 switch. It was still in place, feeding a bunch of access layer switches and a few remaining servers. Until last night that is. My coworker and I had the sad duty of powering down the switch, removing it from the rack, and replacing it with two Cisco 3650-48 switches. (If you know what the annual SmartNET maintenance is on a 6509 switch, then you know the ROI on replacing it with these 3650’s is a huge no-brainer!!)

Everything went very smoothly, the new switches are in place, and the 6509 is sitting on a cart waiting to be sold. I sure wish I could afford it…it would look GREAT in my lab at home. Of course, my electrical bill would be sky-high, not to mention trying to keep the room cooled.

Before...

Before…

And after...

And after…

But…to be very honest…I’m going to miss my 6509. There is nothing like a big chassis class switch to brighten one’s day. I know it did for me.

Time for Another Set of Eyes

It happens to us all at some time or another. You’ve been working all morning at a remote site several hours away, and you’re making substantial changes on the network infrastructure. As you are finishing up, you realize you can get to the Internet, but you can’t FTP to the Internet. After going over the config several times, it’s time to bring in another set of eyes. And so my phone rings.

This ended up being rather simple. We have PBR (Policy Based Routing) in effect for normal web traffic (ports 80 and 443), and a default route for all other Internet destined traffic (such as FTP). I checked the routing table and found this…

show ip route

show ip route

As you can see, gateway of last resort is not set. So for any Internet bound traffic that is not port 80 or 443, the router does not know where to go. My co-worker checked and found that he had mis-typed the entry for the default next-hop path. Once he fixed it, everything worked as it should.

Another example was earlier this summer…I was having some stability issues with one of my VTI (Virtual Tunnel Interface) sites, and was not able to nail down the cause. My co-worker looked over the related configs, and found that I had forgot to set a particular filter on the perimeter firewall. That fixed it.

So if you find yourself staring at a configuration, unable to find the problem, call a fellow co-worker and get a fresh set of eyes on the problem. Two heads are better than one!

Security – Additional Shellshock Resource – Cisco

As a followup to my previous post concerning the BASH bug, known as Shellshock, Cisco has confirmed that a number of it’s products are vulnerable. You can read Cisco’s Security Bulletin for information that may end up affecting some of your network infrastructure.

As more vendors check their equipment, you will see additional bulletins posted. Stay on top of this…active exploits are already being seen out in the Internet.