Tag Archives: protocols

Office Phones Down on Monday Morning – DHCP Issue

If you are a Network Engineer, then having your phone ring early on a Monday morning is never a good thing. And that’s what happened this morning. Seems that none of our Cisco phones were working at the Corporate office…yep, none, as in 150 employees. (And it always seems to be the big offices…why does nothing go down at a little remote office that only has 3 employees?)

After talking to a couple of users (via cell phone of course), I realized that all of the phones were trying to get an IP address but were unable to.  (Phones were displaying:  Configuring IP). Well that is very interesting…and I recalled that the DHCP server was just replaced this past Friday. (Grrrrrr….how do you mess up a DHCP server?)

Well, it’s easy to mess things up if you don’t use the same IP address on the new server as the old server. Remember, DHCP is a layer two broadcast mechanism…the device booting up sends out a DHCP Request packet destined for all F’s, and the server responds.  Now, if the server is NOT on the same local network, then you need an IP helper statement such as this to properly forward the DHCP request…

interface Vlan11
description VoIP VLAN for Corporate office
ip address
ip helper-address

As you can see the switch will take any DHCP requests and forward as a unicast packet to the DHCP server. However, the new DHCP server had a different IP address. Why? I don’t know…however the server guy did fix the issue quickly, which was much appreciated. He simply changed the IP address of the new server to that of the old server…and boom, all of the phones started registering. You may be wondering about all of the PC’s on the network…they were working just fine as they were on the same local network as the DHCP server.

I then was able to finish shaving and get ready for the day. You just have to love Mondays…

Packet Capture on Both Sides of a Conversation

Greetings everyone!! Yes, it’s been a while…sorry about that. Been busy with life…I’ll just leave it at that.

So at work, for the last couple of days, the Citrix admin has been having an issue with users at one of our larger remote sites…seems they are intermittently unable to connect to a Citrix server after being redirected by the Citrix license server. He brought me into the problem this morning once he realized the problem was only occurring at this one site. Very interesting!

First step was to capture some of the traffic and see what’s going on. I have a Linux server at the Data Center running Snort, watching most all of the traffic into and out of the Data Center, so this comes in very handy when I need to capture some traffic. I started a TCPDUMP on the server, specifying the interface in question that has the traffic from the remote site, and dumped it into a file…the command was…

tcpdump -vv -nn -s 0 -i eth3 -w /root/pcapfiles/citrix_issue.pcap host

The Citrix admin was remoted onto a PC at the site, and he attempted to connect into Citrix. After I captured the data, I moved the file to my laptop and opened it up with Wireshark…I saw the 3 way handshake (SYN – SYN/ACK – ACK), and then some data going back and forth, but the session never started up and it timed out. Weird.

I also captured a Wireshark session from my own laptop so I could see how it was supposed to work. I still could not see what the issue was…very strange.

I then decided that I needed to see packet captures from both sides of the conversation. So I remoted into the PC and installed Wireshark on it…I then started up Wireshark on the PC and TCPDUMP on my Linux server, and then tried the Citrix client again. After it timed out, I opened up both PCAP files in Wireshark and examined them side by side, packet by packet. On the PC, I saw this…(PC =, and Citrix server =…

Packet capture on PC side showing initial SYN packet

Packet capture on PC side showing initial SYN packet

I found the matching packet on the capture from the Data Center…

Packet capture from Citrix server side showing full handshake

Packet capture from Citrix server side showing full handshake

Say what????? I see a completed 3-way handshake…including the SYN/ACK from the server back to the PC (which was not in the PC capture), and another packet from the PC to the server completing the handshake…also not in the PC capture. VERY bizarre!!!

Then it hit me…Riverbed!! The only way a handshake could be completed in this manner was from another device sitting in-between…and we do have that. We have Riverbed devices sitting at the Data Center and at most large remotes sites handling WAN optimization. Since no other sites were reporting any issues, then the Riverbed at this remote site must be “kinked”. So we restarted the Citrix application optimization process on the Riverbed, and that fixed everything!

VERY cool…and very interesting. This took some time to figure out, but once I got visibility at both ends of the conversation, the answer was easy. Remember…Wireshark is your friend.

DHCP Scope Configuration – Oops

So for the last couple of days, I’ve been in Nevada at one of our remote sites. (On a side note, the “middle of nowhere” pretty much describes all of Nevada!!) I was setting up a wireless bridge to connect separate parts of a large aggregate plant…it was a very busy few days. We kept running into problems, which took up a lot of time to resolve, but eventually we got things working. As I was testing the new subnet hanging off the bridge, I noticed that DHCP was not working…hmmm, very strange. I’ve configured DHCP many times over the years, and it just works. Time to troubleshoot…

First test was easy…I configured a static IP on my laptop and everything worked great. Next I drove over to the other end which housed the main switch and router, and plugged into a port configured for the new VLAN….and no DHCP. Say what? Hmmm…I must have made a mistake on my configuration…but the DHCP pool looks good…

Config for the DHCP pool

Config for the DHCP pool

And the subinterface configuration looks good too…

Sub-interface config looks good too

Sub-interface config looks good too

Very interesting…the only thing left was the DCHP excluded-address config, but that’s so easy, I know that’s not the problem.  But I checked it out anyways…

DHCP excluded-address config....oops

DHCP excluded-address config….oops

Say what?? How could I have messed that up? But I have to say, the configuration was doing exactly what I asked it to do…basically not handing out any IP’s!! So after a quick edit, everything was working properly…

The proper excluded-address configuration

The proper excluded-address configuration

So remember, most of the time, it will be the simple things that get you.

Installing DIG on a Windows Machine

First up…a quick apology. Back on April 10th, I posted Verifying Proper Email Routing – MX Records. At the end of the post I mentioned I would talk about the trailing dot shown in the DNS records in my next post. Well, I basically forgot. Life just got too busy.

But before I do that, you really need to get DIG installed on your Windows PC (if you have Linux then you should already have DIG). So today, lets get DIG installed and tested…this is a great tool to have, and you will be surprised at how often you will use it.

The package we are going to install is BIND (Berkeley Internet Name Domain) from ISC (Internet Systems Consortium). ISC has been around for years, and most DNS servers on the Internet run BIND (whether Windows or Linux/UNIX based). Do the following…

  • Go to https://www.isc.org/downloads/bind
  • There is a lot of good info on this page concerning BIND and DNS…I would recommend reading this when you have a chance.
  • Click on Download BIND button
  • Expand the BIND menu
  • Click on the Download button to the right of the GREEN “Current-Stable” status
  • In the window that pops open, choose the correct version for your system…(click the appropriate blue box)
  • Next, create a directory in your C: drive called “bind”, move the BIND ZIP file to that directory, then double-click on the ZIP file. Extract all of the files in the \bind directory.

Now, lets test. Open up a command prompt and change to the BIND directory, then type in “dig” and press enter.  You should see something like this…(this example is showing DIG returning a list of the ROOT servers on the Internet)…

Running DIG at the command prompt

Running DIG at the command prompt

If it doesn’t work, there are a couple of common errors which are easily fixed…

32 bit installation error:  You may get an error stating “The application has failed to start because its side-by-side configuration is incorrect”.  If that’s the case, run the file “vcredist_x86.exe” which is located in the same “bind” directory. Then try running “dig” again…it should work.

64 bit installation error:  If you get the error message stating that MSVCR110.DLL is missing, then do a search on the C: drive for that file, and copy it to the “bind” directory. Running “dig” should now work.

Now you can play with DIG…some examples…

Some DIG examples

Some DIG examples

DIG is very powerful, and will quickly become your go-to tool when you have any DNS issues. I will talk more about using DIG in a future post.