As a followup to my post last week concerning troubleshooting a problem T1 circuit, it looks like we are finally making some progress. After working on the issue yesterday, a cable specialist was dispatched out to work on narrowing down exactly what and where the problem lies along the cable span. The carrier sent me an update stating that the LEC (Local Exchange Carrier) has:  “dispatched a cable specialist who has determined there is an unbalanced signal between the last repeater in the local loop distribution plant and the customer premises”.

An “unbalanced signal”….well, that’s a new one for me. But hey, as long as it gets fixed, I’m fine with that.

When I first arrived at work this morning, I checked my stats on the router and verified the circuit was still taking heavy errors…

T1 circuit still taking heavy errors

Later in the day I received an update from the carrier stating that a repair had been made, and that they showed the circuit running clean for the last two hours. Hmmm…really…let me check…

Circuit running clean for the last two hours

Well, well…the circuit really is running clean for the last two hours (8 intervals). NICE!!

One more thing to do…clear the counters on the serial interface. Take a look at the stats on the serial interface…LOTS of accumulated errors over the last 2+ years….so lets clear all those stats and keep track of the circuit from this point on…

Show interface information, and clearing the stats

I’ll be keeping track of this T1 over the next several days to see if the repair made by the carrier really did fix the issue. Fingers are crossed!!

For the last several weeks, I’ve been having a T1 circuit issue at one of my remote sites. The carrier has been working the problem, but the issue is intermittent and difficult to narrow down. This site is way out in the boonies, and I think some of the cable span is old and some moisture has leaked into the cable. So, what can you do to see the health of a T1 circuit? Take a look at the controller stats using the command…

show controller t1 0/0/0         (use the appropriate card slot numbering for your interface)

Each Cisco router keeps a log of the errors on a T1 circuit for the past 24 hours, in 15 minute blocks…so 96 “intervals” as we say. Take a look at this snippet of a clean running T1 circuit…

Example of a clean running T1 circuit

The first data interval is for the current 15 minute block, and shows the elapsed time…in this case 351 seconds. After that, each interval is a full 15 minutes, and this sample shows a very clean running T1 circuit. Notice the last block of data shows the summary of all errors for the preceding 24 hours (96 intervals). I sure wish all my T1’s ran this clean.

Now, here is a snippet from my problem T1 taken earlier today…

Controller stats of a T1 circuit having physical layer issues

A bit messy wouldn’t you say? The first 3 intervals show a circuit up and running, but VERY poorly…few, if any, applications would work properly over this type of circuit (and they weren’t, which my end customer could vouch for). Take a look at interval 17…there are 900 unavailable seconds, which is how many seconds there are in 15 minutes. So for this interval, the circuit was completely down. And notice the Total Data for all intervals…this circuit is indeed in very poor health.

What does this information tell you? Basically, with this kind of high error rate, the problem is almost always with the carrier (issues with the cable span, NIU, or Central Office equipment). In all my years of troubleshooting T1 circuits, I’ve only had a few times where the issue was on my side (it was cabling issues with my extended DMARC usually). And remember, you can copy this information and send it to the carrier to help prove your case.

Hope this helps!

Most all networks have T1 circuits, the most common being either an MPLS T1 or Internet T1. There will be times when one of your T1 circuits will be acting up in a sporadic manner, causing “slowness” for your end users, and will require you to be more proactive in troubleshooting the root cause. This post will talk about how to stress test a T1 using PING.

First off, understand that using a PING command with the default parameters will tell you if the circuit is up or down, and it may show problems such as large latency or excessive drops. But to really test a T1, you need to modify the use of PING to perform a more thorough test. Common actions are to increase the packet size and frequency of pings to better test throughput, and to use specific data patterns to better test the operation of the T1.

You can use Cisco’s PING which is part of IOS.  Here is an example of an extended PING where we increase the packet size to the max MTU of 1500 bytes, and run all 1’s (which will provide additional stress on the circuit)…

Using Cisco’s IOS ping command

A much better PING to use though is Linux, with the “flood” option, as this will allow you to really hammer the T1 circuit. (Note…you need to be root to use the flood option.) The difference is this…Cisco’s PING will send an echo-request, but will wait for the echo-reply before it can send another echo-request. This greatly reduces the amount of ping traffic IOS can send across the T1. Linux however, will immediately start sending as many packets as it can, up to 100 per second. For each echo-request packet it sends, it prints a “.” (dot) on the screen. For each echo-reply it receives, it prints a back-space. So if you only see a couple of dots, then the circuit is handling your ping flood easily. However, if you start seeing dots race across the screen, then there are problems. Here is a Linux PING flood example with 1500 byte packets and running all 1’s…

Using Linux ping -f (flood) option to stress a T1 circuit

As you can see, there are only three dots…7356 packets were sent and 7353 were received. That leaves 3 missing packets. This T1 easily handled this test. Plus a Linux ping flood will typically load up a T1 in the range of 700-900 Kb (about 1/2 of a T1 circuit). If you really want to fully load up a T1, run two different instances of ping flood, and you will see a T1 circuit fully saturated (or near so). Of course, do NOT do this during normal business operations…you will heavily impact the end users, and they will not be happy. When running the Linux ping flood shown above, the resulting bandwidth impact on the T1 was…

Using “show interface” to see bandwidth impact of ping flood

In my next post I will give an example of how I used ping flood to troubleshoot a T1 circuit whose performance was impacted by a unique problem.