Author Archives: sr71rocks

Outside Plant Cable Replacement Project – Part 2

Here is the followup to my post on Friday concerning the 100 pair feeder cable replacement. Things went very smoothly, for the most part. There were some issues, but we handled them as they came up, and we finished the project by 3 PM Saturday afternoon.

My main area of concern was removing the old 100 pair cable and getting the new 50 pair cable successfully installed. Most of the conduit is 4″, but there is one 400 foot section that is only 2.5″. With both the 100 pair and fiber installed, there is no way we can get the 50 pair installed first. We needed to remove the 100 pair to make room, which means cutting the 100 pair, and that means no going back.

So we did get the new cable installed from the main building through the first two conduits (all 4″), prior to making any cuts on the old cable. Here is a picture of a pullbox mounted on the side of the main building…note the yellow pull rope already installed. That was nice of the previous vendor many years ago. However, we used the pull rope to install mule tape, which I like better…it doesn’t stretch, plus it is more resistant to creating heat which could burn through any existing cable, such as the fiber Internet circuit (that would not be a good thing at all).

100 pair cable and yellow pull rope

100 pair cable and yellow pull rope

The next section was the 2.5″ conduit….we then cut the 100 pair cable on both ends of the conduit, and pulled it out (and used it to pull in a heavy duty mule tape). We then pulled in the new cable with no issues. There was just one section of 4″ conduit left to go, which went quickly. We ended up getting the entire cable installed Friday evening. Here is a picture of the spool of new cable, and one of the vaults along the path…

Cable spool and vault

Cable spool and vault

Here is a closeup of the spool hanger, which makes everything MUCH easier…

Spool hanger up close

Spool hanger up close

Saturday was termination and testing, which my vendor completed around 1 PM. I then had to move any phone lines that had been terminated on pairs 51 – 100 of the old cable, since they no longer existed. After that, I then tested every single line and the PRI circuit, and verified proper operation. No issues were encountered. I’ll return on Monday to complete my documentation and a bit of remaining cleanup.

And yes, I am breathing a good bit better today…

Outside Plant Cable Replacement Project – Part 1

Yes, I’m back. Sorry for the long absence…I’ve been swamped with a variety of life events, some good and some not so good. Yet in all things, to God be the Glory.

So, what is outside plant (OSP) cable? Well, it’s cable designed to be outside (or perhaps underground), exposed to the weather. The copper or fiber strands are surrounded by a very thick and hard plastic covering, and certain types of OSP cable contain a jelly filling, which would ooze outwards in the event of a tiny hole and not allow water in. Very cool.

At one of my remote sites, we have a 100 pair copper cable that feeds both data and voice services to another building about 1500 feet away. This cable was installed many years ago, and has a number of splice cans along it’s path. Over the last year, the cable has deteriorated substantially, and keeping the many analog lines up and running is almost impossible. (This site is still a large user of faxes and modems.)

Here is one of several pull boxes located along the path…note the conditions of the vault and the splice can located on the right side…

Wet and muddy pullbox

Wet and muddy pullbox

So I hired a vendor to remove the entire 100 pair cable, and install a new 50 pair cable in one long continuous feed, no splice cans at all. The vendor is also going to clean out all of the vaults and layer in some gravel to aid in drainage. We start at 3 PM this afternoon, working until sunset, and will return tomorrow morning. We hope to have the whole project completed by Saturday afternoon.

Some of you might be wondering why I’m going with a smaller, 50 pair cable, and not another 100 pair. Well, the 100 pair cable was installed way back when every desk phone required it’s own pair of copper. Back then, all of the phones were fed through this 100 pair cable. Plus we had multiple T1 circuits for data needs too. Now, though, our phone system is Cisco VoIP, which is handled via the data network and a single PRI circuit, plus the data T1’s have been replaced by a separate fiber network, so the need for 100 pairs is not there, and will never be there. A 50 pair cable is smaller, cheaper and easier to pull, and it will give us plenty of spare pairs if our needs grow.

I will take pictures of the process and will let you know how it goes in a couple of days. Have a great weekend!!

Cool Tool – Table Top Rack

Sometimes a great tool can be both inexpensive and very cool, all at the same time. Take a look at this…

12U Table Top Rack

12U Table Top Rack

This is a great tool for a network engineer….it’s a table top network rack, 12U in height. It’s about 23″ tall, and will hold a number of network devices (depending on size and weight), and will support up to 75 pounds. As you can see, I have several devices sitting in it, handling different jobs, along with a rack-mount power strip at the top.

Now, the rack isn’t perfect…although it is advertised to be the standard 19″ width of a network rack, it’s just a hair under that. (The rack is really designed to hold musicians equipment.) Network equipment does work in the rack, but you might have to squeeze it in a bit, for a certain few types of devices. Plus, I would not want to load it up to the full 75 pound limit. Otherwise, it works great.

For our lab at work, I have a 50/10 Mbps fiber Internet circuit, with a /29 public IP subnet (about 6 usable IP’s) as follows…

  • The Cisco router is handling some testing I’m doing for VTI and DMVPN tunnels
  • The ASA5505 firewall is handling the departments internal LAB network
  • And the Linksys router allows a plain Internet port to sit at each of our PC Techs desks so they can test users laptops for VPN connectivity, and other Internet related issues.

I bought it at CablesAndKits (and here is a link to the rack). I would recommend that you check out CablesAndKits…I have used them for the last couple of years, having purchased some equipment for both my work and church, and my own lab at home. They are great to deal with, and I highly recommend them. (And no, I receive nothing for this.)

Hope you are having a great week!

Technology – Too Fast, Too Soon?

Technology is progressing at an ever faster rate, I think we can all agree to that. And there is nothing necessarily wrong with it. But we do need to be cautious and careful…we need to make sure we are fully in control of all this technology. Yes, perhaps I’m being a bit overly dramatic…but take a look at this…

Boston Dynamics “Spot” (Youtube) – Note…Google purchased Boston Dynamics back in 2013.

Isn’t that AMAZING??!! Incredible!! Didn’t you flinch just a bit when Spot was kicked? And the potential uses for Spot is huge…carrying supplies to soldiers in the battle field, searching for bad guys inside buildings, etc…endless possibilities. And it will not be long before there will be a Spot that can run much faster than a human. (Edit: They already do…I just came across this.) And in the wrong hands, Spot could be very deadly too. Just think about 2 or 3 Spots working together.

Edit: Take a look at this video, also by Boston Dynamics. I sure hope they don’t change their name to Cyberdyne Systems.

What about other disruptive technologies coming our way? How about driverless cars. Seriously, it will not be long before they will be on the roads in large numbers. What’s wrong with that? Perhaps nothing. But what if you are a taxi driver? Don’t you just know that a lot of taxi companies are just looking to the day when they can have a fleet of cars, and not have to pay a single driver.

How about airline pilots? Right now, planes can take off, fly to their destinations, and land themselves. Pilots are there to do a bit of taxiing and to take over should something go wrong. In fact, we have an issue now in which some pilot’s flight skills are “rusty” due to the constant use of the autopilot. And, when all of the planes are controlled by computers, guess what happens next? We don’t need all of those flight controllers on the ground….a computer can easily replace them too.

Ok…just one more example…drones…those small remote controlled copters flying everywhere holding a camera. Just an annoyance now, but drone technology is rapidly advancing. In fact, software is being developed that will allow many drones to operate together, in a “hive”, with a single purpose. So lots of cameras flying around? I don’t think so. What if each drone had a small firearm of some type? Ugh…I don’t want to think about this.

Am I being too alarmist? I sure hope so…I really do. And I will keep telling myself that.

Troubleshooting – Update on My T1 Circuit Issue

As a followup to my post last week concerning troubleshooting a problem T1 circuit, it looks like we are finally making some progress. After working on the issue yesterday, a cable specialist was dispatched out to work on narrowing down exactly what and where the problem lies along the cable span. The carrier sent me an update stating that the LEC (Local Exchange Carrier) has:  “dispatched a cable specialist who has determined there is an unbalanced signal between the last repeater in the local loop distribution plant and the customer premises”.

An “unbalanced signal”….well, that’s a new one for me. But hey, as long as it gets fixed, I’m fine with that.

When I first arrived at work this morning, I checked my stats on the router and verified the circuit was still taking heavy errors…

T1 circuit still taking heavy errors

T1 circuit still taking heavy errors

Later in the day I received an update from the carrier stating that a repair had been made, and that they showed the circuit running clean for the last two hours. Hmmm…really…let me check…

Circuit running clean for the last two hours

Circuit running clean for the last two hours

Well, well…the circuit really is running clean for the last two hours (8 intervals). NICE!!

One more thing to do…clear the counters on the serial interface. Take a look at the stats on the serial interface…LOTS of accumulated errors over the last 2+ years….so lets clear all those stats and keep track of the circuit from this point on…

Show interface information, and clearing the stats

Show interface information, and clearing the stats

I’ll be keeping track of this T1 over the next several days to see if the repair made by the carrier really did fix the issue. Fingers are crossed!!

Network Engineer – An Invisible Job…Until Things Break

A lot of jobs within the IT field are somewhat invisible…what I mean is most of the company never really sees you. And one of the most invisible positions is that of a Network Engineer. Few people know we exist, and that’s ok by most of us, as we are usually very private by nature. Because of that, though, most people don’t really understand what we do and how important our jobs are.

When you have a few minutes, please read All Systems Down (pdf) by Scott Berinato for CIO magazine back in 2003. It chronicles a major network crisis which occurred in late 2002 at a large medical center in Boston. It is an incredible read!! In fact, I make sure I read it once a year. Why?

Because it reminds me of how important my job is. Sure, I don’t work at a hospital where lives are at stake, but still…the job I do is vitally important to over a thousand people every day. Their ability to get their jobs done in a quick and efficient manner relies largely on the stability and availability of my network. Plus, I learn a lot from this article in terms of attacking problems and working together. And I have to say a big “Thank you” to the CIO (John Halamka) which shared this story…the lessons learned is something every Network Engineer needs to take heed of.

Let me know what you think.

(Note: The article was not written by a Network Engineer, but a CIO journalist. As such, some of his descriptions are not 100% technically correct. But it does not in any way degrade the content and impact of the article.)

Troubleshooting – T1 Circuit Errors and Controller Stats

For the last several weeks, I’ve been having a T1 circuit issue at one of my remote sites. The carrier has been working the problem, but the issue is intermittent and difficult to narrow down. This site is way out in the boonies, and I think some of the cable span is old and some moisture has leaked into the cable. So, what can you do to see the health of a T1 circuit? Take a look at the controller stats using the command…

show controller t1 0/0/0         (use the appropriate card slot numbering for your interface)

Each Cisco router keeps a log of the errors on a T1 circuit for the past 24 hours, in 15 minute blocks…so 96 “intervals” as we say. Take a look at this snippet of a clean running T1 circuit…

Example of a clean running T1 circuit

Example of a clean running T1 circuit

The first data interval is for the current 15 minute block, and shows the elapsed time…in this case 351 seconds. After that, each interval is a full 15 minutes, and this sample shows a very clean running T1 circuit. Notice the last block of data shows the summary of all errors for the preceding 24 hours (96 intervals). I sure wish all my T1’s ran this clean.

Now, here is a snippet from my problem T1 taken earlier today…

Controller stats of a T1 circuit having physical layer issues

Controller stats of a T1 circuit having physical layer issues

A bit messy wouldn’t you say? The first 3 intervals show a circuit up and running, but VERY poorly…few, if any, applications would work properly over this type of circuit (and they weren’t, which my end customer could vouch for). Take a look at interval 17…there are 900 unavailable seconds, which is how many seconds there are in 15 minutes. So for this interval, the circuit was completely down. And notice the Total Data for all intervals…this circuit is indeed in very poor health.

What does this information tell you? Basically, with this kind of high error rate, the problem is almost always with the carrier (issues with the cable span, NIU, or Central Office equipment). In all my years of troubleshooting T1 circuits, I’ve only had a few times where the issue was on my side (it was cabling issues with my extended DMARC usually). And remember, you can copy this information and send it to the carrier to help prove your case.

Hope this helps!

Know Your Network – Documentation (Part1) – WAN Drawing

This post is part of a series that covers what I feel to be the main (or core) tasks that a Network Engineer is responsible for. See my Know Your Network – Introduction page for more details.

Documentation is a big subject, and can be overwhelming. So lets keep this simple. First things first…you need a document that shows all of your wide area network (WAN) circuits. This document should include the information needed to troubleshoot any issues, open tickets with the carriers when things do go wrong, and basically include all information needed to manage your network. Here is a very sanitized copy of my WAN drawing…(I removed about half of the drawing and dummied up the remaining confidential information)…

ACME_Network_Diagram (pdf)

Things to include for each circuit…

  • Carrier circuit ID (also LEC ID when possible)
  • Bandwidth
  • IP addressing (both public and private)
  • Location (remote site)
  • LAN subnet(s) at remote site
  • Router make/model
  • Secure modem information (for those sites that have out-of-band management)

When creating the document, I would also recommend…

  • Use Microsoft Visio. It is uniquely designed for this type of work, and has a number of templates and objects which will make the process easier. Also, most vendors (such as Cisco, HP, Juniper, etc) have created their own Visio objects of their hardware, and put them in files called “stencils” which you can download and add to Visio. This is VERY convenient! (Example: just Google “cisco stencils”)
  • Use color!! By this, I mean do all IP addressing in red, circuit ID’s in blue, PRI circuits in black…etc. Choose your own colors, but stick with it…make it a standard.
  • Create separate layers within the Visio drawing. One layer will be IP addressing, another layer Carrier circuit ID’s, and another circuit bandwidth….you get the idea. This does take time to initially setup, but once it is done, you will have a document that you can easily customize. As an example…you have a vendor meet and they request a network diagram. You don’t want to give them a drawing that shows any confidential information, so prior to printing or saving as a PDF, you can turn off certain layers, such as IP addressing and circuit ID’s. This makes sharing your network diagram both easy and secure, and you don’t have to maintain multiple documents.
  • When creating host names for your network devices (routers, switches, etc) I would use a standard naming convention that has meaning. For example, look at the host names for the construction offices on my sample drawing…the first three letters are all “con” for construction. The next three letters show location (city), and then the final three characters show what the device is (rt-router, sw-switch, fw-firewall, ap-accesspoint, etc), followed by a number (1,2,3, etc) for each device, as there could be more than one at a location, such as switches.
  • Include a Legend that shows all carrier contact information and drawing color definitions.
  • Make it easy to read…print it out on standard tabloid paper (11″x17″). You can fold it in half for easy storage in your laptop case. Also, save as PDF and send to your smartphone and tablet.

This is one of the most important documents you will create and it will greatly aid you in managing your network. If you take your time and do it right, it will serve you well over the years, and will be easy to maintain and update.

Hope this helps…let me know what you think!

Run the Race – Don’t Live as the World Lives

Over the last several months, I have really noticed how easy it is to live as the world lives. I’m not talking about the big things, like buying lots of stuff, having an affair, abusing drugs and alcohol…things like that. I’m talking about the little things that are easy to slip into…things like…

  • Lack of empathy – It’s way too easy to not see those that are in need around you. And it’s not just the homeless. With the current economy, there are many people that are just barely making ends meet. One flat tire or broken refrigerator could turn their lives upside down.
  • Lack of respect – I see this in the way we all drive. People become practically robots, focused on just getting to their destination. We cut people off, flip people the “bird”, and are just plain rude to everyone in our path.

As Christians, we need to reflect Christ to the world around us. And this means in all of the little things too. If you have a little extra cash, help someone fix their tire or change their oil. In the middle of the afternoon traffic, give them the extra space to pull in. Be courteous and respectful in everything you do. And just smile.

Romans 12:2  Don’t copy the behavior and customs of this world, but let God transform you into a new person by changing the way you think… (NLT)

You know, if all of us Christians just smiled and said a heartfelt “hello” to everyone we meet…the world would sit up and notice! And they would want to know why we are so nice…and we could tell them about Christ and his love for them. Just think of the lives that would be changed!

As for me, I’m going to try and do a better job of reflecting Christ to the world.

Know Your Network – Introduction

Finally. My life has calmed down (a bit anyway), and I’m able to get back to my website and do some posting.

I’m going to start a series of posts having to do with the key responsibilities of a network engineer. If you are a new network engineer and just starting out, what are the main tasks you should concentrate on? Or, perhaps you have been a network engineer for a while, but work is keeping you so busy that you are concerned about forgetting to do key tasks in managing the network. I also have seen some network engineers so busy playing with the latest cool toys, that they end up neglecting their main responsibility. Either way, what are the key responsibilities and/or tasks that need to be done to properly manage a network? Here are my key areas that I make sure and focus on…

  1. Documentation – Know what networks you have (carriers, circuit id’s, support information, IP address assignments, etc.). Updated: WAN Drawing
  2. Backups – Maintain proper backups of all your key network infrastructure (router configs and IOS images, switch configs, firewall configs and filter descriptions, along with backup/VMDK files of network related servers).
  3. Logging – this includes SYSLOG’s from your network devices for user access tracking, alerting on device failures, configuration changes, power outages, etc.
  4. Network Outages – Proactively monitor your network for any outages, and be ready to respond quickly and accurately. (It’s very cool to call a remote site letting them know of a network outage, and they haven’t even noticed it yet.)
  5. Circuit Utilization – Know what traffic is running across your network, and be able to quickly identify applications that might be hogging too much bandwidth or be misbehaving. This will also give you the ability to perform capacity planning for future needs.
  6. Perimeter Protection – For the most part, this covers your firewall and any perimeter router(s) you may have in place. Tighten down the security on these devices per best practices.
  7. Cool Tools – Once you have the basics down, then you can start looking at some of the new tools and applications that can assist you in maintaining a robust and secure network (IE: Intrusion Protection Systems (IDS/IPS), Security Information and Event Management (SIEM’s), etc.)

Over the next several weeks I will take a more detailed look into each of the above items, and show you what I use to handle these tasks. Let me know what you think.

Thanks!