Monthly Archives: January 2015

Network Engineer – An Invisible Job…Until Things Break

A lot of jobs within the IT field are somewhat invisible…what I mean is most of the company never really sees you. And one of the most invisible positions is that of a Network Engineer. Few people know we exist, and that’s ok by most of us, as we are usually very private by nature. Because of that, though, most people don’t really understand what we do and how important our jobs are.

When you have a few minutes, please read All Systems Down (pdf) by Scott Berinato for CIO magazine back in 2003. It chronicles a major network crisis which occurred in late 2002 at a large medical center in Boston. It is an incredible read!! In fact, I make sure I read it once a year. Why?

Because it reminds me of how important my job is. Sure, I don’t work at a hospital where lives are at stake, but still…the job I do is vitally important to over a thousand people every day. Their ability to get their jobs done in a quick and efficient manner relies largely on the stability and availability of my network. Plus, I learn a lot from this article in terms of attacking problems and working together. And I have to say a big “Thank you” to the CIO (John Halamka) which shared this story…the lessons learned is something every Network Engineer needs to take heed of.

Let me know what you think.

(Note: The article was not written by a Network Engineer, but a CIO journalist. As such, some of his descriptions are not 100% technically correct. But it does not in any way degrade the content and impact of the article.)

Troubleshooting – T1 Circuit Errors and Controller Stats

For the last several weeks, I’ve been having a T1 circuit issue at one of my remote sites. The carrier has been working the problem, but the issue is intermittent and difficult to narrow down. This site is way out in the boonies, and I think some of the cable span is old and some moisture has leaked into the cable. So, what can you do to see the health of a T1 circuit? Take a look at the controller stats using the command…

show controller t1 0/0/0         (use the appropriate card slot numbering for your interface)

Each Cisco router keeps a log of the errors on a T1 circuit for the past 24 hours, in 15 minute blocks…so 96 “intervals” as we say. Take a look at this snippet of a clean running T1 circuit…

Example of a clean running T1 circuit

Example of a clean running T1 circuit

The first data interval is for the current 15 minute block, and shows the elapsed time…in this case 351 seconds. After that, each interval is a full 15 minutes, and this sample shows a very clean running T1 circuit. Notice the last block of data shows the summary of all errors for the preceding 24 hours (96 intervals). I sure wish all my T1’s ran this clean.

Now, here is a snippet from my problem T1 taken earlier today…

Controller stats of a T1 circuit having physical layer issues

Controller stats of a T1 circuit having physical layer issues

A bit messy wouldn’t you say? The first 3 intervals show a circuit up and running, but VERY poorly…few, if any, applications would work properly over this type of circuit (and they weren’t, which my end customer could vouch for). Take a look at interval 17…there are 900 unavailable seconds, which is how many seconds there are in 15 minutes. So for this interval, the circuit was completely down. And notice the Total Data for all intervals…this circuit is indeed in very poor health.

What does this information tell you? Basically, with this kind of high error rate, the problem is almost always with the carrier (issues with the cable span, NIU, or Central Office equipment). In all my years of troubleshooting T1 circuits, I’ve only had a few times where the issue was on my side (it was cabling issues with my extended DMARC usually). And remember, you can copy this information and send it to the carrier to help prove your case.

Hope this helps!

Know Your Network – Documentation (Part1) – WAN Drawing

This post is part of a series that covers what I feel to be the main (or core) tasks that a Network Engineer is responsible for. See my Know Your Network – Introduction page for more details.

Documentation is a big subject, and can be overwhelming. So lets keep this simple. First things first…you need a document that shows all of your wide area network (WAN) circuits. This document should include the information needed to troubleshoot any issues, open tickets with the carriers when things do go wrong, and basically include all information needed to manage your network. Here is a very sanitized copy of my WAN drawing…(I removed about half of the drawing and dummied up the remaining confidential information)…

ACME_Network_Diagram (pdf)

Things to include for each circuit…

  • Carrier circuit ID (also LEC ID when possible)
  • Bandwidth
  • IP addressing (both public and private)
  • Location (remote site)
  • LAN subnet(s) at remote site
  • Router make/model
  • Secure modem information (for those sites that have out-of-band management)

When creating the document, I would also recommend…

  • Use Microsoft Visio. It is uniquely designed for this type of work, and has a number of templates and objects which will make the process easier. Also, most vendors (such as Cisco, HP, Juniper, etc) have created their own Visio objects of their hardware, and put them in files called “stencils” which you can download and add to Visio. This is VERY convenient! (Example: just Google “cisco stencils”)
  • Use color!! By this, I mean do all IP addressing in red, circuit ID’s in blue, PRI circuits in black…etc. Choose your own colors, but stick with it…make it a standard.
  • Create separate layers within the Visio drawing. One layer will be IP addressing, another layer Carrier circuit ID’s, and another circuit bandwidth….you get the idea. This does take time to initially setup, but once it is done, you will have a document that you can easily customize. As an example…you have a vendor meet and they request a network diagram. You don’t want to give them a drawing that shows any confidential information, so prior to printing or saving as a PDF, you can turn off certain layers, such as IP addressing and circuit ID’s. This makes sharing your network diagram both easy and secure, and you don’t have to maintain multiple documents.
  • When creating host names for your network devices (routers, switches, etc) I would use a standard naming convention that has meaning. For example, look at the host names for the construction offices on my sample drawing…the first three letters are all “con” for construction. The next three letters show location (city), and then the final three characters show what the device is (rt-router, sw-switch, fw-firewall, ap-accesspoint, etc), followed by a number (1,2,3, etc) for each device, as there could be more than one at a location, such as switches.
  • Include a Legend that shows all carrier contact information and drawing color definitions.
  • Make it easy to read…print it out on standard tabloid paper (11″x17″). You can fold it in half for easy storage in your laptop case. Also, save as PDF and send to your smartphone and tablet.

This is one of the most important documents you will create and it will greatly aid you in managing your network. If you take your time and do it right, it will serve you well over the years, and will be easy to maintain and update.

Hope this helps…let me know what you think!

Run the Race – Don’t Live as the World Lives

Over the last several months, I have really noticed how easy it is to live as the world lives. I’m not talking about the big things, like buying lots of stuff, having an affair, abusing drugs and alcohol…things like that. I’m talking about the little things that are easy to slip into…things like…

  • Lack of empathy – It’s way too easy to not see those that are in need around you. And it’s not just the homeless. With the current economy, there are many people that are just barely making ends meet. One flat tire or broken refrigerator could turn their lives upside down.
  • Lack of respect – I see this in the way we all drive. People become practically robots, focused on just getting to their destination. We cut people off, flip people the “bird”, and are just plain rude to everyone in our path.

As Christians, we need to reflect Christ to the world around us. And this means in all of the little things too. If you have a little extra cash, help someone fix their tire or change their oil. In the middle of the afternoon traffic, give them the extra space to pull in. Be courteous and respectful in everything you do. And just smile.

Romans 12:2  Don’t copy the behavior and customs of this world, but let God transform you into a new person by changing the way you think… (NLT)

You know, if all of us Christians just smiled and said a heartfelt “hello” to everyone we meet…the world would sit up and notice! And they would want to know why we are so nice…and we could tell them about Christ and his love for them. Just think of the lives that would be changed!

As for me, I’m going to try and do a better job of reflecting Christ to the world.

Know Your Network – Introduction

Finally. My life has calmed down (a bit anyway), and I’m able to get back to my website and do some posting.

I’m going to start a series of posts having to do with the key responsibilities of a network engineer. If you are a new network engineer and just starting out, what are the main tasks you should concentrate on? Or, perhaps you have been a network engineer for a while, but work is keeping you so busy that you are concerned about forgetting to do key tasks in managing the network. I also have seen some network engineers so busy playing with the latest cool toys, that they end up neglecting their main responsibility. Either way, what are the key responsibilities and/or tasks that need to be done to properly manage a network? Here are my key areas that I make sure and focus on…

  1. Documentation – Know what networks you have (carriers, circuit id’s, support information, IP address assignments, etc.). Updated: WAN Drawing
  2. Backups – Maintain proper backups of all your key network infrastructure (router configs and IOS images, switch configs, firewall configs and filter descriptions, along with backup/VMDK files of network related servers).
  3. Logging – this includes SYSLOG’s from your network devices for user access tracking, alerting on device failures, configuration changes, power outages, etc.
  4. Network Outages – Proactively monitor your network for any outages, and be ready to respond quickly and accurately. (It’s very cool to call a remote site letting them know of a network outage, and they haven’t even noticed it yet.)
  5. Circuit Utilization – Know what traffic is running across your network, and be able to quickly identify applications that might be hogging too much bandwidth or be misbehaving. This will also give you the ability to perform capacity planning for future needs.
  6. Perimeter Protection – For the most part, this covers your firewall and any perimeter router(s) you may have in place. Tighten down the security on these devices per best practices.
  7. Cool Tools – Once you have the basics down, then you can start looking at some of the new tools and applications that can assist you in maintaining a robust and secure network (IE: Intrusion Protection Systems (IDS/IPS), Security Information and Event Management (SIEM’s), etc.)

Over the next several weeks I will take a more detailed look into each of the above items, and show you what I use to handle these tasks. Let me know what you think.


Another Example of Physical Damage by a CyberAttack

As a follow-up to an earlier post concerning real (physical) damage from cyber-attacks, check out this post on Wired about damage done to a steel mill in Germany. Talk about scary…and it’s just going to get worse, I’m afraid. At least until people understand that infrastructure and control networks MUST be separated and secured from the Internet and other Internet facing networks/systems. In the simplest form, you can tie the two networks together and remove the connecting cable….leave it unconnected, except only when needing to perform patches, etc. And then, lock the connecting ports up in a box of some sort, and only the CIO and Admin have the keys. (I’m not kidding folks.)

Yes, I know….this is a bit too simplistic and perhaps not viable in the real world. But, we need to take this seriously. With the escalation of nation state “cyberwar”, you will be seeing more examples of this over the next couple of years. I’m worried…are you?

A New Year – And Time for the CCNP

So, 2014 is done. And a new year has begun. I wish all of you a Happy New Year for 2015!

For me, this year will be busy as I’m going to start studying for the Cisco CCNP certification. I actually decided this a while back, plus my work has made this a requirement for being a Senior Network Engineer. I was going to start this past summer, but then Cisco revamped the CCNP (known as CCNP v2), so I decided to wait until the new v2 study materials were released (which occurred in December). I just received the new ROUTE, SWITCH and TSHOOT hard cover books and eBooks over the last couple of weeks….wow, there is a LOT of material to learn. (You can learn about the new CCNP here.)

So why get a certification? I was asked that recently by my friend Shane (he has a GREAT blog that you should check out). There are many different views about certifications…they are great, they are a waste of time, them mean nothing, or they are a great demonstration of ones abilities. To a degree, all of these views are true. So, here is my take on it…

For most people, I think certifications are a good idea. In certain areas of the country (such as Northern California where I live), certifications are either required or really desired. They might not get you a job, but they can get you an interview, and that’s half the battle. BUT, you need to know what you are talking about…no certification will take the place of experience. They both go hand in hand. As you get the experience, also start obtaining relevant certifications.

Also, as in my case, the study required for a cert makes me a better Network Engineer and employee. It fills in some gaps I may have in terms of technical knowledge and keeps me up to date on new equipment and industry solutions, which may come in handy at work some day.

Now, you may not need or want a certification…that’s ok too. But you do need to keep learning…the Networking industry is always changing and moving forward, and it is easy to get left behind if you are not careful. Obtaining certifications may help with this.

Hope this helps if you are thinking about going for a certification. Let me know what you think.