It is the last Friday in July, and we all know what that means…yes, it’s SysAdmin Day!!

Lesson: Don’t mess with us SysAdmin’s!!

So, did anyone at work tell you Happy SysAdmin Day? Didn’t think so. We really are the unsung heroes. No one thinks about us at all until something breaks…then it’s “The Internet is down!!”. Like, really???…the whole Internet is down? Ugh…

But we fix the problem anyway. Everyday. Because that’s our job…and we love our job!! At least I do…I enjoy being a Network Engineer. I hope you are enjoying your job too!! And I hope someone at work today told you “Happy SysAdmin Day”, and thanked you for all of your hard work.

Have a great weekend!!

A lot of jobs within the IT field are somewhat invisible…what I mean is most of the company never really sees you. And one of the most invisible positions is that of a Network Engineer. Few people know we exist, and that’s ok by most of us, as we are usually very private by nature. Because of that, though, most people don’t really understand what we do and how important our jobs are.

When you have a few minutes, please read All Systems Down (pdf) by Scott Berinato for CIO magazine back in 2003. It chronicles a major network crisis which occurred in late 2002 at a large medical center in Boston. It is an incredible read!! In fact, I make sure I read it once a year. Why?

Because it reminds me of how important my job is. Sure, I don’t work at a hospital where lives are at stake, but still…the job I do is vitally important to over a thousand people every day. Their ability to get their jobs done in a quick and efficient manner relies largely on the stability and availability of my network. Plus, I learn a lot from this article in terms of attacking problems and working together. And I have to say a big “Thank you” to the CIO (John Halamka) which shared this story…the lessons learned is something every Network Engineer needs to take heed of.

Let me know what you think.

(Note: The article was not written by a Network Engineer, but a CIO journalist. As such, some of his descriptions are not 100% technically correct. But it does not in any way degrade the content and impact of the article.)

This post is part of a series that covers what I feel to be the main (or core) tasks that a Network Engineer is responsible for. See my Know Your Network – Introduction page for more details.

Documentation is a big subject, and can be overwhelming. So lets keep this simple. First things first…you need a document that shows all of your wide area network (WAN) circuits. This document should include the information needed to troubleshoot any issues, open tickets with the carriers when things do go wrong, and basically include all information needed to manage your network. Here is a very sanitized copy of my WAN drawing…(I removed about half of the drawing and dummied up the remaining confidential information)…

ACME_Network_Diagram (pdf)

Things to include for each circuit…

• Carrier circuit ID (also LEC ID when possible)
• Bandwidth
• IP addressing (both public and private)
• Location (remote site)
• LAN subnet(s) at remote site
• Router make/model
• Secure modem information (for those sites that have out-of-band management)

When creating the document, I would also recommend…

• Use Microsoft Visio. It is uniquely designed for this type of work, and has a number of templates and objects which will make the process easier. Also, most vendors (such as Cisco, HP, Juniper, etc) have created their own Visio objects of their hardware, and put them in files called “stencils” which you can download and add to Visio. This is VERY convenient! (Example: just Google “cisco stencils”)
• Use color!! By this, I mean do all IP addressing in red, circuit ID’s in blue, PRI circuits in black…etc. Choose your own colors, but stick with it…make it a standard.
• Create separate layers within the Visio drawing. One layer will be IP addressing, another layer Carrier circuit ID’s, and another circuit bandwidth….you get the idea. This does take time to initially setup, but once it is done, you will have a document that you can easily customize. As an example…you have a vendor meet and they request a network diagram. You don’t want to give them a drawing that shows any confidential information, so prior to printing or saving as a PDF, you can turn off certain layers, such as IP addressing and circuit ID’s. This makes sharing your network diagram both easy and secure, and you don’t have to maintain multiple documents.
• When creating host names for your network devices (routers, switches, etc) I would use a standard naming convention that has meaning. For example, look at the host names for the construction offices on my sample drawing…the first three letters are all “con” for construction. The next three letters show location (city), and then the final three characters show what the device is (rt-router, sw-switch, fw-firewall, ap-accesspoint, etc), followed by a number (1,2,3, etc) for each device, as there could be more than one at a location, such as switches.
• Include a Legend that shows all carrier contact information and drawing color definitions.
• Make it easy to read…print it out on standard tabloid paper (11″x17″). You can fold it in half for easy storage in your laptop case. Also, save as PDF and send to your smartphone and tablet.

This is one of the most important documents you will create and it will greatly aid you in managing your network. If you take your time and do it right, it will serve you well over the years, and will be easy to maintain and update.

Hope this helps…let me know what you think!

Finally. My life has calmed down (a bit anyway), and I’m able to get back to my website and do some posting.

I’m going to start a series of posts having to do with the key responsibilities of a network engineer. If you are a new network engineer and just starting out, what are the main tasks you should concentrate on? Or, perhaps you have been a network engineer for a while, but work is keeping you so busy that you are concerned about forgetting to do key tasks in managing the network. I also have seen some network engineers so busy playing with the latest cool toys, that they end up neglecting their main responsibility. Either way, what are the key responsibilities and/or tasks that need to be done to properly manage a network? Here are my key areas that I make sure and focus on…

1. Documentation – Know what networks you have (carriers, circuit id’s, support information, IP address assignments, etc.). Updated: WAN Drawing
2. Backups – Maintain proper backups of all your key network infrastructure (router configs and IOS images, switch configs, firewall configs and filter descriptions, along with backup/VMDK files of network related servers).
3. Logging – this includes SYSLOG’s from your network devices for user access tracking, alerting on device failures, configuration changes, power outages, etc.
4. Network Outages – Proactively monitor your network for any outages, and be ready to respond quickly and accurately. (It’s very cool to call a remote site letting them know of a network outage, and they haven’t even noticed it yet.)
5. Circuit Utilization – Know what traffic is running across your network, and be able to quickly identify applications that might be hogging too much bandwidth or be misbehaving. This will also give you the ability to perform capacity planning for future needs.
6. Perimeter Protection – For the most part, this covers your firewall and any perimeter router(s) you may have in place. Tighten down the security on these devices per best practices.
7. Cool Tools – Once you have the basics down, then you can start looking at some of the new tools and applications that can assist you in maintaining a robust and secure network (IE: Intrusion Protection Systems (IDS/IPS), Security Information and Event Management (SIEM’s), etc.)

Over the next several weeks I will take a more detailed look into each of the above items, and show you what I use to handle these tasks. Let me know what you think.

Thanks!

I had an interesting lunch today. I went to my favorite fast-food place (Chic-Fil-A….they have the best ice tea!!). Anyway, when I arrived the parking lot was full of California Highway Patrol vehicles…I remember thinking that lunch today will be very safe! It was also raining (much needed here in CA), and so I put my iPad under my trenchcoat, up under my left arm-pit, and clamped down on it with my left arm. As I walked into the restaurant, I walked down an aisle full of CHP officers, all chatting and laughing with their peers.

So…it was about then that I decided to remove my iPad.  (Yeah, I know…what was I thinking.) So I put my right hand into my trenchcoat and grabbed the iPad. It was at that time that a whole bunch of eyes were instantly fixed on me as I pulled out the iPad. As soon as they saw the iPad, their eyes went back to their friends. What was interesting is they never stopped talking or laughing…being this aware and alert was normal for them. I bet most of them didn’t even realize what they just did. In other words, they have trained to be alert and aware, and it worked.

So, why bring this up? As network engineers, we need to practice being aware and alert to what is around us. When passing some users, did you hear them mention something about slow Internet? Make a mental note. Did you notice the SSH session pausing momentarily as you scrolled through some configurations from a remote router? Make a mental note. (This happened to me recently….it didn’t feel right, so I tested and found out this circuit was experiencing an above average packet loss.) Heard some users complaining about how slow the ERP application was? Make a mental note.

As you start making these notes, you may see a picture emerging that could very well point to an issue in your network. Or perhaps an issue higher up the protocol stack…maybe not even your problem, but you could alert the correct department and let them run with it. (Years ago, at another company, I noticed some people randomly complaining of slow Internet response. I started doing some testing, and was able to confirm that a random issue was occurring. It ended up being some old BIND DNS servers that were using old “hints” files. We downloaded updated hints files from ISC, and the problem was fixed…and the users even noticed the quicker Internet response!!)

As network engineers, we are uniquely positioned to notice most any issue with the network…as long as we are alert and aware. So if being aware is not second nature to you, start practicing it everyday. Don’t tune the world out…but listen to it. You will end up being a better network engineer.

Hello again. As a follow-up to last weeks post (read it here), this past week was once again way too busy. I spent the entire week at the new office, working 12 hour days, getting it ready for move in and go-live. The following tasks were accomplished…

• All network cabling was installed, labeled and tested. This consisted of about 60 workstations, running two data cables to each workstation. The cabling vendor is a company I’ve worked with for many years…they know what they are doing and it shows in the final product. No worries here.
• A solid wall of backboard was installed in the MPOE, and on that was mounted a swing-out Chatsworth rack. A bit pricey but worth the extra money…the whole rack can pivot to the side, giving access to the rear of the equipment. (Check out Chatsworth’s Swing Gate if you are interested.)
• Network router (Cisco 2851), Cisco switch stack (3750’s), and several Cisco Access Points were installed, configured and tested.
• The new PRI circuit was tested and 100 DID’s (Direct Inward Dialing) were ported over from our old PRI circuit.
• The wireless broadband is working well, but I am still keeping my eye on it. Not sure if the vendor fine tuned it or not, but I am seeing better performance.
• Security cameras and a key-fob access system was installed.

It was a long, but successful week. I am also glad this type of project does not occur often.

I hope you had a great weekend!!

Sorry about the lack of posts this week…I have just been way too busy, and working some long hours.  I will get back on track this weekend. Here is a quick summary of my week…

• Suffered a network outage at one of our busiest District Offices. I had to travel to the location and work with the carrier (a major fiber and Internet carrier), and troubleshoot with them over the phone. As always, they said the issue was with my equipment. (Carriers almost ALWAYS say the issue is with your equipment.) And like always, I have to prove to them that it’s their issue…which it was. Somehow, the VLAN carrying my traffic was changed which brought my network down. We finally got the circuit back up at 2 AM, twelve hours after it went down. Ugh.  And 3 days later, they still cannot tell me how that happened. I’m like “Is there really that many people that can make those types of changes? Don’t you track your changes?”  I guess they don’t.
• I’m the PM (Project Manager) for the IT part of a new District Office which is going live in a couple of weeks. Yes, this is the location in which we have had major issues with the LEC (Local Exchange Carrier). Check out some earlier posts (Part 1 and Part 2) which talks about these challenges. We did finally get a PRI circuit installed, but no fiber Internet. I ended up using a vendor that offers high speed wireless broadband. I was onsite for a couple of days, bringing this up and testing. The circuit is 15 Mb, up and down. It’s working relatively good, but I’m seeing a bit of an issue with large packets (over 1100 bytes)…I have a consistent packet loss of between 1-2%. I know that does not sound like much, but when you are moving large files around, that ends up pushing your through-put way down to around 6 Mb. I will say this…the vendor is very easy to work with, and they already are going to work with me next week to resolve this.
• I had an MPLS T1 circuit at a very remote site giving me fits all week long. It was taking errors pretty much 24×7, and even going down for several hours at a time almost every day. The carrier dispatched out multiple times before finally getting the issue resolved. (They had to replace multiple jumpers, and redo some splices.) It’s now been running clean for almost 48 hours straight. My thanks to the technician who hung in there and got this fixed.
• We recently opened up a temporary site out in the boonies…like way out. This site has no copper facilities at all…no phones, no network circuits…nada. However, it is located right next to a major Interstate, and there is a Verizon tower nearby. I was tasked with getting a Cradlepoint router (with a 4G Verizon card attached) to run DMVPN (Dynamic Multipoint VPN), and connect with a Cisco router at my Data Center. This was a challenge, especially since Verizon likes to run double NAT’ing in their 4G networks. Yep, the 4G card gets a valid public IP address, but that’s not what’s seen on the Internet. Somewhere upstream, still within Verizon’s network, it gets NAT’d again with a different public IP. (Way to go Verizon.) Well, I did get DMVPN to work after much trial and error. We are testing now to see how stable it is, and hope to install it at the site in the next week or two.

As you can see, it was a busy week. And next week will be just as busy.  I’m going to be down at the new District Office most of the week, overseeing all of the cabling, cutting over to the new PRI, installing the network equipment, and working on resolving the packet loss issue. Wish me luck.

And, have a great weekend!!

So, if you’ve been following any of my recent experiences with an unnamed carrier, you know how difficult it can be dealing with them. (You can read those posts here…Part 1 and Part 2.) However, there are good carriers out there, and TWtelecom is one of them. Sure, no carrier is perfect, but I’ve been dealing with TWtelecom for over 6 months now and they are a pleasure to work with. I actually look forward to calling their Support Center, and that says a lot! Take today for an example…

I have three district offices that use TWtelecom for their WAN connectivity (via IPsec tunnels), and all three offices took a quick hit this morning for about 5 minutes. Let me tell you, when you start getting a bunch of text messages all showing various offices going down, it REALLY gets your attention! I grabbed my WAN document (you do have all your WAN circuits documented, right??), and quickly realized that all three locations had TWtelecom as their carrier. So that tells me it’s not a core router issue, and that no more sites should be going down.

BTW…what’s the next thing I did? No, not call TWtelecom. I walked over to the Help Desk area and let them know what happened and which sites were down, and to expect a bunch of calls.

And one more thing I realized…knowing TWtelecom, my sites should be coming back up rather quickly. An outage like this usually means an upstream device or circuit took a hit, and most times they will recover quickly (unless a backhoe was involved). As I was starting to call TWtelecom, the circuits all came back up. (Whew!)

I still called TWtelecom…they needed to know what happened, and I wanted to make sure it wasn’t the start of a recurring issue. And like always, it was wonderful chatting with them…their support personnel are very polite, know what they are talking about, and are quickly able to route tickets to the appropriate department. Within 15 minutes I received a call back confirming they saw the outage, and that it was due to a local LEC issue with one of their aggregate fiber circuits.  And now, about 3 hours later, things are still stable.

I wish all carriers were as pleasant to work with as TWtelecom. (And no, I was not paid to endorse TWtelecom. Unfortunately.)