I had an interesting lunch today. I went to my favorite fast-food place (Chic-Fil-A….they have the best ice tea!!). Anyway, when I arrived the parking lot was full of California Highway Patrol vehicles…I remember thinking that lunch today will be very safe! It was also raining (much needed here in CA), and so I put my iPad under my trenchcoat, up under my left arm-pit, and clamped down on it with my left arm. As I walked into the restaurant, I walked down an aisle full of CHP officers, all chatting and laughing with their peers.
So…it was about then that I decided to remove my iPad. (Yeah, I know…what was I thinking.) So I put my right hand into my trenchcoat and grabbed the iPad. It was at that time that a whole bunch of eyes were instantly fixed on me as I pulled out the iPad. As soon as they saw the iPad, their eyes went back to their friends. What was interesting is they never stopped talking or laughing…being this aware and alert was normal for them. I bet most of them didn’t even realize what they just did. In other words, they have trained to be alert and aware, and it worked.
So, why bring this up? As network engineers, we need to practice being aware and alert to what is around us. When passing some users, did you hear them mention something about slow Internet? Make a mental note. Did you notice the SSH session pausing momentarily as you scrolled through some configurations from a remote router? Make a mental note. (This happened to me recently….it didn’t feel right, so I tested and found out this circuit was experiencing an above average packet loss.) Heard some users complaining about how slow the ERP application was? Make a mental note.
As you start making these notes, you may see a picture emerging that could very well point to an issue in your network. Or perhaps an issue higher up the protocol stack…maybe not even your problem, but you could alert the correct department and let them run with it. (Years ago, at another company, I noticed some people randomly complaining of slow Internet response. I started doing some testing, and was able to confirm that a random issue was occurring. It ended up being some old BIND DNS servers that were using old “hints” files. We downloaded updated hints files from ISC, and the problem was fixed…and the users even noticed the quicker Internet response!!)
As network engineers, we are uniquely positioned to notice most any issue with the network…as long as we are alert and aware. So if being aware is not second nature to you, start practicing it everyday. Don’t tune the world out…but listen to it. You will end up being a better network engineer.
I agree totally. Chic Fil A does have the best sweet tea! 🙂 Apart from that, I agree with the post. Look, listen, and smell. Always be aware of blinking lights, colors, etc. Always be aware of the sounds (switch fans, server reboots, etc). Always be aware of the smells of the closet/data center. Very good post you have here.
I totally agree!
Get to know your network and pay attention to it. Just like your car, you can ‘feel’, hear, see, smell if something is different. Then track it down.
Your network will let you know if something is wrong, pay attention to it.
Great post, Brad!