Author Archives: sr71rocks

How to Stress Test a T1 Circuit

Most all networks have T1 circuits, the most common being either an MPLS T1 or Internet T1. There will be times when one of your T1 circuits will be acting up in a sporadic manner, causing “slowness” for your end users, and will require you to be more proactive in troubleshooting the root cause. This post will talk about how to stress test a T1 using PING.

First off, understand that using a PING command with the default parameters will tell you if the circuit is up or down, and it may show problems such as large latency or excessive drops. But to really test a T1, you need to modify the use of PING to perform a more thorough test. Common actions are to increase the packet size and frequency of pings to better test throughput, and to use specific data patterns to better test the operation of the T1.

You can use Cisco’s PING which is part of IOS. Here is an example of an extended PING where we increase the packet size to the max MTU of 1500 bytes, and run all 1’s (which will provide additional stress on the circuit)…

Using Cisco’s IOS ping command

A much better PING to use though is Linux, with the “flood” option, as this will allow you to really hammer the T1 circuit. (Note…you need to be root to use the flood option.) The difference is this…Cisco’s PING will send an echo-request, but will wait for the echo-reply before it can send another echo-request. This greatly reduces the amount of ping traffic IOS can send across the T1. Linux however, will immediately start sending as many packets as it can, up to 100 per second. For each echo-request packet it sends, it prints a “.” (dot) on the screen. For each echo-reply it receives, it prints a back-space. So if you only see a couple of dots, then the circuit is handling your ping flood easily. However, if you start seeing dots race across the screen, then there are problems. Here is a Linux PING flood example with 1500 byte packets and running all 1’s…

Using Linux ping -f (flood) option to stress a T1 circuit

As you can see, there are only three dots…7356 packets were sent and 7353 were received. That leaves 3 missing packets. This T1 easily handled this test. Plus a Linux ping flood will typically load up a T1 in the range of 700-900 Kb (about 1/2 of a T1 circuit). If you really want to fully load up a T1, run two different instances of ping flood, and you will see a T1 circuit fully saturated (or near so). Of course, do NOT do this during normal business operations…you will heavily impact the end users, and they will not be happy. When running the Linux ping flood shown above, the resulting bandwidth impact on the T1 was…

Using “show interface” to see bandwidth impact of ping flood

In my next post I will give an example of how I used ping flood to troubleshoot a T1 circuit whose performance was impacted by a unique problem.

Killing Those Pesky Child Processes in Linux

Linux is awesome! It is solid and dependable, and you can do most anything you want with it. I use it everyday…for Network management purposes mainly (Nagios, MRTG, SWATCH, SYSLOG, NMAP, etc). If you have not used Linux yet, I would highly encourage you to do so. I will post a feature on Linux soon on how Linux can play a large role in helping you manage your network. But for now, lets kill some pesky child processes.

Although I use Linux a lot, I am in no way a Linux guru. I write simple scripts and hack my way through stuff. However, when I kill a process, I sometimes find one or more child processes that remain. So I researched different ways to take care of these, and there are many ways to accomplish this. For me, this works best…

To help troubleshoot an ISP issue with one of my Internet fiber links, I’m running a ping against the public IP address on a per second basis (very granular), and adding a time/date stamp to each ping reply. Here is the script…(IP address has been changed to protect the guilty)…

Simple Ping Script with Date/Time Stamp

Here is a snippet from the log file showing what the ping replies look like…

Ping results from log file

When I’m notified of a network bounce, I’m able to dig through the file and see if the Internet circuit did indeed take a hit, or was it just my Virtual Tunnel interface bouncing. Here is an example showing an outage that lasted a bit over 2 minutes…take a look at the timestamp and also the gap in the sequence numbers…

Quick circuit outage lasting a bit over 2 minutes

Anyway, when I kill the script, I am ending up with two child processes remaining. I found out that I need to kill the PGID (Process Group ID) to properly take care of any child processes. To find the PGID, you can run “ps -ejH”, which shows you a process tree where you can find the PGID (in column two). Then you can kill the PGID using “kill — -PGID”. Here is an example…

Finding the PGID and killing it

This works well for me. And as for Linux, give it a try.

Cool Network Tools – Kiwi SyslogGen

There is a great network tool I’ve been using for years, from Kiwi (part of SolarWinds), called SyslogGen. I simply love this tool…it’s flexible and powerful, yet easy to use and learn. Basically it’s a tool used to send test syslog messages to a log server so you can verify proper operation of logging and alerting. I used it today at work…

I’m in the middle of troubleshooting an issue at one of my remote sites. I have a fiber Internet circuit feeding the site, and I’m seeing very intermittent short outages (about 2 each day), lasting approximately 90 seconds. During the outage, both the Internet and my VTI (Virtual Tunnel Interface) drops. I’m working with the carrier, but it’s slow going, as I have to prove to them there is actually a problem.

Continue reading →

Targeted Phishing – Educate Your Users…Today!

If you read some of my security postings lately, especially this one, then you should already be signed up to receive CERT notifications. (If not, WHY?) You would have received a US-CERT alert about a “Phishing Campaign Linked with ‘Dyre’ Banking Malware”. Have you read it? AND acted on it? (Here is the US-CERT Alert if you have not read it.)

The most important action item is to educate your users. Yes, you have firewalls and antivirus configured, and perhaps a URL filtering service. And that’s good. But the best defense against phishing is an educated user community. You should be sending out an email on a regular basis, perhaps quarterly, educating your users on what phishing is, and how to recognize it. (It would make more impact if this email came from your CIO or IS-VP.) AND, make sure and let them know that banks and other institutions will never be asking for sensitive data via emails. You should include a sample phishing email (with attachments and embedded links removed, of course).

How to get a sample phishing email? Well, if you have trained your users properly, they will be sending them to you on a regular basis. If they forward these emails to you with a note such as “Received this today…it looks fishy, so I just deleted it, but wanted to let you know”, then you have done well in your training!! Otherwise, just check the inbox of your upper management and finance personnel. Believe me, they are getting them on a regular basis, because they are being targeted. Hackers and Scammers (otherwise known as “Slimy Scum-Bags”) are not emailing the whole world anymore…instead, they are sending their mucky-muck to the people that have the access and power. And this strategy is working. Make sure and educate these users….frequently!

Let me repeat: The BEST defense against phishing is an educated user community!

Make it so. (In my best Captain Picard voice.)

Time for Another Set of Eyes

It happens to us all at some time or another. You’ve been working all morning at a remote site several hours away, and you’re making substantial changes on the network infrastructure. As you are finishing up, you realize you can get to the Internet, but you can’t FTP to the Internet. After going over the config several times, it’s time to bring in another set of eyes. And so my phone rings.

This ended up being rather simple. We have PBR (Policy Based Routing) in effect for normal web traffic (ports 80 and 443), and a default route for all other Internet destined traffic (such as FTP). I checked the routing table and found this…

show ip route

As you can see, gateway of last resort is not set. So for any Internet bound traffic that is not port 80 or 443, the router does not know where to go. My co-worker checked and found that he had mis-typed the entry for the default next-hop path. Once he fixed it, everything worked as it should.

Another example was earlier this summer…I was having some stability issues with one of my VTI (Virtual Tunnel Interface) sites, and was not able to nail down the cause. My co-worker looked over the related configs, and found that I had forgot to set a particular filter on the perimeter firewall. That fixed it.

So if you find yourself staring at a configuration, unable to find the problem, call a fellow co-worker and get a fresh set of eyes on the problem. Two heads are better than one!

Some Great Geek Humor Over at UserFriendly

Us geeks have a great sense of humor! Of course, most of the world doesn’t really understand our humor, but that’s ok with me. You either get it or you don’t. Check this out…

Mind Your Wildcards

This is so funny!! I’ve been following User Friendly for a number of years, and you should too!

Using Google for Better Website Searching

Here is a tip that really helps when searching the Internet…

When you are visiting a website and use that website’s Search function, you may not always get the desired results. It happens a lot, let me tell you. A great example is Cisco’s website. They have a search function in the upper right corner of their webpage, but when using it, you tend to get a lot of marketing materials or other documents unrelated to what you are searching for. It drives me crazy sometimes. Here is a better way to handle this…

Let’s say you are wanting to research configuring port channels on a Cisco switch. You could search on Cisco’s website for “switch portchannels” and see what you get. (Go ahead and try it.) Next, go to Google and search for “switch portchannels site:cisco.com”. That last search option tells Google to limit the search to just cisco.com. Try it and compare the two searches. Yes, this is a simple example, but let me tell you this technique works great. I use it frequently, and not just for Cisco, but for any search that I need targeted to a particular site.

So remember the “site:” parameter when using Google…it will give you better and quicker results.

If you are interested in other Google tips and tricks, check out their related webpage.

iPhone 6 Plus – Just Too Large, Right?

As you know from a couple of previous posts (Part 1 here, and Part 2 here), I’m trying to decide between the two new iPhone 6 smartphones. Both are incredible examples of engineering and design, typical of Apple, and like their previous models, they just work. Everyday. With no hassles or complications. When you depend on your smartphone for work-related communications (voice and messaging), you need a reliable phone. And if it’s engineered the way Apple does, so much the better.

My current iPhone is two years old, and it’s time to upgrade. I’ve been leaning towards the iPhone 6, as I just think the Plus will be a bit too big.

But then…I made a big mistake. I stopped by the Apple store this evening, and played with both a 6 and a 6 Plus. WOW…the Plus is stunning. The screen size seems just right, lots of real-estate to surf the web, read documents (which I have a ton of), and watch videos. Amazing clarity. Light weight, thin, and so well engineered.

I’m hooked. I’m getting a 6 Plus. Here is a picture of the 6 Plus I was playing with at the Apple store…

Getting my Geek on

Yes, the picture does not do it justice…but I’m a geek, and I have to post a picture.

I’ll order the phone through my work next week. No telling how long it will take to arrive. But I am a patient geek.

Geek Stuff – Google Search Metrics by Language

I’m a geek and I’m always on the lookout for things that are interesting and technically “cool”. I ran across this recently….it’s one of the ways that Google is presenting information in a different and visually eye-catching manner, what Google is calling “geographic data visualization”. Take a look at Google Search by Language….it plots out Google search volume by language around the globe. VERY cool stuff!

Google also makes the code available for anyone to plot out whatever they want. Go to Google’s Chromeexperiments to see other visualization examples.

Enjoy!!

The OSI Seven Layer Model – Why Learn It?

The number one reason I started this blog was to provide help and encouragement to new and prospective Network Engineers. Most of the time, I’ll be passing on experiences that I have in the field of networking. However, as I have time, I also want to post articles that I hope will take your understanding of networks up to the next level. I will include practical examples and exercises you can do at home which will give you a better understanding of networks, and I hope will make you a better Network Engineer. Today is my first post of this type. Please let me know if this is helpful (or not). Here goes…

The OSI Seven Layer Model. (I can you see wincing!!) Why bother, right? Most people I know learned just enough about the OSI model to answer the questions on some written test, and most likely have not given it a thought since. It’s just a bunch of theory, and doesn’t really help in the real world. Well, if that’s what you think, then you are missing a powerful tool in understanding networks AND in troubleshooting them. The best network engineers know the OSI model. And you can too.