Sorry about the lack of posts this week…I have just been way too busy, and working some long hours.  I will get back on track this weekend. Here is a quick summary of my week…

• Suffered a network outage at one of our busiest District Offices. I had to travel to the location and work with the carrier (a major fiber and Internet carrier), and troubleshoot with them over the phone. As always, they said the issue was with my equipment. (Carriers almost ALWAYS say the issue is with your equipment.) And like always, I have to prove to them that it’s their issue…which it was. Somehow, the VLAN carrying my traffic was changed which brought my network down. We finally got the circuit back up at 2 AM, twelve hours after it went down. Ugh.  And 3 days later, they still cannot tell me how that happened. I’m like “Is there really that many people that can make those types of changes? Don’t you track your changes?”  I guess they don’t.
• I’m the PM (Project Manager) for the IT part of a new District Office which is going live in a couple of weeks. Yes, this is the location in which we have had major issues with the LEC (Local Exchange Carrier). Check out some earlier posts (Part 1 and Part 2) which talks about these challenges. We did finally get a PRI circuit installed, but no fiber Internet. I ended up using a vendor that offers high speed wireless broadband. I was onsite for a couple of days, bringing this up and testing. The circuit is 15 Mb, up and down. It’s working relatively good, but I’m seeing a bit of an issue with large packets (over 1100 bytes)…I have a consistent packet loss of between 1-2%. I know that does not sound like much, but when you are moving large files around, that ends up pushing your through-put way down to around 6 Mb. I will say this…the vendor is very easy to work with, and they already are going to work with me next week to resolve this.
• I had an MPLS T1 circuit at a very remote site giving me fits all week long. It was taking errors pretty much 24×7, and even going down for several hours at a time almost every day. The carrier dispatched out multiple times before finally getting the issue resolved. (They had to replace multiple jumpers, and redo some splices.) It’s now been running clean for almost 48 hours straight. My thanks to the technician who hung in there and got this fixed.
• We recently opened up a temporary site out in the boonies…like way out. This site has no copper facilities at all…no phones, no network circuits…nada. However, it is located right next to a major Interstate, and there is a Verizon tower nearby. I was tasked with getting a Cradlepoint router (with a 4G Verizon card attached) to run DMVPN (Dynamic Multipoint VPN), and connect with a Cisco router at my Data Center. This was a challenge, especially since Verizon likes to run double NAT’ing in their 4G networks. Yep, the 4G card gets a valid public IP address, but that’s not what’s seen on the Internet. Somewhere upstream, still within Verizon’s network, it gets NAT’d again with a different public IP. (Way to go Verizon.) Well, I did get DMVPN to work after much trial and error. We are testing now to see how stable it is, and hope to install it at the site in the next week or two.

As you can see, it was a busy week. And next week will be just as busy.  I’m going to be down at the new District Office most of the week, overseeing all of the cabling, cutting over to the new PRI, installing the network equipment, and working on resolving the packet loss issue. Wish me luck.

And, have a great weekend!!