Hello again. As a follow-up to last weeks post (read it here), this past week was once again way too busy. I spent the entire week at the new office, working 12 hour days, getting it ready for move in and go-live. The following tasks were accomplished…

• All network cabling was installed, labeled and tested. This consisted of about 60 workstations, running two data cables to each workstation. The cabling vendor is a company I’ve worked with for many years…they know what they are doing and it shows in the final product. No worries here.
• A solid wall of backboard was installed in the MPOE, and on that was mounted a swing-out Chatsworth rack. A bit pricey but worth the extra money…the whole rack can pivot to the side, giving access to the rear of the equipment. (Check out Chatsworth’s Swing Gate if you are interested.)
• Network router (Cisco 2851), Cisco switch stack (3750’s), and several Cisco Access Points were installed, configured and tested.
• The new PRI circuit was tested and 100 DID’s (Direct Inward Dialing) were ported over from our old PRI circuit.
• The wireless broadband is working well, but I am still keeping my eye on it. Not sure if the vendor fine tuned it or not, but I am seeing better performance.
• Security cameras and a key-fob access system was installed.

It was a long, but successful week. I am also glad this type of project does not occur often.

I hope you had a great weekend!!

Sorry about the lack of posts this week…I have just been way too busy, and working some long hours.  I will get back on track this weekend. Here is a quick summary of my week…

• Suffered a network outage at one of our busiest District Offices. I had to travel to the location and work with the carrier (a major fiber and Internet carrier), and troubleshoot with them over the phone. As always, they said the issue was with my equipment. (Carriers almost ALWAYS say the issue is with your equipment.) And like always, I have to prove to them that it’s their issue…which it was. Somehow, the VLAN carrying my traffic was changed which brought my network down. We finally got the circuit back up at 2 AM, twelve hours after it went down. Ugh.  And 3 days later, they still cannot tell me how that happened. I’m like “Is there really that many people that can make those types of changes? Don’t you track your changes?”  I guess they don’t.
• I’m the PM (Project Manager) for the IT part of a new District Office which is going live in a couple of weeks. Yes, this is the location in which we have had major issues with the LEC (Local Exchange Carrier). Check out some earlier posts (Part 1 and Part 2) which talks about these challenges. We did finally get a PRI circuit installed, but no fiber Internet. I ended up using a vendor that offers high speed wireless broadband. I was onsite for a couple of days, bringing this up and testing. The circuit is 15 Mb, up and down. It’s working relatively good, but I’m seeing a bit of an issue with large packets (over 1100 bytes)…I have a consistent packet loss of between 1-2%. I know that does not sound like much, but when you are moving large files around, that ends up pushing your through-put way down to around 6 Mb. I will say this…the vendor is very easy to work with, and they already are going to work with me next week to resolve this.
• I had an MPLS T1 circuit at a very remote site giving me fits all week long. It was taking errors pretty much 24×7, and even going down for several hours at a time almost every day. The carrier dispatched out multiple times before finally getting the issue resolved. (They had to replace multiple jumpers, and redo some splices.) It’s now been running clean for almost 48 hours straight. My thanks to the technician who hung in there and got this fixed.
• We recently opened up a temporary site out in the boonies…like way out. This site has no copper facilities at all…no phones, no network circuits…nada. However, it is located right next to a major Interstate, and there is a Verizon tower nearby. I was tasked with getting a Cradlepoint router (with a 4G Verizon card attached) to run DMVPN (Dynamic Multipoint VPN), and connect with a Cisco router at my Data Center. This was a challenge, especially since Verizon likes to run double NAT’ing in their 4G networks. Yep, the 4G card gets a valid public IP address, but that’s not what’s seen on the Internet. Somewhere upstream, still within Verizon’s network, it gets NAT’d again with a different public IP. (Way to go Verizon.) Well, I did get DMVPN to work after much trial and error. We are testing now to see how stable it is, and hope to install it at the site in the next week or two.

As you can see, it was a busy week. And next week will be just as busy.  I’m going to be down at the new District Office most of the week, overseeing all of the cabling, cutting over to the new PRI, installing the network equipment, and working on resolving the packet loss issue. Wish me luck.

And, have a great weekend!!