That did not go as well as I might have hoped.
Got into the office around 7:45pm last night (Friday). Got home at 7:30am this morning (Saturday). This is a sign that things did not go as well as I might have hoped.
To begin the night, I did the second Opt-E-MAN circuit cut. This one went much more smoothly than the one on Thursday night. Was done in under an hour (less than half of the total time window I had set aside for the event). This was good.
Then, at 12:15 this morning (Saturday) I got started on the firewall (NAT) service migration from old datacenter to new datacenter. Rriding on the success of the Opt-E-MAN cut I was highly optomistic. I rolled into the datacenter right on time with my 18v screwdriver-drill. I started tearing out cables [1] and rack bolts. Everything was going fine until I ran into a pair of cross-threaded bolts on the very first PacketShaper I was unracking. That was incredibly time consuming and I couldn't get past the issue, so I moved on to the second unit. Again, cables went flying and so did rack bolts... though thankfully I had no issues pulling THIS unit from the rack. I found a lonely, abandoned chair with castors on it in the corner of the datacenter (It's amazing how.... empty and torn-apart that facility is now... it's sort of depressing) and tossed the unit I had successfully unracked on top of it. Then it was out to the truck to get my channel locks, my pliers and my side-cutters. 20 minutes of sweating and swearing later I was pushing the chair with both PacketShapers out to my truck and loading them into the back of my pickup. At 12:45 in the morning, when you're loading $50,000+ worth of equipment into the back of your personal vehicle from the facility that your $EMPLOYER is abandoning.... it's hard not to feel a little bit like a grave robber or a looter. :P
I got on the road at around 12:50am and by 12:59am I was at the new datacenter facility, where I began the arduous task of lugging both network appliances from my truck in the parking lot, into the building, through the mantrap, up the stairs and to our cage without aid of a rolling cart or chair. Thankfully I had a box with handles in the back of my truck so I could carry both devices at once, which was mixed blessing. On one hand, I only had to make the trip once. On the other hand, the two devices each weigh over 40 pounds so carrying them plus their accessories all at once made for a long, slow effort.
Racking the gear took even longer. By the time I had everything racked, cabled and powered up it was 2:00am. Half my maintenance window for the work had gone out the, well, window.
A failed media converter that refused to see link/carrier on the copper port took me 20 minutes to troubleshoot (I was, at first, convinced a wiring error had been committed). Tick. Tock.
At 2:30 in the morning I left to go back to the office to work remotely and finish the configuration. I knew I was behind schedule but still didn't have any fear. Until around 3:10am, when I couldn't figure out why the external interface on the userNAT firewall #1 wasn't showing link and why I couldn't ping the IP of the second PacketShaper unit. Then fear set in. It was a rush back to the datacenter and by 3:19 or so I was up the stairs to our cage. Hurried manual reading, some attempted hack-arounds and time later I realized I simply wasn't going to make this work and that I had to execute some sort of fail-back plan. So I grabbed a screwdriver, ran out the bolts on one of the PacketShapers and lugged its unweildy self back to my truck for a rush return to the old datacenter.
I got the unit back into the old datacenter and situation (without bolts) in a rack by around 4:15. And that's when I realized that... I had no idea which fiber cables were the ones I needed and which one was the "inside" feed and which one was the "outside" feed to packet shaper. So then it was a rush to find an available port in our datacenter, plug my laptop into that so I could try to chase down the cables... hey, why doesn't my laptop pull a DHCP address anymore? Oh crap, they took down the old DHCP servers, didn't they? Okay, now I'll just have to assign my own....
Eventually I gave up on trying to chase fibers (it's a complete tangle of cable snarls in there, even now at the bitter end of the decomissioning process) and ran some new fiber lines to the location. A bit more jockeying of VLAN assignments on the switch ports and I had a functioning link to the Internet once again. The time? 4:54 AM -- 54 minutes after my window had been scheduled to end.
[1] This was one of my first mistakes. I have no idea why I didn't label the cables before I disconnected them. I guess it never occured to me that things could go so badly I'd actually have to fail back to the facility we're decommisioning.
To begin the night, I did the second Opt-E-MAN circuit cut. This one went much more smoothly than the one on Thursday night. Was done in under an hour (less than half of the total time window I had set aside for the event). This was good.
Then, at 12:15 this morning (Saturday) I got started on the firewall (NAT) service migration from old datacenter to new datacenter. Rriding on the success of the Opt-E-MAN cut I was highly optomistic. I rolled into the datacenter right on time with my 18v screwdriver-drill. I started tearing out cables [1] and rack bolts. Everything was going fine until I ran into a pair of cross-threaded bolts on the very first PacketShaper I was unracking. That was incredibly time consuming and I couldn't get past the issue, so I moved on to the second unit. Again, cables went flying and so did rack bolts... though thankfully I had no issues pulling THIS unit from the rack. I found a lonely, abandoned chair with castors on it in the corner of the datacenter (It's amazing how.... empty and torn-apart that facility is now... it's sort of depressing) and tossed the unit I had successfully unracked on top of it. Then it was out to the truck to get my channel locks, my pliers and my side-cutters. 20 minutes of sweating and swearing later I was pushing the chair with both PacketShapers out to my truck and loading them into the back of my pickup. At 12:45 in the morning, when you're loading $50,000+ worth of equipment into the back of your personal vehicle from the facility that your $EMPLOYER is abandoning.... it's hard not to feel a little bit like a grave robber or a looter. :P
I got on the road at around 12:50am and by 12:59am I was at the new datacenter facility, where I began the arduous task of lugging both network appliances from my truck in the parking lot, into the building, through the mantrap, up the stairs and to our cage without aid of a rolling cart or chair. Thankfully I had a box with handles in the back of my truck so I could carry both devices at once, which was mixed blessing. On one hand, I only had to make the trip once. On the other hand, the two devices each weigh over 40 pounds so carrying them plus their accessories all at once made for a long, slow effort.
Racking the gear took even longer. By the time I had everything racked, cabled and powered up it was 2:00am. Half my maintenance window for the work had gone out the, well, window.
A failed media converter that refused to see link/carrier on the copper port took me 20 minutes to troubleshoot (I was, at first, convinced a wiring error had been committed). Tick. Tock.
At 2:30 in the morning I left to go back to the office to work remotely and finish the configuration. I knew I was behind schedule but still didn't have any fear. Until around 3:10am, when I couldn't figure out why the external interface on the userNAT firewall #1 wasn't showing link and why I couldn't ping the IP of the second PacketShaper unit. Then fear set in. It was a rush back to the datacenter and by 3:19 or so I was up the stairs to our cage. Hurried manual reading, some attempted hack-arounds and time later I realized I simply wasn't going to make this work and that I had to execute some sort of fail-back plan. So I grabbed a screwdriver, ran out the bolts on one of the PacketShapers and lugged its unweildy self back to my truck for a rush return to the old datacenter.
I got the unit back into the old datacenter and situation (without bolts) in a rack by around 4:15. And that's when I realized that... I had no idea which fiber cables were the ones I needed and which one was the "inside" feed and which one was the "outside" feed to packet shaper. So then it was a rush to find an available port in our datacenter, plug my laptop into that so I could try to chase down the cables... hey, why doesn't my laptop pull a DHCP address anymore? Oh crap, they took down the old DHCP servers, didn't they? Okay, now I'll just have to assign my own....
Eventually I gave up on trying to chase fibers (it's a complete tangle of cable snarls in there, even now at the bitter end of the decomissioning process) and ran some new fiber lines to the location. A bit more jockeying of VLAN assignments on the switch ports and I had a functioning link to the Internet once again. The time? 4:54 AM -- 54 minutes after my window had been scheduled to end.
[1] This was one of my first mistakes. I have no idea why I didn't label the cables before I disconnected them. I guess it never occured to me that things could go so badly I'd actually have to fail back to the facility we're decommisioning.