Fun with NSPs, part #864235
So yesterday, our Colo provider's Chicago main router crashed, for no apparent reason. I noticed because one of my off-site monitors paged me. I called, they said it was unknown cause... and it came back online in about 10 minutes. Yeah, no redundancy, but all in all a blip.
That's when the fun begins. Some hours later they "informed" us that we weren't reachable from their monitoring systems, and must be blocking ICMP from them. Except, um, we're not.
Little history of events:
a) Provider's router crashes
b) Because their router is down, Provider's monitoring notes we are unavailable...
c) Provider does not notify us
d) 4 hours later, Provider writes us a nastygram because, based on one alarm, they could not reach us.
e) When we ask if perhaps it's related to the outage, Provider again blames us with a form letter
f) Provider "restores monitoring" when a clued-in tech is confronted with the full information
Golly, I am sure glad we have our upstream provider monitoring us! Not only did they not notify us of their outage (which made us unavailable), they blamed us for being unavailable and never re-checked if we were in fact back. I wonder what they would do if we really did crash? Mail us some dog poop?
That's when the fun begins. Some hours later they "informed" us that we weren't reachable from their monitoring systems, and must be blocking ICMP from them. Except, um, we're not.
Little history of events:
- Provider's router crashes
- Provider's monitoring notes we are not available, meanwhile I call them based on my own monitoring system.
- Provider's router comes back up about 10 minutes later
- 4 hours later, a Provider tech writes us to tell us their monitoring is being blocked and they will simply stop monitoring us if we don't open it up.
- I ask what ranges they monitor us from - since we did this before and opened up some ranges for them - and ask whether perhaps the alarm is related to the outage?
- 8 hours later, I receive back a form letter telling me they need those IP ranges open for ICMP. Which I confirm they are. And I see Provider ICMP coming in just fine, too. The form letter makes no comment on the outage,
- I call. I get a tech. I have him check the ticket time. Ticket was generated during the outage. Niiiice.
- I explain briefly why this is particularly annoying. This tech is savvy and gets it. I start to wonder about how they vet their techs. Monitoring is restored. Joy.
a) Provider's router crashes
b) Because their router is down, Provider's monitoring notes we are unavailable...
c) Provider does not notify us
d) 4 hours later, Provider writes us a nastygram because, based on one alarm, they could not reach us.
e) When we ask if perhaps it's related to the outage, Provider again blames us with a form letter
f) Provider "restores monitoring" when a clued-in tech is confronted with the full information
Golly, I am sure glad we have our upstream provider monitoring us! Not only did they not notify us of their outage (which made us unavailable), they blamed us for being unavailable and never re-checked if we were in fact back. I wonder what they would do if we really did crash? Mail us some dog poop?