Power Outage at PCCC's Collocation - Some Services Affected
May 20, 2011
At approximately 1:07PM on May 20th, 2011, we had a systems outage due to an unknown power disruption. The power disruption occurred at the power distribution level despite isolated power with battery backup and diesel generators.
As of 1:25PM, the systems are back online and service restoration is pending integrity checks on the file system.
PCCC engineers will perform a post mortem in the coming days for an after incident report. However, temperature was normal and amperage drawn on the affected circuit was normal.
UPDATE 1: Systems are online for customers as of 2:16PM and remaining services are coming online as of 2:39PM.
UPDATE 2: As of 3:23PM, all services are back online except IMAP.
UPDATE 3: As of 3:30PM, all systems including IMAP are operational. Some slowness may be experienced due to integrity checks. Full operational status is expected no later than 6PM EDT.
UPDATE 4: There was no data loss from the outage.
UPDATE 5: As of 5:45PM, all integrity checks have completed and server has returned to 100% operational capability. If you continue to experience issues, please call 703-359-9700 or email firstname.lastname@example.org.
POSTMORTEM: Power for one of our circuits was electronically disabled without rhyme or reason. The Power Distribution Unit is being replaced proactively to prevent future issues.