Power Issue - Toronto 2009-12-30

[jmo]

We are currently experiencing a power issue in our 4th floor, Toronto 151 Front data center. We are working quickly to resolve this issue. Further details to come as more information becomes available. We thank you for your patience.

[jmo]

During our scheduled maintenance on the UPS in Suite 402, UPS shorted to ground, fire alarm was tripped, power was shutdown, and the technician was injured. We are doing everything we can to get power restored quickly. The technician is currenly en-route to the hospital and we have all staff dispatched to assist. We will keep our forums up to date.

[jmo]

The fire department is currently on site investigating the issue before power is returned to the UPS unit. Access to the suite is being restricted until they deem it safe to enter.

[cpugsley]

Fire, police, and local hydro are on site. We are currently waiting for them to do an investigation before allowing us back into the building. Just as soon as this has been completed we will re-enter the building and begin working as quickly as possible to restore services.

[cpugsley]

We have been cleared by the fire and police department to enter the suite, however we are awaiting the decision by the Ministry of Labor to re-energize since there were injuries on site. Injuries appear to be non-life threatening.

[cpugsley]

We have been cleared by the Ministry of Labor and are preparing to restore power. We will post another update once clients are able to access the suite.

[cpugsley]

Power has been restored. We have a full staff on site for remote hands. For clients experiencing issues getting their equipment back online, please contact us:

24/7 Support: 1.866.484.2588 or 604-484-2588
E-mail: support@peer1.com

[cpugsley]

On December 30th, at approximately 20:46 EST, PEER 1's facility monitoring system was alerted that power had been lost in the 151 Front St. Suite 402 data center.
At 20:00 EST an Eaton service technician was onsite to replace a failed fan in the redundant 225kVA module in the Eaton Powerware 9395 550kVA UPS during a scheduled emergency maintenance. During the maintenance at ~20:46PM EST there was an arc flash appearing to have been caused by the technicians screwdriver shorting a lug to ground in the UPS causing 2nd and 3rd degree burns to the Eaton service technician.

The wrap around bypass was inspected for damage by our electrical service company and engaged after given final approval by the building, fire department, Toronto Hydro and the MOL restoring power to the data center suite at ~23:31 EST. After power was restored PEER 1 technicians began working with customers to bring their services back online.

Today after conversations with the ESA, the MOL, Eaton and our electrical company we have started repairs on the UPS and removed the 225kVA module that the service technician was repairing last night. We will be powering up the UPS to allow Eaton to fully test the remaining 2 UPS modules and if everything checks out 100% then we will transfer the critical load back onto the UPS. This transfer should not cause any power loss to the cabinets, but as with any load transfer there is always a small risk of power loss.

All of us at PEER 1 sincerely regret the inconveniences associated with this incident, and will fully stand behind our Service Level Agreement. We apologize for this disruption and thank you for your continued patronage and understanding.

If you have any questions regarding this issue, please contact our 24/7 Network Operations Centre at 1-866-484-2588 (+1-604-484-2588 internationally) or via e-mail at support@peer1.com.

Regards,

Ryan Murphey
Vice President
Facilities & Data Center Operations