Toronto Data Center Issue 2009-07-05

[cpugsley]

At 11:26pm PT, PEER1's monitoring systems alerted to an issue in our 151 Front Data Centre. There was a fire in the 151 Front Street facility and the local fire department shut down power to the 7th and 8th floor for safety reasons. PEER1 technicians were immediately dispatched and are currently working with local fire authorities to gain access and restore services.

Updates will be posted as they are received.

If you have any questions, please call 1.866.484.2588.

Thank you.

[cpugsley]

All network and power has been restored on the 4th and 7th floor. We have an electrical contractor on site now investigating. Our technicians have been kept from the suites due to a high level of carbon dioxide and are receiving updates slowly at this time.

We will be updating this thread as information is being received.

[cpugsley]

At 23:26PM PT (02:00AM ET), PEER 1's monitoring systems alerted to an issue in our Toronto 151 Front Street Data Centre. There was a fire which was localized to the 802 Suite UPS room within the 151 Front Street Facility. As such, power was shutdown to to the 7th and 8th floor suites for safety reasons, as the Fire Department mitigated the fire with dry chemical and CO2 extinguishers. No water was used, and all PEER 1 equipment (with the exception of the UPS), and ALL client equipment is unharmed.

PEER 1 staff, building engineers, as well as electrical contractors were immediately dispatched and are currently working with local fire authorities to gain access and restore services to the building.

As of 01:00AM PT (04:00AM ET), power has been restored to Suites 709 and 816. 4th floor clients experienced a drop in network connectivity ONLY, and should be back on-line now.

We are still working with the Fire Department as well as building engineers, and electrical contractors, regarding power restoration to Suite 802.

We appreciate your patience and understanding in this matter.

If you have any questions and or concerns, please feel free to contact our Network Operations Center at 1.866.484.2588

[cpugsley]

All customers not affected by the power outage will be restored as quickly as possible. Multiple staff members are on site for remote hands.

It will take approximately an hour to bypass the UPS to restore power as it has to be manually bypassed. Further updates will be provided as information is received.

We thank you for your patience.

[cpugsley]

We are currently troubleshooting networking issues on the 7th floor and hope to have them resolved as quickly as possible.

Thank you for your patience.

[cpugsley]

It will take approximately an hour to bypass the UPS to restore power as it has to be manually bypassed. Further updates will be provided as information is received.

We thank you for your patience.

Just a quick update - we are still on target for the above posted update regarding the 1 hour bypass.

[cpugsley]

Customers within Suites 802 and 816 still affected by the power outage. PEER 1 staff, building engineers, and electrical contractors, are working on manually bypassing the Suite 802 UPS in order to restore power to Suite 802 and Network Connectivity to Suite 816.

If you have any questions, please feel free to contact our Network Operations Center at 1.866.484.2588.
We thank you for your patience and understanding.

[cpugsley]

Circuits are being energized and power is being restored at this time. An official follow up will be sent out via e-mail to all customers that were affected by this. Clients who continue to experience issues, please contact our NOC at 1.866.484.2588.

We thank you for your patience.

[Dot Williams]

Dear Valued Customer,

As you are aware, over the weekend the 8th floor of PEER 1’s Toronto data center located at 151 Front Street experienced a fire with a UPS cabinet that set off a chain of events that resulted in power outages in multiple suites and network outages in others. I have detailed the events below.

PEER 1 is extremely thankful that no one was injured during the fire. The fire itself was contained within a single UPS cabinet and the Toronto fire department was able to extinguish it with the use of dry chemicals. No water was used, which significantly reduced the damage and aided in our ability to restore services to all data centers as quickly as possible.

I was personally involved in PEER 1’s response to the situation, and would like to take a moment to apologize to you for the information contained within our first few email alerts regarding the outage. When we were first organizing the team, we received an early report from our on-site staff that all power was out at 151 Front Street. That was not accurate. The fire caused a power outage in PEER 1’s suite 802. The local fire department cut power to PEER 1’s suite 709 as a preventative and safety measure to contain and extinguish the fire. The network gear providing connectivity for Suite 816 are kept in a cage in Suite 802. As a result, Suite 816 suffered a network outage. Power in the rest of the building was unaffected throughout the incident. There is a fine balance to providing information during a crisis, and we do our best to make sure updates to our customers are, quick, transparent, and accurate. In no way would we try to misrepresent the situation.

PEER 1 manages all of the equipment it provides to customers with safety and performance in mind. The UPS mentioned above has been in service for approximately four years. Its last service inspection was on July 2nd and it was only at 65% load. We do not yet have the root cause, but there was nothing in the unit’s maintenance record to indicate there was potential for a problem. Our service contractor and manufacturer are both involved in the investigation.

As part of the process to restore power quickly, we had to bypass the affected UPS. This means that suite 802 is running without UPS power until we can schedule delivery and replacement. We will expedite the process and keep customers in this suite up-to-date with the latest information on the timeline.

Outage Details:

(Timelines represent impact to “most” clients. Some specific network devices failed due to power loss and restore took longer.)

· Suite 802 – Power and network outage. Start 23:32PT End 03:10PT

· Suite 816 – Network outage. Start 23:32PT End 03:10PT

· Suite 709 – Power and network outage. Start 23:32PT End 00:50PT

· Suite 402 – Limited network outage (some latency/packet loss issues). Start 23:32PT End 00:50PT

· Firewall Clients. Network outage. Start 23:32PT End 03:10PT



Ted Smith
SVP, Operations

PEER 1
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
tsmith@peer1.com
T: (678)365-2896
F: (678)365-2956

http://www.peer1.com



NETWORK | CO-LOCATION | MANAGED HOSTING