Data center leaks

image

Leaking data center

We had a problem in our data center: water, where it does not belong!

Our data storage systems had water coming out of them.  Data isn’t wet; this was not a leak of private information, although it put data at risk in other ways.

The problem is that our roof is old, and it was raining.  Not particularly hard; our data center remained dry after Tropical Storm Irene (which brought widespread flooding after >10 inches of rain in the area).  But there is a seam in the roof just in behind these systems, and this was the time it started to leak.  The water you see actually comes from the other side of the rack.

Of course we shut down our systems — electronic systems can often survive water as long as they don’t have electricity flowing through them while they are wet.  We covered the areas where water seemed to be coming, and set up fans to help dry out wet electronics.  The normal intense air flow from the air handlers also helped to dry the systems, as well as the wet areas forming under the floor.

By the next morning, everything had dried sufficiently to bring the systems back on-line.  There were a few places where ceiling tiles had collapsed; these were soon repaired, and the data center once again in good condition.  No data were lost!image

This entry was posted in Life, Technical. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *