Strategically parked over a manhole on a downtown Manhattan street, John Meko spent months working inside a 20-foot storage trailer and learning about resiliency the hard way.
Meko was wrestling with the immediate aftermath of Hurricane Sandy in 2012. His employer at the time was an internet service provider (ISP) that supported about 200 office buildings in New York City, primarily in the Financial District. The massive storm and flooding had destroyed electrical and internet infrastructure throughout large sections of the city, and businesses were desperate to regain service so they could resume their operations.
“Every ISP was down and customers were extremely frustrated and losing money daily,” he said. “Tenants were paying exorbitant amounts of money to relocate offices in Manhattan, spending millions of dollars for backup generators, and they were willing to break any contract they had and sign up for new internet service in order to get back up and running.”
By establishing a temporary office and internet hub in that small trailer over a manhole that accessed his company’s fiber, Meko and his colleagues were able to restore service to numerous buildings while they dug into the months-long task of repairing or replacing damaged equipment in office buildings.
“It was a wild experience to go through. It was not fun to be part of at the time but it definitely taught me a lot,” said Meko who is now the North American director of engineering for WiredScore and an expert on how to create more resilient digital services in commercial buildings.
As e-commerce, cloud computing, file sharing, remote working, distance learning and other online functions have grown, resilient internet service has become crucial to most companies’ operations and their bottom lines. On average, one hour of internet downtime costs a Fortune 1000 company $300,000 in lost productivity, Meko said. And as a recent string of hurricanes, floods, tornadoes and earthquakes have demonstrated, internet infrastructure is vulnerable.
Some practices for hardening that infrastructure are obvious. Moving telecomm equipment out of the basement and to a floor that’s above potential flood levels will reduce a building’s total leasable space, but improve reliability of telephone and internet service to its tenants.
Yet many property owners — even those hit hard by Hurricane Sandy — have not embraced that practice.
“There were landlords who forced us back into the basement” after Sandy, Meko said. “They were pumping water out of a room then putting electronics back into it. Their mindset was that Sandy was a 100-year flood so we would be okay for another 100 years. The problem is that 100-year floods can happen every five or 10 years now.”
Other standard resiliency measures include installing backup generators and sufficient fuel on a rooftop or other flood-proof location and ensuring those generators are connected to the building’s telecomm infrastructure; sourcing a building’s grid electricity from more than one substation (a measure that could preserve at least partial power for a building during a storm or brownout); and outfitting a building with multiple, separate ISP lines. Following Hurricane Sandy, some property owners and tenant companies discovered that their efforts to source multiple ISPs had failed because all of the carriers were using the same Verizon fiber, Meko said.
In areas vulnerable to earthquakes, some developers are installing flexible conduit and reinforced cabinets for telecomm equipment. In Richmond, Washington, Nintendo outfitted its new North American headquarters and accompanying 5,000-square-foot data center with seismic base isolation and rotary uninterrupted power supply systems.
Some developers are pushing internet and electrical resiliency to new heights. Concerned about brownouts in New York, owners of the 18-million-square-foot Hudson Yards development installed a 13.3-megawatt, natural-gas-fired cogeneration plant and linked it to a thermal loop and microgrid that serve all buildings on the 28-acre campus. Further, they worked with Con Edison to install an interconnection that enables the Hudson Yards’ microgrid to disconnect from region’s electrical grid during an outage and continue to provide power to Hudson Yards buildings.
Most individual commercial developments lack the space or funds to install similar systems. The Hudson Yards cogeneration plant, microgrid, thermal loop and interconnection cost $200 million. However in an article about the project, Related Hudson Yards President Jay Cross urged property owners to form partnerships and create communal resources. “We believe the development community in general must stop thinking about raising one building at a time and start thinking about how to create blocks, neighborhoods and cities that can withstand the tests of resiliency that are sure to come.”
Meanwhile, in individual buildings, owners increasingly need to prepare for tenants seeking greater internet and electrical resiliency.
“Tenants are driving this conversation,” Meko said. “We are seeing tenants with lease requirements for backup generation onsite, for at least 400 square feet of floor space where they can put their own generator. We are hearing questions about redundancy of telecomm services in the building and requirements for primary and backup connections from different streets and different manholes… Service reliability is becoming a make-or-break issue for tenants.”