For 23 years, I operated a web hosting service on the Internet, providing server space and support for clients that included national trade associations, political candidates, philosophical organizations and retail operations.
After selling it last year, I kept one web server to service most of my personal web sites, the blogs for some friends and couple of small local businesses — a far cry from the operation that included multiple servers at different locations, real-time backup and a “mirroring service” that made sure that if a connection to one server went down, the sites would stay up through other ways.
As a data services provider for more than two decades, I understood the need for redundant backup operations. It came at a price that cut into the already thin profit margins of such operations but it helped give customers a needed level of satisfaction.
As now a customer and not a provider, I learned that lesson Wednesday afternoon when a fiber cable cut to the data center in San Antonio, Texas, stopped all connection to thousands of servers there.
Two of those servers belonged to me. One provided news and information to Capitol Hill Blue, the oldest web-based political news service on the Internet and the other served blogs like Fred First’s Fragments from Floyd Colleen Redmond’s Loose Leaf Notes, two businesses and several of my personal web sites.
Blue Ridge Muse, this website, did not lose connection. It runs on a separate server and provider.
The fiber line cut came at 2:20 p.m. EST Wednesday. Technicians restored some connectivity a couple of hours later but full restorations took several more hours to complete.
The Capitol Hill Blue server has a duplicate “mirror server” at another place but the electronic switch that directs traffic to that site could not send out its instructions because of the fiber cable cut.
Lessons learned. I will move the DNS for that website today. Our data communications infrastructure is impressive when things work but is dead in the water from a simple glitch like a cut fiber cable.