Can You Recover in 10 Minutes?
My alma-mater may be better known for its football team, but the engineering fraternity Theta Tau hosts a pretty wicked egg drop competition. Some entries are really just an excuse to publicly drop something fun out of an eight-story window in front of an audience. Computers are regular victims.
What would happen if two engineering students broke into your data-center, grabbed a server, and chucked it off the roof? Would that be a problem (aside from cost of hardware)? Would there be an outage? How long would it be? Is that kind of outage acceptable?
While I doubt anyone has engineering students on staff for the purpose of throwing boxes off the room, Netflix is known for intentionally making internal applications destructive—they need to test their resilience just like you do. If you aren’t ready for a box to be pulled and chucked off the roof, you aren’t ready for real world failures. Instead, your app is just an egg enjoying the ride, thankful that this second wasn’t the one where it encountered the ground.
*image from http://www.flickr.com/photos/amagill/129773302/
* a similar image was used by @hchaight in his IOM keynote reminding me of this awesome tradition
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)