Cloud Zone is brought to you in partnership with:

Seth is the CTO at NuoDB. His main areas of focus are on the administration, security and resource management models, automation and the tools that drive these pieces. Seth is a DZone MVB and is not an employee of DZone and has posted 42 posts at DZone. You can read more from them at their website. View Full User Profile

Testing Network Failure on AWS

12.18.2013
| 3679 views |
  • submit to reddit

Recently, Dan wrote a great piece on testing network failures with NuoDB's support for geo-distribution. If you haven't read it, then go do that right now. It's cool, and it illustrates pretty clearly how you can tune the rules for durability based on awareness of regions.

We've made our tools region-aware: you can define where your database is running and what the criteria are for acknowledging commit back to a client. So in Dan's test, he simulated a network partition that cut off the West and East coasts. No writes were lost. Consistency was maintained before, during and after the partition. Like I said, I think that's pretty cool.

The only thing about this test is that it wasn't physically running in Boston, New York and Los Angeles. It was running on systems physically close to each other that were simulating the topology of separate data centers. It proved that our region-aware management features work, but it didn't really stress our software in all the ways that a real deployment might. So yesterday morning I grabbed Dan and asked if we could re-run the same test, this time on Amazon across physically locations.

The Setup

Like with Dan's original test, we ran in three regions: Virginia (us-east-1), California (us-west-1) and Oregon (us-west-2). We setup one host in Oregon and two hosts in each of the other two regions to mimic the original layout:

SQL> select address, type, georegion from system.nodes order by georegion;

            ADDRESS                TYPE     GEOREGION  
 ------------------------------ ----------- ---------- 

 54.234.101.56                  Storage     us-east-1  
 ip-10-180-203-206.ec2.internal Transaction us-east-1  
 54.193.65.101                  Storage     us-west-1  
 54.193.64.233                  Transaction us-west-1  
 54.203.145.204                 Storage     us-west-2  
 54.203.145.204                 Transaction us-west-2 

All instances were m1.xlarge; no additional services or optimizations were used. We spun up a sixth instance in Oregon to drive the jepsen client-load and partition simulation. As with Dan's original test we ran with --commit region:2 to ensure that all commits are made durable in at least two regions. See his previous post for details about how we started each TE/SM.

In this version of the test, Virginia plays the role that LA had before. The East coast region gets partitioned from the West coast. When that happens the two hosts in California and the one in Oregon can continue to talk with each other but the Virginia processes should fail. This time we've got real-world latency and real-world network-translation involved.

The Results

As you can probably guess, the results were exactly what we'd want to see:

0 unrecoverable timeouts
Collecting results.
Writes completed in 200.077 seconds

2000 total
1995 acknowledged
1995 survivors
all 1995 acked writes out of 2000 succeeded. :-)

The Virginia processes failed as expected, and could be re-started with no trouble once the network partition was fixed. In the meantime we could re-direct load as needed and maintain consistency and durability of the data. Again, pretty cool.

Expect to see some follow-up discussion about the next tests we're thinking about. We're also going to be putting up some recipes that make it easier for you to try running this set of tests (or other AWS experiments) yourself. In the meantime, we were pretty jazzed to be running real-world, cross-country failure handling tests so we wanted to share our excitement with all y'all.

Published at DZone with permission of Seth Proctor, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)