Enterprise Integration Zone is brought to you in partnership with:

Mitch Pronschinske is a Senior Content Analyst at DZone. That means he writes and searches for the finest developer content in the land so that you don't have to. He often eats peanut butter and bananas, likes to make his own ringtones, enjoys card and board games, and is married to an underwear model. Mitch is a DZone Zone Leader and has posted 2574 posts at DZone. You can read more from them at their website. View Full User Profile

Scalable Architectures: Taming the Twitter Firehose

03.23.2013
| 5133 views |
  • submit to reddit

Scalable architectures: Taming the Twitter Firehose - Lorenzo Alberton from GrUSP on Vimeo.

Handling lots of real-time streams of information, when Twitter alone is producing 250+ million tweets a day and 27 million links to news and media, can be a daunting task.

This talk will cover how to develop a platform that can deal with billions of items per day, perform complex analysis and serve thousands of customers in real-time, by dissecting a large scale architecture into small components and explaining what processes to put in place, what pitfalls to avoid and how to keep the system running. We will walk through several scalability patterns, clever techniques and cutting-edge technologies, high-throughput message queues, nosql databases, monitoring tools, each accompanied by one or more concrete examples.