Insight into Hotmail's infrastructure

A Conversation with Phil Smoot - An engineer at Hotmail discusses the challenges of keeping one of the Web’s largest and oldest Internet services running 24/7. A good overview of their challenges and methodologies of large scale system.

Summary:

• 10000 servers, 100 admins (across the world)
• Primarily Microsoft shop. (Visual studio, SQLServer etc.)
• billions of email messages (mostly spam?!!) per day

Key challenges:

• Dealing with abuse and spam
• Shipping features every 3 and 6 months
• How to release complex changes over a set of multiple releases
• Best practices
• Fully automated rollouts
• Distribute new version to first 10 servers, and then 100 and then 1000 and so on. Ensure multiple versions can co-exist
• Load balancing across the site. (transactions + user capacity)
• Focus on specs of changes, so as to be able to predict the effects and measure them. Be able to have right instrumentation for measurements.
• Predict cost of change. Even a penny counts (over millions of users.)
• Be able to operationally you can isolate a broken or failing hardware component or server or service.