High-Level Architecture Design for Twitter System

Twitter's architecture emphasizes a read-heavy design, requiring efficient caching and sharding to manage vast user interactions and data scaling.

### Designing Twitter: A High-Level Architectural Overview
* 🛠 Designing the architecture of Twitter involves understanding its core functionalities, including user interactions like **following others**, **creating tweets**, and **viewing a news feed**, which is primarily **read-heavy**.
* 📊 With an estimated **500 million total users** and around **200 million daily active users**, Twitter handles about **20 billion tweet reads per day**, resulting in a staggering **20 petabytes of data read daily**.
* 💾 The system needs to accommodate a **high volume of reads versus writes**, with approximately **50 million tweets created daily**, suggesting a need for **efficient storage solutions and caching mechanisms** to optimize performance.
* 📈 To scale the application, a **stateless architecture** with **load balancers** and **read-only database replicas** is essential, while sharding based on user IDs allows for **efficient data retrieval** tailored to users' **follow relationships**.
* 📦 Media storage must be handled separately using **object storage solutions** like Amazon S3, while a **Content Delivery Network (CDN)** can facilitate **faster access to static assets** such as images and videos.
* 🔄 For real-time updates, a message queue or Pub/Sub system can **asynchronously generate and cache user feeds**, ensuring that users receive **timely content** without overwhelming the system during peak loads.
* ⚖ Ultimately, the design must balance complexity with performance, recognizing that while Twitter's architecture is intricate, it is crucial for **delivering a seamless user experience** amidst the challenges of **scalability and latency**.
* 📚 Engineers are encouraged to explore official Twitter engineering papers to gain **deeper insights** into the **evolving design and architecture** of this large-scale system.

Design Twitter - System Design Interview

NeetCode

Introduction and Context of Twitter System Design

Overview of a system design interview focusing on Twitter's architecture.

Emphasis on discussing trade-offs and demonstrating knowledge, not exact replication of Twitter.

Twitter's nature as a social network with follower relationships and varying user activity levels.

Identification of Twitter as a read-heavy system due to its user behavior.

Discussion of prioritizing key features for design in a time-constrained interview setting.

Functional and Non-functional Requirements

Prioritized features: following users, creating tweets, and viewing news feeds.

Clarification of tweet size limits and inclusion of images and videos in the design.

Scale considerations: 500 million total users, 200 million daily active users.

Estimation of data read volume: 20 billion tweet reads per day, averaging 1 megabyte per tweet.

Estimation of data write volume: 50 million tweets created per day.

Client interaction with application servers for various actions.

Database choice considerations: relational vs. NoSQL, sharding options.

Implementation of a caching layer for read optimization and separate storage for media.

Use of a CDN for media distribution and discussion of pull-based CDN approach.

API design for creating tweets, getting feeds, and following users.

Data storage: relational database with tables for tweets and follows.

Indexing strategy for the follows table based on the follower for read optimization.

Tweet data structure and media referencing.

Addressing the read bottleneck with read-only database replicas.

Handling write scaling with sharding, using user ID as the shard key.

Asynchronous news feed generation using a message queue and a processing cluster.

Challenges in updating feeds efficiently for high-follower users.

Acknowledgement of design complexities and limitations.

Emphasis on the iterative nature of large-scale system design.

Recommendation to explore Twitter's official engineering papers for deeper understanding.