Today we're going to explore API pagination. When building APIs that return large sets of data, we need a way to break that data into smaller, manageable chunks. That's what pagination is. It's a way to split our results into pieces and let clients request them one at a time.

Instead of sending thousands of records at once, we send them in batches of maybe 10 or 100. This helps manage server load, reduces network traffic, and keeps our application responsive. Let's look at two main ways to handle pagination: offset-based and cursor-based. With offset-based

pagination, we have two forms. First, this page-based where we set how many items we want per page, then calculate where to start. Second, we have direct offset where we use offset and limit parameters directly. Here's what a basic offset query looks like in SQL. Offset-based pagination is easy to

build, but it has problems with large data sets. When the offset gets bigger, our query gets slower. This happens because the database has to process every row up to our offset before it can return results. There's an even bigger problem when a data set changes quickly. Offset-based pagination can miss records

or show them twice. If new items are added, they shift everything around. This means we might not get consistent results between requests. Cursor-based pagination fixes these problems. Let's walk through how we implement cursor-based pagination. First, we pick an index column, like an ID, as our cursor. Then, we hash

the cursor value for security. Next, the client sends their last seen cursor value. We use this cursor to filter and fetch the next batch. After that, we send back the result plus a new cursor for the last item. Finally, the client uses this new cursor for their next

Request cursor based pagination works really well with large, fast-changing data sets. It stays consistent even when records are added or deleted between requests. This makes it perfect for real-time feeds or any data that changes often. Some cursor-based implementations are key set pagination and time-based pagination.

Key set pagination uses index keys, such as primary keys, to efficiently retrieve the next set of results by directly accessing rows without scanning all preceding ones. Time-based pagination uses timestamps as cursors to segment and retrieve records within specific time ranges. This approach is practical for applications

dealing with time series data. Now, here's a key takeaway: use cursor-based methods for large and fast-changing data sets. They're more consistent and perform better than offset-based approaches. While more complex to implement, it's worth the effort if you like videos you might

like, a system design newsletter as well. It covers topics and trends in large-scale system design, trusted by a million readers. Subscribe at blog.bio.com

API Pagination: Making Billions of Products Scrolling Possible

Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview book series. This channel is managed by Alex Xu and Sahn Lam.

To master system design, get our 158-page System Design PDF for free by subscribing to our weekly newsletter (10-min read): https://bit.ly/3tfAlYD

Take our system design online course: https://bit.ly/3mlDSk9


ByteByteGo

API 페이지 매김과 그 이점 소개

API 페이지 매김은 큰 데이터 집합을 작은 덩어리로 나누어 효율적으로 데이터를 검색할 수 있도록 합니다.

페이지 매김은 서버 부하를 관리하고 네트워크 트래픽을 줄이며 애플리케이션 응답성을 개선합니다.

두 가지 주요 페이지 매김 방법은 오프셋 기반과 커서 기반입니다.

오프셋 기반 페이지 매김: 방법 및 단점

오프셋 기반 페이지 매김은 '오프셋' 및 '제한' 매개변수와 함께 페이지 기반 또는 직접 오프셋 방법을 사용합니다.

오프셋 기반 방식은 오프셋까지 모든 행을 처리하기 때문에 데이터 집합이 클수록 속도가 느려집니다.

새로운 항목으로 인해 데이터가 급변하는 데이터 세트에서는 일관성 없는 결과가 발생합니다.

커서 기반 페이지 매김: 구현 및 이점

커서 기반 페이지 매김은 인덱스 열(예: ID)을 커서로 사용하며, 보안을 위해 해시 처리되는 경우가 많습니다.

클라이언트는 마지막으로 본 커서를 전송하여 다음 배치를 필터링하고 가져오는 데 사용합니다.

커서 기반 페이지 매김은 데이터가 변경되어도 일관성이 유지되므로 실시간 피드에 이상적입니다.

예를 들면 키 세트 페이지 매김(인덱스 키 사용) 및 시간 기반 페이지 매김(타임스탬프 사용) 등이 있습니다.

빠르게 변화하는 대규모 데이터 집합에는 커서 기반 방법을 사용하여 성능과 일관성을 개선하세요.

API Pagination: Making Billions of Products Scrolling Possible

API Pagination: Making Billions of Products Scrolling Possible

스크립트