> For the complete documentation index, see [llms.txt](https://xzhu0027.gitbook.io/blog/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://xzhu0027.gitbook.io/blog/misc/index/load-balancing.md). # Load Balancing ### #1: Consistent Hashing Only using consistent hashing is suboptimal because it balances loads about as well as choosing a random server for each request, when the distribution of requests is equal. But if some content is much more popular than others, it can be worse than that. Consistent hashing will send all of the requests for that popular content to the same subset of servers ### #2: Round-Robin/Least-Connection These two simple strategies works great for stateless services since they guarantee that each server will get equal amount of requests. However, for stateful services, they are still less-than-ideal: without hashing, these two solutions do not fully utilize the caches. One way to mitigate this cache problem is to use global(regional) cache. More specifically, the servers keep local caches, but also fall back to a regional shared cache. Writes will be send to both in-memory cache and the regional cache. On read, the server will first check its local cache and, if not found, try to fetch from the regional cache. Unfortunately, such strategy scales poorly, the shared cache traffic grows linearly with the number of servers. ### #3: Consistent Hashing with bounded load --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://xzhu0027.gitbook.io/blog/misc/index/load-balancing.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.