> For the complete documentation index, see [llms.txt](https://dianadarie.gitbook.io/system-design-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dianadarie.gitbook.io/system-design-notes/system-design-patterns/segmented-log.md).

# Segmented Log

### Motivation

A single log can become difficult to manage: 

* it can become a performance bottleneck, especially when it is read at the startup
* older logs need to be cleaned up periodically or, in some cases, merged
* doing these operations on a single large file is difficult to implement.

### Solution

Break down the log into smaller segments for easier management, such that the log data is divided into equal-sized log segments. The system can roll the log based on a rolling policy (by time e.g. every 4 hours or size every 1GB)

### Applications

* **Cassandra** uses the segmented log strategy to split its commit log into multiple smaller files instead of a single large file for easier operations. As we know, when a node receives a write operation, it immediately writes the data to a commit log. As the Commit Log grows in size and reaches its threshold in size, a new commit log is created. Commit log segments are truncated when Cassandra has flushed corresponding data to SSTables. A commit log segment can be **archived**, **deleted**, or **recycled** once all its data has been flushed to SSTables.
* **Kafka** uses log segmentation to implement storage for its partitions.