Patterns of Distributed Systems

Problem

A write-ahead log maintains every update to the persistent store. It can grow indefinitely over time. Segmented Log ensures smaller files, but the total disk storage can grow indefinitely if not checked.

Solution

Have a mechanism to tell logging machinery which portion of the log can be safely discarded. The mechanism gives the lowest offset, or low-water mark, before which the logs can be discarded. Have a task running in the background, in a separate thread, which continuously checks which portion of the log can be discarded and deletes the files on the disk.

Click here to view code image

class WriteAheadLog... this.logCleaner ...