Build a durable event writer

What is group commit, and why does it help?

A sync is the slow call, and a design with one sync per event makes every caller wait out every other caller's sync. Group commit lets every caller that arrives while a sync is running write its record and wait for the next sync together. The caller that finds nobody syncing notes how many records are written, flushes, lets go of the lock, syncs once, and wakes everyone. A test that holds the first sync until four more writers have written records two syncs for five appends, and nobody is acknowledged before their own record is synced.

How do you make file order match position order?

Hand out the position under the same lock that adds the event to the pending batch, so the batch is always in position order. Then let only one caller write at a time, and have it take the whole batch in one step. Batch after batch lands in the order the positions were given, and a test can check it by sorting the acknowledgements by position and comparing them with the file. Positions handed out anywhere else, or two writers allowed at once, and the promise is gone.

Does write() returning mean the event is on the disk?

No. A write into a buffered Python file lands in Python's own buffer; flush() forces those bytes into the raw stream, which is the operating system; and os.fsync() is what forces the file's data to the disk. The Python documentation says to do flush() first and then os.fsync(f.fileno()). Bytes in Python's buffer are lost if the process crashes; bytes in the page cache survive that and are lost on a power failure or a kernel crash. A writer that acknowledges before the sync returns has told its caller something a crash can make false.

What should the writer do when a sync fails?

Fail every caller whose event was in that batch with an exception that names the failure, and chain the original error to it. Then decide what later appends get. This build refuses them: after a failed sync nobody can say which of the batch's bytes reached the device, and handing out further positions would put events after a hole of unknown size. A caller that got the exception has to treat its event as unknown rather than lost, because some of the bytes may have landed.

How does recovery cope with a write torn by a crash?

Frame every record with its length and a CRC-32 checksum of its body. Recovery reads a header, checks that the whole body is there and that its checksum matches, and stops at the first record that fails either test. Then it truncates the file at the end of the last whole record and syncs the cut, so the next append is not written after torn bytes that recovery would stop at. A record that was never acknowledged may still be read back if all its bytes happened to land, which is allowed: the promise is that every acknowledged event is in the file, not the other way round.