Kill it, fill its disk, freeze it, cut its power: what a Semurg node gives back
30 October 2026 · Semurg · 2 min read · Markdown
durabilityreliabilityself-hostedtesting
A database earns trust by what it gives back after something goes wrong. So we break a running node the ways real machines break, then read back every write the node acknowledged. These are the results on the release published on 29 October 2026, and each one sits on our durability page beside the release it was measured on.[1]
A killed process
Thirty-two writers kept writing while we killed the node with kill -9 21 times in 140 seconds, restarting it each time.[1] Afterwards we read back all 32.2 million acknowledged writes, each at its newest acknowledged version. None was lost.[1]
A full disk
We kept writing until the data volume filled. As it neared full, the node refused new writes by name (HTTP 507, "insufficient storage") and stayed healthy, and all 2,183,488 writes it had acknowledged read back.[1] We then grew the volume while the node kept running: writes were accepted again with no restart, and all 6,190,464 acknowledged writes read back.[1]
A slow disk
We throttled the disk to 200 MB/s, then 40 MB/s, then 8 MB/s, under write load.[1] The node got slower and never wrong: every acknowledged write read back at every level.
A frozen process
We paused the node 5 times for 15 seconds each while it was under load, and all 15.2 million acknowledged writes read back.[1] During the pauses, 80 client calls timed out.[1] None of those was an acknowledged write that went missing: a call that times out was never acknowledged, so the client sends it again, and every call sent again was stored.
A power cut
We replayed sudden power loss at the block device, on a datacenter drive with power-loss protection, 21 cuts in each of two data shapes.[1] Everything acknowledged before the final seconds survived every cut, and nothing that read back was ever damaged. Writes acknowledged in the last seconds before a cut can be lost on ext4: at most 9.4 seconds in one shape and 4.7 seconds in the other, inside the 10-second bound we test against.[1] Our goal there is zero.
What it means if you run it
Your data stays on your own server, and the node keeps what it told you it kept: through a crash, a full disk, a slow disk or a frozen process, with the power-cut limit above. When it cannot keep a write, it refuses that write by name at the moment you send it, so nothing is accepted now and lost later.
The tests run again on new releases, and the durability page names the release behind every result. To try a node yourself, install one on Linux.
This post was written with AI assistance. Every result in it is checked against a published release or record before the post goes out.
Evidence
- Release r11.138.20261029.4109aec, published on one.semurg.io (release:r11.138.20261029.4109aec; the record as JSON)