Blog / How-to guides

Semurg as a streaming log: topics you append to and read in order

23 October 2026 · Semurg · 3 min read · Markdown

Last checked on r11.126.20261025.70c5311 (25 Oct 2026): every command below ran on a fresh node of that release. The current release is r11.127.20261026.f8fafdc (26 Oct 2026), and the commands have not been re-checked on it yet (how)

streamingdatabasehow-to

An event log is often its own system, beside the database the events describe. In Semurg a topic lives in the same store as everything else, so an order event can carry the id of the customer record it belongs to, and both are read through the same API with the same token. A topic is a named log: you append messages, and a consumer reads everything after its cursor, in order.

A producer appends messages to a topic and each is acknowledged once it is readable; a consumer reads everything after its cursor and can wait for the next message instead of polling; a count answers in one read.Your producerA topic: an orderedlogA consumer: everythingafter its cursorWait for the nextmessageHow many messages: onereadacknowledged once readableno polling

Each block puts your token in the shell first; replace <your token> with yours.

Append messages

A message is text (values) or bytes (values_b64). One call appends up to 65536 messages and up to 64 MiB, in the order given. A topic name is up to 249 characters of letters, digits, ., _ and -, the same names Kafka accepts.

bash
export SEMURG_API_TOKEN=<your token>
curl -sS http://localhost:4000/v1/query \
  -H "Authorization: Bearer $SEMURG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"op":"stream_produce","topic":"payments","values":["payment authorised","payment captured","payment settled"]}'

Expected: the sequence numbers of the first and last of the three messages, and the topic's cursor

The answer comes only after the messages are readable: a message you were told about is never missing from a read that follows.

Read in order, from a cursor

A consumer keeps the cursor it was given and asks for everything after it. max bounds one read (1024 by default, up to 65536).

bash
export SEMURG_API_TOKEN=<your token>
curl -sS http://localhost:4000/v1/query \
  -H "Authorization: Bearer $SEMURG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"op":"stream_read","topic":"payments","cursor":0,"max":10}'

Expected: the messages after cursor 0, each with its sequence number and its value in base64, and the cursor to ask from next

Sequence numbers start at 1, so cursor 0 reads from the beginning; pass the cursor from each answer to the next read. Values come back base64 (value_b64) whether you sent text or bytes, and count is how many messages the topic holds, not how many this read returned.

To wait for new messages instead of polling, add "wait_ms" (up to 600000): the read returns as soon as a message arrives after your cursor, or empty when the wait ends. Over the WebSocket, a subscriber is woken when a topic moves.

Count

How many messages each topic holds, for one topic or many in one call:

bash
export SEMURG_API_TOKEN=<your token>
curl -sS http://localhost:4000/v1/query \
  -H "Authorization: Bearer $SEMURG_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"op":"stream_count","topics":["payments"]}'

Expected: the count for the payments topic

When something is refused

  • 422: a bad topic name, value, cursor, max or wait_ms, named in the message.
  • 413: more messages or bytes in one call than the bounds above; send them in more calls.
  • 503 on an append: nothing was acknowledged; send it again.
  • 501: a release without streaming.

What it does not do yet

  • Partitions: a topic is one ordered log, so a single very busy topic takes appends more slowly than Kafka does today. Many topics append side by side.
  • Consumer groups and committed offsets: a consumer keeps its own cursor.
  • Message keys, a retention policy, or the Kafka wire protocol.

Run the streaming benchmark yourself

The benchmark runs Semurg and the incumbents (Kafka, Redpanda, Pulsar and RabbitMQ) on your machine, each on all of its cores, checks that every engine answered the same question the same way, and only then times it:

bash
curl -fsSL https://one.semurg.io/bench | sudo bash -s -- --streaming

Add --only kafka to compare against one engine. The runner checks the free disk it needs before it downloads anything, and refuses with the figure if your disks are short. Streaming results are being measured on the current release; they will appear on the benchmark page, wins and losses alike, with the release and the machine beside each figure.

Next

How these commands were checked

Our test farm ran each of the 3 command blocks marked runnable, exactly as written, with the answer checked against the expectation printed under it, against a fresh node of the published release r11.126.20261025.70c5311. A post is published only when every one of them passed there.

Related