--- title: "Semurg as a streaming log: topics you append to and read in order" slug: semurg-as-a-streaming-log date: 2026-10-23 author: Semurg section: how-to tags: [streaming, database, how-to] summary: Named topics you append messages to and read back in order, on the same store as your graph, vectors and memory. How to use them, what they do not do yet, and how to run the streaming benchmark yourself. status: published covers: [route:POST /v1/query, op:stream_produce, op:stream_read, op:stream_count] constants: [65536, 64, 249, 1024, 600000, 0, 1] --- > Last checked on r11.126.20261025.70c5311 (25 Oct 2026): every command below ran on a fresh node of that release. The current release is r11.128.20261026.4964735 (26 Oct 2026), and the commands have not been re-checked on it yet. An event log is often its own system, beside the database the events describe. In Semurg a topic lives in the same store as everything else, so an order event can carry the id of the customer record it belongs to, and both are read through the same API with the same token. A topic is a named log: you append messages, and a consumer reads everything after its cursor, in order. > Diagram streaming-log: A producer appends messages to a topic and each is acknowledged once it is readable; a consumer reads everything after its cursor and can wait for the next message instead of polling; a count answers in one read. Each block puts your token in the shell first; replace `` with yours. ## Append messages A message is text (`values`) or bytes (`values_b64`). One call appends up to 65536 messages and up to 64 MiB, in the order given. A topic name is up to 249 characters of letters, digits, `.`, `_` and `-`, the same names Kafka accepts. ```bash runnable kind=shell-node needs=token export SEMURG_API_TOKEN= curl -sS http://localhost:4000/v1/query \ -H "Authorization: Bearer $SEMURG_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"op":"stream_produce","topic":"payments","values":["payment authorised","payment captured","payment settled"]}' ``` ```expect expect: the sequence numbers of the first and last of the three messages, and the topic's cursor assert: json:(.last - .first)==2 ``` The answer comes only after the messages are readable: a message you were told about is never missing from a read that follows. ## Read in order, from a cursor A consumer keeps the cursor it was given and asks for everything after it. `max` bounds one read (1024 by default, up to 65536). ```bash runnable kind=shell-node needs=token export SEMURG_API_TOKEN= curl -sS http://localhost:4000/v1/query \ -H "Authorization: Bearer $SEMURG_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"op":"stream_read","topic":"payments","cursor":0,"max":10}' ``` ```expect expect: the messages after cursor 0, each with its sequence number and its value in base64, and the cursor to ask from next assert: json:(.messages|length)>=3 and .count>=(.messages|length) and .cursor==(.messages|last|.seq) ``` Sequence numbers start at 1, so cursor 0 reads from the beginning; pass the `cursor` from each answer to the next read. Values come back base64 (`value_b64`) whether you sent text or bytes, and `count` is how many messages the topic holds, not how many this read returned. To wait for new messages instead of polling, add `"wait_ms"` (up to 600000): the read returns as soon as a message arrives after your cursor, or empty when the wait ends. Over the WebSocket, a subscriber is woken when a topic moves. ## Count How many messages each topic holds, for one topic or many in one call: ```bash runnable kind=shell-node needs=token export SEMURG_API_TOKEN= curl -sS http://localhost:4000/v1/query \ -H "Authorization: Bearer $SEMURG_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"op":"stream_count","topics":["payments"]}' ``` ```expect expect: the count for the payments topic assert: json:.counts[0]>=3 ``` ## When something is refused - `422`: a bad topic name, value, cursor, `max` or `wait_ms`, named in the message. - `413`: more messages or bytes in one call than the bounds above; send them in more calls. - `503` on an append: nothing was acknowledged; send it again. - `501`: a release without streaming. ## What it does not do yet - Partitions: a topic is one ordered log, so a single very busy topic takes appends more slowly than Kafka does today. Many topics append side by side. - Consumer groups and committed offsets: a consumer keeps its own cursor. - Message keys, a retention policy, or the Kafka wire protocol. ## Run the streaming benchmark yourself The benchmark runs Semurg and the incumbents (Kafka, Redpanda, Pulsar and RabbitMQ) on your machine, each on all of its cores, checks that every engine answered the same question the same way, and only then times it: ```bash curl -fsSL https://one.semurg.io/bench | sudo bash -s -- --streaming ``` Add `--only kafka` to compare against one engine. The runner checks the free disk it needs before it downloads anything, and refuses with the figure if your disks are short. Streaming results are being measured on the current release; they will appear on the [benchmark page](/benchmark), wins and losses alike, with the release and the machine beside each figure. ## Next - [Objects, streams, search and places](/blog/objects-streams-search-and-places): the same route for four kinds of data. - [Talk to a Semurg node over plain HTTP](/blog/connect-over-rest): the token and every route.