DeviceChain Download

Engineering blog

Ingest got cheaper: filling the batches

The short version

  • On a mixed load of about 6,200 events per second on Google Kubernetes Engine, platform CPU per 1,000 events went from 2.60 CPU-seconds on the last build before the writer changes to 2.06 with them.
  • Under a steady trickle the batches did not fill: ten writers per pod waited on the same channel, so each one committed a handful of rows. Collection is now taken in turns, and device-state writes each hot upsert as one statement per tenant, with one array per column.
  • What did not move: in the MQTT-only steps on this shape, with event envelopes off, one broker server was the busiest resource when the acknowledgement time passed its bound.

The problem

The two services that write every event, event-management (into the TimescaleDB event store) and device-state (live state, in the relational database), each run ten writer goroutines per pod. A writer takes what is waiting on an internal channel, optionally lingers for more, and commits the lot in one transaction. That is the textbook way to amortize a commit.

It was not amortizing much. On the “before” build, the event store’s INSERT into events averaged 8.4 rows per statement (measured with the broker on the services nodes, a different placement from the rest of the results below), so every few rows paid the overhead of a statement and a transaction again.

The reason is arithmetic. With ten writers waiting on one channel and no backlog, an arriving message goes to whichever writer is waiting, so the trickle is split up to ten ways. A batch ends up holding about

rate × (linger + commit time) ÷ writers

events. The linger, which exists to let a batch grow, buys little, because the events that arrive during it are taken by the other writers. (device-state had no linger at all by default.) Each small batch then pays the per-statement and per-transaction overhead again: the round trip, planning, the commit, and the client-side work of assembling the statement. Under a real backlog batches do reach their cap; the problem was the steady state.

What we changed

Collection is serialized; everything else stays parallel. A pod’s writers now share one collection token (messaging.Collector, #2142). One writer holds it while it receives raw messages, so its batch fills at the full arrival rate until it reaches the cap or the linger runs out. It releases the token before decoding, tenant admission and the commit, so those can still run concurrently across the writers, and while one writer commits, another can collect. device-state’s projection.lingerMillis now defaults to 10 ms (it was 0), so its batches have time to grow.

Before and after: how a pod's writers collect a batch Left: one channel feeds ten writers at once, so each writer's batch holds a few events. Right: one writer at a time holds the collection token and collects until the cap or the linger runs out, then releases the token before decode and commit, which can still run concurrently across writers. Before: ten writers share one trickle After: collection taken in turns channel … ten writers, each a small batch batch ≈ rate × (linger + commit) ÷ writers channel holds the token: receives only waits for the token decode, admit, commit decode, admit, commit batch ≈ rate × linger, up to the cap
Before, ten writers split one trickle between them. After, one writer at a time collects, then releases the token before decode and commit.

device-state writes one statement per tenant and operation, one array per column. Its hot writes (heartbeat, lock-or-create, the folded write-back, latest measurement and latest location) used to go through the ORM as multi-row VALUES inserts. They now use a column-array upsert in the shared database layer (#2143, #2144): INSERT … SELECT unnest(…) ON CONFLICT … DO UPDATE, with the tenant bound once and, on PostgreSQL, every other column bound as a single array. The statement text and its parameter count are the same for any number of rows, so there is nothing to chunk and very little to build per row. Because ON CONFLICT cannot touch the same row twice in one statement, several updates to one key (one device’s state, say) are folded into one row before the statement runs. The locked path for presence claims still reads before it writes.

Multi-tenancy did not get a shortcut. The new mode binds the tenant from the request context, requires the conflict target to name the tenant column, and validates its update guards against an allow-list grammar, so it carries the same tenant isolation as the insert it sits beside. An equivalence test runs the old ORM statements and the new one over the same batches, on SQLite and on PostgreSQL, and compares what each leaves in the table.

The event store skips work it rarely needs. On PostgreSQL, a grouped batch is first written with a plain INSERT and falls back to ON CONFLICT DO NOTHING only when a duplicate turns up, which removes the second log record the conflict check writes for every row (#2154). The common event-id path now builds the id directly rather than through a JSON document, byte-for-byte the same as before. A new counter, devicechain_eventmanagement_persist_insert_fallbacks_total, shows how often a batch takes the fallback.

The device-state batch cap went from 32 to 64 (#2153). A batch is split by tenant and each tenant’s share is one upsert, so with several tenants running side by side, a cap of 32 left only a fraction of it in each statement.

How we measured

Both columns below use the same shape and harness (one exception, the rows-per-INSERT baseline, is noted under the table). Google Kubernetes Engine in us-east4-b: three services nodes (t2d-standard-4), three database nodes (n2d-standard-4), one load-generator node (n2d-standard-4), and the broker (upstream, unmodified) on its own pool of three t2d-standard-4 nodes, tainted for the broker. 100 GB standard persistent boot disks, premium-rwo as the default storage class, installed with dcctl install --ha and dcctl bootstrap --no-tls --no-escrow, event envelopes off, the optional MQTT acknowledgement gate off (its default).

The load was a mix: 500 HTTP devices at 5 events/s (2,500 events/s), 1,500 MQTT sessions over TLS at 1 event/s each, 500 detection background devices at 4 events/s (2,000 events/s) plus probes, and commands, for about 6,200 events/s stored, measured over a 15-minute steady window after a 4-minute warm-up, with one tenant per kind of load and the test tenants’ ingest ceiling raised for the test.

The “before” column is not the published v1.0.0 images. It is the last build on the main branch before the writer changes. The “after” column is a development build (7917eae7) taken before the release was cut; it carries the collection, column-array and plain-INSERT changes, but not the cap of 64, which was measured on its own (below).

Results

Mixed load, about 6,200 events/s Before (last build before the writer changes) After (development build 7917eae7, before v1.1.0)
Platform CPU per 1,000 events 2.60 CPU-s 2.06 CPU-s (21% lower)
Relational database CPU per 1,000 events 0.402 CPU-s 0.241 CPU-s (40% lower)
Event-store (TimescaleDB) CPU per 1,000 events 0.377 CPU-s 0.193 CPU-s (49% lower)
device-state CPU per 1,000 events 0.154 CPU-s 0.064 CPU-s (58% lower)
Rows per INSERT into events 8.4 (measured with the broker on the services nodes) 41.9
Persist latency, median / 99th percentile 21 ms / 167 ms 17.6 ms / 107 ms

Google Kubernetes Engine, the shape above, mixed load of about 6,200 events/s, 15-minute steady window. In the “before” run the broker’s CPU is an upper bound because of uneven leader placement, so the platform total flatters the change somewhat. The 8.4 rows-per-INSERT baseline was measured with the broker on the services nodes, a different placement from the rest of the column. The “after” build still had the device-state batch cap at 32.

Bigger batches usually cost latency. On this load persist latency went down instead, at the median and at the 99th percentile. The separate batch-cap comparison did show a latency cost: at 4,500 MQTT events/s on the same shape, a cap of 64 against 32 raised the rows per device_states batch from 31.4 to 49.4, lowered device-state CPU by 10% and relational database CPU by 15%, and raised the median device-state hand-off time from 8.0 to 13.2 ms, because batches wait to fill, with the 99th percentile unchanged. That is one run of each setting, not a stable average. Set projection.maxBatch: 32 to keep the previous cap.

What it did not fix

The broker. On the same shape, MQTT over TLS at one event per session per second, each step on its own fresh tenant after a 3-minute warm-up and over an 11-minute steady window, stayed under a 50 ms 99th-percentile acknowledgement time up to 5,250 events/s. At 6,000 it passed it: 60.1 ms over the run (55.7 ms as the mean of the steady window’s 10-second 99th percentiles). At that point the busiest of the three broker nodes ran at 86 to 90% CPU, while the services nodes ran at 17 to 22% and the database nodes at 14 to 32%. Cheaper writers did not remove that broker bottleneck. These are single runs against a latency bound, not a maximum sustainable rate, and they describe this shape, not a capacity guarantee.

Many small tenants. Batches are split per tenant, so a load spread over many tenants still sees smaller statements than a single-tenant one. The larger cap narrows that gap without closing it.

Envelopes are not measured on a cloud cluster yet. v1.1.0 also ships event envelopes, off by default, which carry several events of one tenant in one broker message and so reduce the broker’s work per event. So far they have been measured on a three-worker local kind cluster only, so we make no throughput claim for them here.

What’s next

In the MQTT steps on this shape, the limit we hit was one broker server, not the services or the databases. That is where we are looking next: measuring envelopes on a cloud cluster, and how a single instance can spread that work across more than one broker server.

The detailed upgrade notes, including the new collection metrics (persist_collect_full_total, persist_handoff_depth, state_collect_full_total, state_handoff_depth) that show whether your batches are filling, are in the v1.1.0 release notes.

Try it / reproduce it

  • Upgrade: dcctl install <provider> --version v1.1.0, repeating the settings the cluster was installed with (such as --ha, --kube-context and --database-node-selector), then dcctl upgrade <provider> <instance> --version v1.1.0. See Releases & Upgrades.
  • The full measured row, with its settings, is in Measured throughput.
  • The cluster module and the run script used for these measurements are in the repository at the tag: deploy/gke/README.md and hack/loadtest-rc-run.sh. The load harnesses are built from source at the tag. The measured build was 7917eae7, with Go profiling and pg_stat_statements switched on; the MQTT steps ran after the broker’s stream leaders were rebalanced.

Sources