A gateway service sits between unreliable devices and downstream reliable infrastructure. If the gateway collapses under burst load or network instability, the whole system suffers. This post outlines an async Rust design that is maintainable under pressure.

1. Core responsibilities of a gateway

Keep the scope explicit:

  • accept device connections and messages
  • validate and normalize payloads
  • apply auth and authorization
  • buffer and route to storage or message bus
  • expose health and operational metrics

Do not mix analytics or heavy business logic into the gateway path.

2. Concurrency model with Tokio

I use task-per-connection and channel-based internal handoff:

  • network acceptor task
  • connection worker tasks
  • bounded channel to parser stage
  • bounded channel to persistence stage

Bounded channels are essential. Unbounded channels hide overload and can trigger memory exhaustion.

let (tx, mut rx) = tokio::sync::mpsc::channel::<IngressMsg>(4096);

while let Some(msg) = rx.recv().await {
    // A single bad message must not tear down the whole drain loop.
    // Match and log-and-continue instead of using `?` here.
    match process_message(msg).await {
        Ok(()) => {}
        Err(e) => {
            tracing::warn!(error = %e, "dropping message after processing failure");
            metrics::counter!("gateway_process_errors_total").increment(1);
        }
    }
}

Using ? inside this loop is a subtle trap: the first transient error would return from the task and stop draining every other message in the channel. The drain loop should survive individual failures; only unrecoverable, structural errors should end it.

Wiring the stages together is a spawn plus a bounded channel. The acceptor owns the tx, a dedicated worker owns the rx:

use tokio::sync::mpsc;

#[tokio::main]
async fn main() {
    let (tx, rx) = mpsc::channel::<IngressMsg>(4096);

    // Persistence/parser stage: owns the receiver, drains until all senders drop.
    let worker = tokio::spawn(drain(rx));

    // Acceptor stage: clone `tx` per connection task.
    let listener = TcpListener::bind("0.0.0.0:9000").await.unwrap();
    loop {
        let (sock, _) = listener.accept().await.unwrap();
        let tx = tx.clone();
        tokio::spawn(handle_conn(sock, tx));
    }
    // dropping the last `tx` lets `drain` return; awaited on shutdown:
    // let _ = worker.await;
}

async fn drain(mut rx: mpsc::Receiver<IngressMsg>) {
    while let Some(msg) = rx.recv().await {
        match process_message(msg).await {
            Ok(()) => {}
            Err(e) => tracing::warn!(error = %e, "message dropped"),
        }
    }
}

send().await on a full bounded channel is where backpressure actually happens — the connection task waits instead of piling messages into memory.

Queue size should reflect memory budget and acceptable burst absorption.

3. Backpressure and overload behavior

Define overload policy early:

  • drop oldest non-critical telemetry
  • reject new low-priority connections
  • preserve control and heartbeat traffic

A gateway without explicit priority policy often fails all traffic equally, including critical paths.

4. Error taxonomy

Create a real error model:

  • protocol errors
  • auth errors
  • transient upstream errors
  • permanent persistence errors

Map each class to handling strategy. Transient errors can retry with jitter. Permanent schema errors should fail fast and increment explicit counters.

5. State and idempotency

Devices can resend messages after timeouts. Design ingestion as idempotent where possible:

  • include device sequence IDs
  • deduplicate in a short sliding window
  • attach trace IDs to downstream writes

Idempotency reduces data corruption during network turbulence.

6. Observability contract

Expose metrics from day one:

  • active connections
  • accepted and rejected messages
  • queue depths
  • processing latency percentiles
  • error counts by type

Without queue depth metrics, you cannot detect slow collapse under sustained load.

7. Graceful shutdown and deploy safety

For deploys and restarts:

  • stop accepting new connections
  • drain in-flight queues with timeout
  • flush pending commits
  • exit with clear status

Dirty termination during deploy is a frequent source of unexplained data gaps.

8. Testing strategy

Minimum test set:

  • parser fuzz tests
  • load test with burst traffic
  • chaos test for upstream outage
  • reconnection storms from thousands of clients

Most gateway bugs appear only under concurrency and partial failures.

Final note

Tokio gives strong building blocks, but architecture discipline is what makes a gateway reliable. Queue bounds, backpressure policy, and clear error classes are the foundation.

Sources

  • Imported from the previous version of the site (gaborl.hu). Original publication around 2024-11-12.