UCAM Cloud: a realtime seat board that never blocks section selection

The feature nobody asked for

UCamCloud is a section-selection system for a university of tens of thousands of students. Three times a year a selection window opens and, for the first five to ten minutes, it looks like Ticketmaster on a major drop: ~15,000 requests per second, all of it hammering the same hot endpoint — POST /courses/sections/{course_code}/select — trying to grab a seat before it's gone.

The requirement was simple: let a student pick a section, atomically, without double-booking a seat. That requirement was met. There was no line item for "show the seats filling up live."

But I kept thinking about the student staring at a static page, hammering refresh, not knowing whether the last seat in their section had just evaporated. So I built a live seat board — a WebSocket stream that shows every section's seat count updating in real time as the window drains. Nobody asked for it. Which meant I got to set exactly one non-negotiable rule for myself:

The rule: the realtime server is a spectator. If it falls over, catches fire, or gets deployed wrong at 3 a.m., section selection must not notice. Not one dropped seat. Not one added millisecond on the hot path.

Everything about the design falls out of that rule. This is how it works.

The constraints

1. Zero coupling to the selection path. The /select handler may tell the realtime world something changed, but it must never wait for it, never fail because of it, never even know if it worked.
2. No business logic, no persistence. The server owns nothing. It allocates no seats, writes no database rows. It renders state it's handed and forgets it.
3. Near-zero cost at idle. ~360 days a year there's no window. The whole thing should cost pocket change when nobody's looking.
4. Survive a reconnect storm. When the window opens, tens of thousands of browsers try to connect in the same few seconds. The front door has to hold.

Architecture at a glance

Four moving parts and one direction of data flow. The selection API publishes a seat-change snapshot to Redis after it has already committed and replied. A long-running Go server subscribes to those Redis channels, keeps an in-memory picture of current state, and renders it to WebSocket clients on a fixed clock. A separate analytics worker eavesdrops on the same firehose to write history. The browser just draws what it's told.

Figure 1 — the selection API commits first, then fire-and-forgets a snapshot to Redis. The realtime hub and the analytics worker are both downstream spectators.

The dashed line from the selection API to Redis is the whole ballgame. It's an arrow that's allowed to fail — and the rest of this post is really just a tour of everything hanging off the end of it.

Why "fire-and-forget" is the entire design

Here's the publish side, living at the tail end of the SelectSection handler — after the database transaction that actually reserves the seat has already succeeded:

// publishSeatUpdate sends a seat update notification via Redis PubSub.
// Updates are published to a per-course channel keyed by the parent course code.
func (s *enrollmentService) publishSeatUpdate(ctx context.Context, section CourseSection, courseCode string) {
	log := s.common.GetLogger()
 
	msg := CourseUpdateMessage{
		SharedCourseCode: section.CourseCode,
		SectionID:        section.SectionCode,
		TotalSeats:       section.TotalSeats,
		SeatsTaken:       section.SeatsTaken,
		SeatInfo:         section.SeatInfo,
	}
	data, err := json.Marshal(msg)
	if err != nil {
		log.Warn("failed to marshal seat update message", zap.Error(err))
		return // give up silently — the seat is already reserved
	}
 
	channel := seatUpdateChannel(s.config.PubSubChannel, section.CourseCode)
	vk := s.common.ValkeyClient()
	if err := vk.Publish(ctx, channel, string(data)); err != nil {
		log.Warn("failed to publish seat update", zap.Error(err)) // <- swallowed
	}
}

Look at what happens when Redis is unreachable: log.Warn, and then... nothing. The function returns. The student's /select call has already returned 200. The seat is already committed to the database. The realtime server can be scaled to zero, mid-deploy, or on fire, and the only symptom is a warning log and a seat board that's briefly stale. That's the rule made real in about four lines.

Fire-and-forget: the publish happens after the response is on the wire, its error is logged and dropped, and the caller never blocks on it. This is the opposite of the instinct most of us have — to make the write "reliable" with a retry, a queue, an ack. Reliability is exactly what I didn't want here. A retry loop on the publish would be a way for the seat board to reach back and add latency to checkout. The most robust thing the realtime feature can do is stay cheap to ignore.

Figure 2 — the seat is committed and the response is sent before the snapshot is published. Everything past the boundary can fail and selection never notices.

The hub is a renderer, not a message bus

My first mental model was wrong, and it's worth confessing because the wrong version is the obvious one.

The obvious design is an event forwarder: a seat changes, you get a message, you forward that message to every connected client. Simple — until it isn't. What happens when a client's connection is briefly slow and it misses one message? Its seat count is now permanently wrong, because the "seat 41 → 42" event is gone forever. What happens at 15,000 RPS, when a single popular course emits hundreds of updates per second? You forward hundreds of tiny messages per second to every viewer of that course. The forwarder scales with event volume, and event volume is exactly the thing that explodes during a window.

So the hub isn't a forwarder. It's a renderer, and the mental model is a graphics engine:

The reframe: seat updates are physics updates. The set of changed sections is the changing world. The clock is the frame clock. Connections are displays. The hub renders a frame of current state on a fixed tick — it never forwards an event.

This works because of one property of the messages: every seat update is a full current-state snapshot of a section, not a delta. TotalSeats: 42, SeatsTaken: 42 — not +1. Two beautiful things fall out for free:

• Coalescing: if a hot section changes 200 times in one 200ms tick, the hub keeps only the latest snapshot and renders it once. Throughput on the wire is bounded by the tick rate, not the event rate.
• Self-healing: a client that misses a frame isn't broken. The next frame carries the whole truth. There's no such thing as a lost delta because there are no deltas.

Here's the core of it — the ingest that coalesces, and the flush that renders:

// ingest records a seat-update snapshot into the dirty map; coalescing keeps
// the latest snapshot per section within a tick.
func (h *Hub) ingest(u SeatUpdate) {
	h.mu.Lock()
	defer h.mu.Unlock()
 
	if _, watched := h.courses[u.Course]; !watched {
		return // nobody on this node is viewing this course
	}
	sections := h.dirty[u.Course]
	if sections == nil {
		sections = make(map[int]protocol.SectionState)
		h.dirty[u.Course] = sections
	}
	sections[u.Section.SectionID] = u.Section // last write within the tick wins
}
 
// flush renders one frame per dirty course and sends it to that course's
// subscribers only. Runs on the student tick (200ms).
func (h *Hub) flush() {
	h.mu.Lock()
	for course, sections := range h.dirty {
		frame := buildFrame(course, sections)
		for id, cl := range h.courses[course] {
			if !cl.conn.Send(frame) { // non-blocking; full buffer = dropped frame
				cl.strikes++
			}
		}
	}
	h.dirty = make(map[string]map[int]protocol.SectionState) // world consumed
	// ...close clients that struck out...
	h.mu.Unlock()
}

The student frame clock ticks every 200ms; the admin aggregate runs on its own slower 750ms clock so a monitoring dashboard stays readable and cheap. Because a full buffer just drops a frame — and the next frame is the whole truth anyway — a slow client can never corrupt anyone's view. Dropping data is safe by construction, which is a rare and lovely thing to be able to say.

Figure 3 — the dirty map coalesces a storm of snapshots into one frame per course per tick. Wire traffic is bounded by the clock, not the event rate.

Presence without a coordinator

The board also shows how many people are looking at each section — the "37 students viewing" counter that makes the pressure visible. Counting live viewers across a fleet of servers is normally where you reach for a coordination service. I didn't want one. So presence is just a Redis sorted set per course, and it's almost embarrassingly simple:

// Each course's viewers live in a Redis sorted set: member = userID,
// score = expiry as Unix-milliseconds.
func (p *redisPresence) Add(ctx context.Context, course, userID string, expiry time.Time) error {
	return p.rdb.ZAdd(ctx, p.key(course), redis.Z{Score: score(expiry), Member: userID}).Err()
}
 
// Count evicts expired members, then returns the live cardinality.
func (p *redisPresence) Count(ctx context.Context, course string, now time.Time) (int, error) {
	cutoff := strconv.FormatInt(now.UnixMilli(), 10)
	if err := p.rdb.ZRemRangeByScore(ctx, p.key(course), "0", cutoff).Err(); err != nil {
		return 0, err
	}
	n, err := p.rdb.ZCard(ctx, p.key(course)).Result()
	return int(n), err
}

Two decisions carry the whole design:

• Member is the user id, score is the expiry. Keying by user id means a student with three tabs open collapses to a single viewer — the count is by distinct human, not by socket. Scoring by expiry (a future Unix-ms timestamp) means counting is cleanup: Count first evicts everything scored at or before now, then reads the cardinality.
• Crash recovery is nobody's job. A live node batch-extends its viewers' expiry every 15 seconds (PresenceTTL is 30s). If a node crashes, its viewers stop getting refreshed and simply fall out of the set on their own after the TTL lapses. There is no cleanup routine, no health check, no "did node 3 die" logic. A crashed node's presence evaporates because nobody renewed it.

The recompute-and-broadcast tick runs every 3 seconds, and only pushes a new count to clients when the number actually changed. Idle courses cost nothing.

Figure 4 — presence is a sorted set scored by expiry. Counting is cleanup; crash recovery is the absence of a refresh.

The firehose only turns on when someone's watching

A student viewing one course only needs that one course's updates, so the hub subscribes to Redis channels dynamically: it subscribes to a per-course channel when it gains the first viewer of that course, and unsubscribes when it loses the last. A node only ever listens to the courses it's actually serving.

Admins are different — a monitoring dashboard wants a system-wide view of every course. For that there's a firehose: a single Redis pattern subscription (<prefix>*) over every course channel. But the firehose is expensive, so it's demand-driven: the hub turns it on only when its first admin connects, and off when its last admin leaves. A student-only node never pays for it.

That "turn on the firehose" moment hides a genuinely nasty race, and it's the gotcha I want to flag:

Gotcha: the instant an admin connects and you PSUBSCRIBE the wildcard, every course that already had a per-course subscription is now covered twice — once by its narrow subscription and once by the firehose. Left alone, every one of those courses double-delivers, and the admin aggregate double-counts activity.

The fix is to make SubscribeAll tear down the narrow subscriptions it's about to make redundant, atomically, so no channel is ever covered by both at once:

// SubscribeAll turns on the admin firehose. The per-course subscriptions are
// closed because the firehose already delivers their courses — keeping both
// would double-deliver.
func (s *redisSource) SubscribeAll() error {
	s.mu.Lock()
	defer s.mu.Unlock()
	if s.firehoseOn {
		return nil
	}
	for course, ps := range s.perCourse {
		_ = ps.Close()            // drop the now-redundant narrow subs
		delete(s.perCourse, course)
	}
	s.firehose = s.rdb.PSubscribe(context.Background(), s.prefix+"*")
	s.firehoseOn = true
	go s.pump("*", s.firehose)
	return nil
}

Figure 5 — narrow per-course subscriptions for students; a single wildcard firehose that only turns on while an admin is connected. The swap is atomic so no channel is ever covered twice.

I found this the un-fun way: an admin dashboard showing selection rates that were suspiciously, consistently double what the logs said. The numbers weren't random-wrong, they were exactly 2× wrong — which, once you see it, is the fingerprint of a duplicate subscription.

Surviving the front-door storm

The rule about not touching selection covers what happens when realtime fails. The reconnect storm is about what happens when it succeeds too well — tens of thousands of browsers all connecting in the same ten seconds. The connection front door has a few layers, all cheap, all before any real work:

The accept limit and the auth check happen before the WebSocket upgrade, so a storm of credential-less or excess connections gets a cheap retryable 503 and backs off, rather than costing an upgrade:

func (s *Server) handleWS(w http.ResponseWriter, r *http.Request) {
	// Token-bucket accept limit caps how fast a node takes on new connections.
	if !s.reg.AllowAccept() {
		http.Error(w, "accept rate exceeded", http.StatusServiceUnavailable)
		return
	}
	dec := conngate.Decide(s.validator, conngate.TokenFromRequest(r))
	// Reject unauthenticated handshakes before the upgrade — cheap.
	if !dec.Accept && dec.HTTPStatus != 0 {
		http.Error(w, dec.Reason, dec.HTTPStatus)
		return
	}
	// Refuse if the node is at its cap or draining, before spending an upgrade.
	if dec.Accept && !s.reg.Admit() {
		http.Error(w, "node unavailable", http.StatusServiceUnavailable)
		return
	}
	// ...upgrade, register with the hub, start read/write pumps...
}

Figure 6 — every storm guard is a cheap, retryable rejection that happens before the upgrade spends any real work.

The drain is the subtle one. When a node shuts down for a deploy, closing 40,000 sockets at once means 40,000 browsers reconnect at the same instant — a self-inflicted storm. So the drain spreads the closes over a 45-second window with server-issued jittered reconnect hints, smearing the reconnect wave across time instead of stacking it into a spike.

Reading it back: the analytics worker

The hub persists nothing — by rule. But the seat firehose and the presence sets are a rich signal I didn't want to throw away, so a separate process samples them into history. The analytics worker is not part of the realtime server; it's a Redis-driven worker (-worker analytics) that pattern-subscribes to the same seat firehose to count selection throughput, and on a fixed 30-second bucket scans the presence sorted-sets to snapshot viewer counts. Each bucket it writes a per-session time-series row into the durable store.

The split matters: the realtime server stays a pure, stateless spectator that can be killed at any time, and the only writer of the history keyspace is a worker that idles to nothing between windows (no traffic → empty buckets → no writes). Sampling read-only with ZCOUNT over the live score range means the worker never races the realtime server's own presence bookkeeping — it looks, it never touches.

Cost, and why it's genuinely cheap

For a production window the whole thing runs on one shared t4g.large — a burstable ARM Graviton box, 2 vCPUs, 8 GiB of memory, and up to 2,780 Mbps of EBS bandwidth — that serves both the production and dev stages at once, isolated by hostname behind a single Traefik that terminates TLS (Let's Encrypt via the TLS-ALPN challenge). One box. Two environments. Between windows it can drop to a smaller size with a one-line resize, so it costs a handful of dollars a month when nothing's happening.

That's affordable because of the architecture, not in spite of it:

• Idle is free. No window means no seat changes, no publishes, no frames, no presence churn. The frame clocks tick over empty maps and return immediately. Graviton burst credits accumulate while nothing happens, and the box can sit at a smaller size until the next window.
• The frame clock caps the work. During a spike, wire traffic per course is bounded by the 200ms tick, not by the 15,000 RPS event rate. A course changing hundreds of times a second still renders one frame per tick. The server's load tracks the number of viewers and courses, not the selection throughput.
• Presence is O(cheap). A sorted set per course, batch-refreshed every 15s, counted every 3s. No coordination service to run or pay for.
• It reuses the Redis that already exists. Pub/sub and presence ride the same Valkey instance already provisioned for sessions and rate limiting. Zero new stateful infrastructure.

And if a pre-window load test ever proved the shared Redis was the bottleneck, there's a single env var (REALTIME_REDIS_URI) to repoint pub/sub and presence at a dedicated instance — no code change. Cheap by default, with an escape hatch bolted on.

Tradeoffs, and what I'd change

This isn't free of compromises, and the honest ledger:

• Presence and rosters are per-node, not global. The live viewer count is global (it's in Redis), but an admin's roster drill-down shows the viewers this node holds. On a single box that's the whole truth; the day this needs to scale horizontally, roster aggregation becomes real work. I punted on it deliberately — one box is plenty for the current window sizes, and a distributed roster for a feature nobody asked for is a bad trade.
• A single box is a single point of failure. Entirely acceptable because of the rule — if the box dies, the seat board goes dark and section selection keeps running without a hiccup. The blast radius of a realtime outage is "no live board for a few minutes," which is exactly the severity a side-car should have.
• Staleness is bounded by the tick, not zero. A student can be up to 200ms behind reality. For a seat board that's imperceptible; if this were a trading system it'd be unacceptable. Know which one you're building.

If I revisited it, the first thing I'd add is a second box behind the load balancer for redundancy during the window — not for capacity, purely so a bad deploy or a hardware blip doesn't blank the board mid-window.

Takeaways

Three things I'd carry to any "extra feature bolted onto a critical system":

1. Make the side-car cheap to ignore. The strongest reliability property a non-essential feature can have is that the essential system doesn't depend on it at all. Fire-and-forget with a swallowed error isn't sloppy — it's the whole safety guarantee. Don't make the nice-to-have reachable from the must-have.
2. Render state, don't forward events. If your messages are full snapshots instead of deltas, coalescing and self-healing come free, a slow client can't corrupt anyone, and your load tracks a clock you control instead of a traffic rate you don't. Reach for the renderer model whenever "current state" matters more than "every event."
3. Let TTLs do your cleanup. Presence-by-expiry means crash recovery is the absence of a refresh, not the presence of a health check. The cheapest distributed-systems code is the code you delete because a timestamp does the job.

The realtime seat board is my favorite thing in this codebase precisely because it's the least important. It taught me that the best way to add something risky to a system that must not break is to build it so it can't — and then it doesn't matter how often it does.