# EduQueue Deep Dive: The Asynq Worker Pool, ChromeDP PDF Pipeline & Live-Reload Settings

> A platform-engineering deep dive into UIU EduQueue's async backbone — Asynq with a 10-permit global pool and a 2-permit ChromeDP semaphore, idempotent PDF generation, presigned R2 URLs, gocron-driven monitoring, and a Redis pub/sub settings service that live-reloads worker behavior without a redeploy.

- Author: [Azraf Al Monzim](https://monzim.com/)
- Published: 2026-04-18
- Updated: 2026-06-13
- Topic: Infrastructure
- Tags: Go, Asynq, Redis, ChromeDP, Maroto, PDF Generation, Cloudflare R2, Presigned URLs, gocron, Live Reload, Pub/Sub, Platform Engineering, Deep Dive, EduQueue
- Canonical: https://monzim.com/blog/eduqueue-async-worker-pool

---

> **Companion post to the [UIU EduQueue platform overview](https://monzim.com/projects/uiu-eduqueue-platform).** This is the long-form version of "how the worker pool actually works."

The HTTP layer of EduQueue is small on purpose. Its job is to validate, enqueue, and return a 202. The interesting code lives downstream of that, in the **Asynq worker pool** that drives every PDF render, every email send, every analytics event, and every cron-style observability tick.

This post walks through that pool — sizing, idempotency, the dual-renderer PDF pipeline, the presigned-URL distribution layer, the gocron observability stack, and the Redis pub/sub `SettingsService` that lets me change worker behavior in production without a redeploy.

---

## The pool, sized

Worker init lives in `server/worker/main.go:86-94`:

<table>
  <thead>
    <tr><th>Setting</th><th>Value</th><th>Reasoning</th></tr>
  </thead>
  <tbody>
    <tr><td>Global concurrency</td><td><code>10</code></td><td>Bound on Postgres + Redis connection use. The DB pool is sized to <code>max&#95;open=20</code>; 10 workers leaves headroom for the HTTP layer.</td></tr>
    <tr><td>PDF semaphore</td><td><code>2</code></td><td>ChromeDP allocates ~150–250MB resident per render. Two concurrent fits comfortably in a 512MB container with the rest of the worker process.</td></tr>
    <tr><td>Bulk email concurrency</td><td><code>5</code></td><td>Sits below the 10/min Azure cap with headroom for retries inside the same window.</td></tr>
    <tr><td>Email queue tick</td><td><code>30s</code></td><td>Long enough to amortize the rate-limit math (one tick = one window check); short enough to feel live to the operator.</td></tr>
    <tr><td>Worker monitor HTTP</td><td><code>:8775</code></td><td>Out-of-band so health endpoints don't share a goroutine pool with task work.</td></tr>
  </tbody>
</table>

The PDF semaphore is the only place I pin a hard internal limit below the global concurrency. ChromeDP is the only resource where the cost of over-subscription is a Linux OOM-kill rather than a backed-up queue. Asymmetric cost ⇒ asymmetric guard.

---

## Registered tasks and their idempotency keys

<table>
  <thead>
    <tr><th>Task type</th><th>Handler</th><th>Retries</th><th>Timeout</th><th>Idempotency key</th></tr>
  </thead>
  <tbody>
    <tr><td><code>pdf:generate</code></td><td><code>HandlePDFTask</code></td><td>2</td><td>50s</td><td><code>SubscriptionEmail(StudentID, ExamName, Type)</code></td></tr>
    <tr><td><code>email:bulk</code></td><td><code>HandleBulkEmailTask</code></td><td>2</td><td>—</td><td>De-duped by <code>EMAIL&#95;COOLDOWN&#95;HOURS</code></td></tr>
    <tr><td><code>campaign:custom-send</code></td><td><code>HandleCustomCampaignSend</code></td><td>2</td><td>—</td><td><code>EmailCampaign.ID</code> + recipient set</td></tr>
    <tr><td><code>monitor:health-check</code></td><td>gocron @ 5min</td><td>—</td><td>—</td><td>—</td></tr>
    <tr><td><code>monitor:daily-report</code></td><td>gocron @ 00:01 UTC</td><td>—</td><td>—</td><td>—</td></tr>
    <tr><td><code>monitor:weekly-report</code></td><td>gocron @ Sun 00:05</td><td>—</td><td>—</td><td>—</td></tr>
  </tbody>
</table>

The single most important property of every task: **idempotent by construction**. Re-running the same `pdf:generate` either no-ops (the cooldown row already exists) or produces a byte-identical PDF in R2 under the same `routines/{studentID}/{filename}-{uniqueID}.pdf` key. This is what lets me `docker compose down/up` mid-batch and not lose anyone's routine.

There is no dead-letter queue in the traditional sense — Asynq's `MaxRetry(2)` parks failed tasks in its own archived state, where they're surfaced on the operator dashboard's "stuck queue" panel. Manual replay is one click.

---

## The PDF pipeline

PDFs are generated by the worker, never by the request handler. The HTTP layer's role is to validate, enqueue a `pdf:generate` task, and return a 202 with a queued message.

### Two renderers, on purpose

<table>
  <thead>
    <tr><th>Renderer</th><th>When it's chosen</th><th>p50 latency</th><th>Knobs</th></tr>
  </thead>
  <tbody>
    <tr><td><b>ChromeDP</b> (headless Chromium)</td><td>Layouts that need real CSS, web fonts, JS</td><td>~3.5s</td><td>A4 viewport, 0.3/0.2/0.2/0.2-inch margins, 5s post-nav settle, 50s ctx timeout, <code>--no-sandbox --disable-dev-shm-usage --disable-gpu --headless</code></td></tr>
    <tr><td><b>Maroto</b> (native Go PDF)</td><td>Tabular reports, simple invoices</td><td>~300ms</td><td>No Chromium dependency; ~10× faster</td></tr>
  </tbody>
</table>

The ChromeDP path lives in `helper/pdf_html_v2.go:20-100`. The Docker flags matter: `--no-sandbox` is required because the worker container doesn't run as a privileged user; `--disable-dev-shm-usage` avoids the `/dev/shm` 64MB default that Chrome treats as fatal under load.

The 5-second post-navigation sleep is a settle window for web fonts and CSS animations. It is empirically calibrated, which is to say: I tried 1s and got broken layouts, tried 2s and got intermittently broken layouts, settled on 5s after a week of test runs.

### R2 distribution, not S3 storage

PDFs land on **Cloudflare R2** at `routines/{studentID}/{filename}-{uniqueID}.pdf`. The metadata is intentional:

```
Cache-Control: public, max-age=604800
```

A week of edge caching means a re-fetch of the same routine is served from Cloudflare's POP nearest the user. The originating R2 bucket sees one read per million students — and R2's egress is free even when it doesn't.

Distribution to the user is via **presigned URLs**, never raw bucket reads. The database stores the R2 object key, never the public URL. Each user-facing link is signed at request time:

```go
url := s3.PresignClient.PresignGetObject(ctx, input,
    func(opts *s3.PresignOptions) {
        opts.Expires = constant.S3PDFExpiry  // default 24h
    })
```

This shape gives me four properties at once:

1. **Time-bounded leakage.** A leaked URL stops working in 24 hours.
2. **No bucket exposure.** The R2 bucket is private; only signed URLs work.
3. **Per-request audit.** I can grep logs for who minted which URL.
4. **Cheap rotation.** Rotating R2 credentials invalidates every signed URL in flight without a database change.

### PIN-gated access

A presigned URL is necessary but not sufficient. The user must also prove ownership with a PIN:

<table>
  <thead>
    <tr><th>Field</th><th>Purpose</th></tr>
  </thead>
  <tbody>
    <tr><td><code>PIN</code></td><td>6-digit, server-generated, required to mint a presigned URL</td></tr>
    <tr><td><code>AccessCount</code></td><td>Incremented per fetch, capped by <code>MAX&#95;ROUTINE&#95;ACCESS&#95;COUNT</code> (default 5)</td></tr>
    <tr><td><code>ExpiresAt</code></td><td>Hard cutoff (<code>ROUTINE&#95;ACCESS&#95;WINDOW&#95;HOUR</code>, default 168h)</td></tr>
    <tr><td><code>EduQueueBlockedStudent</code></td><td>SQL-side blocklist consulted in the access query</td></tr>
  </tbody>
</table>

PIN, view counter, and expiry are *three independent guards*. Any one of them can fail open — say, a database trigger that doesn't fire — and the other two still hold. That is the platform-engineering instinct: **defense in depth where the cost of a leak is asymmetric**.

---

## Cache-driven products on top of the same pool

The same worker pool that renders PDFs also keeps the **currency module's** cache layers warm. The currency converter at `/currency` is a public read-heavy product that never goes to upstream Open Exchange Rates on a hot path — the worker hydrates Redis on a schedule, and a Postgres `ExchangeRateData` table provides a warm fallback when both cache and upstream miss.

![EduQueue currency converter page showing a USD-to-BDT result of 123.16, the quick-convert grid, currency comparison bars, batch converter, historical trend chart, popular currency pairs, and the live exchange-rate table.](https://storage-portfolio.monzim.com/uploads/1776543399526-03-currency-converter.webp)

_The currency converter is one of the products driven by this worker pool. Three cache layers (Redis hot → Postgres warm → upstream API cold) mean the page always renders something, even when upstream is down._

The pattern matters more than the product: **a worker pool that handles long-running tasks is also the right place to keep latency-sensitive caches warm**. Same goroutine pool, same Postgres connection budget, same observability surface. No second daemon.

---

## The SettingsService — live reload via Redis pub/sub

Every value that I might want to change in production at 2am without a redeploy lives in `EduQueueSettings` and is fronted by `SettingsService` (`worker/settings_service.go:24-152`).

The service:

1. Loads the table into an in-memory `map[string]string` on init.
2. Exposes `GetBool`, `GetInt`, `GetDuration` helpers with env-default fallbacks.
3. Subscribes to the Redis channel `eduqueue:settings:update`.
4. Updates its in-memory cache on every published `key:value` message.

The HTTP layer's `PUT /api/eduqueue/admin/settings` writes to Postgres and then publishes:

```go
rdb.Publish(ctx, "eduqueue:settings:update",
    fmt.Sprintf("%s:%s", key, value))
```

…and within the next moment, every worker process honors the new value. No restart. No deploy. No race.

Things wired through `SettingsService`:

<table>
  <thead>
    <tr><th>Key</th><th>Default</th><th>Effect</th></tr>
  </thead>
  <tbody>
    <tr><td><code>email&#95;processor&#95;enabled</code></td><td><code>true</code></td><td>Pauses the email tick on the next 30s boundary</td></tr>
    <tr><td><code>email&#95;rate&#95;limit&#95;per&#95;minute</code></td><td><code>30</code></td><td>Caps total send-rate independent of provider limits</td></tr>
    <tr><td><code>tracking&#95;enabled</code></td><td><code>true</code></td><td>Toggles open/click pixel injection (privacy override)</td></tr>
    <tr><td><code>provider&#95;chain&#95;order</code></td><td><code>aws,azure,http</code></td><td>Live re-orders the failover chain</td></tr>
  </tbody>
</table>

The `provider_chain_order` toggle is the one I love most. When AWS SES throttles me during exam week, I can demote it to tertiary from my phone, in three taps, while Postgres is still under load.

---

## Observability — Discord as a stack

EduQueue does not run Datadog. EduQueue does not run Grafana. The observability stack is a `gocron` scheduler in the worker (`worker/monitoring.go:22-47`) emitting structured messages to a Discord webhook.

<table>
  <thead>
    <tr><th>Cadence</th><th>What it emits</th><th>Discord behavior</th></tr>
  </thead>
  <tbody>
    <tr><td>Every 5 min</td><td>Health check vs CPU 80% / Mem 85% / Disk 90% / API 2s</td><td>Silent if green; @here on breach</td></tr>
    <tr><td>Daily 00:01 UTC</td><td>24h roll-up: subscribers, sends, opens, clicks, top campaigns</td><td>Posted to #eduqueue-reports</td></tr>
    <tr><td>Weekly Sun 00:05</td><td>Long-form report</td><td>Chunked at 2000-char Discord limit (<code>monitoring.go:1028-1102</code>)</td></tr>
    <tr><td>On-demand</td><td><code>POST /api/v1/monitor/trigger</code> with <code>&#123;"job&#95;name": "..."&#125;</code></td><td>Manual report fire</td></tr>
  </tbody>
</table>

This is intentional. Discord is a notification surface I check anyway. The chunking, the threshold logic, and the on-demand trigger together cover 95% of what I would want from a paid observability tool, at zero monthly cost. When the platform earns paid observability, the upgrade path is a Vector or OpenTelemetry collector tailing the same webhook payload format.

The metrics endpoint (`GET /api/v1/monitor/metrics`) returns the JSON the Discord posts are built from — `GoRoutines`, `HeapAlloc`, `DBConnections`, plus the per-task Asynq counters. Anyone with the session token can pull it; the payload is small enough to render inline in the operator dashboard.

---

## What I'd change

Three things this subsystem will want before the next scale milestone:

1. **Per-queue isolation.** Right now all task types share one Asynq queue. Pulling `pdf:generate` into its own queue with its own concurrency knob would prevent a slow ChromeDP run from blocking an email retry.
2. **Structured task tracing.** Every task gets a UUID; carrying it through to the Discord report and the analytics layer would let me reconstruct any single user's journey in one query.
3. **Worker autoscaling.** Right now I run a fixed 1-replica worker. A simple "add a replica when queue depth > N" loop in the deploy workflow would give me free elasticity without Kubernetes.

None of these are urgent. All are queued.

---

*Part of the [UIU EduQueue platform case study](https://monzim.com/projects/uiu-eduqueue-platform).*