# Three bugs that hung my PDF service | DocPipe

> My HTML-to-PDF service hung under load. The fix was three problems compounding — a zombie reaper, a chromedp lifetime gotcha, and a synchronously-fired event I was waiting for asynchronously.

- Author: [Azraf Al Monzim](https://monzim.com/)
- Published: 2026-05-19
- Updated: 2026-06-13
- Topic: Infrastructure
- Tags: go, chromedp, docker, pdf, debugging
- Canonical: https://monzim.com/blog/docpipe-v2-three-bugs

---

I run a small service called DocPipe. It does one thing: take HTML, return a PDF. The CASCK job portal uses it to generate admit cards — a couple of thousand per hour during exam season, near-zero the rest of the time.

For a year, the v1 implementation was fine. Sixty lines of Go, one HTTP handler, chromedp to drive a headless Chromium. It worked. Then one Tuesday in April it stopped working, and stayed stopped, and the only way back was a container restart.

This is what I learned debugging it, what I rebuilt, and three real bugs I hit while rebuilding that I want to remember.

## What v1 looked like

The whole renderer was this, roughly:

```go
func generatePDF(w http.ResponseWriter, r *http.Request) {
    var input HTMLInput
    json.NewDecoder(r.Body).Decode(&input)

    htmlBytes, _ := base64.StdEncoding.DecodeString(input.Base64HTML)
    htmlContent := string(htmlBytes)

    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()
    ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    var pdfBuffer []byte
    chromedp.Run(ctx,
        chromedp.Navigate("about:blank"),
        // assign the decoded HTML into the page body via Evaluate
        // (this is wrong; we'll come back to it)
        injectHTMLIntoBody(htmlContent),
        chromedp.Sleep(10*time.Second),
        chromedp.ActionFunc(func(ctx context.Context) error {
            buf, _, err := page.PrintToPDF().
                WithPrintBackground(true).
                WithPaperWidth(8.27).
                WithPaperHeight(11.69).
                Do(ctx)
            pdfBuffer = buf
            return err
        }),
    )

    w.Header().Set("Content-Type", "application/pdf")
    w.Write(pdfBuffer)
}
```

The `injectHTMLIntoBody` step mutated `document.body` directly via `Evaluate` — which discards everything in `<head>`: doctype, stylesheets, web fonts. We'll come back to that. But the structural problem, the one that makes it eventually hang, is that **every request spawns a fresh Chromium process**.

Here's the actual failure mode, in order:

1. Request arrives.
2. `chromedp.NewContext(context.Background())` spawns a Chromium process and its grandchildren (renderer, GPU, zygote, crashpad).
3. We render, write the PDF, and `cancel()` returns.
4. The Chromium process exits, but its grandchildren do not get reaped immediately. They become orphans.
5. The Go binary is PID 1 in the container. PID 1 has a special responsibility: reaping orphaned children. The Go runtime doesn't do this.
6. After tens of thousands of requests, the kernel runs out of process IDs in the container's namespace, `fork()` starts failing, and the next `chromedp.NewContext` blocks forever waiting on a Chromium that will never start.

I knew about the PID 1 thing — there are blog posts about it going back to 2016 — but I'd convinced myself my service didn't need it because the parent process exits cleanly. It doesn't.

## The fix isn't one thing

If you read the literature on this you'll find three independent fixes, each presented as if it alone solves the problem:

- "Use `tini` as PID 1."
- "Reuse a single browser process via a shared allocator."
- "Don't sleep — wait on a load event."

The annoying truth is that you need all three, and missing any one of them keeps the bug latent.

`tini` alone reaps grandchildren but you're still paying full Chromium startup cost per request, which limits sustained throughput. A shared allocator alone reduces the number of processes but doesn't reap the ones that *do* die. Switching from `Sleep(10s)` to event-based waiting alone just makes the failure happen faster.

So the v2 rebuild had three load-bearing changes:

1. One long-lived `chromedp.ExecAllocator` for the lifetime of the process. Per-request tabs derive from a shared parent context.
2. `tini` as PID 1 in the container.
3. Event-based waits — `Page.loadEventFired`, optionally with a 500ms network-idle debounce — instead of `Sleep`.

And then while implementing those three fixes, I hit three more bugs.

## Bug one: the chromedp parent-context lifetime gotcha

The pattern I started with was canonical:

```go
allocCtx, allocCancel := chromedp.NewExecAllocator(ctx, opts...)
parentCtx, parentCancel := chromedp.NewContext(allocCtx)

// Warm the browser
probeCtx, cancelProbe := context.WithTimeout(parentCtx, 20*time.Second)
defer cancelProbe()
if err := chromedp.Run(probeCtx, chromedp.Navigate("about:blank")); err != nil {
    return err
}

// Now ready for per-request tabs
b.parentCtx = parentCtx
```

I built the service. I started it in Docker. The browser launch succeeded. The first render came in, and failed in 1.7ms with `context canceled`.

I spent the better part of an hour adding logs. The supervisor probe was passing, then failing, then succeeding again. The parent context's `Err()` was non-nil within milliseconds of `spawn()` returning. But nothing was calling `parentCancel()`. What was canceling it?

The chromedp source. Specifically, this line of behavior that isn't called out in the package docs:

> The browser is created lazily on the first `Run`. The browser process's lifetime is bound to the context passed to that first `Run`.

`probeCtx` was a child of `parentCtx` via `context.WithTimeout`. When I called `chromedp.Run(probeCtx, ...)`, the browser process was attached to `probeCtx`, not `parentCtx`. When my deferred `cancelProbe()` fired at the end of `spawn()`, it canceled `probeCtx` — and took the browser with it. The `parentCtx` was just a wrapper that pointed at a dead browser.

The fix: never pass a child of `parentCtx` to the *first* `Run`. Bound the launch a different way:

```go
launchErr := make(chan error, 1)
go func() {
    launchErr <- chromedp.Run(parentCtx, chromedp.Navigate("about:blank"))
}()
select {
case err := <-launchErr:
    if err != nil {
        return err
    }
case <-time.After(20 * time.Second):
    return errors.New("browser launch timed out after 20s")
}
```

Now the browser is attached to `parentCtx` directly. The timeout-via-select bounds the launch without ever giving the cancellation gun to a child context.

The thing that hurts about this bug is that **the canonical pattern works for one-shot scripts** (`NewContext` → `Run` → `cancel`) because the cancel that kills the browser is intentional. It only breaks when you try to keep the browser alive across multiple `Run` calls and want a timeout on the launch.

## Bug two: a load event that fires while you're not looking

With the browser alive, the next request came in. It hung for the full timeout. Then failed.

The wait strategy was supposed to be "load": fire when `Page.loadEventFired` arrives. The implementation was:

```go
// inside runRender
chromedp.Run(ctx,
    chromedp.Navigate("about:blank"),
    setDocumentContentAction(html),
    waitForLoad(opts.WaitTimeout),
    printPDFAction(opts, &pdf),
)

func waitForLoad(timeout time.Duration) chromedp.Action {
    return chromedp.ActionFunc(func(ctx context.Context) error {
        loaded := make(chan struct{})
        chromedp.ListenTarget(ctx, func(ev any) {
            if _, ok := ev.(*page.EventLoadEventFired); ok {
                close(loaded)
            }
        })
        select {
        case <-loaded:
            return nil
        case <-time.After(timeout):
            return fmt.Errorf("page load timeout")
        }
    })
}
```

The actions in `chromedp.Run` execute sequentially. So `setDocumentContent` runs, then `waitForLoad` runs, then `printPDF` runs. Looks fine.

It isn't fine. `Page.setDocumentContent` *fires `loadEventFired` synchronously inside the call*. By the time the next action installs its listener, the event has already passed. The listener will wait forever — or until our timeout fires.

I worked this out by reading chromedp event listener semantics very carefully and matching them against CDP's `Page` domain docs. The fix is to install the listener *before* the action that triggers it. Since `chromedp.Run` runs actions sequentially, you can't do that across two separate actions — you have to bundle them inside one `ActionFunc`:

```go
func renderWithWait(html string, opts Options, out *[]byte) chromedp.Action {
    return chromedp.ActionFunc(func(ctx context.Context) error {
        // 1. Install the listener BEFORE the action that fires the event.
        ready, cleanup := startWait(ctx, opts)
        defer cleanup()

        // 2. Load HTML — this synchronously fires loadEventFired.
        if err := loadHTML(ctx, html); err != nil {
            return err
        }

        // 3. Wait for the listener to signal.
        select {
        case err := <-ready:
            if err != nil { return err }
        case <-ctx.Done():
            return ctx.Err()
        }

        // 4. Print.
        return printPDF(ctx, opts, out)
    })
}
```

The first time I ran this, the render completed in 120ms. The first render had been waiting 3 seconds to time out on a load event that fired 2.999 seconds before the listener attached.

Three lessons compressed into that bug:

1. **`chromedp.Run` runs actions one at a time.** Listeners attached in action N can't catch events fired in action N-1.
2. **`Page.setDocumentContent` is not asynchronous like a real navigation.** It's an in-frame mutation that fires load events synchronously.
3. **If your event-driven wait sometimes times out and sometimes returns instantly, you're racing.** Test under load before declaring victory.

This was also the moment I went back and fixed v1's other mistake: that `injectHTMLIntoBody` step from earlier was assigning HTML directly into the body element, which silently discards the whole `<head>` — doctype, stylesheets, web fonts, the lot. The fix is to call CDP's `Page.setDocumentContent` against the root frame ID, which replaces the entire document. v1 had been producing cosmetically wrong PDFs the whole time and no-one had noticed because the admit-card template inlined its CSS.

## Bug three: Alpine's broken Chromium

With the renderer working in local tests against my system's Chromium, I built the production Docker image. The base I'd planned to use was Alpine 3.20 with the `chromium` package. Multi-stage build, `tini` as PID 1, non-root user — by the book.

The container started. The first render returned 500 instantly. Container logs:

```
chrome_crashpad_handler: --database is required
Try 'chrome_crashpad_handler --help' for more information.
```

`chrome_crashpad_handler` is the out-of-process crash reporter that ships with Chromium. In Alpine 3.20, the `chromium` package (version 131) is built such that this subprocess is launched unconditionally — and when launched without `--database`, it aborts immediately, taking the parent Chromium with it.

I tried every flag I could think of:

- `--disable-crash-reporter` — no effect
- `--no-crash-upload` — no effect
- `--disable-features=Crashpad` — no effect
- `--disable-breakpad` — no effect
- `--noerrdialogs` — no effect
- `--user-data-dir=/tmp/c` (give crashpad a writable directory) — no effect
- `--single-process` (skip the subprocess fork entirely) — no effect

Every invocation died the same way. I went looking at the actual `chromium-launcher.sh` wrapper, the chromium config files, the binary itself — nothing pointed at a flag that would disable the crashpad subprocess. It's an Alpine packaging bug.

The two fixes I considered:

1. **Downgrade to Alpine 3.19.** Ships Chromium 121, which doesn't have the bug.
2. **Switch base image** to `chromedp/headless-shell` — a curated Debian-based image the chromedp project maintains specifically for running headless Chromium with chromedp.

I tried 3.19 first. Same failure. So I switched to `chromedp/headless-shell`, added `tini` and the Noto fonts on top, and it worked first try. The image is about 280MB — comparable to Alpine + Chromium — and it's a known-good combination tested by the chromedp maintainers.

I documented the decision in the Dockerfile:

```dockerfile
### Why chromedp/headless-shell instead of alpine + chromium:
#
Alpine 3.20's chromium 131 packaging is broken — chrome_crashpad_handler is launched unconditionally and aborts with "--database is required" before any flag we pass is honoured. Alpine 3.19 (chromium 121) shows the same failure. The chromedp/headless-shell image ships a curated headless chromium that's known-good with chromedp and is what the upstream project recommends for production.
```

Comments are usually a code smell, but a comment explaining a non-obvious dependency choice — especially a "we tried the obvious thing and it doesn't work" — is exactly the kind of comment that belongs in code. The next person who tries to slim the image by switching to Alpine will hit the same wall I did, three months from now, with less context.

## Validating the fix

The point of v2 was to stop hanging. Three independent fixes each plausibly help, but I wanted evidence. So the soak test is the regression gate:

- 1000 sequential requests
- Then 50-wide parallel waves for 10 minutes
- At the end, assert:
  - `/readyz` still returns 200 (browser process still healthy)
  - RSS within 100MB of baseline (no leak)
  - **Zero zombie processes inside the container** — this is the v1 hang's signature
  - `/v1/stats.totals.requests` matches what we sent

I ran a compressed version against the built image. 101 requests, 0 failures, 0 zombies, RSS flat. Then I ran the full version, twice. Same result.

The thing I want to remember about soak tests is that **the failure mode you're guarding against often doesn't manifest at low volume.** v1's hang showed up after tens of thousands of requests over weeks. Reproducing it deterministically in 10 minutes required pushing process creation rates well above production. If I'd only tested correctness at low volume, all three of my new bugs would have shipped and I'd have rediscovered them in production.

## What I'd do differently next time

A few notes for future-me:

1. **Read the package source, not just the docs.** The chromedp parent-context lifetime behavior is exactly the kind of thing docs gloss over and source code makes obvious. I lost an hour to assumptions; the source took five minutes.

2. **Listener-then-trigger is a strong default.** Whenever I'm wiring an event listener, I now ask: "Could the event fire before the listener attaches?" If the answer is anything other than no, the listener goes first.

3. **`tini` is free; use it always.** Anyone reading a Dockerfile in 2026 and seeing the application binary as PID 1 should reach for `tini`. There's no downside, the image cost is tiny, and the failure mode without it is invisible for weeks before it kills you.

4. **Don't trust a base image without exercising it.** I would have caught the Alpine/crashpad bug on day one if I'd done a smoke test of `chromium-browser --headless about:blank` inside the chosen image before writing any application code.

5. **Comment the "why this dependency" decisions.** Code quality folklore says don't write comments — and that's right for "what this does" comments. But "we tried X and it doesn't work because Y" is institutional knowledge that lives in your head until you write it down.

## Where DocPipe is now

v2 has been running in production for a few weeks. The dashboard shows ~3 requests/second average with peaks near 40/s during exam events, p95 latency around 800ms, zero browser restarts after the first day, and the analytics file on disk is roughly the size of a screenshot.

The legacy v1 endpoint still works — `POST /api/html-to-pdf` accepts the old payload, returns `admit_card.pdf`, and emits `Deprecation: true` + `Sunset` headers so the portal team knows to migrate. When they do, that handler comes out and v3 stops being a thing I have to think about.

The image lives at [`ghcr.io/monzim/docpipe`](https://ghcr.io/monzim/docpipe). The source is at [github.com/monzim/docpipe](https://github.com/monzim/docpipe). The complete writeup with architecture diagrams and the full request lifecycle is in the repo's `ARCHITECTURE.md`.

You can check the deployed dashboard from here [https://docpipe.services.monzim.com/v1/stats/dashboard](https://docpipe.services.monzim.com/v1/stats/dashboard)

If you're building anything similar — synchronous, browser-backed, supposed to stay up forever — read `internal/render/browser.go` first. The fixes there are the load-bearing ones.