Error Monitoring

Alchemify uses the Sentry React SDK on the frontend and the Sentry Node SDK on the backend to capture errors and ship them to a self-hosted Bugsink instance — a single-container, Sentry-protocol-compatible event store. No data leaves the infra.

Configuration

Frontend (apps/web) Vite environment variables:

Variable	Default	Notes
VITE_SENTRY_DSN	unset	Bugsink project DSN. Leave unset to disable monitoring entirely.
VITE_SENTRY_ENV	development	Tag for filtering events. Set to production for prod builds.

Backend (apps/server and apps/chat) environment variables, loaded from ~/.config/alchemify/server.env and chat.env respectively:

Variable	Default	Notes
SENTRY_DSN	unset	Bugsink project DSN. Leave unset to disable backend capture. Same DSN as the frontend is fine — events are tagged service: server or service: chat for filtering.
SENTRY_ENV	development	Tag for filtering events.

When VITE_SENTRY_DSN is unset (or in EXPORT_MODE self-hosted bundles), the frontend SDK is never fetched at runtime — @sentry/react is dynamically imported only when monitoring is on, so the disabled path adds no initial-chunk bundle weight. The Sentry chunk is still emitted to dist/; it just isn’t loaded by the browser unless a DSN is configured.

Ad-blocker tunnel

Browser-side error events from @sentry/react POST envelope payloads. uBlock Origin’s default lists (EasyPrivacy + uBlock built-in) match on the sentry_key= query param and the /api/N/envelope/ path pattern regardless of hostname — the same rules trip on every Sentry-protocol server (Sentry, Bugsink, GlitchTip). Brave Shields and Firefox ETP-Strict do similar. With those filters on, a meaningful fraction of error events are dropped at the browser before they leave.

To work around this, the SDK is configured with tunnel: "/_t/e" — a same-origin path with no token-bearing query params and no envelope substring, so none of the heuristic filter rules match. The actual rewrite to Bugsink’s ingest path is handled in nginx (admin repo, dev only — prod rollout tracked in #658):

limit_req_zone $binary_remote_addr zone=sentry_tunnel:1m rate=30r/m;

location = /_t/e {
    limit_req zone=sentry_tunnel burst=10 nodelay;
    client_max_body_size 250k;
    proxy_pass https://bugsink.dev.alchemify.ai/api/<PROJECT_ID>/envelope/;
    proxy_set_header Host bugsink.dev.alchemify.ai;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

The block lives in both the hq.dev.alchemify.ai and the *.dev.alchemify.ai server blocks (the Builder and tenant frontends).

What gets captured

• Unhandled errors and unhandled promise rejections.
• React render errors (via a custom root RootErrorBoundary in apps/web/src/main.tsx, which forwards caught exceptions to Sentry.captureException via the lazy-loaded SDK reference).
• console.error and navigation breadcrumbs.
• fetch / xhr breadcrumbs (with bodies dropped — see scrubbing below).

Performance tracing (BrowserTracing) and session replay are disabled — error monitoring only.

What is scrubbed

Outgoing events run through beforeSend and beforeBreadcrumb in apps/web/src/lib/error-monitoring.ts:

• Headers — Authorization and Cookie are removed (case-insensitive). Referer/Referrer is rewritten to scrub sensitive query params.
• Request URLs — query params with these keys have their values replaced with [scrubbed]: token, code, email_change_token, magic_link_token, password. The same scrub applies to navigation from/to and fetch breadcrumb URLs.
• Request bodies — for all non-GET requests, the body is replaced with [scrubbed] regardless of endpoint. This covers /sql (queries + params), /function/*, /chat, wizard, auth, and file uploads.
• Error messages and exception values — both event.message and each entry in event.exception.values[].value are passed through a message scrubber that:
  • rewrites PostgreSQL Key (col)=(value) constraint-violation details to Key (col)=([scrubbed]),
  • replaces email-shaped substrings with [email].
• Console breadcrumbs — console.* arguments and breadcrumb messages run through the same message scrubber.

Residual PII risk

The message scrubber covers the common PG-constraint-violation and bare-email patterns, but cannot catch every form of free-text leakage. Stack frames, custom error messages, and extra payloads can still surface PII unintentionally. Operators should periodically review captured events and either:

• Add patterns to ignoreErrors in error-monitoring.ts, or
• Extend scrubMessage / beforeSend to redact additional recognised patterns before send.

Sandbox (TSX-in-DB) pages

Custom pages compile and run inside a sandboxed iframe at an opaque origin. Errors caught by the iframe’s internal error boundary are forwarded to the parent via postMessage, where SandboxedPage calls captureSandboxError({ slug, phase, message }). Captured events carry:

• tags.sandbox = "true"
• tags.slug — the page slug
• tags.phase — render | data | unknown | watchdog

Stack traces from sandbox errors are intentionally not forwarded across the postMessage boundary.

Backend capture

apps/server and apps/chat initialize @sentry/node at startup when SENTRY_DSN is set. Both apps tag events with service: server or service: chat via initialScope, and sendDefaultPii: false prevents the SDK from attaching IPs/user agents by default.

Two capture seams:

• with-role and with-auth wrappers — most backend failures (SQL errors, RLS violations, JWT issues, function-call exceptions) are caught inside packages/db/src/with-role.ts and with-auth.ts and converted to JSON 4xx/5xx responses. An optional onError callback fires once per caught exception (skipping AppSchemaError, which is validation, not a crash). apps/server/src/server.ts and apps/chat/src/server.ts wire Sentry.captureException as that callback when monitoring is enabled.
• Express error handler — Sentry.setupExpressErrorHandler(app) catches anything that escapes the wrappers (synchronous throws in route handlers, middleware errors).

Startup failures in main().catch() capture + flush before process.exit(1).

Backend scrubbing

@sentry/node defaults capture incoming request URLs and bodies, which on this server would mean SQL payloads, chat prompts, JWT-bearing headers, and auth cookies. Each backend app’s beforeSend hook drops:

• request.data → [scrubbed] (request body)
• request.query_string → [scrubbed]
• Authorization and Cookie headers (case-insensitive) → removed
• Sensitive query params (token, code, email_change_token, magic_link_token, password) in request.url → values replaced with [scrubbed]

It also runs event.message and each event.exception.values[].value through the same message scrubber as the frontend (PG Key (col)=(value) constraint detail and bare email addresses).

Known gap — local route catches

apps/server’s auth.ts/admin.ts/files.ts route handlers convert exceptions to JSON responses inside their own try/catch blocks, so those failures don’t currently route through the with-role/with-auth onError seam. They do still hit the Express error handler if anything bubbles past the local catch. If finer-grained capture for those paths is needed, the next step is wiring onError into apps/server/src/shared.ts’s withTransaction / withClient helpers.

The chat service’s two SSE endpoints (POST /conversations/:id/chat and POST /conversations/:id/continue) similarly catch prepareChatContext failures locally before the withAuth seam fires. Their catches do call onError for non-ChatContextError (genuine exception) paths, so capture works for those. Validation-style ChatContextError 4xx returns are intentionally not captured.

Running Bugsink

OVH dev

Prereq: sudo apt install -y python3-venv (Bugsink runs in a Python venv).

bash ops/install-bugsink.sh
INCLUDE_BUGSINK=true pm2 start ecosystem.config.cjs --only bugsink,bugsink-snappea
pm2 save

The install script creates a venv at ~/bugsink/.venv, runs pip install bugsink, generates ~/bugsink/bugsink_conf.py from the upstream singleserver template (bugsink-create-conf --template=singleserver), runs both migrate and migrate snappea --database=snappea, then prompts for a Django superuser. The pm2 entries (bugsink, bugsink-snappea) are gated on INCLUDE_BUGSINK=true so they only start when you opt in. The --only filter avoids touching the running hq/server/chat/provision processes.

~/.config/alchemify/bugsink.env is sourced by the install + run scripts and is the place for runtime overrides (e.g. SMTP creds). With the singleserver template, SECRET_KEY, BASE_URL, and BEHIND_HTTPS_PROXY are baked into the generated bugsink_conf.py rather than read from env. Changes to bugsink.env only take effect after pm2 restart bugsink bugsink-snappea. bugsink_conf.py is generated only on first install — to pick up template changes or a different host, delete the file and rerun ops/install-bugsink.sh.

nginx vhost lives in the admin repo at config/etc/nginx/sites-available/alchemify (bugsink.dev.alchemify.ai block). The *.dev.alchemify.ai wildcard cert at /etc/letsencrypt/live/dev.alchemify.ai/ already covers the bugsink hostname — no certbot work needed.

DSN wiring for the dev frontend: write apps/web/.env.local (gitignored, persists across git pull) with VITE_SENTRY_DSN=... and VITE_SENTRY_ENV=development. The DSN is build-time config — to apply, rerun ~/admin/scripts/release.sh (which builds and publishes apps/web/dist → /opt/alchemify/web). A bare pnpm build:web only updates apps/web/dist; nginx serves /opt/alchemify/web, so an unpublished build won’t be visible to browsers.

AWS prod

The Bugsink service definition lives in the admin repo’s docker-compose.yml (alongside hq, server, chat, provisioning), not in this monorepo’s compose file (which targets self-hosted exports). The full prod rollout — service definition, DNS, nginx reverse proxy, EBS volume, secret management, backup rotation, and DSN wiring into the frontend build — lives in #658.

Known gaps

• Source maps — not yet uploaded to Bugsink, so production stack frames reference minified bundles. Tracked as a follow-up.
• Local dev — no Bugsink instance runs locally. Local builds set VITE_SENTRY_DSN to empty; the SDK is a no-op. Test scrub behaviour via error-monitoring.test.ts.