How OpportunitySignal is actually built.

Keeping the catalogue current: the ingestion agent

The catalogue can't be hand-curated forever. A source like Devpost alone lists thousands of items, mostly noise. So ingestion is an agentic loop: on a schedule (or on demand), the system discovers new candidates from a source adapter, decides which are worth keeping, enriches the survivors into structured records, and stages them for a human. The source is just an adapter — Devpost is the first; the pipeline below is source-agnostic.

How decisions are made — a two-stage gate

Relevance is judged by the organizing body, not the theme. The gate is deliberately cheap-first:

• Allowlist short-circuit. A substring match of the organizer against a curated set of trusted bodies accepts an item with no model call. Most real opportunities hit this path, so the common case is free.
• LLM gate, only for unknowns. Everything else goes to a small LLM with a constrained prompt: explicit PASS/REJECT checklists, few-shot anchors, and a single JSON verdict { relevant, confidence, reason }. It is fail-closed — an ambiguous or malformed verdict counts as reject.

Why hybrid: the allowlist buys deterministic precision and zero cost on the items we're surest about; the LLM buys cheap coverage on the long tail without us maintaining rules for every organizer.

Keeping the LLM honest — grounding and structure

The model never generates an opportunity; it only transforms text we already fetched (listing JSON plus the scraped detail page). That single constraint removes most of the hallucination surface. On top of it:

• No-invention prompt. Extraction runs at temperature 0 with an explicit rule: if a field — a date especially — isn't stated in the source, return null. Never guess a year, never infer a deadline.
• One canonical schema. Both the agent and the manual bulk importer emit the same field contract, so there's a single definition of "valid".
• Validation as a hard gate. Output is requested as JSON, parsed leniently (code fences stripped), then checked against a zod schema — enums constrained to the real category set, dates coerced to real timestamps. Invalid output is discarded, not written.
• Traceability. Every record stores the source URL, so a curator can verify any claim against the original in one click.

Why a human stays in the loop

Auto-publishing was a real option and we rejected it. The reasoning is a set of trade-offs, not dogma:

• Cost of error is asymmetric. This is student-facing — a wrong deadline or eligibility line does active harm and erodes trust. Validation catches type errors; it can't catch a plausible-but-wrong date. A human can.
• The review cost is trivial. The gate already filters to a handful of relevant items per cycle, so a human approves the few that survive — seconds of work against a high downside.
• It lets us run a cheap model. Because a person and a schema absorb residual errors, we don't need a frontier model for accuracy — a small, inexpensive LLM is enough, which keeps ingestion close to free.
• Rejection is durable curation. A rejected item is deduped out forever, so the human's judgement compounds instead of being re-litigated every run.

Net: the machine handles volume and structure; the human owns publish-worthiness. Each does the part it's actually good at.

The publish lifecycle

Visibility is a small state machine. Drafts are invisible to the public site and to email — every public query is published-only — so a pending or rejected item simply cannot leak.

High-level design

The ingestion agent isn't a separate service — it's a shared library invoked two ways: by the scheduler for the unattended run, and by an admin endpoint for on-demand runs. Reads are server components hitting Mongo directly; writes go through route handlers. External calls reach only the source adapters and the LLM.

Beyond the agent

The parts that make the rest of the product hold together:

• Two-tier auth. Product routes require a NextAuth (Google) session, enforced centrally in middleware. The admin area runs a separate hand-rolled session — credentials exchanged for a signed JWT (jose) in an httpOnly cookie, verified both in middleware and per write route. Different trust levels, different mechanisms.
• Status is derived, not stored. An opportunity's lifecycle (upcoming → open → closed → ongoing → ended) is computed at read time from its dates — open and close and end and event dates — with an optional manual override. Nothing to keep in sync, no stale badges; countdowns and "closing soon" fall out of the same logic.
• Always-fresh reads. Listing pages render dynamically and sort by urgency (soonest deadline first), with a published-only filter applied everywhere so drafts never surface.
• Email as its own pipeline. A weekly digest cron (secured by a shared bearer secret) and per-opportunity alert blasts use Resend + React Email; unsubscribe links carry a signed token, so opting out needs no login.
• Discovery on the client. A global search spans opportunities and company profiles; categories stay separated so the feed never turns into an undifferentiated list.
• Design tokens + SEO. Theming is CSS variables consumed through Tailwind, with class-based dark mode; metadata, sitemap, robots and a generated OpenGraph image ship with the routes.

Watching it run

For demos and transparency, a curator can trigger the agent from the admin panel and watch every step live. The endpoint iterates the same pipeline generator and streams each typed event as a line of NDJSON; the browser renders the flow as it arrives — same code as the cron, just a different consumer.