Signals search

Superseded. Read Signals Search instead. That page is the current design. It replaces the split between Sonar company discovery and Headhunter people discovery with one bounded workflow.

This page records the earlier Inngest component. It names workflow.signals_search, components/signals/ and src/agents/signals/, which the agentic platform replaces. Read it for history, not to build from.

Status: superseded · Superseded: 2026-08-28 · Original date: 2026-08-12

1. What we're building

On-demand search for buying signals across companies and people, packaged as a reusable component rather than a standalone app.

This is the on-demand search producer from the decided signals intelligence database. It defines no new storage. It writes into the existing intel_signals spine alongside the manual, monitor and scraper producers.

A signal on its own is a fact with no address. "Acme raised a Series B" is useful only when we hold Acme's domain, its firmographics, and the people to contact. So the workflow does not stop at the write. It hands every new subject to company search, people search and enrichment. That follow-up is §6.

Out of scope, deliberately: per-org scoring and feed materialization (lives above the store), and the caller surfaces (workflow step / chat / user-composed workflow — designed for, not built yet).

2. The core decision: route by detection mechanism

The twelve signal types don't differ by topic. They differ by how the fact is detected, and only one of the two mechanisms is a search problem.

An agent with web search reads published prose. It cannot establish an absence or a delta. started_meta_ads means the company had no ads yesterday and has ads today; no quantity of web search proves the "yesterday" half. Same for a creative-count jump, a tech-stack change, a profile change. An agent handed one of those types emits plausible output with no way to know it's wrong — the worst failure mode available, because nothing downstream can catch it either.

Routing those types away from the agent is a ~20-line rule and it deletes the entire failure class. It is a larger quality lever than any tooling, prompting or model choice applied to the agent itself.

2.1 Taxonomy by lane

The subject is a second axis, not a second lane. Both subject types appear in both lanes, so the routing rule stays one rule: it splits by detection mechanism only. The subject decides which follow-up runs in §6, and nothing else.

A type asked of the wrong lane is rejected, never guessed. A caller passing signal_types: [started_meta_ads] to lane A gets an explicit rejection in rejected_types, not an invented signal.

On tenure_milestone: it's arithmetic, so it looks like a third "derived" lane. It isn't. The arithmetic is trivial; the role start date is what goes stale. Derived from a cached intel_people row it is confidently wrong — the person may have moved months ago, and we'd nudge them about a job they left. It belongs in lane B, derived from the same live provider fetch that already supplies job_change and promotion. Freshness is a precondition on the read, not a lane in the taxonomy.

3. Architecture

The ASCII below is the spine only.

Four Inngest functions, each invoked through step_invoke_typed, plus three workflows it calls:

workflow/signals_search.requested
        ↓
  workflow.signals_search
    1 prepare + preflight        budget guard, mock flag
    2 start run                  workflow_runs + SSE
    3 route signal_types[]       split by detection mechanism
    4 invoke lanes               ctx.group.parallel, one lane may degrade alone
    5 invoke signals.ingest      one write path for every lane
    6 follow up                  resolve + enrich each new subject   ← §6
    7 finalize                   counts, funnel, SSE done
        ↓
  ┌─ A · agent ──────────┐   ┌─ B · collectors (4) ─┐   ┌─ manual entry ─┐
  └──────────┬───────────┘   └──────────┬───────────┘   └───────┬────────┘
             └──────────────────────────┴───────────────────────┘
                                  ↓
                        component.signals.ingest
                     normalize → dedup → append (INSERT only)
                                  ↓
              intel_sources · intel_signals · intel_signal_sources
                                  ↓
                      follow-up, by subject_type
              company_search · people_search · enrichment

The split is what makes it reusable: callers see typed payloads, never internals. The agent inside lane A can be swapped for a deterministic Exa/Parallel fan-out without touching a caller or the write path.

4. Lane A — the agent component

4.1 Why an agent here and nowhere else

The five lane-A types share one property: evidence is unstructured prose scattered across sources with no common API. A funding announcement lives in a press release, a trade publication, a company blog, or a filing. A hand-written query planner underfits that variety — exactly the case an agent handles well. Everything in lane B has a real API, and where a real API exists you want deterministic code driving it, not a model.

4.2 Prompt strategy — one prompt, five sections

## funding_round
sources: crunchbase, techcrunch, press releases, regulatory filings
NOT a match: a VC fund closing its own fund (the investor, not the target)
date: the announcement date, not the article publish date
payload: { amount, currency, round, investors[] }

One session covers every requested type. The prompt's active sections are the requested types; the agent plans at least one query family per requested type before deepening any one, and labels an extracted signal by the section rules — never by which query surfaced the source. Both rules exist because a multi-type call otherwise degenerates: the agent burns the search cap on the first productive type, or it mislabels a funding article as an exec change because that's what it was looking for.

No per-type managed-agent skills yet. Skills earn their provisioning cost when a type needs its own eval gate; each one adds a provision_all cycle and drift risk. Extract a section into a skill when:

• a type regresses and the failure can't be localized without an isolated eval
• its rules outgrow ~15 lines and start crowding the shared grounding rules
• the type count passes ~8 and the prompt stops being readable

Same call already made on the sonar extractor: narrow prompt, eval-gated, no skill layer.

4.3 Tools — two, shared, not per type

Two tools in the agent's context, three providers behind them:

search(query, objective, lookback_days, allowed_domains[], category?)
   └─ Exa + Parallel, deterministic fan-out, merged and deduped by URL
fetch(url)
   └─ Firecrawl, via the sanitized wrapper

The agent picks the query, not the vendor. It has no feedback loop on which index recalls better — it never sees what the other one would have returned — and the independent benchmark puts Exa, Parallel and Firecrawl in a statistical tie, so there is no correct choice to learn. Exposing three search tools would add tool-definition tokens, a decision per call, and a hidden variable in every eval. Exa's category filter is exposed as the category parameter and routed internally: capability, not vendor.

Fan-out is keyed to depth, so the policy is deterministic and evaluable:

Why fetch is Firecrawl. Lane A reads press releases, company blogs and trade pubs — JS-heavy and often unscrapeable. src/shared/company/sanitized_firecrawl_tools.py already handles that surface: full markdown, injection sanitizing (HTML comments + zero-width chars stripped), error absorption so the agent doesn't retry-loop on a dead URL, a 50k char cap, firecrawl_rate_limit at 5/s, and FIRECRAWL_SCRAPE_COST_PER_PAGE in vendor_pricing. Exa/Parallel contents cover search results, not arbitrary fetches.

Not the hosted MCP servers, the vendor CLIs, or the model-native web_search. See §9 — vendor limits are reactive (Exa 10 QPS, Parallel 600/min), our limiters sit under them (exa_rate_limit 8/s, parallel_rate_limit 5/s) and are Redis-backed so one account budget divides across concurrent runs; and provider spend has to reach ai_usage_log for data we resell. An uncapped Exa burst turned 38s of work into a 443s retry stall (run a1e68be1). Model-native web_search fails separately: its results come back as encrypted_content, so it cannot populate intel_sources.payload at all.

Two prerequisites, both in src/clients/parallel_client.py: it wires no rate limiter (parallel_rate_limit() exists but its only caller is workflow_engine/.../people_parallel_tools.py) and no sanitizer (Exa has SanitizedExaTools, Firecrawl has its wrapper; Parallel excerpts reach the agent raw). Both land before §12 step 4.

4.4 Guardrails

effort · task_budget · a max-search count stated in the prompt · the existing agent-run reaper. Cost is otherwise unbounded, and that is the main risk this lane carries relative to the deterministic pipelines we've been converging on.

4.5 What the agent must never do

• Emit a type outside the 12-value intel taxonomy. Distinct from the 13-value CRM SignalType in src/shared/signal_types.py; needs its own module + alias table on the same pattern.
• Assert a field the fetched source doesn't state. No inferred country, size, funding total, revenue.
• Resolve entity identity. It emits company_domain / person_linkedin_url; resolution is deterministic.
• Write to the database.

Returns two artifacts: sources.json (provider responses verbatim → intel_sources.payload, the replay and takedown record) and signals.json (extracted rows referencing sources by index).

5. Lane B — the collectors

Each polls a real API, stores a snapshot, emits signals from the diff. All deterministic, no model.

These are the same mechanism the monitor and scraper producers need, so they were always going to be built. This design only asserts the search agent shouldn't duplicate them.

5.1 On LinkedIn

There is no legitimate LinkedIn API or MCP for post search or profile-change feeds. The official APIs don't expose them; anything that does is a scraper wrapper — a ToS violation, aggressively blocked, and a poor foundation for a proprietary database we resell. The path for job_change / promotion / tenure_milestone is a licensed provider (CoreSignal-class, already wired) polled on a schedule and diffed.

For these types the binding constraint is data licensing, not agent capability — no skill, tool or subagent moves it.

6. Follow-up — make the signal actionable

A signal is a fact about a subject. Nobody can act on it until the subject has an address.

funding_round on acme.com is useful when we hold the firmographics and the people to contact. job_change on a LinkedIn URL is useful when we hold the new employer and a work email. The ingest step writes the fact. The follow-up step makes it usable.

The follow-up runs after ingest, and only for a subject that needs it.

Three rules keep the cost bounded.

1. Skip a subject we already hold and that is fresh. Enrichment already owns the freshness rule, so the follow-up passes refresh: stale and lets it decide. A well-known company costs one read.
2. The search runs in resolve mode, not discovery mode. It takes a name or a URL and returns one identity. It runs no planner agent and no vendor search. Discovery mode is for a brief; a signal already names its subject.
3. The follow-up is a dispatch, not a block. It fires after complete_workflow_run, in the same shape as the two trailing lanes on Sonar and Headhunter. The signal roster appears at once. The addresses fill in behind it.

A person signal also enriches the employer. job_change names a new company. That company is often new to us. The roll-up resolves it and enriches it, so the person signal seeds a company we can sell to.

follow_up: none turns the whole stage off. A monitor that re-checks known companies every night does not need it.

7. The shared write path

component.signals.ingest, identical for every producer:

1. Normalize — resolve company_domain → intel_companies, person_linkedin_url → intel_people. Stamp subject_type + subject_id. Write related_company_id so a person signal rolls up to the employer feed.
2. Dedup — dedup_key = hash(subject · type · observed window), enforced by a unique index. MISS appends a new intel_signals row. HIT appends no signal and instead attaches the new outlet as an intel_signal_sources link, so a second outlet reporting the same raise never doubles the fact.
3. Append — INSERT only. No UPDATE, no DELETE. Decay applied at read time.

Adding a collector therefore adds no write code.

8. Data contracts

A call carries a list of types, not one type — see §9. The accounting fields are therefore per-type, or a five-type run reports one number and no one can tell which section failed.

class SignalSearchInput(InngestPayload):
    run_context: ComponentRunContext
    query: str
    signal_types: list[IntelSignalType] | None   # None → agent picks within lane A
    lookback_days: int = 30
    geo: str | None = None
    target_per_type: int = 10    # per requested type, NOT a run total
    depth: Literal["quick", "standard", "deep"] = "standard"
    require_company: bool = True
    emit_progress: bool = True
    follow_up: Literal["none", "resolve", "enrich"] = "enrich"   # §6


class ExtractedSignal(BaseModel):
    signal_type: IntelSignalType
    description: str
    observed_at: date            # when the event happened, NOT when it was published
    source_ref: int              # index into sources[]
    snippet: str
    company_domain: str | None
    person_linkedin_url: str | None


class PerTypeResult(BaseModel):
    signal_count: int
    queries_run: int
    zero_result_reason: str | None   # why THIS type came back empty


class SignalSearchResult(InngestPayload):
    sources: list[RawSource]                        # verbatim provider payloads
    signals: list[ExtractedSignal]
    per_type: dict[IntelSignalType, PerTypeResult]  # attribution per requested type
    queries_run: int                                # run total
    rejected_types: list[IntelSignalType]           # asked for, but not lane A
    zero_result_reason: str | None                  # whole-run failure only
    follow_up: FollowUpSummary                      # subjects resolved / enriched / skipped
    usage: AgentUsage

target_per_type is per requested type because a run total silently shrinks each type's quota as the list grows — five types against a total of 25 is five each, and deep stops meaning anything.

depth maps to agent controls so callers never touch them. Budget and search cap scale with the number of requested types, for the same reason:

task_budget  = base × (1 + 0.5 × (n_types − 1))    capped at 2× base
max_searches = base × (1 + 0.5 × (n_types − 1))    capped at 2× base

Sub-linear on purpose: types share a corpus, so the second type costs far less than the first. The 2× cap keeps a five-type call from being five times the bill. Both the multiplier and the cap are guesses until §10 Q4 is measured.

9. Decisions and rejected alternatives

10. Open questions

1. Identity creation policy. When the agent surfaces a signal for a company absent from intel_companies, does normalize create the entity or drop the signal? Dropping loses real signals; creating means an agent extraction seeds the intelligence DB. The §6 follow-up shifts this: normalize can create an identity-only row and let the follow-up fill it from real sources, so the seeded row is never an agent's guess for long. It does not remove the question, because a wrong identity still creates a row that the follow-up then fails to resolve.
2. Undated signals. Decay runs off observed_at. When a source states no date: drop, fall back to source publish date, or store with a confidence marker? A silent fallback corrupts decay.
3. Ingest provenance and feed materialization. The decided DB design includes intel_signal_ingests (one row per discovery, so a run can list every signal it surfaced including re-finds) and intel_org_signal_feed (per-org scored copy). Neither is written by this component today. Without the ingest log, a dedup hit leaves no record that this run found the signal — so the Signals app can't list its own results. Either both come back into scope, or a run-scoped results table replaces them.
4. Cost ceiling per run. Sets task_budget and the search cap, and calibrates the §8 per-type scaling multiplier. Unset, lane A is the only unbounded cost in the design.
5. Cap on len(signal_types)? Moot while lane A holds five types — asking for all of them is the normal case. It stops being moot if the taxonomy grows: at some type count a single session's context can no longer hold the sections plus the fetched corpus, and that is the point where per-type fan-out becomes right after all. Revisit at the same threshold as the §4.2 skill-extraction trigger (~8 types).

11. Package layout

src/inngest_functions/
  workflows/signals_search/
    signals_search.py            orchestrator
    steps/{prepare,route,follow_up,finalize}.py
  components/signals/
    search.py                    lane A, agent
    ingest.py                    normalize → dedup → append
    collect/{meta_ads,job_boards,tech_stack,people_diff}.py
    schemas.py
src/agents/signals/
    open_web_search.py           agent spec + prompt
src/shared/
    intel_signal_types.py        the 12-value vocabulary + aliases

12. Build order

Resolve open questions 1 and 4 before step 4. Question 3 blocks any UI that lists a run's own results. Step 7 depends on enrichment build step 4; without the freshness rule the follow-up cannot correct an employer that a job_change signal just invalidated.

13. References

• Exa — rate limits (/search 10 QPS, /contents 100 QPS)
• Parallel — API rate limits (Search 600/min; GET reads don't count)
• Exa MCP — the rejected option
• Parallel Search MCP — basic mode, ~25k chars/call, filters go in query text
• Parallel CLI — official, agent-targeted; also rejected (§9)
• Anthropic web search tool — results return as encrypted_content; rejected (§4.3)
• AIMultiple agentic search benchmark — Brave 14.89 · Firecrawl 14.58 · Exa 14.39 · Parallel 14.21/13.50; top four statistically tied
• Firecrawl pricing — subscription credits, so FIRECRAWL_SCRAPE_COST_PER_PAGE assumes a tier
• Signals intelligence database — the decided store this writes into