fix(connectors): validate and repair the knowledge-base connector fleet - #6757

Five review passes over all 61 knowledge-base connectors, validated against each provider's live documentation or machine-readable spec. Passes 3 through 5 reviewed the previous pass's edits, which is where most of what follows was caught — including regressions the audit itself introduced.

The dominant bug class

shouldReconcileDeletions hard-deletes any stored document absent from a listing that did not set syncContext.listingCapped. So any path that turns a failure into an empty or short listing is a silent mass-delete. Most fixes here are instances of that one shape:

• confluence — the CQL path paginated with start and read totalSize; neither exists on /content/search. hasMore could never be true, so it only ever returned the first page, and that truncated listing fed deletion reconciliation on every sync.
• gmail — a failed labels.list was swallowed, producing a query matching nothing. An empty page arriving with a nextPageToken also ended the listing early, producing a short-but-complete-looking listing that reconciliation acted on. Separately, the labels selector stores label ids while validateConfig compared against label names, so any user label was rejected at save time and could never be configured at all (system labels, where id equals name, worked).
• google-calendar — listingCapped was set nowhere, and maxEvents defaults to 500. Any calendar with more than 500 events in the configured window listed only the first ~500 and reconciliation hard-deleted the rest — on the default configuration, with nothing for the user to misconfigure.
• wordpress and zendesk — both capped their listings while setting listingCapped nowhere, so every stored document beyond the cap was hard-deleted on every run. wordpress additionally paged by pure offset, which re-numbers mid-run when a post is published, skipping posts.
• zoom — formatDate used local-time getters against an API documented in UTC, so on any non-UTC host the window boundary shifted by a day and that day's recordings could fall outside every window and never be listed. Separately, both bounds are inclusive but the span was computed as to - WINDOW_DAYS, producing 31-day windows that overlapped by a day at each seam — seam-day recordings were listed twice and consumed the budget twice.
• salesforce — a hardcoded LIMIT bounded the entire result set with no nextRecordsUrl, so any org larger than that limit had the remainder hard-deleted every sync.
• github — listingCapped was set nowhere, so a repo truncated by the Trees API's own 100,000-entry / 7 MB limit, or by a maxFiles cap, produced a short listing and reconciliation hard-deleted the stored documents outside it.
• sharepoint — the cap gate missed in-page truncation. Now documents.length < stubs.length, mutation-verified.
• discord — a non-numeric maxMessages produced NaN, the loop never ran, and the empty listing hard-deleted the stored document.

Other production faults repaired

• salesforce — API_VERSION was 'v62.0' substituted into a {version} placeholder that already carries the v, yielding /services/data/vv62.0/. Every REST call 404'd.
• notion — block-fetch failures were swallowed and the walk never recursed, so pages with tables, toggles, callouts, columns or nested lists were indexed empty.
• airtable — the content hash included attachment URLs, which expire every two hours, so attachment-bearing records re-indexed on every sync forever.
• google-docs / google-drive — incompleteSearch was read in code but never named in the Drive fields mask, so the partial-corpus guard was dead code.
• jira — listingCapped was set nowhere, so any source with maxIssues below its real issue count listed only the top N by updated DESC and hard-deleted everything that fell out of that window, every sync. (Correcting an earlier draft of this description: the /search/jql migration is not in this diff — staging already used that endpoint. Only PAGE_SIZE changed.)
• github — a 30-item batch size capped listings at 15,000 files, so larger repos hit the page ceiling and had reconciliation blocked permanently.

Cap-at-exhaustion, and the deletion wave it implies

The inverse error: setting listingCapped when the cap lands exactly on source exhaustion permanently blocks reconciliation, so documents deleted upstream are stranded in the knowledge base forever. Fixed across gmail, ashby, azure-devops, grain, granola, greenhouse, incidentio and others.

This has a visible consequence on the first sync. Those connectors have been over-flagging, so deletion reconciliation has been suppressed on them — possibly since setup. Once the gate is correct, the documents that were stranded become eligible for removal and will reconcile away. That is the fix working as intended, and it is two-phase (tombstone, then delete only on a second consecutive absence, behind the suspect-listing guard), but it will read as a deletion wave.

Two connectors were worse than over-flagging: hubspot and intercom never set listingCapped at all, so every capped sync hard-deleted every stored document beyond the cap, on every run. Both now set it.

Pagination

"Last-page precision" (shrinking page size to the remaining budget) is only valid when the cursor is position-independent. Over an opaque cursor token no provider documents that a cursor survives a changed page size, and the cap was already enforced server-side — so it bought nothing and risked a mid-sync iterator error. Reverted on airtable, asana, ashby and confluence.

To be precise about scope, since an earlier draft of this description overstated it: no taper was ever committed for greenhouse or intercom. Both were audited as page-number APIs (Harvest documents page as "the n-th chunk of per_page objects", and Intercom's /articles is the same shape), and both already held their page size constant on staging. The only change there is a comment recording why it must stay constant — there was nothing to revert.

Regressions this audit introduced and fixed within it

Net zero versus staging. Listed because the record should be honest about which findings were pre-existing and which were our own — a later pass reviewed each earlier pass's edits, which is the only reason these did not ship.

• fireflies — an early pass added .catch(() => null) on the JSON parse plus data?.data ?? {}, turning an unparseable 2xx into a confident empty listing. Staging returned data.data bare and threw loudly, so this was never a production fault.
• github — an early pass narrowed matchesExtension in a way that dropped dotfiles, and separately added a utf-8 blob branch justified by a misattributed quote. Both reverted; net versus staging is TSDoc only.
• airtable, asana, ashby — page-size tapering over opaque cursors, added then reverted. Staging always sent a constant page size.
• greenhouse — a 403 classified as transient, appending :partial to a hash that then never converged.
• google-meet — a 404 → break in the new fetchParticipants; staging had no such function.
• google-docs — the response byte cap was briefly 800 MB before landing at 100 MB.
• hubspot — a loose HTML-detection regex, before the shared anchored helper; staging did no detection at all.
• typeform — an undocumented started enum member; staging omitted the parameter entirely.
• secure-fetch — retry headers attached enumerably, so the logger's own-enumerable copy wrote upstream Set-Cookie values into sync logs. Staging attached no headers at all.

Security

SecureFetchHeaders declares private setCookies: string[], but TypeScript private is compile-time only — at runtime it is an own enumerable property. @sim/logger copies own enumerable properties into the formatted line and the retry loop logs { error } on every failed attempt, so newly-attached response headers wrote upstream Set-Cookie values into connector-sync logs. Now attached non-enumerably. Scope is the dev/colorize branch only — production reduces errors to .message.

SFTP now requires a host-key fingerprint on all sources, failing closed before the credential is sent (verified by mutation: 7 tests go red). DocuSign's OAuth host is allowlisted to demo/production so a misconfigured value cannot redirect the flow and its bearer token to another origin.

Fabricated doc citations

Eight TSDoc comments quoted provider sentences that do not exist (github ×2, jira, jsm, linear, google-meet, dropbox, youtube) — two of them introduced by this audit. One inverted the usual failure mode: google-meet's code was correct and the comment invented a sentence to justify it, which nearly got a correct fallback reverted. Worth noting that two of three "unverifiable" flags carried into the final pass turned out to be fully documented — re-checking beat reverting on suspicion.

Removed

Evernote, entirely — its auth flow was non-functional, so this is removal rather than repair. Verified complete: the only surviving references are prose in agent-tooling markdown; both icon registries are de-registered. No knowledge-base source, no deployed workflow, and no executed workflow references it.

Customer impact

One-time re-index, deliberate. externalId is unchanged in all three, so these are in-place updates — no deletes, no re-creates, no re-auth:

Other behaviour changes worth watching on first sync:

• airtable — records with attachment/collaborator/barcode/button cells re-index once. This ends permanent churn rather than causing it.
• asana — archived projects are now excluded; tasks living only under archived projects reconcile away.
• github — file ceiling rises 15,000 → 100,000. Repos previously hitting it had deletions permanently suppressed and will now reconcile normally. An add wave, not a delete wave.
• google-drive — getDocument now throws instead of returning null, so persistent content-read failures become visible failed rows. Expect a one-time rise in reported failures; this is the intended invariant, not a defect.
• confluence — sources whose listing was previously truncated to a single page will index their full corpus once.
• monday — consolidating MONDAY_API_VERSION moves four call sites (both selector routes, and webhook create/delete) from an inline 2024-10 pin to 2026-04. To be precise: 2024-10 is in Maintenance, not deprecated, so it was still being served as declared — this is not a deprecation fix. What was genuinely wrong is the divergence: the selector routes listed boards at one schema version while the tools operating on those boards ran at another, inside a single user-visible flow. The consolidation is right, but it jumps roughly five version bumps at those four sites, and monday fails silently here (an unrecognized version resolves to Current rather than erroring). Worth a reviewer's eye rather than being read as a pure refactor.
• reddit, microsoft-teams, outlook — smaller deliberate one-time re-indexes (edit detection, threaded replies, and diverged conversation dates respectively).
• s3 — drops rtf from the default extension list and excludes keys containing ./.. path segments. Both are intentional scope filters, so they correctly do not set listingCapped and the affected documents reconcile away.

Operational notes for the first post-deploy sync

Two things will look alarming in logs and are not incidents.

notion's re-index will not finish in one task window. The corpus must re-hydrate against a documented ~3 req/s average while the engine hydrates 5 documents concurrently (SYNC_BATCH_SIZE = 5) against a 30-minute budget (maxDuration: 1800). Expect the run to be killed mid-sync with heavy 429 backoff first. It is not destructive and needs no intervention: batches commit as they go, a page that exhausts its retries rejects and is therefore excluded from deletion reconciliation, and a re-hydrated page classifies unchanged next run. It should converge over several scheduled syncs.

jira's deletion reconciliation will run for the first time on mid-sized projects. PAGE_SIZE 50 → 100 raises the effective listing ceiling from 25,000 to 50,000 (MAX_PAGES = 500). A project in the 25,001–50,000 band previously always hit the ceiling, which set listingTruncated — an unconditional, non-overridable block on reconciliation — so stale rows have accumulated since setup. Those projects now list completely and will tombstone that backlog on the first full sync. This is the correct outcome and it is two-phase behind the suspect-listing guard, but note the honest framing: raising the page size enables deletion for that band rather than fixing a deletion bug. A project larger than 50,000 truncates again and still fails safe.

Final validation sweep

A last read-only /validate-connector pass ran over all 59 changed connectors plus the shared files, verifying every quoted claim against provider specs. It found six real defects, all now fixed:

• ashby, azure-devops — getDocument returned null on an API-shape fault. Both connectors set contentDeferred, so these paths were live: a fulfilled null records no failure and no log, and the document silently vanishes.
• dropbox — 409 covers Dropbox's entire LookupError union, and restricted_content / locked both mean the file still exists. Only not_found is absence now.
• docusign — fetchFormValues swallowed every non-OK status, baking a permanently incomplete document (the hash is metadata-only, so the next sync sees it as unchanged forever).
• typeform — all sent response_type=started,partial,completed, but Typeform documents only partial and completed. An unknown enum member risks a 400 that fails the whole sync, and staging omitted the parameter entirely — so this shipped as a regression. Now sends the widest documented set, which is strictly wider than staging's effective behaviour.
• github — a utf-8 blob branch was added on a misattributed quote. That sentence describes the encoding request parameter of Create a blob; the GET response is documented as always base64. Branch removed, and two comments corrected that were hiding a real drop (>1 MB files under vnd.github+json 403 rather than returning encoding: none).

The sweep also confirmed the htmlToPlainText blast radius independently: exactly microsoft-teams and mintlify both decode HTML and hash the result, verified by intersecting the htmlToPlainText importers against the computeContentHash callers.

The corollary matters more than the re-index does: for the other ~22 connectors that call htmlToPlainText, the hash is metadata-keyed, so the entity-decode fix is forward-only — already-indexed documents keep their literal ’ text until the upstream record's own timestamp changes. The same is true of the Jira ADF extraction improvements, since both jira and jsm hash on metadata. Correcting the existing corpus would need a deliberate rehydrate, not this deploy.

hubspot additionally moved to a shared anchored HTML-detection helper. The loose pattern matched angle-bracketed prose (Reply from John <john@acme.com>), and htmlToPlainText deletes the span and collapses line structure — so a false positive loses data rather than passing it through. Worth fixing now rather than later precisely because the hubspot:v2: bump rewrites every affected document once.

The one item I'd want a decision on before merge

sentry — the endpoint migration narrows scope and will delete issues. Moving from the deprecated /projects/{org}/{project}/issues/ to /organizations/{org}/issues/?project= is correct on its own terms, and it fixes a genuine bug alongside it (staging sorted on Sentry's default date, a mutable key, so under cursor paging an issue whose lastSeen advanced mid-sync jumped the cursor, was skipped, and was hard-deleted as absent).

But the organization endpoint always resolves a date range, and with no statsPeriod/start/end it defaults to roughly 90 days. Issues last seen before that window are absent from the listing and reconcile away. That is a scope narrowing this PR introduces, not one it inherits — the in-code comment currently rationalizes it as the pre-existing "aged out of the query window" semantic, which understates it.

I did not set an explicit window because I could not verify Sentry's maximum accepted value, and a wrong one 400s the endpoint. There are no live Sentry sources, so nothing is at risk today. Options are: accept it as intended scope, pin an explicit widest-supported window, or set listingCapped when the window is the limiting factor. Worth an owner's call rather than my guess.

Known gaps

All 61 connectors have been through the final pass, verified against the four invariants — cap gating, getDocument absence semantics, contentHash parity, and pagination shape.

isScopeSatisfiedBy (lib/oauth/utils.ts) was the one shared-OAuth item flagged as needing scrutiny — it treats a granted non-readonly scope as satisfying its .readonly variant by string suffix, which would be unsound if any provider had a foo.readonly scope where foo is not a superset. Checked and clear. Exactly three .readonly requiredScopes exist fleet-wide, all Google, and all three appear verbatim in their provider scope lists, so exact match already satisfies them. The suffix rule fires for only one: ediscovery (bare) is grantable and is a genuine documented superset of ediscovery.readonly. The other two cannot trigger it — bare meetings.space and forms.responses are not real scopes and appear in no provider list. No false-positive path exists.

Issues found and deliberately not fixed here, each with a reason:

1. outlook routes a correctness-critical value between listDocuments and getDocument through an untyped syncContext key that connectors/types.ts documents as a cache. Correct today — the engine has exactly one getDocument caller — but the fallback fails silently and permanently if that ever changes.
2. "Full resync" is destructive on capped sources — it overrides listingCapped and skips the tombstone grace period, hard-deleting hidden documents in one pass. Correct semantics; deserves a UI warning.
3. gong maps a 404 to an empty page unconditionally, which can hard-delete a source over two consecutive syncs. The fix requires matching a documented error string that could not be verified — Gong's API reference is not publicly fetchable — and guessing at it would be worse than the known behaviour.
4. rootly swallows timeline-fetch failures into an empty array, baking a permanently timeline-less document (the hash is updated_at-keyed, so it never self-repairs). Pagerduty solved this class with an explicit failure flag; rootly is the remaining instance.
5. gitlab and notion flag listingCapped when a count merely reaches the cap without confirming more remained. Errs safe — over-suppression, never over-deletion — and pre-existing.
6. servicenow sorts by a mutable key under offset pagination, so a record updated mid-sync can slide past the window and read as deleted. Sentry's equivalent was fixed here by switching to an immutable sort key.
7. youtube builds contentHash from different fields on the stub and hydrated paths. Tracing the engine shows it settles to docsUnchanged rather than re-indexing, so the cost is a wasted API call per video per sync — quota burn, not churn.

Verification

29 audits (check:audits), 24/24 type-check, 1,332 tests, CodeQL clean, biome clean.