frozen: true step: AIP-a model: "Thinking (thinking-high)" date: 2026-08-24 branch: feat/automation-ingest-policy-a status: thinking-freeze-2026-08-24 supersedes: "Phase C + Lane D remain the machine credential and propose-path contract. This freeze adds per-account ingest rules so cron agent writes are not forced into Review as generic proposals. Does not authorize AIP-b Auto until freeze-review pass. Does not edit Scooling. Does not wire VideoFactory (AIP-c). Does not flip evaluation policy, billing enforce, or enable Born Free templates globally." evidence: "Knowtation PRIMARY 2026-08-24 after KN-AUTH-LANE-D propose-path land. Cron agents with kt_agent_ still land every write in Review. Born Free / VideoFactory trend scout is the first integration test, not a hard-coded Hub hack."
AUTOMATION-INGEST-POLICY — per-account ingest rules
Ground truth for AIP-b Auto. Downstream Auto may treat this document as ground truth without re-deriving. Phase C (docs/DURABLE-AGENT-AUTH-PHASE-C-FREEZE.md) remains the token-shape and mint/exchange contract. Lane D remains the health + isolation + one machine path contract. This freeze does not edit Scooling, change SEC-SEAM / personal self-apply / T5 fingerprints, grant vault:write on the cron JWT, let agent_access call POST api/v1/proposals/:id/approve, bypass elevated triggers or auto_flag_reasons, enable Born Free templates on every Hub, or implement VideoFactory (AIP-c).
phase: AIP-a
outputs:
- id: automation-ingest-policy
path: docs/AUTOMATION-INGEST-POLICY-FREEZE.md
frozen: true
notes: Per-sub ingest rules (match + disposition). Router first-match. POST api/v1/automation/ingest. Scope ingest:automation. direct_note is server-trusted write. proposal_auto_apply is a server hook, not approve. Born Free pack disabled. REST-only v1.
frozen_inputs:
- id: agent-credential-core
path: hub/lib/agent-credential-core.mjs
notes: ALLOWED_AGENT_SCOPES, DEFAULT_AGENT_SCOPES, applyScopeCeiling, agentScopesPermitMethod, PROPOSE_CREATE_PATHS, JWT cid + agent claims at exchange
- id: access-token-authz
path: hub/gateway/access-token-authz.mjs
notes: subFromVerifiedPayload, isAgentAccessPayload, isSessionBoundActor, roleFromMcpAccessScopes, mayApplyAdminAllowlistOverride
- id: gateway-get-user-id
path: hub/gateway/server.mjs
notes: getUserId uses effectiveRequestPath so propose allowlist sees api/v1/proposals not the Express suffix
- id: request-path
path: hub/gateway/request-path.mjs
notes: effectiveRequestPath is the only path string Auto may pass to agentScopesPermitMethod on gateway and self-hosted ingest/hook routes
- id: review-triggers
path: lib/hub-proposal-review-triggers.mjs
notes: applyReviewTriggers; elevated + auto_flag_reasons; packaged default + data/ override
- id: personal-self-apply
path: lib/hub-proposal-personal-self-apply.mjs
notes: roleEligibleForPersonalSelfApply refuses agent_access; T5 fingerprints unchanged
- id: proposal-create-augment
path: lib/hub-proposal-create-augment.mjs
notes: stripClientEvaluationFields + triggers + E1 after triggers
- id: proposal-lifecycle
path: docs/PROPOSAL-LIFECYCLE.md
notes: States, roles, personal self-apply, review_queue / review_severity / auto_flag_reasons
- id: phase-c-freeze
path: docs/DURABLE-AGENT-AUTH-PHASE-C-FREEZE.md
notes: propose is not approve; propose path allowlist; default scopes propose + vault:read; consumer trend-agent is follow-on
- id: audit-log
path: hub/audit-log.mjs
notes: Existing JSONL hub_audit.log; AIP extends actions, does not invent a second log family
- id: billing-middleware
path: hub/gateway/billing-middleware.mjs
notes: note_write vs proposal_write; ingest handler chooses op after route
- id: list-notes
path: lib/list-notes.mjs
notes: filterNotesByListOptions; content_class filter is additive
- id: hub-provenance
path: lib/hub-provenance.mjs
notes: mergeProvenanceFrontmatter kind agent already exists
- id: proposals-store
path: hub/proposals-store.mjs
notes: createProposal + updateProposalStatus for self-hosted auto-apply mark
review_stamp:
reviewed_at: '2026-08-24T14:05:38Z'
verdict: pass
reviewer_mode: agent
reviewer_model: thinking-high
reviewer_provider: local
kit_version: 0.1.0
artifact_digest: sha256:9fded978386543865225f5cc2bc0f04f09e777f8de96a56ef5d516c995a2793c
downstream:
- id: AIP-b
model: Auto
consumes_as_ground_truth: true
notes: Implement router, ingest route, scope, Settings CRUD, Research filter, audit, seven-tier tests. Starts only after freeze-review pass. Auto does not edit Scooling. Auto does not enable Born Free templates. Auto does not claim production smoke.
- id: AIP-c
model: Thinking → Auto
consumes_as_ground_truth: true
notes: VideoFactory trend-scout wire. Not this Hub Auto. Appendix A is the contract edge only.
tier3_gates:
- T1 Muse main or muse-mirror to GitHub main (SD-14) outside SD-21 land hygiene
- T2 Production ingest smoke on api.knowtation.store (Operator; record PASS or FINDINGS)
- T3 Enabling packaged Born Free templates as enabled-true in the default JSON (global opt-in)
- T4 Changing SEC-SEAM / personal self-apply / T5 fingerprints or admitting agent_access to E1
- T5 Granting vault:write on DEFAULT_AGENT_SCOPES or requiring vault:write for cron ingest
- T6 Teaching agent_access to call POST api/v1/proposals/:id/approve
- T7 Flipping BILLING_ENFORCE or inventing unscoped long-lived API keys
- T8 Editing Scooling or VideoFactory from this Knowtation tip
- T9 Feature branch to GitHub main / non-muse-mirror head
Auto must not build until freeze review pass. This Thinking tip does not implement routes. This Thinking tip does not flip any env.
Review record
| Round | Reviewer | Verdict | Resolution |
|---|---|---|---|
| 0 | Thinking (this session) | draft | Freeze authored from Phase C + Lane D + review triggers + self-apply + getUserId propose-path + billing + list-notes |
| 1 | Freeze-review loop (thinking) | findings | R1-F1–F7 fixed below. Mechanical dry-run was already pass. |
| 2 | Freeze-review loop (thinking) | findings | R2-F1–F3 fixed below. |
| 3 | Freeze-review loop (thinking) + ok review --freeze + ok check-ok --path |
pass | R1–R2 hold. Interfaces, fail-closed, seven-tier matrix, Tier-3 gates present. No open design decisions for Auto. No escalating category. Digest in review_stamp.artifact_digest. Cleared for AIP-b Auto. Auto must not edit Scooling, enable Born Free templates, or claim production smoke. |
Round 1 findings (cited — file+line)
| ID | Sev | Cat | Citation | Finding | Fix |
|---|---|---|---|---|---|
| R1-F1 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:465 (prior “Auto may call”) | Hosted auto-apply mark was unbound. Auto could invent Motoko or skip status flip. | D23 locks gateway fetch to canister approve with gateway auth only; D23 failure code. |
| R1-F2 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:443 (prior “override or helper”) | Billing hook was an “or”. Auto could double-charge or skip storage cap. | D24 locks opts.operation on runBillingGate. |
| R1-F3 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:348 (prior store-or-frontmatter) | content_class persist unbound. Auto could edit proposals-store ad hoc. |
D25 frontmatter only. |
| R1-F4 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:450 (prior “GET existing note”) | D22 I/O unbound. | D22 names readNote and canister GET. |
| R1-F5 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:434 (prior execute “may”) | Policy vs I/O split unbound; D26 later conflicted. | D26 pure exports; execute in route wrappers. |
| R1-F6 | MINOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:574 (prior PATCH sugar) | Auto could add PATCH. | PATCH forbidden in v1. |
| R1-F7 | MINOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:640 (prior skip audit) | Hosted audit could no-op. | Same dataDir as augment; always append. |
Round 2 findings (cited — file+line)
| ID | Sev | Cat | Citation | Finding | Fix |
|---|---|---|---|---|---|
| R2-F1 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:226 (prior D26 store-only) | Idempotency persist names unbound. Auto could invent a third file. | D26 names getIngestIdempotency / putIngestIdempotency on the store module. |
| R2-F2 | MAJOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:748 (prior “if a new blob”) | Netlify blob globals unbound. Auto could reuse billing/agent blobs. | D27 two dedicated blobs + globals. |
| R2-F3 | MINOR | completeness | docs/AUTOMATION-INGEST-POLICY-FREEZE.md:729-736 (prior “if needed”) | File list still had optional edits. | access-token-authz and audit-log are do-not-edit; D8 hook is required. |
Citation discipline
Every freeze-review finding MUST cite file+line (OVERSEER-KIT-SPEC §6). Do not
trust uncited review output. HTTP routes in this doc omit the leading slash
(api/v1/…) so the freeze mechanical gate does not treat them as absolute machine
paths. Cross-repo paths use ~/scooling/…. Never leading-slash absolute paths.
1. Plain-language summary
Robots with kt_agent_ credentials can already create proposals. Every one of those writes still lands in Review as a generic proposal. Operators need a Settings page of ingest rules: for matching robot writes, choose write the vault note now, create a proposal and apply it on the server, or leave it in Review (today’s default). Elevated legal/security triggers still win. Robots never get a blanket vault-write password. Born Free is a disabled template pack the operator turns on for one account — not three rules baked into every Hub.
Technical summary
AIP adds ingest:automation to the Phase C scope vocabulary; a first-match per-sub rule router; POST api/v1/automation/ingest as the preferred cron entrypoint; a required agent-only hook on POST api/v1/proposals when the body matches the ingest contract (D8); server-trusted direct_note (canister or writeNote, billing note_write); proposal_auto_apply as create-then-server-apply (never the approve HTTP route; never E1); review_queue via existing augment + triggers; content_class list filter; JSONL audit actions on hub_audit.log. MCP is out of v1. REST only.
2. Ground truth — what the code does today (file+line)
Every row was read in this session.
| # | Fact | Citation |
|---|---|---|
| G1 | Allowed agent scopes are vault:read, propose, vault:write only |
hub/lib/agent-credential-core.mjs:27 |
| G2 | Default mint scopes are propose + vault:read |
hub/lib/agent-credential-core.mjs:29 |
| G3 | applyScopeCeiling always keeps propose; vault:write needs role write |
hub/lib/agent-credential-core.mjs:110-116 |
| G4 | propose without write allows only three POST paths |
hub/lib/agent-credential-core.mjs:411-433 |
| G5 | vault:write (or admin) short-circuits agentScopesPermitMethod to allow all |
hub/lib/agent-credential-core.mjs:427-429 |
| G6 | Exchange JWT already carries cid and agent (credential name) |
hub/gateway/agent-credential-routes.mjs:226-236 |
| G7 | subFromVerifiedPayload enforces aud, typ, and agentScopesPermitMethod |
hub/gateway/access-token-authz.mjs:173-178 |
| G8 | isSessionBoundActor is true only for type: session |
hub/gateway/access-token-authz.mjs:69-70 |
| G9 | Gateway getUserId passes effectiveRequestPath (Lane D propose-path fix) |
hub/gateway/server.mjs:1959-1967 |
| G10 | Gateway catch-all app.use('api/v1') bills then proxyToCanister |
hub/gateway/server.mjs:4466-4469 |
| G11 | Self-hosted jwtAuth verifies JWT only — no agentScopesPermitMethod |
hub/server.mjs:395-406 |
| G12 | Self-hosted POST api/v1/proposals inlines triggers; does not call augmentProposalCreateRequestBody |
hub/server.mjs:3194-3231 |
| G13 | Hosted create runs augmentProposalCreateForHosted → augment + E1 |
hub/gateway/server.mjs:3790-3808 |
| G14 | applyReviewTriggers sets forcePending, queue, severity, auto_flag_reasons |
lib/hub-proposal-review-triggers.mjs:119-165 |
| G15 | E1 runs after triggers; elevated or auto-flag stay pending | lib/hub-proposal-create-augment.mjs:78-84 |
| G16 | roleEligibleForPersonalSelfApply refuses tokenType === 'agent_access' |
lib/hub-proposal-personal-self-apply.mjs:437-440 |
| G17 | Hosted approve path refuses agents via humanActor: false + tokenType: agent_access |
hub/gateway/server.mjs:3575-3616 |
| G18 | Phase C denies approve/discard/notes write even with propose |
docs/DURABLE-AGENT-AUTH-PHASE-C-FREEZE.md:330 |
| G19 | Billing maps POST api/v1/notes and PUT notes to note_write; proposals to proposal_write |
hub/gateway/billing-middleware.mjs:26-40 |
| G20 | filterNotesByListOptions has no content_class key |
lib/list-notes.mjs:57-119 |
| G21 | Vault filter filter-content-scope is notes vs approval logs only |
web/hub/index.html:180-184 |
| G22 | Audit append is JSONL on hub_audit.log |
hub/audit-log.mjs:18-30 |
| G23 | Provenance kind already includes agent |
lib/hub-provenance.mjs:37 |
| G24 | Self-hosted approve writes via writeNote; store can updateProposalStatus |
hub/server.mjs:3378; hub/proposals-store.mjs:353-368 |
| G25 | Canister note IO uses x-user-id, x-actor-id, x-vault-id, plus canisterAuthHeaders() |
hub/gateway/server.mjs:3663-3670 |
| G26 | Settings tabs today: backup, team, vaults, integrations, appearance, billing, consolidation — no Automation tab | web/hub/index.html:705-713 |
| G27 | Mint UI checkboxes: propose, vault:read, vault:write | web/hub/index.html:1448-1450 |
| G28 | Phase C consumer contract still posts api/v1/proposals (not ingest) |
docs/DURABLE-AGENT-AUTH-PHASE-C-FREEZE.md:381 |
| G29 | PROPOSAL-LIFECYCLE says agents create proposed rows via POST api/v1/proposals |
docs/PROPOSAL-LIFECYCLE.md:18 |
Consumer evidence (operator-reported, not in this workspace): Born Free / VideoFactory trend scout posts Hub proposals after kt_agent_ exchange. Lane D unblocked POST api/v1/proposals. Those rows still queue as generic Review items. Trend-agent source is not a Knowtation tree. AIP-b does not edit VideoFactory. Appendix A freezes the Hub-side contract edge only.
3. Frozen product goal
An operator signed into Hub can:
- Open Settings → Automation → Ingest rules.
- See packaged templates (Born Free pack) as disabled. Enable or copy a template for this account only.
- Order rules by priority (lower number wins). First matching enabled rule decides the write.
- Choose per rule: direct vault note, server auto-apply proposal, or Review queue.
- Mint a cron credential with
ingest:automation+vault:read(novault:write). - Point the robot at
POST api/v1/automation/ingest(preferred). ExistingPOST api/v1/proposalsstill works for humans; agents with an ingest-shaped body may hit the same router. - Filter Vault by Research (
content_class=research).
Success metric: a matching Born Free trend write can land as a research note or an auto-applied proposal only after the operator enables a rule; unmatched or flagged writes stay in Review; agent_access still cannot approve.
4. Decisions (lock — do not reopen in Auto)
| ID | Decision | Recorded default |
|---|---|---|
| D1 | Router | First-match. Sort enabled rules by priority ascending, then rule_id lexicographic. First AND-match wins. No specificity score. |
| D2 | Default disposition | No enabled match → review_queue (today’s behavior). |
| D3 | Match empty | Forbidden. At least one predicate required. Blank match is 400 INGEST_RULE_MATCH_EMPTY. |
| D4 | Match combinator | All specified predicates AND. Unspecified keys are wildcards. |
| D5 | Scope | New ingest:automation. Additive like propose in applyScopeCeiling. Not in DEFAULT_AGENT_SCOPES. |
| D6 | Cron mint | Recommended ingest:automation + vault:read. Checkbox default off. Never require vault:write. |
| D7 | Ingest path allowlist | POST + normalized path api/v1/automation/ingest requires ingest:automation (or vault:write, existing short-circuit). propose alone does not open the ingest route. |
| D8 | Legacy proposals hook | On for agent_access only when the body matches the ingest contract (D14). Session / human callers never enter the router. |
| D9 | Elevated / flags | After match, before execute, run applyReviewTriggers. If forcePending or review_severity === 'elevated' or auto_flag_reasons.length > 0 → force review_queue and audit ingest_elevated_override. All three dispositions. |
| D10 | Evaluation vs auto-apply | If evaluation policy would leave evaluation_status pending (E1 does not apply to agents — G15, G16) → cannot proposal_auto_apply; fall back to review_queue. Do not invent ingest-E1. Do not stamp evaluation_status: passed for robots. |
| D11 | Approve HTTP | agent_access never calls POST api/v1/proposals/:id/approve. Auto-apply uses writeNote / canister note POST + updateProposalStatus / hosted status flip without forwarding the agent Bearer to approve. |
| D12 | direct_note vs JWT | Server-trusted write only. JWT does not need vault:write. |
| D13 | MCP | REST-only v1. No MCP tool, prompt, or resource. |
| D14 | Ingest contract body | See §6.2. Required source_fingerprint. Marker is ingest === true or a known content_class. |
| D15 | Idempotency precedence | If header X-Ingest-Idempotency-Key is non-empty after trim → that is the store key. Else key = source_fingerprint. See §8. |
| D16 | Born Free pack | Packaged templates, all enabled: false. Never copy into a new sub as enabled. Operator enable is per-sub CRUD. |
| D17 | Settings chrome | New Settings tab Automation (data-settings-tab="automation"). Ingest rules live there. Pointer line under Integrations → Agent credentials. |
| D18 | Rule cap | Max 32 enabled+disabled user rules per sub (templates in the pack do not count until copied). |
| D19 | Self-hosted agent auth | Ingest route and the proposals hook must call agentScopesPermitMethod with effectiveRequestPath(req) from hub/gateway/request-path.mjs (G11 is a gap). Do not rewrite all of jwtAuth in this Auto. |
| D20 | Hosted mount | Dedicated app.post for ingest before the catch-all at hub/gateway/server.mjs:4466. |
| D21 | Scooling | No edits. No T5 fingerprint changes. |
| D22 | Path clobber | direct_note / auto-apply write to an existing path is allowed only on idempotent replay (same fingerprint) or if existing frontmatter source_fingerprint equals the request. Else 409 INGEST_PATH_CONFLICT. Self-hosted existence check is readNote in lib/vault.mjs. Hosted existence check is GET canister api/v1/notes/<path> with the same headers as G25. Missing note is not a conflict. |
| D23 | Hosted auto-apply mark | Gateway fetches ${CANISTER_URL} + api/v1/proposals/<id>/approve with POST, canisterAuthHeaders(), x-user-id, x-actor-id, x-vault-id, JSON {}. No inbound agent Authorization header is copied. Incoming agent HTTP to that path still hits assertHostedProposalApproveDiscard (G17). If this mark fails after the note write, respond 500 INGEST_APPLY_FAILED with proposal_id + path; do not delete the note; audit ingest_apply_failed. |
| D24 | Billing hook | Extend runBillingGate with opts.operation. When set, that string replaces operationFromRequest. Ingest calls it after route: proposal_write for §9.1; note_write for §9.2 and §9.3. When opts.operation === 'note_write', also run the existing storage-cap branch (getNoteCount). Do not add a path regex for ingest that always maps to note_write. |
| D25 | content_class persist |
Stamp on frontmatter only (note and proposal). Do not add a proposals-store column. List filter reads n.content_class ?? n.frontmatter?.content_class. Response content_class is computed. |
| D26 | Module exports | lib/automation-ingest-policy.mjs (pure) exports routeAutomationIngest, isIngestContractBody, normalizeIngestBody, listPackTemplates. hub/gateway/automation-ingest-store.mjs exports loadIngestRulesForSub, saveIngestRulesForSub, getIngestIdempotency, putIngestIdempotency (file + blob). Execute I/O stays in hub/gateway/server.mjs and hub/server.mjs wrappers — not in the pure module. |
| D27 | Netlify blobs | netlify/functions/gateway.mjs provisions two new dedicated blobs: gateway-automation-ingest-rules → globalThis.__knowtation_gateway_ingest_rules_blob and gateway-automation-ingest-idempotency → globalThis.__knowtation_gateway_ingest_idempotency_blob. Never reuse gateway-agent-credentials or gateway-billing. |
5. Rule schema (frozen)
5.1 Rule object
{
"rule_id": "ingr_<16 hex>",
"enabled": false,
"priority": 100,
"pack_id": null,
"label": "Trend scout → Review",
"match": {
"credential_id": null,
"credential_name": null,
"credential_name_prefix": null,
"path_prefix": "inbox/trends/",
"intent": null,
"content_class": "research"
},
"disposition": "review_queue",
"content_class": "research"
}
| Field | Type | Rules |
|---|---|---|
rule_id |
string | Server-minted ingr_ + 16 lowercase hex. Client cannot supply on create. |
enabled |
boolean | Disabled rules are skipped by the router. |
priority |
integer | 0–10000. Lower wins. Default 100. |
pack_id |
string or null | born_free_v1 when copied from the pack; else null. |
label |
string | 1–128 chars. Required. |
match |
object | At least one non-null predicate (D3). |
disposition |
enum | Exact: direct_note | proposal_auto_apply | review_queue. |
content_class |
enum or null | Stamp when request omits class. Closed set §7. |
5.2 Match predicates
| Key | Match | Source |
|---|---|---|
credential_id |
exact, case-sensitive | JWT claim cid (G6). If claim missing → predicate fails (no store lookup required in v1). |
credential_name |
exact, case-sensitive | JWT claim agent (G6). Missing claim → fail. |
credential_name_prefix |
agent starts with prefix (case-sensitive) |
Same claim. |
path_prefix |
notePathMatchesPrefix (lib/write.mjs:49-52) |
Request path. Normalize slashes; no leading slash. |
intent |
exact | Request intent. |
content_class |
exact | Request content_class after defaulting (§7). |
Null / omitted / empty-string predicates are skipped (wildcard). After skip, zero remaining predicates → invalid rule (D3).
5.3 Disposition enum
| Value | Meaning |
|---|---|
direct_note |
Server writes the note. No Review row unless D9/D22 forces queue. |
proposal_auto_apply |
Create proposal (augment + triggers) then server-apply if D9 and D10 allow. |
review_queue |
Create proposal only. Today’s path. |
Unknown disposition on save → 400 INGEST_DISPOSITION_UNKNOWN.
6. Router (frozen)
Export routeAutomationIngest(input, rules) from lib/automation-ingest-policy.mjs. Pure. No I/O.
6.1 Algorithm
- Take
rulesforinput.subonly (caller loads storage). - Drop
enabled !== true. - Sort by
priorityascending, thenrule_idascending (UTF-8). - For each rule, evaluate §5.2 AND. First true → candidate.
- If none → candidate
{ rule_id: null, disposition: 'review_queue', content_class: input.content_class }. - Run
applyReviewTriggerson{ path, body, intent, labels }(G14). If D9 trips → rewrite disposition toreview_queue, setelevated_override: true, keeprule_idof the candidate (or null). - If candidate disposition is
proposal_auto_applyand D10 trips (evaluationRequired === trueand E1 will not stamp — agents never session-bound) → rewrite toreview_queue, setevaluation_block: true. - Return
{ rule_id, disposition, content_class, elevated_override, evaluation_block, trigger_result }.
content_class on the result: request class if known; else rule content_class; else general.
6.2 Ingest request body (preferred route)
POST api/v1/automation/ingest
Auth: agent_access with ingest:automation (D7) or session editor/admin (operator test from Settings). Vault: X-Vault-Id must pass assertAgentVaultAllowed for agents (Phase C §7.4).
{
"path": "inbox/trends/example.md",
"body": "markdown",
"frontmatter": {},
"intent": "videofactory.trend_scout.ingest",
"labels": [],
"source": "automation_ingest",
"source_fingerprint": "sha256:…-or-opaque-8-to-128",
"content_class": "research",
"ingest": true
}
| Field | Required | Rules |
|---|---|---|
path |
yes | Vault-relative. No ... No leading slash. Max 512. Must end .md. |
body |
yes | String. Max 512 KiB. |
frontmatter |
no | Object. Reserved provenance keys stripped (lib/hub-provenance.mjs). |
intent |
no | String max 256. |
labels |
no | String array, max 32, each max 64. |
source |
no | Default automation_ingest. Max 64. |
source_fingerprint |
yes | Trimmed 8–128 chars. Charset [A-Za-z0-9._:/-]. |
content_class |
no | §7. Unknown → 400 INGEST_CONTENT_CLASS_UNKNOWN. |
ingest |
no | Boolean. Ignored on the ingest route (always ingest). Required as a marker on the legacy hook unless content_class is known (D14). |
Headers
| Header | Required | Rules |
|---|---|---|
Authorization |
yes | Bearer session or agent_access. |
X-Vault-Id |
no | Default default; must be in vault_ids for agents. |
X-Ingest-Idempotency-Key |
no | Trimmed 8–128. Same charset as fingerprint. Precedence D15. |
6.3 Success response
201 on first apply/create. 200 on idempotent replay (replayed: true).
{
"disposition": "review_queue",
"rule_id": "ingr_ab12cd34ef567890",
"outcome": "proposal",
"path": "inbox/trends/example.md",
"content_class": "research",
"proposal_id": "prop-…",
"note": null,
"replayed": false,
"elevated_override": false,
"evaluation_block": false
}
outcome |
When |
|---|---|
note |
direct_note write succeeded |
proposal |
Review row created (queue or auto-apply-created) |
note_and_proposal |
proposal_auto_apply wrote the note and marked approved |
proposal_id required when a proposal exists. note is { "path": "…" } when a note was written.
6.4 Error codes (closed)
| HTTP | code |
|---|---|
| 400 | INGEST_PATH_INVALID, INGEST_BODY_REQUIRED, INGEST_FINGERPRINT_REQUIRED, INGEST_FINGERPRINT_INVALID, INGEST_CONTENT_CLASS_UNKNOWN, INGEST_CONTRACT_REQUIRED (legacy hook body mismatch — should not appear on the ingest route), INGEST_RULE_MATCH_EMPTY, INGEST_DISPOSITION_UNKNOWN |
| 401 | UNAUTHORIZED |
| 403 | AGENT_VAULT_FORBIDDEN, FORBIDDEN |
| 409 | INGEST_IDEMPOTENCY_CONFLICT, INGEST_PATH_CONFLICT |
| 402 | billing codes from runBillingGate unchanged |
| 429 | existing agent rate limit if any |
| 500 | INGEST_APPLY_FAILED (D23 only) |
| 503 | AGENT_CREDENTIAL_STORE_UNAVAILABLE / store I/O on rules or idempotency |
Do not invent extra codes in Auto.
6.5 Legacy POST api/v1/proposals hook (D8)
Runs only when all hold:
- Verified payload
type === 'agent_access'(G8 inverse). - Method
POSTand normalized path is exactlyapi/v1/proposals(not…/approve, not task/flow facades). - Body matches ingest contract (D14): non-empty valid
source_fingerprintand (ingest === trueor knowncontent_class).
Then the handler calls the same router + execute as the ingest route (scope already allowed via propose on this path — G4). Response shape is the ingest success/error envelope (not the raw createProposal object) so cron clients can share a parser. HTTP status 201/200 as §6.3.
If (1) or (3) fail → existing create path unchanged (G12 / G13). Session callers with source_fingerprint set still use existing create. That is intentional (do not hijack Hub UI / CLI humans).
Task / media / flow / path facade POSTs are not ingest. Do not hook them.
7. content_class + Research filter
Closed enum (lowercase):
| Value | Meaning |
|---|---|
research |
Research / scout notes. Vault filter Research. |
ops |
Operational automation. |
general |
Default when omitted. |
Stamp:
- Note frontmatter key
content_class(string). Server sets; client cannot override reserved provenance but can sendcontent_class(not in RESERVED — G23). Server overwrites with the routed class. - Proposal
frontmatter.content_classonly (D25). Do not invent a store column. Do not add a fakeresearchlabel unless the client sent that label.
List filter: add content_class to filterNotesByListOptions (G20). Exact match on n.content_class ?? n.frontmatter?.content_class. Query GET api/v1/notes?content_class=research.
Hub UI: new <select id="filter-content-class"> immediately after #filter-content-scope (G21). Options: All classes, Research, Ops, General. Client-side mirror in hub.js (hosted list already filters client-side — web/hub/hub.js near the filter-content-scope branch).
Do not overload content_scope (notes vs approval logs).
8. Idempotency (frozen)
Store file: data/automation_ingest_idempotency.json (self-hosted / gateway data dir). Hosted blob name gateway-automation-ingest-idempotency, persist key automation-ingest-idempotency-v1. Shape:
{
"version": 1,
"entries": {
"<sub>\\t<vault_id>\\t<key>": {
"source_fingerprint": "…",
"path": "…",
"result": { },
"created_at": 0,
"expires_at": 0
}
}
}
| Rule | Value |
|---|---|
| Key | D15 |
| TTL | 30 days from first write. Expired entries ignored (treat as miss). |
| Replay | Same key and same source_fingerprint and same path → return stored result with replayed: true, no second billing deduct, no second write. |
| Conflict | Same key, different fingerprint or path → 409 INGEST_IDEMPOTENCY_CONFLICT. |
| In-flight | v0 last-write-wins on the JSON file. No 409 INGEST_IN_PROGRESS. |
result is the §6.3 JSON object (without forcing a new proposal_id).
9. Execute paths (frozen)
I/O stays in gateway / hub/server.mjs wrappers that call routeAutomationIngest then §9.1–§9.3. The policy module stays pure (D26). Do not duplicate trigger logic.
9.1 review_queue
- Build create payload: path, body, frontmatter (stamp
content_class,source_fingerprint,source), intent, labels,proposed_by= actor sub. - Hosted: existing
augmentProposalCreateForHosted(G13). Self-hosted: must callaugmentProposalCreateRequestBody(close G12 for this path only — do not leave ingest on the inline trigger copy). - Persist proposal (
createProposal/ canister POST proposals). - Do not write the note.
- Audit
ingest_review_queued. - Billing: D24 —
runBillingGate(..., { operation: 'proposal_write' }).
E1 still runs inside augment and still fails for agent_access (G15, G16). That is correct.
9.2 direct_note
- D9 already applied. If override → §9.1 instead.
- D22 path-conflict check (
readNoteself-hosted; canisterGET api/v1/notes/<path>hosted). Compare existing frontmattersource_fingerprint. - Merge frontmatter via
mergeProvenanceFrontmatter(..., { sub, kind: 'agent' })(G23). Stampcontent_class,source_fingerprint,source. - Self-hosted:
writeNote(vaultPath, path, { body, frontmatter })— same function asPOST api/v1/notes(hub/server.mjs:2574). Do not go throughrequireRole('editor')on the agent JWT; the ingest handler is the trusted caller. - Hosted:
POST${CANISTER_URL}/api/v1/noteswith headersAccept,Content-Type,x-user-id= owner effective id,x-actor-id= uid,x-vault-id,...canisterAuthHeaders()(G25). Body{ path, body, frontmatter }. Do not forward the agent Bearer to the canister as a write grant. - Billing: D24 —
runBillingGate(..., { operation: 'note_write', getNoteCount: getNoteCountForUser }). - Audit
ingest_direct_note. - No proposal row.
9.3 proposal_auto_apply
- D9 / D10 already applied. If either trips → §9.1.
- Create proposal as §9.1 (so Review/Activity has an audit row) with
source: automation_ingest. - Write the note as §9.2 steps 2–6 (same trusted write).
- Mark proposal
approved(D23):- Self-hosted:
updateProposalStatus(dataDir, id, 'approved')(G24). Do not run ExpressPOST api/v1/proposals/:id/approveorrequireApproveRole. - Hosted: gateway
fetchto canister approve path per D23 (gateway auth only). Incoming agent HTTP still hits G17. - Mark failure after a successful note write →
500INGEST_APPLY_FAILED(D23). Do not delete the note.
- Self-hosted:
- Do not call
applyPersonalSelfApplyEvaluationE1with fake sessionBound. Do not addagent_accesstoroleEligibleForPersonalSelfApply. - Audit
ingest_auto_appliedon full success;ingest_apply_failedon D23 failure. - Billing: D24
note_writeonce. Do not also chargeproposal_writeon the same ingest.
9.4 What must not happen
agent_accessHTTPPOST api/v1/proposals/:id/approve→ still 401/403 (G4, G17, G18).direct_notegrantingvault:writeon mint or JWT.- Skipping
applyReviewTriggers. - Using E1 / T5 fingerprints for ingest.
- Enabling pack templates in the packaged JSON.
10. Scope + mint (frozen)
10.1 Vocabulary
Add ingest:automation to ALLOWED_AGENT_SCOPES (G1). Keep FORBIDDEN_AGENT_SCOPES unchanged.
DEFAULT_AGENT_SCOPES stays propose + vault:read (G2). Do not add ingest to the default.
applyScopeCeiling: treat ingest:automation like propose — always keep it if requested (G3). Members can mint it.
10.2 agentScopesPermitMethod
Keep G5 short-circuit (vault:write / admin → allow).
Add before the propose-path check:
- If
POSTandnormalizeAgentRequestPath(path) === 'api/v1/automation/ingest'→ allow iff list includesingest:automation.
Then existing: safe → vault:read; mutating → propose + PROPOSE_CREATE_PATHS (G4).
ingest:automation does not add ingest to PROPOSE_CREATE_PATHS. It does not allow notes POST, approve, discard, enrich, or evaluation write.
CRUD routes (api/v1/automation/ingest-rules) are session only. agent_access on those paths → agentScopesPermitMethod false → getUserId null → 401.
10.3 Mint UI
Add checkbox id="agent-cred-scope-ingest" label ingest:automation, unchecked. Warning line (show when checked): ingest lets this robot use the ingest router per Settings rules; it does not grant vault:write.
hub.js mint body includes the scope when checked (mirror of G27).
10.4 Interaction with propose + vault:read
| Mint set | Ingest route | Legacy proposals hook | Reads | Direct notes POST |
|---|---|---|---|---|
ingest:automation + vault:read |
yes | no (no propose) |
yes | no |
propose + vault:read |
no | yes if D14 body | yes | no |
propose + ingest:automation + vault:read |
yes | yes if D14 body | yes | no |
vault:write |
yes (G5) | n/a | yes | yes (existing; not the AIP cron path) |
Cron default documentation: row 1. Do not require row 4.
11. Storage + CRUD (frozen)
11.1 Files
| Kind | Path |
|---|---|
| Packaged templates | hub/automation-ingest-rules-default.json |
| Self-hosted / gateway file overlay | data/automation_ingest_rules.json (under KNOWTATION_GATEWAY_DATA_DIR on hosted file fallback) |
| Hosted blob | name gateway-automation-ingest-rules, key automation-ingest-rules-v1 |
Overlay shape:
{
"version": 1,
"subs": {
"<sub>": {
"rules": [],
"updated_at": 0
}
}
}
Packaged file shape:
{
"version": 1,
"templates": [ ]
}
Load: templates from packaged JSON; user rules from overlay subs[sub].rules. Router sees user rules only. Templates appear in GET as templates and are never enabled in the packaged file (D16).
Netlify missing blob global: same fail-closed spirit as Lane D agent store — do not silently mint an empty overlay that wipes a missed blob. If overlay I/O throws → 503. If overlay missing → treat as empty user rules (templates still list). Do not write the overlay on GET.
11.2 Session CRUD
All require session JWT (type: session or legacy session). editor or admin. agent_access → 401.
| Method | Path | Behavior |
|---|---|---|
| GET | api/v1/automation/ingest-rules |
{ rules, templates } for caller sub |
| PUT | api/v1/automation/ingest-rules |
Replace rules (full list). Validate §5. Cap D18. Re-mint missing rule_ids. |
| POST | api/v1/automation/ingest-rules |
Append one rule. Server mints rule_id. |
| POST | api/v1/automation/ingest-rules/from-template |
Body { template_id }. Copy that pack row into rules with new rule_id, enabled: false unless body { "enable": true } and the operator is session (explicit opt-in). |
| DELETE | api/v1/automation/ingest-rules/:rule_id |
Remove from user list. |
template_id on pack rows is the packaged rule_id (stable, e.g. tmpl_born_free_trend_review).
No PATCH in v1. PUT replace is the update path.
11.3 Born Free pack (disabled)
Pack id born_free_v1. All enabled: false. Match on credential name + path (not global).
rule_id (template) |
label | match | disposition | content_class |
|---|---|---|---|---|
tmpl_born_free_trend_review |
Born Free trend → Review | credential_name = videofactory-trend-agent, path_prefix = inbox/trends/ |
review_queue |
research |
tmpl_born_free_trend_direct |
Born Free trend → vault note | credential_name = videofactory-trend-agent, path_prefix = inbox/trends/accepted/ |
direct_note |
research |
tmpl_born_free_trend_auto |
Born Free trend → auto-apply | credential_name = videofactory-trend-agent, path_prefix = inbox/trends/drafts/ |
proposal_auto_apply |
research |
These are examples the operator enables. They are not live on a new Hub. Do not add a second hidden match on born-free prefix in v1 (keeps first-match predictable). Operators who mint a different name copy the template and edit credential_name.
12. Settings UI (frozen)
New tab button: data-settings-tab="automation" label Automation, visible to every signed-in user (not hidden behind admin-only). Panel id="settings-panel-automation".
Section Ingest rules:
- Intro: matching robot writes can go to the vault, auto-apply, or Review. Elevated triggers still force Review. Born Free templates stay off until you add them.
- Table of
rules: label, match summary, disposition, enabled, priority, Enable/Disable, Delete. - Add rule form: label, priority, disposition select, match fields, content_class, Save (POST).
- Pack list: three templates + Add to my rules (from-template, stays disabled unless the operator checks Enable on add).
- Reorder: number input for priority (no drag required in v1).
Pointer under Agent credentials (after G27 warning): “Ingest routing lives under Settings → Automation.”
Unit tests assert element ids from Hub HTML (same pattern as Lane D banner ids). e2e is HTTP, not a browser driver.
13. Audit JSONL (frozen)
Reuse appendAudit (G22). New action values (closed):
action |
When |
|---|---|
ingest_routed |
Router returned (every ingest, including replay) |
ingest_direct_note |
Note written |
ingest_auto_applied |
Note written + proposal approved via hook |
ingest_review_queued |
Proposal created, not applied |
ingest_elevated_override |
D9 rewrote disposition |
ingest_idempotent_replay |
Store hit |
ingest_apply_failed |
D23 mark failed after note write |
proposalId on the existing field when a proposal exists; else proposalId: ''.
detail object (no secrets, no raw kt_agent_, no JWT):
{
"rule_id": "ingr_…-or-null",
"disposition": "review_queue",
"source_fingerprint": "…",
"path": "inbox/trends/example.md",
"content_class": "research",
"vault_id": "Business",
"credential_id": "cid-from-jwt-or-null",
"elevated_override": false,
"evaluation_block": false,
"replayed": false
}
Hosted and self-hosted both append using the same dataDir already passed to augmentProposalCreateForHosted / createProposal (G13, G12). Do not skip. Do not invent a second log file.
14. MCP (frozen)
Out of v1. docs/AGENT-INTEGRATION.md must say ingest is REST-only. Do not add an MCP tool that wraps ingest. Phase C §7.6 (MCP optional) is unchanged.
15. Docs Auto must amend (same PR as code)
| Doc | Change |
|---|---|
docs/openapi.yaml |
POST api/v1/automation/ingest, CRUD ingest-rules, content_class query on notes list |
docs/AGENT-INTEGRATION.md |
Machine path: exchange → ingest route; cron scopes; REST-only; no JWT-as-env |
docs/PROPOSAL-LIFECYCLE.md |
Cross-link: agent ingest may skip Review only via AIP rules; approve HTTP still denied; elevated triggers still apply |
docs/HUB-API.md |
Ingest section + honesty that templates ship disabled |
No docs-only PR to main.
16. Test matrix (seven tiers)
| Tier | File (Auto creates) | Must prove |
|---|---|---|
| unit | test/automation-ingest-unit.test.mjs |
normalize/match/first-match/priority; D3 empty match; D9 override; D10 evaluation block; D15 key precedence; scope allowlist ingest path; propose cannot ingest route; ingest:automation cannot approve path |
| integration | test/automation-ingest-integration.test.mjs |
load pack disabled; PUT rules; route + execute review_queue create; from-template stays disabled unless enable |
| e2e | test/automation-ingest-e2e.test.mjs |
HTTP ingest on test gateway/self-hosted helper: session CRUD + agent ingest → 201 envelope |
| stress | test/automation-ingest-stress.test.mjs |
32 rules first-match under cap; 33rd PUT fails; 100 sequential ingests no unbounded growth beyond TTL map |
| data-integrity | test/automation-ingest-data-integrity.test.mjs |
replay same key+fingerprint+path; conflict on fingerprint change; pack JSON all enabled: false; no secret in audit detail |
| performance | test/automation-ingest-performance.test.mjs |
routeAutomationIngest p95 budget on 32 rules (document threshold in test, local only) |
| security | test/automation-ingest-security.test.mjs |
agent cannot approve; agent cannot CRUD rules; session proposals with fingerprint do not auto-route; elevated body cannot direct_note; vault:write not required; roleEligibleForPersonalSelfApply still refuses agent_access; Scooling fingerprint constants unchanged (source scan) |
Security tier must fail against pre-AIP code for: ingest route 401/404; ingest:automation unknown scope; missing content_class filter.
Smoke: scripts/verify-automation-ingest-smoke.mjs — default target is local/test gateway. Production URL is Operator T2. Script must refuse to print credentials.
17. Fail-closed rules (checklist)
- No
ingest:automationonDEFAULT_AGENT_SCOPES. - No pack template
enabled: trueinhub/automation-ingest-rules-default.json. - No
agent_accesson approve / discard / ingest-rules CRUD. - No E1 / T5 / SEC-SEAM predicate edits.
- No trigger skip on any disposition.
- No
evaluation_status: passedstamp for ingest actors. - No Scooling edits. No VideoFactory edits.
- No MCP ingest tool.
- No second audit file format (extend
appendAudit). - Ingest route mounted before gateway catch-all.
- Self-hosted ingest + proposals hook call
agentScopesPermitMethod. - Idempotent replay does not bill twice.
- D23 mark failure does not delete the note and does not return 201.
18. AIP-b file list (mechanical)
Auto creates or edits only these (plus tests/docs in §15–16):
lib/automation-ingest-policy.mjs(schema, route, execute orchestration)hub/automation-ingest-rules-default.jsonhub/lib/agent-credential-core.mjs(scope + allowlist)hub/gateway/access-token-authz.mjs— do not edit (re-exportsagentScopesPermitMethodalready)hub/gateway/server.mjs(ingest + CRUD + mount order + D8 proposals hook — required)hub/gateway/billing-middleware.mjs(opts.operationper D24 only)hub/server.mjs(parity routes + hook +agentScopesPermitMethod+effectiveRequestPathon those routes)hub/gateway/automation-ingest-store.mjs(D26)lib/list-notes.mjs+ hosted notes list client filter inweb/hub/hub.jsweb/hub/index.html+web/hub/hub.js(Automation tab, mint checkbox, Research filter)hub/audit-log.mjs— do not edit (emptyproposalIdstring is already valid)netlify/functions/gateway.mjs(D27 two blob globals)scripts/verify-automation-ingest-smoke.mjs
Do not edit lib/hub-proposal-personal-self-apply.mjs except if a test import needs a comment — no predicate changes.
19. SD-21 land path (document only)
After AIP-b BV pass, with no live posture/env flip, secrets, real money, or Delegation write env in the diff:
- Muse merge/FF
feat/automation-ingest-policy-b→ Musemain ./scripts/muse-bridge-deploy.sh- GitHub PR
muse-mirror→main(merge commit) - Never
git push origin main. Never feature → GitHubmain.
Production smoke (T2) is Operator, recorded in docs/reviews/<date>-automation-ingest-live-smoke.md as PASS or FINDINGS: rule_id, disposition, Review proposed-count delta, HTTP status + JSON code only. Auto must not claim that smoke.
20. Operator production smoke record (not Auto)
After land + deploy, Operator records:
- PASS or FINDINGS
rule_id(ornullif default queue)disposition- Review proposed-count delta
- HTTP status + JSON
code - No secrets, no JWTs, no
kt_agent_material
Appendix A — VideoFactory trend scout (contract edge; AIP-c implements)
Not Hub Auto. Knowtation does not contain trend-agent source (Phase C §2 same fact).
| Item | Frozen Hub expectation |
|---|---|
| Hub URL | KNOWTATION_HUB_URL (hosted https://api.knowtation.store) |
| Vault | KNOWTATION_HUB_VAULT_ID (operator; Lane D smoke used Business) |
| Secret | KNOWTATION_HUB_AGENT_CREDENTIAL (kt_agent_ family) |
| Exchange | POST api/v1/auth/agent/token → short agent_access |
| Preferred write | POST api/v1/automation/ingest with ingest:automation + vault:read |
| Interim write | POST api/v1/proposals with ingest contract fields (D8, D14) if the agent still has propose |
intent |
videofactory.trend_scout.ingest |
path prefix |
inbox/trends/ (accepted / drafts subfolders if using pack templates) |
content_class |
research |
source_fingerprint |
Stable id of the scout item (URL + published time, or item id). Same item → same fingerprint. |
Credential name |
videofactory-trend-agent to match pack templates without editing |
| Fingerprint (consumer) | intent + path prefix + credential name + content_class=research |
| Forbidden | ktn_refresh / session JWT in cron env; POST api/v1/proposals/:id/approve; mint vault:write for this cron |
| Full wire | AIP-c (VideoFactory repo). This appendix is the Hub contract only. |
Appendix B — What Auto must not treat as open
If a later reviewer asks “should we score specificity?” — no (D1).
If “should default mint include ingest?” — no (D5).
If “should evaluation-on vaults auto-apply?” — no (D10).
If “should MCP grow a tool?” — no (D13).
If “should pack enable itself?” — no (D16).
)