# annum Agent Guide

Base URL: `https://annum.aisloppy.com`

## Purpose

Annum is a small, general-purpose **workflow engine**. You submit a DAG as
declarative JSON (nodes + deps); Annum computes ready nodes, dispatches each as
a task to your **outbound worker**, stores each node's output + metadata, tracks
per-node progress, and releases downstream nodes. It also fans a DAG out over
many entities (one run each), and **hosts the shared frontend widgets** (progress
bars, infoviz, chat, etc.) at `/widgets/`.

The service never runs your code. Your worker polls for tasks, executes locally,
and reports back. The boundary is explicit: a node is `{name, task_type, params}`;
your worker maps `task_type` -> a local handler.

## Authentication

- **Machine clients (workers + producers):** send `X-API-Key: <key>`. Keys are
  AuthReturn app-scoped keys verified against the `annum` slug. Get Annum's key
  from `/opt/secrets/annum.json`, or issue your own app key via AuthReturn.
- **Browser/UI:** Cognito JWT via `Authorization: Bearer <jwt>` (AuthReturn login).
- Every `/api/*` workflow endpoint accepts **either**.

## Producer API (submit work, read status)

- `POST /api/dag-templates` `{name, dag}` -> `{dag_template_id}`
  - `dag = {"nodes": [{"name","task_type","params"?,"deps"?}], "edges"?: [[from,to]]}`
  - Validated for unknown deps + cycles on submit.
- `POST /api/runs` `{dag_template_id, entity?, inputs?}` -> `{run_id}`
- `POST /api/runs/:id/extend` `{nodes[]}` -> append-only extension preserving existing node state
- `POST /api/runs:batch` `{dag_template_id, entities[], inputs?}` -> `{run_ids[]}` (fan-out)
- `GET  /api/runs/:id` -> run + per-node `{status, pct, elapsed, expected_seconds, error, logs, artifact_ref}`
- `GET  /api/runs?dag_template_id=&entity=` -> shallow run list
- `GET  /api/runs/by-artifact?node=&key=&value=` -> `{run_ids[]}` reverse provenance lookup: runs whose `node` artifact has `data[key]==value`, newest first (e.g. find the run that produced a given output id)
- `GET  /api/dag-templates/:id` -> template + dag
- `GET  /api/artifacts/:id` -> `{node, data, metadata, ...}`

## Worker API (outbound workers)

- `POST /api/workers/claim` `{worker_id, task_types[]}` -> `{task}` or `{task:null}`
  - `task = {node_id, run_id, name, task_type, params, entity, inputs:{run, deps}}`
  - Atomic; sets a claim expiry. Resolves dep node outputs into `inputs.deps`.
- `POST /api/tasks/:id/heartbeat` -> extend the claim during long runs
- `POST /api/tasks/:id/complete` `{output, metadata?, logs?}` -> store + release downstream
- `POST /api/tasks/:id/fail` `{error, logs?}` -> mark node + run failed

Reconciliation: any node stuck `running` past its claim expiry is requeued on the
next claim/read. Nothing stays stuck.

### Handler contract versioning

Treat `task_type` as the exact worker-handler contract identifier, not merely a
display label. When a handler's accepted inputs, returned output, or validation
contract changes incompatibly, replace it with a new task type (for example,
`video.select_highlights.v2`) in both the DAG and worker registration. Do not keep
the old handler registered for compatibility: an old worker must be unable to
claim the new node.

Keep one explicit workflow-contract version shared by the producer DAG and worker
registrations. Bump it only when task inputs, outputs, or semantics change
incompatibly; ordinary application releases must not fragment task timing history.

Worker processes load consumer code once at startup, but they must remain independent
of ordinary producer/web restarts so active work survives. Make the worker watch the
consumer's deployment version: after finishing its current node, it exits before the
next claim if that version changed; systemd's `Restart=always` then loads the new code.
Versioned task contracts prevent the draining old worker from claiming incompatible
new work.

## Worker SDK

Use `annum_worker.Worker` (see the repo). Register handlers, call `.run()`:

```python
from annum_worker import Worker
w = Worker("https://annum.aisloppy.com", api_key=KEY, worker_id="myapp-1")
w.register("pull_repo", pull_repo)   # fn(inputs, params) -> output | (output, metadata)
w.register("count_loc", count_loc)
w.run()                              # claim -> heartbeat -> complete/fail loop
```

`inputs = {"run": {...run inputs...}, "deps": {dep_name: dep_output}}`.

## Forecasting / observability

`expected_seconds` per node is a rolling average of completed durations for that
`(flow, task_type)`, so a frontend can animate a live timer. `pct` is derived
from elapsed vs expected. The run read-model also carries a run-level
`expected_total_seconds`: the critical path (longest dependency chain) through
the run's DAG — feed it to a time-estimate-driven progress bar (e.g.
BrightWrapper `LLMProgress`). It is `null` until every node type in the run has
timing history (no fabricated partial totals). The dashboard at `/` shows live
per-node progress.

## Widget hosting

Annum is the canonical home of the shared frontend widgets. `GET /api/widgets`
lists them; load any at `/widgets/<name>` (e.g. `/widgets/llm-progress.js`).
LLM time estimates still come from BrightWrapper's `/api/estimate` (its job as
the LLM wrapper); Annum owns the widget code.

### Run-progress widget (`/widgets/annum-runs.js`)

A generic, embeddable live view of **any** flow's runs — cards of per-node
progress bars (name + task_type + status + pct + elapsed/`~expected_seconds`),
animated between polls. It has no per-flow knowledge; point it at run ids (or a
flow + entities) and a data base URL.

```html
<script src="https://annum.aisloppy.com/widgets/annum-runs.js"></script>
<div id="pipeline"></div>
<script>
  const handle = AnnumRuns.mount('#pipeline', {
    apiBase: '/api/annum',            // where to read runs (see auth note)
    runIds: [12, 13],                 // explicit runs, OR
    flowId: 1, entities: ['a', 'b'],  // resolve latest run per entity
    pollInterval: 2500,
    disconnectDeadline: 60000,        // bounded reconnect window; then show Retry
    title: 'My pipeline',
    onSettled: (runs) => {},          // fires once when all runs are done/failed
    adaptive: true,                   // chain → rows; branch/merge → AnnumDag
    nodePlugin: {                     // optional inspector owned by a specific row/node
      ownsResult(node) { return node.task_type === 'my.prompt.step'; },
      label(node) { return 'Prompt'; },
      render(el, ctx) {},
      unmount(el) {},
    },
  });
  // handle.stop() / handle.refresh()
</script>
```

It reads `{apiBase}/runs/:id` (and, for the flow+entities form,
`{apiBase}/runs?flow_id=&entity=`) expecting Annum's run JSON shape.
With `adaptive:true`, a single run is classified from its actual dependencies:
one root with exactly one successor per step stays in the compact row view
(redundant transitive dependency edges are ignored);
branches or merges use `AnnumDag` when that asset is loaded.
Polling updates an existing DAG instance in place, preserving its settled
orientation, animation state, expanded groups, and pinned node.
The linear view follows the same restart-safe contract: run identity and task
state remain in Annum, the widget keeps the last confirmed bars visible through
temporary server/proxy outages, and polling resumes sequentially without
overlapping requests. After `disconnectDeadline` it stops automatic retries,
shows a terminal disconnected state, and offers an explicit Retry action.
Remounting after a browser or host-app restart resumes from the supplied durable
`runIds`; consumers must persist those IDs with their own job record.

### DAG run view (`/widgets/annum-dag.js` + `/widgets/annum-dag.css`)

The full graph view of **one** run: an animated pan/zoom DAG (Dagre layout, FLIP
expand/collapse transitions) with per-node progress rings driven by
elapsed/`expected_seconds`, fan-out replicas (`"<step>#<n>"` node names) grouped
into collapsible sample clusters, a hover/pin inspector sidebar (each node's
stored artifact rendered as a folding JSON tree), and a cancelled/failed run
banner. Self-contained IIFE — React and dagre are bundled, the host page needs
no other script. No per-flow knowledge.

```html
<link rel="stylesheet" href="https://annum.aisloppy.com/widgets/annum-dag.css">
<script src="https://annum.aisloppy.com/widgets/annum-dag.js"></script>
<div id="dag"></div>
<script>
  AnnumDag.render(document.getElementById('dag'), {
    run,                       // Annum run JSON (poll it; call render again per poll)
    entity: 'AAPL',            // header label
    variant: 'live',           // 'live' | 'resting' (bounded-height at-rest panel)
    direction: 'auto',         // 'auto' | 'LR' | 'TB'; override when the host fixes aspect ratio
    api: {                     // host proxy URL builders (see auth note above)
      runUrl: (id) => `/api/annum/runs/${id}`,        // "run #N ↗" links
      artifactUrl: (id) => `/api/annum/artifacts/${id}`, // sidebar result trees
    },
    explainNode: (id) => '…',  // optional per-step prose for the sidebar
    nodePlugin: {              // optional host extension for the sidebar card
      prepare({ tasks, entity, runId, runStatus }) {}, // per poll — dedupe internally
      ownsResult(node) { return false; },  // true = replace the default artifact tree
      render(el, ctx) {},      // mount host UI into a DOM slot — use your own React
                               // copy freely; ctx = { node: {id, label, operation,
                               // status, artifactRef}, entity, runId, runStatus,
                               // buildActive, artifact: {loading, data} }
      unmount(el) {},
    },
  });
  AnnumDag.unmount(host);
  AnnumDag.demo(host, { api });  // synthetic fan-out harness (?dir=TB, ?expand=…)
</script>
```

Run-level progress bars are deliberately **not** part of the widget: hosts mount
a real time-estimate-driven bar (e.g. BrightWrapper `LLMProgress` fed by the
run's `expected_total_seconds`) above it instead of a simulated percent.

### Explanatory system model (`AnnumDag.model`)

The same DAG view can explain a concept without creating or executing an Annum
run. Pass a static acyclic model; selecting a node opens its description and
structured notes in the existing inspector.

```html
<link rel="stylesheet" href="https://annum.aisloppy.com/widgets/annum-dag.css">
<script src="https://annum.aisloppy.com/widgets/annum-dag.js"></script>
<div id="model"></div>
<script>
  AnnumDag.model(document.getElementById('model'), {
    model: {
      title: 'Recommendation system',
      description: 'Select a concept to inspect its mechanism and tradeoffs.',
      nodes: [
        {
          id: 'source',
          label: 'Source',
          description: 'Build the candidate universe.',
          notes: [
            {label: 'Design choices', body: 'Graph, interactions, or content.'},
            {label: 'Failure if wrong', body: 'Relevant items never enter the pool.'},
          ],
        },
        {id: 'rank', label: 'Rank', description: 'Order eligible candidates.'},
      ],
      edges: [['source', 'rank']],
    },
    direction: 'LR',
  });
</script>
```

The model contract rejects missing or duplicate node IDs, unknown dependencies,
malformed edges, and cycles. A linear DAG renders as compact rectangular cards
with selectable descriptions and click/keyboard note disclosure; branch and join
structure uses the full adaptive Dagre canvas. See `/model-demo.html`.

**Auth note (cross-origin embeds):** Annum's `/api/*` requires a key/JWT and the
widget JS is served cross-origin. The recommended pattern is for the host app to
**proxy** Annum through its own backend (e.g. expose `GET /api/annum/runs/:id`
forwarding to Annum with the app's key) and set `apiBase` to that proxy — the key
stays server-side with no cross-origin API calls. Same-origin pages (Annum's own
dashboard) can set `apiBase:''` and pass a Bearer token via `fetchOptions`.

## Integration Rules

- Fail fast on errors; do not silently degrade. Read `PORT` from environment.
- Use network APIs between services; no cross-app imports.
- Long node-tasks belong in your worker, not in a request handler.

## Requesting an API key (self-service)

New clients can mint a key without filesystem/AuthReturn access:

```
POST /api/keys/request
{ "email": "<annum-app account>", "password": "<password>", "force": false }
```

Returns `{api_key}` (shown once) — store it and send as `X-API-Key`. AuthReturn
issues **one** non-expiring key per user per app; pass `"force": true` to revoke
and regenerate the existing one.