Roadmap
Status: 2026-11-08. What the library does today, what is planned, and what is deliberately not planned. The reasons behind the behaviours are in Decisions.
Available today
| Capability | What you get |
|---|---|
| Ask | ask(): a grounded answer on a thread, text or structured (outputSchema), Markdown or plain, with the call's own instructions and context, files of the question, sampling temperature, a debug transcript on request |
| Run | run(): an agent loop with typed tools, a policy per tool that allows, denies or hands a call to a human (policy), per-call tool narrowing and per-call tools, guarded workflows (a tool after its requirement), a tool offered only in some steps (visibleWhen), a tool that ends the run (endsRun), timeouts and output caps on tools |
| Skills | Instructions the model loads on demand (ISkill {name, description, body}), on the profile (IAIProfile.skills) or on the call (skills): a run() lists them by name and description and loads a body with the built-in dyna_aigw_load_skill tool when a task needs it, an ask() gets the bodies whole; they sit in the cached part of the system prompt |
| Human in the loop | Approval of a tool call (the tool's policy, or per tenant in the profile), the text the model reads on a rejection decided per call, the model's questions to the user as a form, resume with approve / edit / reject / answer, review of the final answer before the user sees it (per tenant in the profile: approve, edit, reject), resume on another profile, cancel, expiry of unanswered pauses; a resumed run cites, traces and remembers like one that never paused |
| Streaming | Every call returns its events (text deltas, tool requests and results, pause, run ended by a tool, done, withdrawn answer) and its response; the application owns the transport |
| Cancellation | abortSignal and timeoutMs on every call: the model call, the agent loop, the tool and the rules stop through one signal; the response is FAILED with call_aborted or call_timed_out, what the call had stored stays |
| Graceful shutdown | shutdown({timeoutMs}): new calls refused with a 503, the calls in flight drained within the grace period, the rest aborted cleanly, the background work landed; paused runs untouched |
| Recovery after a crash | recoverStuckRuns({tenantId}), a scheduled job: the runs a dead server left running are failed, their tool call closed, their thread open again |
| Profiles and models | A model catalog with capabilities and cost; profiles per tenant that bundle model, cheap helper model, tools, approvals, prompt addendum and limits; the profile chosen per call; models and profiles changed live; a model switched off or removed, a tenant's profiles listed, a profile removed unless a paused run waits on it |
| Prompt templates | Versioned templates per tenant with one active version, history paged, the shipped seed as fallback |
| Memory | The thread's history per profile as a sliding window (the newest turns that fit the model's window, verbatim) or a summary buffer (a summary and facts plus the last turns), or the host's own IWorkingMemory; the user's facts across threads; both with a required switch; memory tools for the model; the facts editable through the gateway |
| Knowledge base | Markdown documents chunked, embedded and stored per tenant → library → book; scope, k and minimum score per call; query rewrite; citations of the chunks given to the model; a book moved between libraries without re-embedding |
| Files | Upload, images and PDFs sent to a model that can read them, a short-lived URL of a file for the screen, removal per thread and per user |
| Threads for a chat screen | The calling user's threads in pages, newest first; a thread's history read whole; rename; bulk delete of threads with everything they left (turns, memory, files, runs, trace entries) |
| Vercel chat client | dyna-ai-gateway/vercel-ui: a server-only module that takes the gateway and a use-case config and speaks the UI message protocol of useChat: chat streams the answer and the approval cards, history gives the messages of a thread, resume streams the continuation of a paused run |
| Guardrails | Built-in input and output rules; the application's rules with evidence, timeouts, severities and a judge call on the cheap or main model |
| Usage | The application's adapter asked before every model call, helper calls included, with the call's factors and a token estimate, and told what the call used after it; no token logic in the gateway; steps per run capped by the profile |
| Prompt caching | The stable part of the system prompt (template, output format, profile addendum) sent as its own block, marked for the provider's cache (Anthropic's marker, OpenAI's cache key); per tenant or shared across tenants, both decided by a required switch; the tokens read from and written to the cache in the usage report and priced at the model's cache rates |
| Trace | One entry per model call with kind, model, template version, usage, cost, chunks and issues |
| Data protection | Delete or anonymise everything the gateway holds about a user |
| Persistence | A repository or adapter interface behind every store, with in-memory implementations of all of them |
| Providers | OpenAI, Anthropic, Google, Mistral and Amazon Bedrock for chat and tools; OpenAI for embeddings |
| Documentation | The guides and the API reference generated from the JSDoc, on the docs site |
Planned
In the order they are expected.
- Deleting threads. Delete one or every thread of a user with all it holds: turns, memory, files, runs, trace entries.
- Real adapters. MongoDB and PostgreSQL repositories, a vector search index, S3 for files, as packages of their own next to the gateway (
dyna-ai-gateway-mongodb,dyna-ai-gateway-postgres,dyna-ai-gateway-s3; the database driver is a peer dependency, the adapter receives a connected client). Each with its collection or table names as configuration and a storage proposal before it is built. - Model settings on the profile. Temperature, reasoning effort and provider options per profile; the per-call temperature stays.
- Repair loop. One retry with the issues of a failed output check handed back to the model, bounded by configuration; the same for a structured answer that misses its schema.
- Retrieval quality seams. A reranker hook.
- Built-in judge rules. Moderation, PII, injection and topic scope as ready-made rules on the judge.
- Learning loop. Rating an answer on its trace entry, running an evaluation set against a profile, pinning a template version per call so two versions can be compared.
- Template overrides per profile.
IAIProfile.templateOverridesis declared and not read yet. - Later. Sub-agents as tools (a tool whose execution runs another profile), export of rated traces as training pairs.
Not planned
- A gateway-owned push transport. The call returns its events; publishing them to a client in another request (WebSocket, pub/sub, a resumable stream) is the application's, with its own channel ids.
- Registering tools or guardrail rules on a running gateway. Both are configuration. A tool for one request is passed on that call.
- An HTML output format. Markdown from the model, converted and sanitised where it is rendered.
- A profile stored on the thread. The caller decides the profile per call.