Improve your chat

Improvements shows issues found in finished conversations and operator lesson drafts. Prompt issues can have a proposed fix to your agent's system prompt. You preview the fix on real visitor messages and publish it as a new agent version when you are happy with it. It does not train model weights, and nothing is published unless an owner or admin clicks Publish.

Tenant owners and tenant admins open Improvements for their tenant. Global Admins use AI & knowledge → Tenant improvements, choose a Target tenant, and review that tenant's improvements. Issues, previews and versions remain scoped to the selected tenant.

Where issues come from

Every conversation from the last 7 days that has ended (Completed, Closed or Abandoned), or that has been idle for 30 minutes, and that contains a visitor message is analyzed automatically. Older conversations are not analyzed. Up to 100 conversations per tenant are analyzed per UTC day, newest first; the page shows the count as conversations analyzed today (UTC).

Each analysis lists what went well, what went wrong and recommendations. Failed tools and failed workflows in the conversation are added as issues too. Jev groups the same problem found in several conversations into one issue row, and the row shows how many conversations are behind it.

An issue is a suggestion to investigate, not a verified fact. Open the Conversations links on the row to read the conversations yourself before you publish anything.

Conversations that could not be analyzed are listed under Conversations that could not be analyzed, with the reason. Click Retry to queue the analysis again. Conversations whose evidence was purged are not listed and cannot be analyzed again.

Preview and publish a prompt fix

Each issue row shows its title, status, the agent, the number of conversations and when it was last seen. Open What happened and the proposed fix to see:

What went well and What went wrong, taken from the conversations behind the issue.
Conversations: links that open each conversation on the Conversations page.
Proposed fix: a before and after comparison of the agent's system prompt.

Then use the buttons on the row:

Click Preview. The agent's own model replays the last 3 visitor messages from up to 3 of the conversations behind the issue, once with the current prompt and once with the proposed prompt, and shows both replies side by side under Preview replies. Jev then judges each pair, and the preview shows whether it passed or did not pass. Preview needs the decision model (Jev): if it is unavailable, the Preview does not pass (the issue shows decision_model_unavailable) and Publish stays unavailable. Preview tests the prompt only: tools and the knowledge base are not run, so a reply that depends on them can differ in real chats. It can take a few minutes.
Click Publish. It is available only after a passing Preview. Confirm the inline prompt ("Publish vN? This changes the live agent."). Publish creates the agent's next version with the new prompt. Only the prompt changes: it never publishes unpublished work from the agent editor. The issue then shows published as vN, and the version appears in the agent's version list.
- If the agent's current version has moved on but some chat configurations are still pinned to the version the fix was drafted from, the fix is published as a new version for those chats only. The agent's current version does not change.
- If someone published from the agent editor after the fix was drafted, and no chat is pinned to the version the fix was drafted from, Publish is refused with "The agent changed since this fix was drafted. Redraft it first." Click Redraft to draft the fix again from the agent's current prompt.
Click Roll back on a published issue to make the version the fix was drafted from current again. If the fix was published for pinned chats only, Roll back moves those chats back instead. It is refused if a newer version was published since. Afterwards the issue is back in Ready for review and can be published again.
Click Redraft to ask for a new proposed fix, for example when the Preview did not pass.
Click Reject to dismiss the issue. You must enter a Reason. Reject is not available once an issue is published; use Roll back first.

Platform issues ("no prompt change can fix this") and knowledge base issues ("correct the source in your knowledge base") have no Publish. Fix a knowledge base issue in your knowledge base; see Create a knowledge base.

Release-based evaluation (advanced)

Click an issue's title to open its detail page. Below the prompt fix, Release-based evaluation (advanced) keeps the regression-tested lane for changes that need it. It is separate from Preview and Publish above. If your platform uses an AI decision model (for routing and yes/no checks), the baseline records which model it was; see Supported changes and limits below.

Use Reject with a reason if the item does not warrant a change.
Click Capture current baseline to capture the supported configuration and agent scope. If the scope has changed, use Rebase to current scope and clear approval, then prepare a new draft and evaluation. A baseline captured now does not establish which settings an older conversation used.
Click Generate draft, then review Edit candidate. Check the Change type, complete agent instructions or callable workflow offer instructions, and any proposed knowledge correction. Use Save draft and clear prior approval after editing.
Verify knowledge sources using Verify source for this candidate when applicable. A plausible answer or generated reference is not sufficient evidence.
Prepare the Regression suite. Add review templates supplies starting cases; adapt them to your tenant's facts and actual workflows. Supply verified expected answers, deterministic assertions and at least one Held-out case, then Verify and save case. Held-out cases are excluded from draft generation.
Click Evaluate. Model-dependent cases run three times against both the baseline and candidate. Critical checks must pass on every candidate run, quality must meet or exceed the baseline, and errors fail evaluation. Inspect the results before approving.
Click Approve, then Publish to activate the approved, evaluated change. In this lane, approval alone does not change live chat behavior.

Editing a candidate clears its evaluation and approval. Changed scope or new evidence can make an earlier evaluation stale and block publishing until it is reviewed again. Accuracy and safety checks take precedence over conversion gains.

Supported changes and limits

Supported executable candidates include agent instructions, verified knowledge corrections within supported document or website scopes, and offer instructions for captured callable workflows. Callable workflows are still subject to their supported node types.

Supported agent skills are capture_data, render_form, suggest_items, request_handoff, end_conversation, schedule_callback and list_products (List products in chat). A release keeps a snapshot of the knowledge base's scanned website pages, which list_products lists from, so replays see the same products. A knowledge base with more than 1,000 scanned pages can't be captured yet.

Some dependencies cannot yet be captured for a reproducible improvement evaluation. These include parent workflows with more than a single AI node, MCP tools, Shopify, WooCommerce or email retrieval, external user information, webhook skills and other built-in skills, enabled integrations, and host page context (Use Page Content for AI Answers; see Page content in AI answers). If capture reports an unsupported scope, ask your platform administrator to review it; preserve the safeguards your chat needs.

The AI decision model (used for routing and yes/no checks, such as page link selection) does not block capture: the baseline records the selected model. If the selected decision model changes later, the captured release is blocked until you capture and evaluate a new baseline.

Workflow graph changes use the existing workflow editor: review the proposal, edit and save the workflow, save its attachment and publish the agent, then capture a fresh baseline. Engineering proposals are review-only; the inbox does not execute generated code or deploy software.

Releases and rollback

This section covers the release-based lane. Published improvement releases apply to future chats. Existing conversations and running workflow calls keep their captured release settings. If active guardrails no longer match a captured release, runtime checks can block it pending staff recovery.

Use Roll back on a published proposal to return to its baseline ancestor release in the same scope. Future chats use that release; existing chats keep their prior settings. Rollback does not undo emails, CRM updates or other external actions already performed. This release rollback is separate from Roll back on an issue row, which makes the agent's previous version current again, and from Versions → Set as Current in the agent editor.

Review outcomes and read the metrics

Under Review a production outcome, select Conversation evidence and record whether the answer is grounded, contains an unsupported claim or repeated clarification, and handles handoff appropriately. Click Save reviewed outcome. Staff reviews provide verified judgments; automated findings do not become reviewed accuracy scores by themselves.

For qualification, choose Also review qualification, enter the Tenant qualification criteria, and record whether the conversation is eligible and qualified under those criteria. Criteria are versioned so incompatible definitions are not combined into one rate.

Recorded outcomes · last 30 days reports observed outcomes for the release:

Offer acceptance uses accepted offers divided by recorded offers.
Form completion uses valid submissions divided by collection starts.
Delivery success uses confirmed delivery outcomes for submitted conversations.
Qualification uses reviewed eligible conversations under the same criteria version.
Reviewed accuracy includes its reviewed sample size. Missing judgments or delivery receipts remain unknown, rather than being treated as zero.
AI credits and model-call latency cover correlated recorded calls; uncorrelated older calls are not counted.

Synthetic evaluation runs are excluded from production outcomes. These metrics describe recorded results; they do not establish that a release caused a conversion increase. Use Chat test for the visitor experience as well as regression evaluation for candidate comparisons.

Review operator lesson drafts

In the mobile app, Create a lesson draft requests an administrator-reviewed draft. Successful handback accepts the request; it does not promise a generated or published lesson. Teaching submissions includes pending, blocked, failed and ready requests, even when no proposal or agent target is available. Owners and administrators can review the submitting operator, source conversation, message and private-note references, redacted evidence, sensitivity outcome and failure reason. Ordinary operators do not gain access to reviewer evidence.

Source release identifies where the evidence came from. Intended publication scope and Target release identify where a reviewed lesson might be used: its agent, version, chat configuration and workflow node. These are separate. Redaction does not guarantee anonymity; review the evidence for unnecessary personal or confidential information.

For a ready draft, open Review lesson and edit the Reusable question and Reusable answer. Save lesson draft and clear prior approval stages a candidate. It is not ready for evaluation until Verify source for this candidate succeeds. Verify the factual claim against the redacted source; the operator's assertion alone is not source verification. Then Evaluate, review the results, Approve, and separately Publish. Publication affects future conversations only in that exact scope; existing chats keep their release pins. Lesson drafts have no prompt-fix Preview or Publish shortcut.

To change scope, enter an existing compatible Target release ID and use Select target release explicitly. Rebase to current exact scope uses the existing pointer for the same agent, version, configuration and node. These actions clear verification, evaluation and approval; they do not create a runtime target or publish anything. If no compatible target exists, the draft stays blocked. A blocked request may need scope review, may have no reusable lesson, or may require further sensitivity review.

Source corrections continue to use CuratedKnowledge and correct an existing knowledge source. A new lesson draft adds a separately reviewed reusable claim. Prompt fixes change instructions and retain their own Preview workflow. Rejection and rollback use the existing release review actions; neither turns a lesson into a prompt fix.

Remove evidence

Purge evidence and derived copies removes the selected evidence and its derived copies and invalidates dependent candidates, evaluations and releases. Treat it as a removal action, not a way to reject a suggestion. Automatic evidence expiry runs only when an explicit retention policy is configured.