Evaluations and Checks

Watch the workflow: Revise a scenario and inspect missing evidence · Authored dummy data

Edit an authored manual scenario in Tensile, clarify the required input and save a revision. Its rule remains unverified and the missing trace evidence keeps Create draft check unavailable. This clip does not show a published pack, calibration, approval or live scoring. No evaluation or provider runs.

Read the walkthrough transcript

• 00:00 — Review the authored scenario in Tensile. Its rule remains unverified.
• 00:05 — Clarify the recorded input a reviewer needs to answer the question.
• 00:11 — Save the revision. Missing evidence still prevents creating a draft check.

Open Executions to browse Packs, All evaluations, Check library, Run history and Analytics. This workspace is available only to enabled organizations. The separate Agent executions destination keeps the source-run index and its original records.

Start with expectations and coverageCopy Anchor Link

In Executions, open First-use setup and coverage. This reuses work setup and the inherited Default Packs. Each Pack shows its exact version, Check readiness, missing policy or source, and existing opt-outs. A ready Check can assess supported work while another Check remains held.

Open a linked Evaluation version to read the intended outcome, permitted and forbidden effects, policy source, and next owner. Use Open exact expectation version to edit through the existing versioned editor. Its saved-run link stays pinned to the selected Evaluation version. A missing result or source leaves coverage and rates unavailable.

Check history across versions is labelled separately from the current configuration. An observation is not a policy Pass. Not applicable requires a known item. No eligible work observed requires verified source coverage; it does not mean zero failures.

Observe, review and configured notification remain separate. Connections and delivery stay in Settings. A named workflow reviewer must inspect the first bounded saved result for comprehension and false alarms before new workflow recurrence. Selecting a Check grants no agent permission. Blocking stays unavailable until a qualified pre-action integration exists.

Browse and authorCopy Anchor Link

• A Pack groups Evaluations. Membership can be direct or related. Related scope does not imply a shared verdict.
• An Evaluation retains its native evaluator definition, versions and results. Existing records without a Pack stay visible as Unassigned. Creating a new Evaluation requires a Pack.
• A reusable Check pins one saved definition version and its Deterministic or Model judge method. A Model judge selects a configured client connection from Settings. The saved Check keeps the reference, never the token. Changing the source definition does not change an existing Check version.
• A draft Evaluation can pin Check versions with Atomic results only or All Checks required. The latter preserves failing, unknown, held and incomplete answers. Saving a composition grants no approval or activation.
• A Default Pack is published by Tensile and distributed to workspaces once it is approved. It appears in Packs beside your own Packs, with its inherited version and its state in this workspace: Enabled, Disabled or Blocked. Its record lists each Evaluation and Check. Disabling the Pack or one Evaluation stops only that path; a Check shared with another enabled Pack keeps running there. To stop a Check everywhere, open it in Check library and choose Disable in every Pack. Each Default Pack reads the policies it lists from your workspace; a Check whose policy is not set stays Blocked and never passes by default. Default Checks are assessed only on work observed after automatic assessment is set up for your workspace.

Older Evaluation pages remain available. Their Open in Executions link opens the same Evaluation and version here. A link to a step that only the older page provides, such as Examples, Shadow, Validation, Approval or Results, says so and links back to that page with its filters intact. A version that does not belong to the Evaluation shows Version not found instead of another version.

Search, sort and page each collection on the server. Open a row for its full record. Breadcrumbs, browser Back and record arrows retain the collection context. Select a saved version explicitly; an unavailable version is not replaced by the latest version.

Schedule saves an Evaluation's timezone and daily, weekly or monthly recurrence. Choose the month-end and missed-occurrence rules explicitly. A missing daylight-saving time is skipped; a repeated time uses its first instant. The backend job checks for due Evaluations every 30 minutes, independently of the saved recurrence. Due work can begin on the next job poll. Schedules do not activate definitions or grant source or model access. Run history records held or failed work with its reason. A saved schedule or held receipt is not a completed run.

Inspect an observed Gmail scanCopy Anchor Link

Open a saved Check result from Run history. The result keeps task, policy, evidence and environment separate. It shows the recorded read coverage, reports, attempts, denied actions, committed effects and evidence gaps. Expand the source details for the exact scope, cutoff, versions and receipt history. Unavailable or revoked evidence stays unavailable.

Task completion and policy compliance remain Unknown · unqualified until their exact conditions and qualification support a claim. An atomic Pass does not complete the task. A retained failure remains visible when another result is Unknown. Observed work does not have an isolated test's fixture or reset qualification.

The run's scan inventory separates included work from exclusions. Included units reconcile to assessed, pending and Unknown units at the same source cutoff. Evidence-gap counts can overlap these result counts. No eligible scans is different from zero failures. A business completion rate remains unavailable without qualified outcomes.

Review this saved finding in Investigations opens the same native result. An exact linked Investigation supplies its assigned reviewer. Otherwise, the result shows Owner missing. Notification connections and delivery remain in Settings → Notifications. Opening the result does not send a notification or save another judgment.

Where the permitted evidence scope or supported execution route is not configured, dispatch remains held. Configure the missing contract with the Evaluation owner. No automatic provider fallback or historical catch-up is implied.

The original definition, Examples, Results, Validation, approval and activation remain separate under Native authoring and review. The older “evaluator pack” term in these native pages means an Evaluation; it does not mean a new topical Pack. Legacy definitions remain available through their original destination and retain their original identity and result meaning.

An evaluator pack is a versioned definition of a workflow and the decision or outcome to assess. It describes what the evaluator may use as evidence, the question it must answer, and what each answer means.

Packs are designed to keep business meaning, evidence, and model judgments inspectable. A proposed pack is a draft: connecting a trace source or creating a pack does not activate evaluator-pack scoring. Existing operational views may still summarize captured event signals such as success, latency, or token use.

The Trace patterns page can surface possible gaps from recorded agent turns and collect reviewer feedback before they become approved checks.

AnalyticsCopy Anchor Link

Analytics reads stored results only. It never runs a model or re-judges work. Choose one of two scopes:

• Historical rubric results uses stored Evals verdicts. By default, the record count and recent table exclude segmented results. Choose Include segmented saved records to inspect those retained verdicts too; rates, charts and the version breakdown still use non-segmented results. Inspect Quality and Safety judge pass rates, recorded judge-human agreement, daily trends and each selected verdict’s rationale. The breakdown reconciles the comparable-verdict count. When several judge models or methods are present, a card shows Mixed versions instead of one combined rate.
• Execution results shows Check results from scheduled Evaluation runs. The pass rate is pass / (pass + fail). Unknown, Not applicable and unmapped answers are counted separately and do not change the rate. A judgment that two Evaluations reuse counts once, with two references.

Each scope shows its UTC window, from inclusive to exclusive, its clock and its unit. Select Pin this exact window to keep a link to the same cutoff. A rate with no pass or fail result shows Unavailable, not 0%. A filter that a scope cannot apply is listed as unavailable.

From a workspace Pack, Evaluation or Check record, select View results for this version to open Execution results for that exact version. The return link goes back to the same record and version.

The Historical rubric results scope uses the same saved views as Evals. Open the view menu beside the page controls to select a personal or organization view. Select Save view to keep the current filters. The view menu also lets you edit a view you can manage, pin it to the sidebar, or set your own view as the default.

A default view applies when you open Historical rubric results without filter parameters. Filters already in a link take priority. Select Default in the view menu to clear the saved view. If you change a loaded view's filters, use Save to update it, Save as new to create another view, or Cancel to restore its saved filters. These controls do not appear in Execution results.

The pack lifecycleCopy Anchor Link

1. Connect evidence. Bring in traces from a supported integration or the ingestion API. Each source exposes different fields and history.
2. Review workflow findings. Confirm or correct what the workflow is trying to complete, which states count as success or failure, what actions matter, and which source facts prove an outcome. Unsupported or contradictory claims remain unresolved.
3. Inspect evaluator proposals. A proposal should identify its unit of evaluation, question, answer criteria, evidence recipe, source support, and unresolved assumptions. Definitions are versioned.
4. Review eligible examples. Read the human question and answer meanings alongside the case evidence. Source and trace provenance are shown when recorded. If a pack has no eligible examples, there is no example queue to label.
5. Assess readiness. Before a pack can be trusted for production use, its labels, calibration, holdout isolation, independent review, coverage, and owner approval must meet the applicable gates.

Set up work by handCopy Anchor Link

From Executions > Native authoring and review > Onboarding, choose Set up work by hand (or Work setup and scenarios for a listed workflow) to describe a workflow without traces or a code repository:

1. Confirm the work setup. Pick the work type (OCR and extraction, unstructured review, structured review, transformation, or actions), the unit of work, the domain, the checkable result, which sources are available, and the KPIs you want to improve. Each saved change is a new version.
2. Review scenarios. A scenario is a question about the work with its rule and version, required input and output, evidence, suggested examples and unknowns. Accept, edit, reject or add scenarios. Starter scenarios for the work type are a starting point, not a complete list; evidence the scenarios do not cover yet stays listed.
3. Create a draft check. An accepted scenario whose evidence sources are available becomes a draft check. If a source is missing, the scenario names it. From the draft, Write the check from this scenario drafts its question, answers and evidence as a new version. A check can only be written once someone stands behind the rule: confirm or edit a starter scenario or an AI suggestion so its rule names your policy, your input or public guidance first.
4. Add reviewed examples. Save real inputs and agent outputs with your answer (correct, failing or unknown), the source, versions and your reason. Suggested examples in a scenario are prompts, not answer keys.

These steps never activate a check, run a model or change your systems.

Code proposals. If a code repository snapshot exists, the page proposes workflows from it. A workflow is Identified only when cited code findings show agent work; a file name alone stays a Candidate. You still confirm the work type, unit of work and checkable result.

AI suggestions. When a source has an approved model, data path and spending limit, you can queue a request for five or six draft scenarios. The panel shows the exact model, what each kind of input sends and where, and requires a budget before anything is queued. Optional inputs are your code findings, trace field names, one page of the website you saved, and public information about your industry: where public search is set up for your workspace, a web search for your industry name and work type only, with up to three result pages read. A quote found on one of those public pages makes the suggestion public guidance, with the page and date; a page found by search is marked so you can check it is an authority. Each request shows what was actually sent and why any input was skipped. Suggestions arrive as proposals marked as an unverified rule. Your customer policy and decisions rank above them.

Create or revise an EvaluationCopy Anchor Link

Under Executions > All evaluations, select New Evaluation. Customer owners, customer admins, and verified Tensile staff who belong to the workspace can create and edit drafts. Other members can read definitions but cannot save changes.

Choose a Pack and enter a name. To start with no definition, keep Blank definition, choose an existing work type, and select Save. The first version is Discovered. Open the versioned definition from the record to edit its structure. Its page shows the requirements that must be met before a complete draft can be saved. This form does not create a work type.

Open Edit structure on the latest version. Choose one source:

• Bundled Quality rubric copies the current Quality text into the draft. Select a pass position from 0, 25, 50, 75, or 100. Scores at or above that position pass. A separate question checks whether there is enough evidence; insufficient evidence stays Unknown.
• Tracked intent copies one stored intent definition. Select the intent, its evidence fields, and the evidence source. A missing definition or failed source read stays Unknown and prevents saving. Use Retry source read after the source is available.

Select the evidence fields and an exact judge model permitted by the workspace. If no permitted model is available, a workspace administrator must configure one before the complete draft can be saved. Selecting a model does not call it.

Save draft version checks the definition and appends a new Draft version. A refused or unchanged edit creates no version. Earlier versions and their copied source text remain unchanged when the source later changes. Open an older version to inspect it; return to the latest version to edit.

Use Edit wording to change the name, reviewer question, or answer wording. Enter a reason and save a new version. All evaluations shows the changed name. To copy a saved definition, create a New Evaluation, choose its Pack and select the definition under Start from an Evaluation before saving.

Creating, copying, or editing a draft does not call a model, activate a check, or change live scores and historical results. A Quality draft does not replace the bundled Quality evaluator. Examples, validation, and approval remain separate requirements.

Define the expected workflow outcomeCopy Anchor Link

On the latest Evaluation version, open Definition. If no expectation is configured, select Configure expectation under Workflow expectation. State the intended outcome and the entry point with its revision. The assessment unit comes from the saved definition.

Add completion conditions, select their evidence fields and map them to exact Check versions. For example, complete scan coverage can require a recorded cutoff and coverage evidence. A permitted partial report can describe which coverage is missing. A valid no-action condition can require evidence that no eligible work was found. These are authored conditions, not verified results.

Record allowed and forbidden effects, the policy source and revision, its owner and authority scope. Keep an unknown policy Unresolved. An owner name or approval reference does not approve the policy. Connections and delivery settings remain in Settings; the form refers to existing response configuration.

Select Save new draft version to append a version with the selected Checks. Earlier versions remain unchanged. Editing wording, Quality or a copy preserves the expectation. Changing an existing Check selection also creates a version; remove a Check's dependent mapping before removing that Check. An unavailable or stale version cannot silently replace the version you selected. An Evaluation that maps requirements to other Checks cannot become one atomic Check. Reuse its individual Checks instead. The original Evaluation and its copies keep their complete expectations.

Incomplete input stays in this tab for the same workspace and version after navigation, when browser storage is available. If a save result is unknown, inspect the saved version before submitting again. Saving grants no action authority and transfers no approval or qualification. Checks retain their individual results until the applicable aggregation is approved.

Review an exampleCopy Anchor Link

When an example is available, the reviewer should be able to:

• read the specific decision question and the criteria for each answer;
• inspect the decisive facts and attributed trace evidence;
• save work in progress and return to it;
• lock an answer and continue through the eligible queue; and
• compare with the model's answer only after locking the human answer.

The reviewer should not have to infer the question from technical field names or from the order of model-generated questions. Missing evidence should remain visible rather than being treated as a successful or failed result.

If the definition lacks a reviewer question, Definition and Examples identify that same prerequisite and link to the wording editor. A wording change creates a new version; example answers stay attached to the version the reviewer saw. Missing source evidence is a separate blocker. Saving an answer becomes available only when both the question and the required case evidence are ready.

Draft review is not live scoringCopy Anchor Link

The current workspace supports draft pack definitions and human review for eligible cases. The production evaluator remains separate from that review experience. An unapproved pack's judgments must not be treated as approved evaluator results or drive customer routing, alerts, or automatic actions.

Promotion requires an independently supported validation path. Depending on the pack, this includes disjoint calibration labels, an untouched sealed set, adequate answer-class coverage, separate reviewer checks, and approval from the authorized owner. A model's own judgments are not independent labels. In a named-class validation request, reviewers lock all their evidence-backed answers before seeing the requester's per-case sampling tags. The app keeps accuracy Unknown until the separate source, holdout, and approval gates pass.

Source-specific setupCopy Anchor Link

• Connect Braintrust to import selected project logs.
• Connect Langfuse to import traces from a Langfuse host.
• Use the ingestion API to send agent events directly.

The source system and trace identity shown in a review must come from recorded provenance. Tensile does not infer a source from an organization or customer name.

Inspect recorded verdicts in EvalsCopy Anchor Link

Evals starts with the last seven days unless the link or a personal saved view specifies another period. Use the period selector beside the Evals heading to change it.

Recent Verdicts shows a bounded set of the latest 100 matching records. Search, Pass/Fail filtering, column sorting, and pagination apply to that loaded set; the page's total matching count can be larger. Rows show rubric, score and verdict; select the first-column details button to inspect the result, rationale and recorded model. On larger screens the judge model and timestamp appear in the table. View session sits beside the verdict when a source timestamp is available; the raw session ID is under Technical details. On small screens, use Back to list to return from the selected verdict.

A judge fail means the evaluator assessed the work as failing its rubric. Human labels, when present, are shown separately. Review the source session and rubric before deciding what action is needed.

Use Evaluator definitions to inspect current tracked definitions and the approved Quality prompt template. Current configuration does not establish the governing version of a historical verdict.

Execution results cover judgments pinned by execution manifests and saved default-Pack automatic work. Follow Analytics from a Pack, Evaluation or Check to inspect the selected version and its source-linked judgments. Automatic Runs retain their saved judgment receipts; scheduled Runs also link to their Evaluation and execution manifest.

Historical result groups retain their recorded rubric prompt version, judge model and evidence method. Different known versions show Mixed versions instead of a combined score. Incomplete version evidence keeps matching counts visible and the affected rates, mean scores and trends Unavailable.