get_eval_case
ChatGPTFetch one eval case in full: what the tester is instructed to do, the criteria it is judged against, its tool policy and limits, the suites it belongs to, and its most recent individual runs with outcome, score, duration, and any error. Use this to understand WHY a case failed before changing anything.
get_issue
ChatGPTFetch one issue (problem cluster): its cause, in-range blast radius (sessions, tools, use cases, harnesses), and example sessions that hit it. Mirrors the Issues drawer.
get_session
ChatGPTFetch one reconstructed session end to end: header (including resolved user identity when recorded), the agent's intent, every event in start-time order with tool calls and search calls, classification, logs, and the latest judgement. Mirrors the session replay drawer.
get_suite_run
ChatGPTFetch a suite run: overall status (running / passed / failed), how many eval cases completed and passed, whether the result still matches the suite's current membership, and the per-case outcome with each underlying individual run. Omit suiteRunId to get the latest suite run for the suite — the answer to "is this MCP ready to ship right now".
list_eval_cases
ChatGPTList the eval cases in one suite — exactly the set run_eval_suite dispatches. Each case carries its criteria count, schedule type, the other suites it belongs to, its last run outcome, and a pass rate over its recent runs. Runs the judge could not score are excluded from that rate, never counted as failures. Omit testSuiteId when the workspace has exactly one suite.
list_eval_suites
ChatGPTList the workspace's active eval suites, each with the MCP server it tests and the number of eval cases it would actually run. Start here when you need a testSuiteId for run_eval_suite or list_eval_cases. The case count reflects what "Run suite" dispatches, so an archived case is already excluded.
list_issues
ChatGPTList the workspace's issues (recurring problem clusters) ranked by blast radius, with session KPIs (successful / failed / pass rate), daily pass-rate buckets, and each issue's cause category and affected use cases. Mirrors the Issues inbox.
list_mcp_servers
ChatGPTList active production MCP servers configured for the organization, including callable MCP targets and SDK analytics-only registrations. Each row's target_kind distinguishes mcp from analytics_ingest; auth-profile and latest connection-test metadata never include secret refs.
list_sessions
ChatGPTList reconstructed agent sessions for the workspace, newest first, with each session's resolved user identity when recorded, primary use case, flag state, and latest judged outcome. Filter by outcome view, use case, MCP server, and time range. Mirrors the Sessions dashboard.
list_use_cases
ChatGPTList the use-case clusters reconstructed from the workspace's sessions, each with volume, pass rate, and share of traffic. Before the classifier has promoted clusters, falls back to raw intent grouping (fallback: "raw_intents"). Mirrors the Use cases dashboard.
run_eval_case
ChatGPTRun one eval case now against its configured tester target, producing a single individual run. Returns the run id immediately — the run executes asynchronously, so poll get_eval_case for the outcome. Use run_eval_suite instead when you want a ship-readiness answer for a whole suite. Requires the editor, admin, or owner role.
run_eval_suite
ChatGPTStart one suite run over every eval case in a suite — the same action as "Run suite" in the Armature dashboard, and the check to run before shipping an MCP change. The suite membership is snapshotted at dispatch, so adding or removing a case afterwards marks this result historical rather than leaving the MCP silently green. Returns the suite run id and how many cases were queued; the individual runs execute asynchronously, so poll get_suite_run for pass/fail. Requires the editor, admin, or owner role.
send_feedback
ChatGPTSend feedback to the developers of this server: a feature request, or something the user asked for that these tools can't do. The user can also ask you to send feedback directly.
