create_benchmark
ClaudeCreate a benchmark for a project.
Developer-Tools
Overview
Bencher is a continuous benchmarking platform that helps systems software engineers catch performance regressions before they merge. How Bencher Works: 1. Run: Run your benchmarks locally or in CI using the exact same bare metal runners and your favorite benchmarking tools. The bencher CLI orchestrates running your benchmarks on bare metal and stores the results. 2. Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure. 3. Catch: Catch performance regressions locally or in CI using the exact same bare metal hardware. Bencher uses state of the art, customizable analytics to detect performance regressions before they merge. For the same reasons that unit tests are run to prevent feature regressions, benchmarks should be run with Bencher to prevent performance regressions. Performance bugs are bugs!
Tools
create_branch
ClaudeCreate a branch for a project.
create_measure
ClaudeCreate a measure for a project.
create_report
ClaudeCreate a report from raw benchmark results for an existing branch and testbed. Prefer
submit_run unless the branch and testbed are already known to exist.create_testbed
ClaudeCreate a testbed for a project.
create_threshold
ClaudeCreate a threshold for a project: a branch, testbed, and measure with a statistical model used to detect performance regressions.
parameters is the filter over variants, and a variant is checked when any entry in the filter is a subset of its parameters; the default checks every variant. metric names the metric the threshold checks and defaults to value. Every threshold that matches a metric row runs, so a row may earn several boundaries and several alerts.list_alerts
ClaudeList performance alerts for a project, generated when a threshold boundary is exceeded.
list_benchmarks
ClaudeList benchmarks for a project.
list_branches
ClaudeList branches for a project.
list_jobs
ClaudeList bare metal runner jobs for a project.
list_measures
ClaudeList measures for a project (e.g. latency, throughput).
list_plots
ClaudeList saved dashboard plots for a project.
list_projects
ClaudeList projects. With a project API key (
bencher_run_), only that project is returned; with a user API key, all projects visible to the user are returned.list_reports
ClaudeList reports for a project, sorted by date time in reverse chronological order by default. Results and alerts are collapsed to counts unless
expand is set to true.list_testbeds
ClaudeList testbeds for a project.
list_thresholds
ClaudeList thresholds for a project. A threshold is a branch, testbed, and measure with a statistical model, checking a filter over the benchmark's variants and one metric name.
perf_image
ClaudeRender a perf plot image (JPEG) for a project. Takes the same query arguments as
query_perf, plus an optional title.query_perf
ClaudeQuery benchmark metrics over time for a project. This is the primary tool for investigating performance history and regressions.
branches, testbeds, benchmarks, and measures are comma-separated lists of UUIDs; use the corresponding list tools to find them. start_time and end_time bound the results in milliseconds since epoch.submit_run
ClaudeSubmit benchmark results to create a run. The
results field takes the raw output of a benchmark harness as strings; the server parses them using the adapter in settings.adapter (defaults to magic auto-detection). Branch, testbed, and thresholds are created on the fly if they do not exist. Prefer the bencher run CLI when benchmarks are executed from a local checkout; use this tool when the results are already in hand.update_alert
ClaudeUpdate an alert. Project API keys (
bencher_run_) can only update the alert status (e.g. acknowledge or dismiss).update_benchmark
ClaudeUpdate a benchmark. Project API keys (
bencher_run_) cannot rename benchmarks.update_branch
ClaudeUpdate a branch. Project API keys (
bencher_run_) cannot rename branches.update_measure
ClaudeUpdate a measure. Project API keys (
bencher_run_) cannot rename measures.update_testbed
ClaudeUpdate a testbed. Project API keys (
bencher_run_) cannot rename testbeds.update_threshold
ClaudeUpdate a threshold's statistical model, or remove the model by setting it to null.
view_alert
ClaudeView an alert, including the boundary that was exceeded. The
alert argument is an alert UUID.view_benchmark
ClaudeView a benchmark. The
benchmark argument accepts a benchmark slug or UUID.view_branch
ClaudeView a branch. The
branch argument accepts a branch slug or UUID.view_job
ClaudeView a bare metal runner job. The
job argument is a job UUID.view_measure
ClaudeView a measure. The
measure argument accepts a measure slug or UUID.view_metric
ClaudeView a single metric with its full context: report, branch, testbed, benchmark, measure, and any threshold boundary. The
metric argument is a metric UUID.view_plot
ClaudeView a saved dashboard plot. The
plot argument is a plot UUID.view_project
ClaudeView a project. The
project argument accepts a project slug or UUID.view_report
ClaudeView a report, including its full results and any alerts.
view_testbed
ClaudeView a testbed. The
testbed argument accepts a testbed slug or UUID.view_threshold
ClaudeView a threshold. The
threshold argument is a threshold UUID.
