Developer Tools

Openlayer

by Openlayer

Overview

Openlayer is the AI evaluation and observability platform. This connector gives Claude access to your Openlayer workspace so you can check how your AI systems are doing and fix problems without leaving the conversation. Ask Claude to list projects and inference pipelines, check whether a data source is receiving traffic, pull recent traces and sessions, and summarize test results for a commit or a production pipeline. When a test fails, Claude can fetch the rows that caused the failure, along with the judge explanations, and help you work out why. For governance, Claude can summarize workspace rule status, list failing rule results, and show the evidence behind them. Claude can also create and update projects, pipelines, tests, rules, and frameworks, and run evaluations. Write and delete tools always ask before they run. Each connection is scoped to one workspace you pick when you sign in. Requires an Openlayer account.

Tools

add_rule_result_evidence

ChatGPT

Attach evidence to a rule result (only for evidence-type rules).

create_framework

ChatGPT

Create a custom compliance/governance framework in a workspace.

create_inference_pipeline

ChatGPT

Create a new inference pipeline in a project.

create_presigned_upload_url

ChatGPT

Get a URL to upload a commit bundle to. Used instead of push_commit on the shared remote server, which can't read a directory on your machine. Tar your project directory yourself, then multipart-POST it to the returned url: include every entry of fields as a form field, and add the tar as a file field LAST in the form (required by S3). Then call finalize_commit with the returned storage_uri.

create_project

ChatGPT

Create a new project in the user's workspace.

create_rule

ChatGPT

Create a governance rule in a workspace.

create_test

ChatGPT

Create a test (goal) in a project.

delete_rule

ChatGPT

Delete a governance rule (custom rules only; built-in rules cannot be deleted).

delete_test

ChatGPT

Permanently delete a test and all of its results. This cannot be undone. Prefer `update_test(test_id, archived=True)` unless the test should truly disappear: archiving hides the test while preserving its result history, and is reversible. Deleting is appropriate for cleaning up a test that was created by mistake.

evaluate_data_source

ChatGPT

Trigger evaluation of every monitoring test on a data source over a time range. The pipeline-wide counterpart to `evaluate_test

, which runs one test at a time. Useful for backfilling results after adding tests or after late data arrives. Runs asynchronously: the response carries a background-task ID and a status URL. The requested timestamps are aligned to the tests' evaluation-window boundaries during execution, so the window actually evaluated may be slightly wider than the one requested. Unlike

evaluate_test, this route takes no overwrite flag -- use evaluate_test` when you need to overwrite existing results for one test.

evaluate_test

ChatGPT

Trigger a one-off evaluation of a monitoring test over a time range.

fetch_data_source_rows

ChatGPT

Fetch rows (traces / inference data points) from a data source. Use this to inspect the structure and contents of production traces -- e.g. to understand which columns exist before proposing tests. Rows come back untrimmed, so they include each test's judge score (`openlayer_metric_score_* / openlayer_llm_rubric_score_*) alongside the judge's explanation in the matching *_explanation_* column. To go the other way -- from a failing test to the rows that caused it -- use fetch_failed_rows_for_goal`.

fetch_data_source_sessions

ChatGPT

Fetch session-level aggregations for a data source. Sessions group related traces/records; use this to analyze conversation or request sessions before proposing session-level tests.

fetch_failed_rows_for_goal

ChatGPT

Fetch the rows that made a test fail, with per-row judge scores and the LLM judge's explanation (reasoning) for each score. This is how you answer "why did this test fail on this row?" -- each returned row carries a `judge_scores list of {score_column, score, test_name, explanation}, where explanation` is the judge's written justification, the same text the UI shows in the record detail panel. Rows are selected using a 3-tier filter strategy: 1) Prefer pre-computed rowsBody.columnFilters from the goal result (update measurement names with llmSettingHash if available; invert operators to target failed rows). 2) Otherwise construct aggregate metric filters (latency/cost/tokens).

finalize_commit

ChatGPT

Create a commit from a bundle already uploaded via create_presigned_upload_url.

get_data_source_health

ChatGPT

Check whether a data source is still receiving data, and how much. Returns when the last sample arrived plus request-volume series. Use it first when a user reports missing data or asks whether an integration is still working: a stale `dateLastSampleReceived

 says the problem is upstream of Openlayer, not in their tests. The volume series come in fixed hourly/daily/weekly/monthly windows computed by the backend -- the route accepts no time range, so an arbitrary window (say, one specific day last week) cannot be requested here. Use

fetch_data_source_rows` with timestamp filters for that.

get_data_source_row

ChatGPT

Fetch one row (trace) from a data source by inference ID, with its OTel steps. Direct lookup for when you already have an inference ID -- from a user, a log, or a previous search -- so you don't have to build a filter to find one trace.

get_data_source_session

ChatGPT

Get the aggregation for one session in a data source. The drill-down for `fetch_data_source_sessions`: returns the session's record count, token usage, cost, duration and scores.

get_data_source_session_summary

ChatGPT

Read the cached AI summary of a session, plus a staleness flag. A whole conversation compressed into a paragraph -- far cheaper to reason over than paging the raw turns. Read-only: it never triggers generation. Summary generation is gated by workspace AI-feature settings and is deliberately not exposed as a tool, so a session with no summary yet reports that explicitly rather than looking like an empty summary. `stale` is true when the session has changed since the summary was written.

get_data_source_user

ChatGPT

Get the aggregation for one user of a data source. Use when you already have a user ID and want that user's session count, record count, token usage and cost without paging the whole cohort.

get_framework_project_rule_stats

ChatGPT

Get per-project rule pass/fail stats for a framework.

get_rule_result_evidence

ChatGPT

Get the evidence attached to a rule result.

get_test_results

ChatGPT

List results for a test.

get_workspace_rule_stats

ChatGPT

Get aggregate rule-result stats for a workspace (compliance summary).

list_commit_test_results

ChatGPT

List the test results for a commit. Each result is shaped to the essentials: test_name, status, threshold, the measured value, and goal_id -- pass that goal_id to `fetch_failed_rows_for_goal` to drill into the failing rows and read the LLM judge's per-row scores and explanations.

list_commits

ChatGPT

List the commits in a project.

list_data_source_user_sessions

ChatGPT

List one user's sessions in a data source. The cohort investigation path: `list_data_source_users` surfaces a user whose cost or latency stands out, then this shows that user's sessions.

list_data_source_users

ChatGPT

List per-user activity aggregations for a data source. Returns users who interacted with the data source along with session counts, record counts, token usage, and costs -- useful for cohort/session analysis.

list_framework_documents

ChatGPT

List the documents (e.g. the underlying standard/policy) for a framework.

list_framework_rules

ChatGPT

List the rules that belong to a framework.

list_frameworks

ChatGPT

List the compliance/governance frameworks in a workspace.

list_inference_pipeline_test_results

ChatGPT

List the test results for an inference pipeline (data source). Each result is shaped to the essentials: test_name, status, threshold, the measured value, and goal_id -- pass that goal_id to `fetch_failed_rows_for_goal` to drill into the failing rows and read the LLM judge's per-row scores and explanations.

list_inference_pipelines

ChatGPT

List the inference pipelines in a project.

list_projects

ChatGPT

Get a list of the projects in the user's workspace.

list_rule_results

ChatGPT

List the per-project results for a rule.

list_rule_tags

ChatGPT

List the rule tags defined in a workspace. Tags categorize rules (e.g. by topic or control family); use them when composing a framework or filtering rules.

list_rules

ChatGPT

List the governance rules in a workspace.

list_test_result_sessions

ChatGPT

List the sessions behind a session-level test result, with their scores. Use this to answer "why did this session test fail?". Each session comes back enriched with the test's session-metric score, its explanation, and pass/fail status. This is the session-level counterpart to `fetch_failed_rows_for_goal, which only reaches row-level failures. A null

 score does not necessarily mean the session was unscored -- it depends on the data source's backend. ClickHouse-native data sources return only sessions that have a recorded score in the result's evaluation window; other backends return every session in the window, with

score` null where the metric had no resolved value.

list_tests

ChatGPT

List the tests defined in a project. Returns a summary per test (name, type, subtype, a one-line threshold, dataset usage). The full threshold/rubric is intentionally omitted here to save tokens.

list_workspace_rule_results

ChatGPT

List rule results across a workspace (the compliance posture).

list_workspaces

ChatGPT

List the workspaces you belong to, and which one this connection uses. This connection can only act on its own workspace. To work in another one, add a second Openlayer MCP connection and pick that workspace when authorizing it.

retrieve_commit

ChatGPT

Retrieve a commit by ID.

retrieve_framework

ChatGPT

Retrieve a framework by ID.

retrieve_framework_document

ChatGPT

Retrieve a framework document's content. The full document can be large (tens of thousands of tokens). Set `sections_only=True` to get just the section titles (the outline), then fetch the full content only when needed.

retrieve_inference_pipeline

ChatGPT

Retrieve an inference pipeline by ID.

retrieve_rule

ChatGPT

Retrieve a rule by ID.

retrieve_rule_result

ChatGPT

Retrieve a single rule result by ID.

retrieve_test

ChatGPT

Retrieve one test in full, including its complete thresholds and rubric. `list_tests

 deliberately summarizes thresholds to one line to save tokens, so this is the way to see what a test actually asserts -- the full threshold list, and for LLM-rubric tests the evaluation prompt itself. Read this before calling

update_test`, so you preserve the fields you don't mean to change.

search_openlayer_docs

ChatGPT

Search the Openlayer documentation for relevant information. Use as a last resort when other tools don't provide what's needed. Returns a list of results, each with 'title', 'link', and 'content'.

update_data_source_row

ChatGPT

Patch fields onto an existing production row -- e.g. backfill ground truth. The main use is labeling: tests that compare predictions against ground truth cannot evaluate rows where the ground-truth column is empty, so filling it in is what makes those rows measurable. This modifies live production data. Only the keys present in `row are written; other columns are left alone. Confirm the target with get_data_source_row` first.

update_framework

ChatGPT

Update a framework -- e.g. activate or deactivate it. Set `enabled=True to activate a framework for the workspace (or False to deactivate). For built-in frameworks only enabled, tags, and project_selector` are mutable.

update_project

ChatGPT

Update an existing project's metadata. Only the fields you pass are changed; anything omitted is left alone. Use this to rename or re-describe a project after it has been created. Git repo, ownership, and Slack alert routing are deliberately not updatable here -- change those in the web app.

update_rule

ChatGPT

Update a governance rule (built-in rules cannot be renamed).

update_rule_result

ChatGPT

Update a rule result -- e.g. deactivate/reactivate it or assign an owner.

update_test

ChatGPT

Update an existing test: rename it, retune its thresholds, archive or pause it. This is how you refine a test rather than creating a near-duplicate. Only the arguments you pass are sent, but `thresholds replaces the whole list, so read the current value with retrieve_test` and send the full edited list -- not just the entry you changed.

add_rule_result_evidence

Claude

create_framework

Claude

create_inference_pipeline

Claude

create_presigned_upload_url

Claude

create_project

Claude

create_rule

Claude

create_test

Claude

delete_rule

Claude

delete_test

Claude

evaluate_data_source

Claude

evaluate_test

Claude

fetch_data_source_rows

Claude

fetch_data_source_sessions

Claude

fetch_failed_rows_for_goal

Claude

finalize_commit

Claude

get_data_source_health

Claude

get_data_source_row

Claude

get_data_source_session

Claude

get_data_source_session_summary

Claude

get_data_source_user

Claude

get_framework_project_rule_stats

Claude

get_rule_result_evidence

Claude

get_test_results

Claude

get_workspace_rule_stats

Claude

list_commit_test_results

Claude

list_commits

Claude

list_data_source_user_sessions

Claude

list_data_source_users

Claude

list_framework_documents

Claude

list_framework_rules

Claude

list_frameworks

Claude

list_inference_pipeline_test_results

Claude

list_inference_pipelines

Claude

list_projects

Claude

list_rule_results

Claude

list_rule_tags

Claude

list_rules

Claude

list_test_result_sessions

Claude

list_tests

Claude

list_workspace_rule_results

Claude

list_workspaces

Claude

retrieve_commit

Claude

retrieve_framework

Claude

retrieve_framework_document

Claude

retrieve_inference_pipeline

Claude

retrieve_rule

Claude

retrieve_rule_result

Claude

retrieve_test

Claude

search_openlayer_docs

Claude

update_data_source_row

Claude

update_framework

Claude

update_project

Claude

update_rule

Claude

update_rule_result

Claude

update_test

Claude

Capabilities

Writes

Example Prompts

Click any prompt to copy it.

Links

Documentation ↗Privacy Policy ↗

App Stats

110

Tools

Prompts

ChatGPT, Claude

Platforms