DeepEval MCP server: hosted, broad, and able to delete
The evaluation platform behind DeepEval is reachable over MCP. The surface is wide, and the delete tools are the part to read first.
Last verified 31 August 2026 · from Confident AI's own MCP documentation, a direct handshake against both documented endpoints, and the GitHub API
This page is one of 90 in a directory of monitoring and developer tools. Each page checks what an AI coding agent can get out of the tool through its MCP server, the connector that lets the agent query the tool directly.
Summary
Confident AI hosts the DeepEval MCP server for the platform built around the library. It exposes 75 tools across 13 areas, and needs nothing installed: clients point at https://mcp.confident-ai.com/mcp and complete an OAuth flow in the browser.
The reach is the platform, not the library. Prompts, datasets, goldens (the individual test cases a dataset holds), traces, threads, spans, annotations, annotation queues, metrics, dashboards and risk assessments are all addressable. DeepEval the Python framework is still where metrics execute.
The surface writes as well as reads. Confident AI's own documentation warns that several of these tools delete data permanently and names which: datasets, goldens, prompt branches, dashboards, widgets and annotation queues.
What is the DeepEval MCP server?
DeepEval is an open-source evaluation framework for large-language-model applications, published by Confident AI. It runs metrics such as faithfulness, relevance and hallucination detection against test cases.
Confident AI is the hosted platform built around it, holding evaluation runs, datasets, prompts and traces. That platform is what the MCP server exposes.
An open-source server repository was announced in March 2026 and no longer resolves. What shipped instead is hosted and first-party, which is why a search of the GitHub organization finds nothing while the product is live.
Confident AI sits in agent observability, and the connector reaches most of what the platform holds.
| Part of the work | What DeepEval has here | Can an agent reach it |
|---|---|---|
| Code, tests & review | Evaluation runs, datasets and goldens | Partial. run_llm_evals and simulate_conversation start evaluations, and the dataset and golden tools manage the cases they run against |
| CI & release | None | No pipeline product and no release tools |
| Production observability | None | No infrastructure telemetry. The tracing here is of model calls, not of hosts or services |
| Agent observability & evals | Traces, threads, spans, annotations, metrics and dashboards | Yes. Nine tools cover traces, threads and spans, with evaluate_trace, evaluate_thread and evaluate_span scoring them in place |
| AI cost management | None | No spend or billing tools are documented |
The connector is strongest where the platform is: scoring and inspecting agent runs after the fact. It does not reach infrastructure telemetry, and it does not replace running DeepEval in Python, which is still where a metric is defined and executed.
What can an AI agent do with DeepEval?
| Task | Tools | Works | Watch out for |
|---|---|---|---|
| Score an existing trace | get_trace evaluate_trace | Yes | evaluate_trace writes: the score lands on the platform record |
| Inspect a thread or span | list_threads get_thread get_span | Yes | Read tools. The tracing here is of model calls, not hosts or services |
| Pull a dataset and its goldens | pull_dataset get_golden | Yes | delete_dataset and delete_golden sit in the same area and delete permanently |
| Start an evaluation run | run_llm_evals simulate_conversation | Yes | Both are writes; they start evaluations rather than define metrics |
| Manage prompt versions and branches | pull_prompt list_prompt_versions push_prompt | Yes | delete_prompt_branch in the same area deletes for good |
| Run a metric definition | No | Metrics are defined and executed in DeepEval the library, not over this server |
75 tools documented across 13 areas, all available once connected
- Projects1
list_projects- Prompts6
list_promptspull_promptinterpolate_promptlist_prompt_versionslist_prompt_commitslist_prompt_branches- Prompts, write5
push_promptcreate_prompt_versioncreate_prompt_branchupdate_prompt_branchdelete_prompt_branch- Datasets4
pull_datasetlist_datasetslist_dataset_versionsget_golden- Datasets, write7
push_datasetdelete_datasetcreate_dataset_versioncreate_goldenupdate_goldendelete_goldenqueue_goldens- Evaluate2Write
run_llm_evalssimulate_conversation- Traces, threads and spans6
list_tracesget_tracelist_threadsget_threadlist_spansget_span- Scoring in place3Write
evaluate_traceevaluate_threadevaluate_span- Annotations2
list_annotationsget_annotation- Annotations, write2
create_annotationupdate_annotation- Annotation queues4
list_annotation_queuesget_annotation_queuelist_annotation_queue_itemsget_next_annotation_queue_item- Annotation queues, write6
create_annotation_queueupdate_annotation_queuedelete_annotation_queueadd_items_to_annotation_queueannotate_queue_itembatch_annotate_queue_items- Test runs2
list_test_runsget_test_run- Metrics3
list_metricsget_metriclist_metric_data- Metrics, write3
create_metricbatch_create_metricsupdate_metric- Metric collections1
list_metric_collections- Metric collections, write2
create_metric_collectionupdate_metric_collection- Dashboards5
list_dashboardsget_dashboardquery_dashboardquery_dashboard_widgetpreview_widget- Dashboards, write6
create_dashboardupdate_dashboarddelete_dashboardadd_dashboard_widgetupdate_dashboard_widgetdelete_dashboard_widget- Risk assessments1
list_risk_assessment_frameworks- Risk assessments, write2
run_risk_assessment_frameworkcreate_risk_assessment- Governance2
list_ai_connectionsassess_governance
Areas and tool names are Confident AI's own, from its MCP documentation page. The write marks are read off the tool names and the documentation's deletion warning, not from server-side readOnlyHint annotations, which the documentation does not publish. No count was captured from tools/list, because the endpoint requires an authenticated OAuth session.
Can DeepEval watch your own AI agents?
Scoring agent runs is what this platform is for, so these signals are the fair test. Each row below is read from the 75 documented tools, retrieved from Confident AI's documentation on 31 August 2026.
| Signal | Covered | Notes |
|---|---|---|
| Eval scores on recorded traffic | Yes | evaluate_trace, evaluate_thread and evaluate_span score a stored run in place. All three write the score back to the platform record |
| Trace and session tracking | Yes | Six read tools cover the hierarchy: list_traces and get_trace, list_threads and get_thread, list_spans and get_span. A thread is the multi-turn session |
| Prompt version tracking | Yes | Versions, commits and branches each have their own read tool, and interpolate_prompt renders one with variables filled in |
| Offline experiments | Yes | run_llm_evals starts an evaluation and simulate_conversation generates one. Test runs are readable afterwards with list_test_runs and get_test_run |
| Human review of agent output | Yes | Annotation queues are a first-class surface, with ten tools covering queue creation, item assignment and single or batch annotation |
| Token and cost tracking | No | No spend or billing tool is documented among the 75, and the cost stage says so |
| Agent trace joins the infrastructure trace | No | Nothing reaches logs, metrics or spans from your own systems. The traces here are the ones Confident AI recorded |
DeepEval MCP server limits: where answers come back incomplete
Two things to watch: one about what an agent can destroy, one about a repository that no longer exists.
An agent removed a dataset, golden, branch, dashboard or queue and it did not come back.
Confident AI's documentation states plainly that several tools delete data permanently, naming datasets, goldens, prompt branches, dashboards, widgets and annotation queues.
There is no documented server-side read-only mode. delete_dataset, delete_golden, delete_prompt_branch, delete_dashboard, delete_dashboard_widget and delete_annotation_queue are all in the surface an agent gets on connection.
Guard: Allowlist the read tools in your client before connecting, or connect a project that holds nothing you cannot rebuild. The permanence is the vendor's own word, not an inference from the tool names.
An agent or a guide points at confident-ai/confident-mcp-server, which returns 404.
An open-source server was announced on 7 March 2026 and described as open-sourced. That repository returned HTTP 404 on 31 August 2026, and the confident-ai organization carries no MCP repositories.
The working route is hosted rather than installed, so an organization search finds nothing while the product is live. A dead repository is not evidence that a product is gone.
Guard: Reach for the documented endpoint, not the announcement. An unauthenticated POST to https://mcp.confident-ai.com/mcp returns HTTP 401, which is how you tell a gated server from an absent one.
Do you need the MCP server at all?
Not for defining or running metrics. DeepEval the library remains the place those happen. An agent with Python execution can score test cases directly, with no protocol in between, and that is how most teams already run it.
Where a second evaluation surface is wanted, Braintrust documents 34 tools including create_evaluator, test_evaluator and run_eval, and returns aggregated experiment metrics against a baseline.
Arize Phoenix covers datasets and experiments across 27 tools with prompt versioning alongside. Keywords AI, trading as Respan, exposes nine evaluator tools.
For tracing rather than scoring, Langfuse and Traceloop both reach model-call spans, which is the other half of the picture evaluation is usually paired with.
Connect Confident AI to inspect and score runs that already exist. Keep DeepEval in Python for defining and executing metrics.
DeepEval MCP server setup
Nothing is installed. The server is hosted, and clients reach it over HTTP at the regional endpoint for your account.
Authentication is OAuth. The first time a client connects, a browser window opens to sign in to Confident AI and approve the connection; later sessions refresh automatically.
Claude Code
Use the EU host instead if your account is in that region.
No key goes on the command line. The OAuth window opens the first time the client connects.
claude mcp add --transport http confident-ai \ https://mcp.confident-ai.com/mcp
Cursor
In ~/.cursor/mcp.json. The OAuth flow opens on first connection.
{"mcpServers":{"confident-ai":{
"url":"https://mcp.confident-ai.com/mcp"}}}Codex CLI
Codex reads TOML, so the JSON blocks above will not transfer.
The entry carries a URL and no token. Codex hands the OAuth sign-in to the browser.
[mcp_servers.confident-ai] url = "https://mcp.confident-ai.com/mcp"
Every other client
Each block below is the configuration for one client, with the file path and the key that client expects.
Claude Desktop
macOS ~/Library/Application Support/Claude/claude_desktop_config.json. Windows %APPDATA%\Claude\claude_desktop_config.json. There is no CLI. Edit through Settings, Developer, Edit Config.
Quit and restart fully for changes to load.
Put no credential in this file. Approve the OAuth prompt in the browser after the restart.
{"mcpServers":{"confident-ai":{"url":"https://mcp.confident-ai.com/mcp"}}}VS Code Copilot
.vscode/mcp.json per workspace, or your user profile.
The block holds the URL alone. Copilot starts the OAuth sign-in when it first reaches the server.
{"servers":{"confident-ai":{"url":"https://mcp.confident-ai.com/mcp"}}}The key is `servers`, not `mcpServers`. VS Code is the only client that uses that name, and copying a config from anywhere else fails silently.
EU and self-hosted
EU accounts use https://eu.mcp.confident-ai.com/mcp. A self-hosted deployment exposes /mcp on its own host. Both were documented on 31 August 2026, and the EU endpoint returned HTTP 401 to an unauthenticated request, the same as US.
Read-only access, permissions and security
Deletes are permanent, and the vendor says so
The documentation states that several tools delete data permanently, naming datasets, goldens, prompt branches, dashboards, widgets and annotation queues.
No documented read-only mode
Nothing in the published documentation describes a server-side flag or a read-only endpoint variant, so restriction has to happen in the client.
OAuth binds the session to a person
Actions arrive as the account that approved the connection, so an agent's writes are attributed to whoever signed in.
DeepEval MCP server: Reference
| Item | Value |
|---|---|
| Server published | Yes, hosted and first-party |
| US endpoint | https://mcp.confident-ai.com/mcp |
| EU endpoint | https://eu.mcp.confident-ai.com/mcp |
| Transport | HTTP |
| Authentication | OAuth, browser sign-in on first connection |
| Tools | 75 across 13 areas |
| Largest areas | Prompts 11 · Datasets 11 · Dashboards 11 · Annotation queues 10 |
| Deleting tools | Datasets, goldens, prompt branches, dashboards, widgets, annotation queues |
| Read-only mode | None documented |
confident-ai/confident-mcp-server | HTTP 404 on 31 August 2026 |
MCP repositories in confident-ai | 0 |
| DeepEval stars | 17,990 |
| DeepEval last push | 30 August 2026 |
| Checked | 31 August 2026 |
What engineers report
DeepEval carries 17,990 stars and was pushed on 30 August 2026. The server built around it is hosted, which is why a repository search finds nothing while the product answers on two continents.
| What was checked | What it shows |
|---|---|
| Tools documented | 75 |
| Areas | 13 |
| Endpoints documented | 3, counting self-hosted |
Unauthenticated initialize against US | HTTP 401 |
Unauthenticated initialize against EU | HTTP 401 |
confident-ai/confident-mcp-server | HTTP 404 |
Both documented endpoints were sent an unauthenticated JSON-RPC initialize on 31 August 2026 and each returned HTTP 401 with an unauthorized JSON body, which establishes a gated server rather than an absent one. The confident-ai GitHub organization was searched for MCP repositories the same day and returned zero, and confident-ai/confident-mcp-server returned HTTP 404.
Should you connect an agent to DeepEval?
Yes. Best for inspecting and scoring existing runs, with the deleting tools allowlisted out.
- Strongest fit
- Teams already on Confident AI who want an agent to read traces, score them in place and manage the datasets behind an evaluation.
- Main advantage
- Hosted and broad: 75 tools reach most of the platform with nothing to install.
- Main weakness
- No documented read-only mode, on a surface with six deleting tools the vendor calls permanent.
- Operational risk
- An agent with the default surface can delete a dataset, a golden, a prompt branch, a dashboard, a widget or an annotation queue for good.
From Oodle
One platform for agent traces and infrastructure
Agent traces usually sit in a different product from the rest of your telemetry, so when a slow database makes an agent slow the symptom is in one tool and the cause is in another. Oodle keeps both in one query surface, from $10 per million spans.
See agent observabilityFrequently asked questions
Does DeepEval have an MCP server?
Yes. Confident AI, the platform behind DeepEval, hosts one at https://mcp.confident-ai.com/mcp with an EU endpoint alongside. It exposes 75 tools across 13 areas.
Why do people say there isn't one?
Because confident-ai/confident-mcp-server returns HTTP 404 and the GitHub organization carries no MCP repositories. The server shipped hosted rather than open-source, so a repository search finds nothing while the product is live.
Can an agent delete my data?
Yes. Confident AI's documentation states that several tools delete permanently and names datasets, goldens, prompt branches, dashboards, widgets and annotation queues. No server-side read-only mode is documented, so restrict the tools in your client.
Does this replace running DeepEval in Python?
No. Metrics are still defined and executed in the library. The server reaches the hosted platform around it: runs, datasets, prompts, traces and dashboards.
How do I know the endpoint is real without an account?
Send an unauthenticated initialize to it. A live gated server answers HTTP 401 with a JSON error body, which is what both documented endpoints returned on 31 August 2026. A missing server does not resolve at all.
