DeepEval MCP server: hosted, broad, and able to delete
The evaluation platform behind DeepEval is reachable over MCP. The surface is wide, and the delete tools are the part to read first.
Checked 6 October 2026 · from Confident AI's own MCP documentation, a direct handshake against both documented endpoints, and the GitHub API
One of 91 guides to monitoring and developer tools. Each covers what an AI coding agent can get out of the tool through MCP, the protocol that lets the agent query the tool directly.
Summary
Confident AI hosts the DeepEval MCP server for the platform built around the library. It exposes 202 tools across 34 areas, and needs nothing installed: clients point at https://mcp.confident-ai.com/mcp and complete an OAuth flow in the browser.
The reach is the platform around the library. Prompts, datasets, goldens (the individual test cases a dataset holds), traces, threads, spans, annotations, annotation queues, metrics, dashboards, reports, red-teaming frameworks and governance policies are all addressable. DeepEval the Python framework is still where metrics execute.
The surface writes as well as reads. Confident AI's tools reference describes 26 tools that delete permanently, including datasets, goldens, dashboards, reports, personas and governance controls.
What is the DeepEval MCP server?
DeepEval is an open-source evaluation framework for large-language-model applications, published by Confident AI. It runs metrics such as faithfulness, relevance and hallucination detection against test cases.
Confident AI is the hosted platform built around it, holding evaluation runs, datasets, prompts and traces. That platform is what the MCP server exposes.
An open-source server repository was announced in March 2026 and no longer resolves. What shipped instead is hosted and first-party, which is why a search of the GitHub organization finds nothing while the product is live.
Confident AI sits in agent observability, and the connector reaches most of what the platform holds.
| Part of the work | What DeepEval has here | Can an agent reach it |
|---|---|---|
| Code, tests & review | Evaluation runs, datasets and goldens | Partial. run_evals and run_dataset_evaluation start evaluations, and the dataset and golden tools manage the cases they run against |
| CI & release | None | No pipeline product and no release tools |
| Production observability | None | No infrastructure telemetry. The tracing here is of model calls, not of hosts or services |
| Agent observability & evals | Traces, threads, spans, annotations, metrics and dashboards | Yes. Nine tools cover traces, threads and spans, with evaluate_trace, evaluate_thread and evaluate_span scoring them in place |
| AI cost management | Model token prices and LLM span cost | Partial. Four Model Costs tools set per-model token prices, and dashboard widgets total the cost of LLM spans. Platform billing is out of reach |
The connector is strongest where the platform is: scoring and inspecting agent runs after the fact. It does not reach infrastructure telemetry, and it does not replace running DeepEval in Python, which is still where a metric is defined and executed.
What can an AI agent do with DeepEval?
| Task | Tools | Works | Watch out for |
|---|---|---|---|
| Score an existing trace | get_trace evaluate_trace | Yes | evaluate_trace writes: the score lands on the platform record |
| Inspect a thread or span | list_threads get_thread get_span | Yes | Read tools. The tracing here is of model calls, not hosts or services |
| Pull a dataset and its goldens | pull_dataset get_golden | Yes | delete_dataset and delete_golden sit in the same area and delete permanently |
| Start an evaluation run | run_evals run_dataset_evaluation | Yes | Both are writes; they start evaluations rather than define metrics |
| Manage prompt versions and branches | list_prompts get_prompt_versions push_prompt | Yes | delete_prompt_branch in the same area deletes a branch. Only main and branches with open pull requests are protected |
| Run a metric definition | No | Metrics are defined and executed in DeepEval the library, not over this server |
202 tools documented across 34 areas, all available once connected. 85 read, and 117 write, start a run or call out
- Projects1
list_projects- Prompts7
get_prompt_branchesget_prompt_by_commitget_prompt_by_labelget_prompt_by_versionget_prompt_commitsget_prompt_versionslist_prompts- Prompts, write5
create_prompt_branchcreate_versiondelete_prompt_branchpush_promptupdate_prompt_branch- Datasets6
get_dataset_ingestion_taskget_dataset_versionsget_goldenlist_dataset_ingestion_taskslist_datasetspull_dataset- Datasets, write11
create_dataset_ingestion_taskcreate_dataset_versioncreate_goldendelete_datasetdelete_dataset_ingestion_taskdelete_goldenpush_datasetqueue_dataset_goldensrun_dataset_evaluationupdate_dataset_ingestion_taskupdate_golden- Evaluate4Write
evaluate_spanevaluate_threadevaluate_tracerun_evals- Traces, threads and spans6
get_tracelist_tracesget_threadlist_threadsget_spanlist_spans- Test runs2
get_test_runlist_test_runs- Annotations2
get_annotationlist_annotations- Annotations, write2
create_annotationupdate_annotation- Annotation queues5
get_annotation_queueget_queue_ingestion_tasklist_annotation_queue_itemslist_annotation_queueslist_queue_ingestion_tasks- Annotation queues, write9
add_annotation_queue_itemsannotate_annotation_queue_itembatch_annotate_annotation_queue_itemscreate_annotation_queuecreate_queue_ingestion_taskdelete_annotation_queuedelete_queue_ingestion_taskupdate_annotation_queueupdate_queue_ingestion_task- Metrics3
get_metriclist_metricslist_metric_data- Metrics, write3
create_metricupdate_metriccreate_metrics_batch- Metric collections2
get_metric_collectionlist_metric_collections- Metric collections, write3
create_metric_collectiondelete_metric_collectionupdate_metric_collection- Evaluation rules2
get_evaluation_rulelist_evaluation_rules- Evaluation rules, write3
create_evaluation_ruledelete_evaluation_ruleupdate_evaluation_rule- Dashboards and widgets5
get_dashboardlist_dashboardsquery_dashboardquery_widgetquery_ad_hoc_widget- Dashboards and widgets, write6
create_dashboardcreate_widgetdelete_dashboarddelete_widgetupdate_dashboardupdate_widget- Reports and templates4
get_reportlist_reportsget_report_templatelist_report_templates- Reports and templates, write6
create_reportdelete_reportupdate_reportcreate_report_templatedelete_report_templateupdate_report_template- Scheduled alerts2
get_scheduled_alertlist_scheduled_alerts- Scheduled alerts, write3
create_scheduled_alertdelete_scheduled_alertupdate_scheduled_alert- Exports and forwarding6
get_export_destinationlist_export_destinationsget_export_schedulelist_export_schedulesget_forwarding_connectorlist_forwarding_connectors- Exports and forwarding, write9
create_export_destinationdelete_export_destinationupdate_export_destinationcreate_export_scheduledelete_export_scheduleupdate_export_schedulecreate_forwarding_connectordelete_forwarding_connectorupdate_forwarding_connector- Classifiers4
get_classifierget_classifier_labellist_classifier_labelslist_classifiers- Classifiers, write7
create_classifiercreate_classifier_labeldelete_classifierdelete_classifier_labelgenerate_classifier_labelsupdate_classifierupdate_classifier_label- Red teaming8
get_risk_categoryget_rt_frameworklist_risk_categorieslist_rt_frameworksget_attack_methodlist_attack_methodsget_vulnerabilitylist_vulnerabilities- Red teaming, write12
create_risk_categorycreate_rt_frameworkdelete_risk_categorydelete_rt_frameworkrun_rt_frameworkupdate_risk_categoryupdate_rt_frameworkreset_attack_methodupdate_attack_methodcreate_vulnerabilitydelete_vulnerabilityupdate_vulnerability- Personas2
get_personalist_personas- Personas, write3
create_personadelete_personaupdate_persona- AI connections and transformers4
get_ai_connectionlist_ai_connectionsget_transformerlist_transformers- AI connections and transformers, write8
create_ai_connectiondelete_ai_connectionping_ai_connectionupdate_ai_connectioncreate_transformerdelete_transformertest_transformer_codeupdate_transformer- MCP servers2
get_mcp_serverlist_mcp_servers- MCP servers, write4
connect_mcp_servercreate_mcp_serverdelete_mcp_serverupdate_mcp_server- Model costs1
list_model_costs- Model costs, write3
create_model_costdelete_model_costupdate_model_cost- Governance11
get_governance_controlget_governance_policyget_governance_policy_skillget_governance_projectlist_governance_control_assessmentslist_governance_control_versionslist_governance_controlslist_governance_policieslist_governance_policy_controlslist_governance_policy_projectslist_governance_projects- Governance, write16
assess_governanceassess_governance_controlassess_governance_policyassign_projects_to_governance_policycreate_governance_controlcreate_governance_control_versioncreate_governance_policydelete_governance_controldelete_governance_policydelete_governance_policy_skillremove_governance_policy_controlsunassign_projects_from_governance_policyupdate_governance_controlupdate_governance_policyupdate_governance_policy_controlsupdate_governance_policy_skill
Areas and tool names are Confident AI's own, from its MCP tools reference, read 6 October 2026. Small related areas are merged here. The write marks are read off each tool's name and description, and cover tools that change state, start a run or call an outside service. The reference publishes no readOnlyHint annotations. The endpoint requires an authenticated OAuth session, so the count is the documentation's own rather than a tools/list capture.
Can DeepEval watch your own AI agents?
Scoring agent runs is what this platform is for, so these signals are the fair test. Each row below is read from the 202 documented tools, retrieved from Confident AI's tools reference on 6 October 2026.
| Signal | Covered | Notes |
|---|---|---|
| Eval scores on recorded traffic | Yes | evaluate_trace, evaluate_thread and evaluate_span score a stored run in place. All three write the score back to the platform record |
| Trace and session tracking | Yes | Six read tools cover the hierarchy: list_traces and get_trace, list_threads and get_thread, list_spans and get_span. A thread is the multi-turn session |
| Prompt version tracking | Yes | Versions, commits and branches each have their own read tool. get_prompt_by_version, get_prompt_by_label and get_prompt_by_commit fetch one exact prompt |
| Offline experiments | Yes | run_evals scores a batch of test cases and run_dataset_evaluation scores a dataset's finalized goldens. Test runs are readable afterwards with list_test_runs and get_test_run |
| Human review of agent output | Yes | Annotation queues are a first-class surface, with 14 tools covering queues, ingestion tasks, item assignment and single or batch annotation |
| Token and cost tracking | Partial | Dashboard widgets total TOTAL_COST over LLM spans, and the Model Costs tools set the token prices behind it. Platform billing is out of reach |
| Agent trace joins the infrastructure trace | No | Nothing reaches logs, metrics or spans from your own systems. The traces here are the ones Confident AI recorded |
DeepEval MCP server limits: where answers come back incomplete
Two things to watch: one about what an agent can destroy, one about a repository that no longer exists.
An agent removed a dataset, golden, branch, dashboard or queue and it did not come back.
Confident AI's tools reference documents 29 delete tools and describes 26 of them as permanent. They reach datasets, goldens, dashboards, reports, personas, classifiers, governance controls and more.
There is no documented server-side read-only mode. delete_dataset, delete_golden, delete_dashboard, delete_report and delete_governance_policy are all in the surface an agent gets on connection. The tools reference lists the full set.
Guard: Allowlist the read tools in your client before connecting, or connect a project that holds nothing you cannot rebuild. The permanence is the vendor's own word, not an inference from the tool names.
An agent or a guide points at confident-ai/confident-mcp-server, which returns 404.
An open-source server was announced on 7 March 2026 and described as open-sourced. That repository returned HTTP 404 on 31 August 2026, and the confident-ai organization carries no MCP repositories.
The working route is hosted rather than installed, so an organization search finds nothing while the product is live. A dead repository is not evidence that a product is gone.
Guard: Reach for the documented endpoint, not the announcement. An unauthenticated POST to https://mcp.confident-ai.com/mcp returns HTTP 401, which is how you tell a gated server from an absent one.
Do you need the MCP server at all?
Not for defining or running metrics. DeepEval the library remains the place those happen. An agent with Python execution can score test cases directly, with no protocol in between, and that is how most teams already run it.
Where a second evaluation surface is wanted, Braintrust documents 34 tools including create_evaluator, test_evaluator and run_eval, and returns aggregated experiment metrics against a baseline.
Arize Phoenix covers datasets and experiments across 27 tools with prompt versioning alongside. Keywords AI, trading as Respan, exposes nine evaluator tools.
For tracing rather than scoring, Langfuse and Traceloop both reach model-call spans, which is the other half of the picture evaluation is usually paired with.
Connect Confident AI to inspect and score runs that already exist. Keep DeepEval in Python for defining and executing metrics.
DeepEval MCP server setup
Nothing is installed. The server is hosted, and clients reach it over HTTP at the regional endpoint for your account.
Authentication is OAuth. The first time a client connects, a browser window opens to sign in to Confident AI and approve the connection; later sessions refresh automatically.
Claude Code
Use the EU host instead if your account is in that region.
No key goes on the command line. The OAuth window opens the first time the client connects.
claude mcp add --transport http confident-ai \ https://mcp.confident-ai.com/mcp
Cursor
In ~/.cursor/mcp.json. The OAuth flow opens on first connection.
{"mcpServers":{"confident-ai":{
"url":"https://mcp.confident-ai.com/mcp"}}}Codex CLI
Codex reads TOML, so the JSON blocks above will not transfer.
The entry carries a URL and no token. Codex hands the OAuth sign-in to the browser.
[mcp_servers.confident-ai] url = "https://mcp.confident-ai.com/mcp"
Every other client
Each block below is the configuration for one client, with the file path and the key that client expects.
Claude Desktop
macOS ~/Library/Application Support/Claude/claude_desktop_config.json. Windows %APPDATA%\Claude\claude_desktop_config.json. There is no CLI. Edit through Settings, Developer, Edit Config.
Quit and restart fully for changes to load.
Put no credential in this file. Approve the OAuth prompt in the browser after the restart.
{"mcpServers":{"confident-ai":{"url":"https://mcp.confident-ai.com/mcp"}}}VS Code Copilot
.vscode/mcp.json per workspace, or your user profile.
The block holds the URL alone. Copilot starts the OAuth sign-in when it first reaches the server.
{"servers":{"confident-ai":{"url":"https://mcp.confident-ai.com/mcp"}}}`.vscode/mcp.json` takes the key `servers`. VS Code also reads mcpServers from a portable .mcp.json at the project root or ~/.copilot/mcp-config.json. Its docs recommend the portable files for new servers.
EU and self-hosted
EU accounts use https://eu.mcp.confident-ai.com/mcp. A self-hosted deployment exposes /mcp on its own host. Both were documented on 31 August 2026, and the EU endpoint returned HTTP 401 to an unauthenticated request, the same as US.
Read-only access, permissions and security
Deletes are permanent, and the vendor says so
The tools reference describes 26 of the 29 delete tools as permanent, from datasets and goldens to reports and governance controls.
Some tools spend usage or reach your systems
run_rt_frameworksends generated attacks to your application and costs usage. The governance assess tools re-run controls at a compute cost.No documented read-only mode
Nothing in the published documentation describes a server-side flag or a read-only endpoint variant, so restriction has to happen in the client.
OAuth binds the session to a person
Actions arrive as the account that approved the connection, so an agent's writes are attributed to whoever signed in.
DeepEval MCP server: Reference
| Item | Value |
|---|---|
| Server published | Yes, hosted and first-party |
| US endpoint | https://mcp.confident-ai.com/mcp |
| EU endpoint | https://eu.mcp.confident-ai.com/mcp |
| Transport | HTTP |
| Authentication | OAuth, browser sign-in on first connection |
| Tools | 202 across 34 areas, as of 6 October 2026 |
| Largest areas | Organization (governance) 26 · Datasets 17 · Annotation queues 14 · Prompts 12 |
| Deleting tools | 29 delete tools, 26 described as permanent. Confident AI's MCP tools reference lists them all |
| Read-only mode | None documented |
confident-ai/confident-mcp-server | HTTP 404 on 31 August 2026 |
MCP repositories in confident-ai | 0 |
| DeepEval stars | 18,648 |
| DeepEval last push | 5 October 2026 |
| Checked | 6 October 2026 |
What engineers report
DeepEval carries 18,648 stars as of 6 October 2026 and was pushed on 5 October 2026. The server built around it is hosted, which is why a repository search finds nothing while the product answers on two continents.
| What was checked | What it shows |
|---|---|
| Tools documented | 202 |
| Areas | 34 |
| Endpoints documented | 3, counting self-hosted |
Unauthenticated initialize against US | HTTP 401 |
Unauthenticated initialize against EU | HTTP 401 |
confident-ai/confident-mcp-server | HTTP 404 |
Both documented endpoints were sent an unauthenticated JSON-RPC initialize on 31 August 2026 and each returned HTTP 401 with an unauthorized JSON body, which establishes a gated server rather than an absent one. The confident-ai GitHub organization was searched for MCP repositories the same day and returned zero, and confident-ai/confident-mcp-server returned HTTP 404.
Should you connect an agent to DeepEval?
Yes. Best for inspecting and scoring existing runs, with the deleting tools allowlisted out.
- Strongest fit
- Teams already on Confident AI who want an agent to read traces, score them in place and manage the datasets behind an evaluation.
- Main advantage
- Hosted and broad: 202 tools reach most of the platform with nothing to install.
- Main weakness
- A surface with 26 deleting tools the vendor calls permanent, and a read-only mode is undocumented.
- Operational risk
- An agent with the default surface can delete a dataset, a golden, a dashboard, a report or a governance policy for good.
From Oodle
One platform for agent traces and infrastructure
Agent traces usually sit in a different product from the rest of your telemetry, so when a slow database makes an agent slow the symptom is in one tool and the cause is in another. Oodle keeps both in one query surface.
See agent observabilityFrequently asked questions
Does DeepEval have an MCP server?
Yes. Confident AI, the platform behind DeepEval, hosts one at https://mcp.confident-ai.com/mcp with an EU endpoint alongside. It exposes 202 tools across 34 areas.
Why do people say there isn't one?
Because confident-ai/confident-mcp-server returns HTTP 404 and the GitHub organization carries no MCP repositories. The server shipped hosted rather than open-source, so a repository search finds nothing while the product is live.
Can an agent delete my data?
Yes. Confident AI's tools reference describes 26 tools that delete permanently, including datasets, goldens, dashboards, reports and governance controls. Confident AI documents no server-side read-only mode, so restrict the tools in your client.
Does this replace running DeepEval in Python?
No. Metrics are still defined and executed in the library. The server reaches the hosted platform around it: runs, datasets, prompts, traces and dashboards.
How do I know the endpoint is real without an account?
Send an unauthenticated initialize to it. A live gated server answers HTTP 401 with a JSON error body, which is what both documented endpoints returned on 31 August 2026. A missing server does not resolve at all.
