RAGAS MCP server: run RAG evaluations from your agent

RAGAS scores RAG pipelines and agents from Python. Community MCP wrappers expose those scores as tools, but two pin an older RAGAS release and two fail on the current MCP SDK.

Checked 3 October 2026 · from The RAGAS docs and repository, PyPI, the MCP Python SDK wheels, four community repositories and a Weights & Biases issue

One of 91 guides to monitoring and developer tools. Each covers what an AI coding agent can get out of the tool through MCP, the protocol that lets the agent query the tool directly.

How RAGAS works with MCP

RAG, retrieval-augmented generation, answers a question from documents fetched at query time. RAGAS scores such a system with metrics like faithfulness, context recall and factual correctness.

Many metrics use a judge model, an LLM that grades each answer. RAGAS runs in your own Python process, so there is no hosted RAGAS service or account to connect to.

The repository moved from explodinggradients/ragas to vibrantlabsai/ragas and has 15,912 stars. The latest PyPI release is 0.4.3, from 13 January 2026.

Every vendor here gets the same four checks: official, maintained, distributed and usable. RAGAS passes 0 of them.

RAGAS produces scores, not telemetry. An agent can ask it to grade answers, but traces of what the pipeline did come from a tracing tool.

What can an AI agent do with RAGAS?

How to configure RAGAS MCP for agents

1. 1

   Pin the SDK

   Any Python MCP wrapper that imports mcp.server.fastmcp needs mcp<2 until it is ported.

2. 2

   Judge model

   Name the evaluator model explicitly and record it with each score. Changing it changes the numbers.

3. 3

   Data exposure

   Evaluation inputs are often real user questions and retrieved documents, and the judge model's provider receives them.

Paste this into your agent’s instructions

When you report RAGAS scores, state:
- The metrics, the RAGAS version and the judge model.
- The number of samples scored and where they came from.
Compare runs only when those settings match.
Say when a wrapper pinned an older RAGAS release.

Do you need the MCP server at all?

Call the library. An agent that can run Python imports ragas, builds an EvaluationDataset and calls evaluate(). That uses the current release, not a pinned 0.2 line.

`mertdurukan/agent-eval-mcp-server`. Five tools, including evaluate_rag, evaluate_agent_trace and detect_eval_drift. It pins mcp==1.27.2, so the 2.0 import break does not apply.

`ThomasZiegenhein/ragas_mcp`. Apache-2.0, last pushed on 2 October 2025, with Docker examples. It imports mcp.server.fastmcp and needs mcp<2 like CaringNihilistic's wrapper.

`dmikey/ragas-mcp`. A single-file FastAPI service built on ragas-experimental, from June 2025. It does not use the MCP Python SDK.

Evaluation platforms with first-party servers. Braintrust, Arize Phoenix and DeepEval (through Confident AI) run or read evaluations without a local wrapper.

Score with the RAGAS library in Python. Use a pinned wrapper only when the agent must call evaluation as an MCP tool.

RAGAS integration setup

The library route needs pip install ragas and a judge model key. The MCP route below runs CaringNihilistic's wrapper with the SDK pinned below 2.0.

from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, LLMContextRecall
from langchain_openai import ChatOpenAI

data = EvaluationDataset.from_list(rows)
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
print(evaluate(dataset=data, metrics=[Faithfulness(), LLMContextRecall()], llm=judge))

pip install -e . "mcp<2"
claude mcp add-json "ragas" '{"command":"/path/to/.venv/bin/python",
  "args":["-m","ragas_mcp.server"],
  "env":{"ANTHROPIC_API_KEY":"<your-key>"}}'

Every other client

Each block below is the configuration for one client, with the file path and the key that client expects.

What engineers report

None of the four wrappers has a GitHub star. The Weights & Biases issue on the mcp 2.0.0 change, opened 31 July 2026, documents the same import failure in a widely used server.

Should you connect an agent to RAGAS?

No for most teams; call the RAGAS library from Python. Use a wrapper only when the agent must score answers as an MCP tool. Best for teams evaluating RAG pipelines in Python who want an agent to run or compare evaluations.

Strongest fit

No for most teams; call the RAGAS library from Python. Use a wrapper only when the agent must score answers as an MCP tool.

Main advantage

RAGAS gives reproducible RAG and agent metrics that a CI job or an agent can compute locally.

Main weakness

Every MCP wrapper is a small community project, and the two most complete pin RAGAS below 0.3.

Operational risk

Unpinned wrappers break at import on mcp 2.x, and the client reports only that the server closed.

Frequently asked questions