RAGAS MCP server: the community one fails to start

An evaluation library with no server, a community wrapper that no longer imports, and the SDK change that explains both.

Last verified 1 September 2026 · from The RAGAS documentation, the community repository and a first-hand run of it

This page is one of 90 in a directory of monitoring and developer tools. Each page checks what an AI coding agent can get out of the tool through its MCP server, the connector that lets the agent query the tool directly.

What is the RAGAS MCP server, and is it usable?

RAGAS evaluates retrieval-augmented generation. It scores answers for faithfulness, relevance and context quality, which is exactly the kind of judgment an agent might want to call during development.

It is a Python library rather than a service. There is no hosted RAGAS to connect to and no account, so a server would have to wrap the library on the machine running it.

The community wrapper attempted that, and it is a casualty of an ecosystem change rather than of neglect. The MCP Python SDK released 2.0.0 and removed mcp.server.fastmcp. Any package that imports that path stopped working.

Every one of the 90 vendors in this directory gets the same four checks. RAGAS passes 1 of them.

RAGAS works in agent observability, and nothing usable connects to it.

RAGAS is a library and is best called as one. An agent that can run Python can import RAGAS and score a dataset in a few lines, which is less indirection than a server would add.

What can an AI agent do with RAGAS?

The community wrapper broke because of an SDK change. That change affects more than this package.

Do you need the MCP server at all?

No, and there is not one to need. Call RAGAS as a library, which is how it is designed to be used. An agent that can run Python can import it, score a dataset and read the metrics back without a protocol in between.

For evaluation with a maintained server behind it, several vendors publish one. Braintrust documents 34 tools including create_evaluator, test_evaluator and run_eval; Keywords AI exposes nine evaluator tools plus experiment score averages; Arize Phoenix covers datasets and experiments across 27 tools.

For tracing the retrieval pipeline rather than scoring it, Langfuse, Traceloop and Arize Phoenix all reach spans and traces from model calls.

If the wrapper is ever fixed, the thing to check first is whether it pins the MCP SDK below 2.0.0. Without that pin, the same failure returns on the next clean install.

Import RAGAS in Python. Connect Braintrust or Arize Phoenix when you want evaluation an agent can call over MCP.

RAGAS MCP server setup

Nothing here installs a RAGAS server. The community one was never published to PyPI and fails at import, so the working route is the library RAGAS actually ships.

RAGAS is a Python package. Evaluation runs in your own code, and there is no protocol in between to configure.

Where the point is an agent that can call evaluation over MCP, two vendors in this directory publish servers for that.

pip install ragas
python evaluate.py    # RAGAS runs inside your own code

claude mcp add-json "phoenix" '{"command":"npx",
  "args":["-y","@arizeai/phoenix-mcp"],
  "env":{"PHOENIX_ENDPOINT":"https://app.phoenix.arize.com",
         "PHOENIX_API_KEY":"<your-key>",
         "PHOENIX_PROJECT":"<your-project>"}}'

Every other client

Each block below is the configuration for one client, with the file path and the key that client expects.

pip install git+https://github.com/CaringNihilistic/ragas-mcp
# ModuleNotFoundError: No module named 'mcp.server.fastmcp'

Read-only access, permissions and security

• No RAGAS credential is created

  With no working server, nothing here asks for a token against RAGAS.

• The library route keeps evaluation data local

  Scores are computed in your own process, and nothing is sent to a connector.

• An unlicensed repository is a legal blocker, not just a technical one

  GitHub reports no license for the community server, so the terms for internal reuse are undefined.

• Evaluation inputs are often production text

  Whatever you score, a hosted evaluation server will hold, so scope the project before connecting one.

• A server that fails at import exposes nothing

  It registers no tools, so the risk here is wasted time rather than access.

Troubleshooting

You cannot find an official RAGAS MCP server

There is not one. The RAGAS documentation index never mentions MCP, and its GitHub organization holds no MCP repository.

`pip install ragas-mcp` finds nothing

It was never published. PyPI returned 404 for ragas-mcp on 24 August 2026, so installation is from the git repository only.

The server fails immediately on start

Installing and running it on 24 August 2026 failed with ModuleNotFoundError: No module named 'mcp.server.fastmcp'. Zero tools are reachable as a result.

The cause is the SDK rather than your environment. The same import failure is corroborated by an open issue on wandb/wandb-mcp-server.

The repository reports no license

GitHub reports none for CaringNihilistic/ragas-mcp, so the terms for reuse are undefined. Treat that as a blocker before any internal fork.

You still want RAGAS scores in an agent workflow

Call the RAGAS library directly from your evaluation code. It is a Python package, and no connector stands between it and a script you already run.

RAGAS MCP server: Reference

What engineers report

The failure is reproducible, and it affects other packages too. The counts below are what the checks found.

The RAGAS documentation index and organization were checked for MCP on 24 August 2026 and carry none. The community package was installed from git and run on the same date; it exited at import with the error above. PyPI was checked for ragas-mcp and returned 404.

Should you connect an agent to RAGAS?

No. A server exists and is not usable. Best for importing RAGAS in Python, or connecting an evaluation platform that publishes a maintained server.

Strongest fit

Nobody. The only server does not start.

Main advantage

None reachable: the server exits before registering a tool.

Main weakness

0 stars, no license, not on PyPI, and broken by an SDK change it did not pin against.

Operational risk

The failure is at import, so a client reports only that the server did not start.

Frequently asked questions