Skip to content
MCPpedia
Submit
MCP server
Registry

evalsmith-mcp MCP Server

Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

evalsmith-mcp scores 38 (Limited), licensed MIT and written in TypeScript. It installs as a local process. It does not declare tools yet. Filed under AI and memory. GitHub shows 1 stars. Install configs are generated for 10 clients. Being listed in the registry is not the same as being vendor-official.

by diegoperezg7MITTypeScript1 stars
InstallView on GitHub
Trust score
38Limited
Trust score 38 of 100, Limited

Limited trust

38 of 100 from six public signals. Nobody can buy a point.

  • OSI-style license: mit.
How we score trust
Scorecard

License, quality and maintenance

How grades work
Trust breakdown

Where the 38 comes from

0 + 20 + 1 + 10 + 4 + 3 = 38. Each factor is capped, so no single signal can carry a listing.

Registry0 / 30

Active in the official MCP registry, plus a known vendor namespace.

Registry: 0 of 30 points.
Maintenance20 / 20

How recently the repository saw a commit or release.

Maintenance: 20 of 20 points.
Popularity1 / 20

GitHub stars and weekly downloads, on a log scale.

Popularity: 1 of 20 points.
License10 / 10

A clear, OSI-style open-source license.

License: 10 of 10 points.
Security4 / 10

Not archived, a secure install path, a pinned version.

Security: 4 of 10 points.
Docs3 / 10

A real summary, a README with depth, listed tools.

Docs: 3 of 10 points.

Install

Pick your client, copy the config, paste it in.

Add to Cursor: evalsmith-mcpInstall in VS Code: evalsmith-mcpInstall in VS Code Insiders: evalsmith-mcp

Needs Cursor or VS Code installed on this device. The button opens the app and asks you to confirm before it adds the server.

~/Library/Application Support/Claude/claude_desktop_config.json
{  "mcpServers": {    "diegoperezg7-evalsmith-mcp": {      "command": "npx",      "args": [        "-y",        "evalsmith-mcp@0.1.0"      ],      "env": {        "EVALSMITH_JUDGE_BASE_URL": "http://localhost:11434/v1",        "EVALSMITH_JUDGE_MODEL": "llama3.1:8b",        "EVALSMITH_JUDGE_API_KEY": "YOUR_EVALSMITH_JUDGE_API_KEY",        "LOG_LEVEL": "info"      }    }  }}
Install guide

Environment variables

NameRequiredNotes
EVALSMITH_JUDGE_BASE_URLOptionalOpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost)
EVALSMITH_JUDGE_MODELOptionalModel used by the optional LLM judge
EVALSMITH_JUDGE_API_KEYSecretOptionalAPI key for the judge endpoint, if it needs one
LOG_LEVELOptionalpino log level; logs go to stderr

Tools

All capabilities

Tools show up when the registry metadata or the repository README lists them. Public READMEs are read directly. A GitHub token is used only when one is already configured.

About

MCP server and CLI to evaluate RAG retrieval, agent tool calls and prompt regressions, with CI gates. No paid keys.

Frequently asked questions

What is evalsmith-mcp MCP Server?

Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

How do I install evalsmith-mcp MCP Server in Claude Desktop, Cursor, or VS Code?

Pick your client in the Install section on this page and copy the generated config into the client's config file. Step-by-step guides: Claude Desktop, Cursor, and VS Code. evalsmith-mcp MCP Server runs locally from the npm package evalsmith-mcp with the command npx -y evalsmith-mcp@0.1.0.

Is evalsmith-mcp MCP Server free and open source?

Yes. evalsmith-mcp MCP Server is released under the MIT license, an OSI-approved open-source license, so you can use, modify, and self-host it under those terms.

Does evalsmith-mcp MCP Server support remote (HTTP) connections?

No remote endpoint is listed for evalsmith-mcp MCP Server. It runs locally on your machine over stdio, started by your MCP client.

How is evalsmith-mcp MCP Server's trust score calculated?

evalsmith-mcp MCP Server scores 38 out of 100 (Limited). The score adds six capped public signals: registry status, maintenance, popularity, license, documentation, and security. Nobody can pay for points. Read the methodology for the full formula.

Reviews

No reviews yet

—Be the first to review

5 star0 reviews at 5 stars
4 star0 reviews at 4 stars
3 star0 reviews at 3 stars
2 star0 reviews at 2 stars
1 star0 reviews at 1 stars

No published reviews yet. Used it? Tell the next person how it went.

Add badge to your README

Paste either snippet into the repository README. The image links to this listing.

Listed on MCPpedia

Top 5 MCP servers

Daily or weekly. Confirm by email, unsubscribe any time.

How often

Manage this listing

Claim this server
Same category
All alternatives
Trust score 78 of 100, Good

Drive a stock iPhone via iPhone Mirroring: no Developer Mode, Xcode or WDA. LLM-free skill replay.

Registry
LicenseQualityMaintenanceLicense: A. Apache-2.0 is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 13 days ago.
268 stars1.5K/wk weekly downloads— tools
Trust score 77 of 100, Good

Knowledge-graph RAG for multi-language codebases: index, query and edit code in natural language.

Registry
LicenseQualityMaintenanceLicense: A. MIT is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 25 days ago.
5.3K stars—/wk weekly downloads— tools
Trust score 77 of 100, Good

Configure and automate Home Assistant with Claude, ChatGPT, and other AI assistants.

Registry
LicenseQualityMaintenanceLicense: A. MIT is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 14 days ago.
5.3K stars—/wk weekly downloads— tools
Mirobody logo
Trust score 75 of 100, Good

Mirobody

by thetahealth · Python

Lab results to LOINC codes and UCUM units, symptoms and diagnoses to ICPC-3. Offline, no API key.

Registry
LicenseQualityMaintenanceLicense: A. Apache-2.0 is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 26 days ago.
1.4K stars—/wk weekly downloads— tools
Trust score 75 of 100, Good

projectmem

by riponcm · Python

Coding agent memory — one local MCP server for every project. Warns before repeating failed fixes.

Registry
LicenseQualityMaintenanceLicense: A. MIT is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 12 days ago.
870 stars—/wk weekly downloads— tools
Trust score 75 of 100, Good

reddit-mcp-buddy

by karanb192 · TypeScript

Browse Reddit RSS; approved API access for search, analysis, and optional posting.

Registry
LicenseQualityMaintenanceLicense: A. MIT is a permissive open source license.Quality: Not graded. No README or tool list has been captured yet.Maintenance: A. Last commit or release 15 days ago.
865 stars—/wk weekly downloads— tools