How do I install polite-crawler?

Pick a client above and copy the generated config. Cursor and VS Code also have a deep link when the server can run locally or remotely.

Is polite-crawler official?

No. Registry-listed is not the same as vendor-official. Read the methodology for the difference.

How is the trust score calculated?

This listing scores 55 (Fair). The formula is published on the methodology page.

MCPpedia

polite-crawler MCP Server | MCPpedia

polite-crawler MCP Server

Name: polite-crawler MCP Server
Author: mark1russell7

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Tools Versions Alternatives

Registrymark1russell7

55Fair

Install

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "mark1russell7-polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler@0.1.0"
      ],
      "env": {
        "POLITE_CRAWLER_DIR": "",
        "POLITE_CRAWLER_CONTACT": "",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": ""
      }
    }
  }
}

Install guide

Related servers

kordoc

83Excellent

Parse Korean documents (HWP, HWPX, PDF, XLSX, DOCX, PPTX) to Markdown; fill forms, diff, render.

Registrychrisryugj2.5K stars44.2K/wk

pagespeed-insights-mcp

75Good

Google PageSpeed Insights and CrUX: six workflow tools for web performance.

Environment

Name	Required	Notes
POLITE_CRAWLER_DIR	No	The folder of the store. The default is ~/.polite-crawler.
POLITE_CRAWLER_CONTACT	No	An email address or URL for the From header of each request.
POLITE_CRAWLER_ALLOW_PRIVATE	No	Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
POLITE_CRAWLER_ALLOW_USER_PROVIDED	No	Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.

polite-crawler MCP Server

Install

Related servers

kordoc

pagespeed-insights-mcp

Environment

Tools

About

FAQ

Reviews

Hive Intelligence MCP

Screaming Frog SEO Spider MCP Server (headless)

Kilotest

Actual Budget