crawl-readiness MCP Server
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
Install
{
"mcpServers": {
"grubbomatic-crawl-readiness-mcp": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp@0.2.5"
],
"env": {
"CRAWL_READINESS_API_KEY": "YOUR_CRAWL_READINESS_API_KEY",
"CRAWL_READINESS_API_BASE": ""
}
}
}
}Related servers
doc-scraper
69GoodCrawl documentation sites into local corpora agents can search, read, and diff fully offline.
chrome-bridge
68GoodBorrow a tab from your own logged-in Chrome, hand it back; 2FA and CAPTCHA stay yours. 67 tools.