robotstxt-ai MCP Server
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Install
{
"mcpServers": {
"sharozdawa-robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp@1.0.1"
]
}
}
}Related servers
kordoc
83ExcellentParse Korean documents (HWP, HWPX, PDF, XLSX, DOCX, PPTX) to Markdown; fill forms, diff, render.
pagespeed-insights-mcp
75GoodGoogle PageSpeed Insights and CrUX: six workflow tools for web performance.