Scrape-LE MCP Server
Analyse robots.txt content and report whether a path may be crawled.
Install
{
"mcpServers": {
"nolindnaidoo-scrape-le": {
"command": "npx",
"args": [
"-y",
"scrape-le-mcp@2.4.0"
]
}
}
}Related servers
doc-scraper
69GoodCrawl documentation sites into local corpora agents can search, read, and diff fully offline.
chrome-bridge
68GoodBorrow a tab from your own logged-in Chrome, hand it back; 2FA and CAPTCHA stay yours. 67 tools.