SiseoBot, the crawler behind siseo audits
If you found SiseoBot in your logs, someone asked siseo.io to audit your site. Here is what it does and how to control it.
What it does
SiseoBot visits a site only when someone asks siseo for an SEO audit of it. It reads pages the way a search engine does, follows links from the homepage, and stops after 200 pages for a free scan or 1000 for a paid report.
How it behaves
- It identifies itself with this user agent:
Mozilla/5.0 (compatible; SiseoBot/1.0; +https://report.siseo.io/bot) - It follows robots.txt, under its own name (
SiseoBot) or the rules for all crawlers (*), including Crawl-delay. - It makes at most four requests at a time, slows down when a server answers slowly or with errors, and never fills in forms, signs in or buys anything.
- It opens a few pages in a headless browser, to compare what visitors see with what crawlers that skip JavaScript see.
- To test whether a firewall turns AI search crawlers away, it requests the homepage once with each AI crawler's user agent. Those requests come from siseo's servers, not from the AI companies.
How to stop it
Add this to your robots.txt and SiseoBot won't crawl the site:
robots.txt
User-agent: SiseoBot
Disallow: /