GeoFlakesBot

The crawler GEOFLAKES uses to read a website when someone audits it: how ready the site is to be read and cited by AI assistants, and the llms.txt and robots.txt files the audit hands back.

How it identifies itself

Every request it makes carries this user agent, and nothing that presents it as a browser:

GeoFlakesBot/1.0 (+https://geoflakes.ai/bot)

Where its requests come from

From cloud servers whose addresses change. There is no fixed list of IP addresses to allow. To let it in, allow the name GeoFlakesBot in your firewall, CDN or bot-protection rules; to keep it out, block that name.

What it reads, and how fast

Your homepage, robots.txt, llms.txt and llms-full.txt, your sitemap, and a sample of the pages your menu and sitemap link to. It reads the HTML your server sends and runs no scripts, as most AI crawlers do.

It asks for at most 6 pages at a time, for at most 60 seconds an audit. When your site answers 429 Too Many Requests, it waits as long as your Retry-After asks (up to 10 seconds) and then asks for at most 2 at a time. A page your site refuses (401, 403 or a bot check) is not asked for again in that audit: it never tries to get past your protection.

robots.txt

The pages it checks while writing the llms.txt and llms-full.txt it hands back follow your robots.txt: a group for GeoFlakesBot, or otherwise the one for every crawler (User-agent: *). To keep it out of them:

User-agent: GeoFlakesBot
Disallow: /

The audit itself reads the homepage, and a few pages, of the site someone asked it to audit, since that audit is what was asked for.

Contact

Questions about GeoFlakesBot, or a request to stop it reading your site, can be sent to support@geoflakes.ai.