HolicAudit

AI crawlers · Logs never uploaded · Pro $5 once

AI crawlers · runs in your browser

See which AI crawlers actually visit your site

Drop your server's access log and see how often GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot and other crawlers came, which pages they fetched, and whether they got a 200, a redirect or an error. Your log is never uploaded: it's analyzed in this browser tab.

Free: one log up to 10 MB. Pro, $5 once: up to 1 GB, several files, .gz and .zip, exports and IP checks. Want to know whether AI bots can read a page at all? Run the free AI visibility check.

Crawlers it recognises

User-agent names from each operator's public crawler documentation. Bots we don't recognise but that call themselves a bot, crawler or spider are listed separately.

User agentOperatorUsed for
GPTBotOpenAITraining
OAI-SearchBotOpenAIChatGPT search
ChatGPT-UserOpenAIFetches for a user's question
OAI-AdsBotOpenAIAd landing page checks
ClaudeBotAnthropicTraining
Claude-SearchBotAnthropicClaude search
Claude-UserAnthropicFetches for a user's question
PerplexityBotPerplexityPerplexity search
Perplexity-UserPerplexityFetches for a user's question
GooglebotGoogleGoogle Search, including AI Overviews
GoogleOther, Google-CloudVertexBot, Google-InspectionTool, Storebot-GoogleGoogleOther Google crawlers
BingbotMicrosoftBing search (also used by Copilot)
ApplebotAppleSiri, Spotlight and Safari search
Meta-ExternalAgentMetaTraining and indexing
Meta-ExternalFetcherMetaFetches for a user's request
Amazonbot, Amzn-SearchBot, Amzn-UserAmazonTraining, search, user fetches
DuckAssistBotDuckDuckGoAI answers
MistralAI-UserMistral AIFetches for a user's question
BytespiderByteDanceTraining
CCBotCommon CrawlPublic crawl used to train many models
DuckDuckBot, YandexBot, Baiduspider, PetalBot, YouBot, DiffbotVariousSearch engines and other crawlers

Google-Extended and Applebot-Extended are robots.txt controls, not crawlers, so they never show up in logs. To change which bots may crawl you, see should you block GPTBot.

AI Crawler Log Analyzer Pro

One-time payment by PayPal or USDT. No subscription, no expiry. Your logs still never leave your browser.

Free

$0 one log up to 10 MB

For a quick look at a day of traffic.

  • nginx, Apache, Cloudflare, Caddy, IIS and similar logs
  • AI crawlers, search engines and other bots
  • Top paths, status codes and timeline per bot
  • Runs in your browser
  • No sign-up
Use it free

Pro Recommended

$5 one-time

For site owners, developers and SEO consultants.

  • Logs up to 1 GB at a time, several files at once
  • .gz and .zip archives unpacked in your browser
  • CSV and JSON downloads, plus a standalone HTML summary
  • IP check against the crawler lists OpenAI, Perplexity, Google, Bing and Apple publish
  • Your own bot patterns and excluded paths, saved in your browser
  • License key with no expiry
Get Pro, $5

Frequently asked questions

Is my log uploaded?

No. The file is read by JavaScript in this browser tab and the results are calculated there. Nothing from your log is sent to us or anyone else, and nothing is stored. The only network requests are the license check (it sends the key only) and, for the Pro IP check, a download of the IP lists that OpenAI, Perplexity, Google, Bing and Apple publish; the matching happens in your browser.

Which log formats work?

nginx and Apache combined or common log format (including variants with a virtual host in front), JSON lines from Cloudflare Logpush, Caddy and most JSON loggers, W3C extended logs from IIS and CloudFront, and most other formats with a quoted request line such as "GET /page HTTP/1.1". Lines we can't read are counted and a few are shown so you can see why.

Where do I find my access log?

On a Linux server with nginx it's usually /var/log/nginx/access.log, with Apache /var/log/apache2/access.log or /var/log/httpd/access_log. cPanel hosts offer Raw Access logs under Metrics. Cloudflare can export HTTP request logs with Logpush (an Enterprise feature). Many managed hosts let you download logs from their dashboard.

How much can the free version read?

One file or one paste, up to 10 MB of plain text: roughly 40,000 to 60,000 lines of a typical combined log, which covers a day of traffic on many small sites. A larger file is read up to 10 MB and the results say so. Pro reads up to 1 GB at a time, from several files, and unpacks .gz and .zip archives in the browser.

Can I trust the bot names?

A user agent is only a claim: anyone can send "GPTBot" in a request. Pro checks each bot's IP addresses against the lists OpenAI, Perplexity, Google, Bing and Apple publish, so you can see which hits really came from them. Anthropic, Meta, ByteDance and others don't publish lists we can check automatically.

Why don't I see Google-Extended or Applebot-Extended?

They aren't crawlers. Google-Extended and Applebot-Extended are robots.txt tokens that control how Google and Apple may use what Googlebot and Applebot already crawled, so they never appear in access logs.

Does crawler traffic mean I'm cited in AI answers?

No. The log shows which pages AI crawlers fetched and whether your server answered with 200, a redirect or an error. It can't show whether an AI answer cited or ranked you. It's most useful for finding blocks (403, 429, 5xx), pages crawlers never reach, and how much training-crawler traffic you serve.

What do I get for $5?

A license key with no expiry: logs up to 1 GB at a time, several files at once, .gz and .zip, CSV and JSON downloads, a standalone HTML summary you can share, the IP check against published crawler lists, and your own bot patterns and excluded paths saved in your browser. It's a one-time payment by PayPal or USDT, no subscription. It's a simple log reader, not a hosted analytics service.

Related