diff options
| author | Leitwolf11 | 2026-11-05 06:48:34 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-05 06:48:34 +0300 |
| commit | 7a9b7891812611f9e45e64b2c585d006a414af97 (patch) | |
| tree | 3f0e1dfbb41a625952c73dbd62956b06b58fbce5 /.github/workflows/codeql.yml | |
| parent | 6f114e3df68d0b20c7e58c5ffb8c46070317008a (diff) | |
| download | tazel.net-7a9b7891812611f9e45e64b2c585d006a414af97.tar.gz | |
Name the crawlers robots.txt lets in
Search crawlers and the crawlers that fetch a page for a person are
allowed: Googlebot, Google-Extended, OAI-SearchBot, ChatGPT-User,
Claude-SearchBot, Claude-User and ClaudeBot. GPTBot, CCBot and
Bytespider are turned away. Every other crawler is allowed, and the
sitemap stays named.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffstat (limited to '.github/workflows/codeql.yml')
0 files changed, 0 insertions, 0 deletions