diff options
| author | Leitwolf11 | 2026-10-03 16:34:06 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-10-03 16:34:06 +0300 |
| commit | befe7555f6b39b384f5d31b502d7f43760338228 (patch) | |
| tree | 654c64a225f57867b93123a816ddd5b0de77b928 /frontend/tests | |
| parent | 721049419bea10c07ca8a4f3b206908c6b673d49 (diff) | |
| download | website-chore/ai-crawler-rules.tar.gz | |
Change which AI crawlers may read the sitechore/ai-crawler-rules
- Google-Extended, Google's token for model training, is now allowed.
- GPTBot, OpenAI's training crawler, is now disallowed. OAI-SearchBot and
ChatGPT-User, which index and fetch pages for answers, stay allowed.
- ClaudeBot, Anthropic's training crawler, is now allowed, next to
Claude-SearchBot and Claude-User, which already were.
Every other rule is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffstat (limited to 'frontend/tests')
0 files changed, 0 insertions, 0 deletions