diff options
| author | Leitwolf11 | 2026-10-03 16:34:18 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-10-03 16:34:18 +0300 |
| commit | 4c803fe10d3120a7dc7f594a1fb2eb205eca34fc (patch) | |
| tree | 2f4e9bc382d8681aee062ae7c1cfd5421995b303 /apps | |
| parent | 282b015e23474c3bb78fe303af627248f5bd2d7a (diff) | |
| download | progmaweb-4c803fe10d3120a7dc7f594a1fb2eb205eca34fc.tar.gz | |
Change which AI crawlers may read the site
- Google-Extended, Google's token for model training, is now allowed.
- GPTBot, OpenAI's training crawler, is now disallowed. OAI-SearchBot and
ChatGPT-User, which index and fetch pages for answers, stay allowed.
- ClaudeBot, Anthropic's training crawler, is now allowed, next to
Claude-SearchBot and Claude-User, which already were.
Every other rule is unchanged.
Diffstat (limited to 'apps')
| -rw-r--r-- | apps/web/app/robots.ts | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/apps/web/app/robots.ts b/apps/web/app/robots.ts index 1a05aae..1bce9a0 100644 --- a/apps/web/app/robots.ts +++ b/apps/web/app/robots.ts @@ -16,13 +16,13 @@ export default async function robots(): Promise<MetadataRoute.Robots> { return { rules: [ { userAgent: "Googlebot", allow: "/" }, - { userAgent: "Google-Extended", disallow: "/" }, + { userAgent: "Google-Extended", allow: "/" }, { userAgent: "OAI-SearchBot", allow: "/" }, { userAgent: "ChatGPT-User", allow: "/" }, - { userAgent: "GPTBot", allow: "/" }, + { userAgent: "GPTBot", disallow: "/" }, { userAgent: "Claude-SearchBot", allow: "/" }, { userAgent: "Claude-User", allow: "/" }, - { userAgent: "ClaudeBot", disallow: "/" }, + { userAgent: "ClaudeBot", allow: "/" }, { userAgent: "CCBot", disallow: "/" }, { userAgent: "Bytespider", disallow: "/" }, { userAgent: "*", allow: "/" }, |