Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/apps
diff options
context:
space:
mode:
authorLeitwolf112026-10-03 16:34:18 +0300
committerLeitwolf112026-10-03 16:34:18 +0300
commit4c803fe10d3120a7dc7f594a1fb2eb205eca34fc (patch)
tree2f4e9bc382d8681aee062ae7c1cfd5421995b303 /apps
parent282b015e23474c3bb78fe303af627248f5bd2d7a (diff)
downloadprogmaweb-4c803fe10d3120a7dc7f594a1fb2eb205eca34fc.tar.gz
Change which AI crawlers may read the site
- Google-Extended, Google's token for model training, is now allowed. - GPTBot, OpenAI's training crawler, is now disallowed. OAI-SearchBot and ChatGPT-User, which index and fetch pages for answers, stay allowed. - ClaudeBot, Anthropic's training crawler, is now allowed, next to Claude-SearchBot and Claude-User, which already were. Every other rule is unchanged.
Diffstat (limited to 'apps')
-rw-r--r--apps/web/app/robots.ts6
1 files changed, 3 insertions, 3 deletions
diff --git a/apps/web/app/robots.ts b/apps/web/app/robots.ts
index 1a05aae..1bce9a0 100644
--- a/apps/web/app/robots.ts
+++ b/apps/web/app/robots.ts
@@ -16,13 +16,13 @@ export default async function robots(): Promise<MetadataRoute.Robots> {
return {
rules: [
{ userAgent: "Googlebot", allow: "/" },
- { userAgent: "Google-Extended", disallow: "/" },
+ { userAgent: "Google-Extended", allow: "/" },
{ userAgent: "OAI-SearchBot", allow: "/" },
{ userAgent: "ChatGPT-User", allow: "/" },
- { userAgent: "GPTBot", allow: "/" },
+ { userAgent: "GPTBot", disallow: "/" },
{ userAgent: "Claude-SearchBot", allow: "/" },
{ userAgent: "Claude-User", allow: "/" },
- { userAgent: "ClaudeBot", disallow: "/" },
+ { userAgent: "ClaudeBot", allow: "/" },
{ userAgent: "CCBot", disallow: "/" },
{ userAgent: "Bytespider", disallow: "/" },
{ userAgent: "*", allow: "/" },