Progmasoft Git

progmaweb

The site and account service of Progmasoft: a Next.js frontend and an ASP.NET Core API.

Clonegit clone https://git.progmasoft.com/progmaweb

Change which AI crawlers may read the site

- Google-Extended, Google's token for model training, is now allowed.
- GPTBot, OpenAI's training crawler, is now disallowed. OAI-SearchBot and
  ChatGPT-User, which index and fetch pages for answers, stay allowed.
- ClaudeBot, Anthropic's training crawler, is now allowed, next to
  Claude-SearchBot and Claude-User, which already were.

Every other rule is unchanged.
Commit
4c803fe10d3120a7dc7f594a1fb2eb205eca34fc
Author
Leitwolf11 ·
Parents
282b015e2347

Files · Download .tar.gz

Changes

 apps/web/app/robots.ts | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/apps/web/app/robots.ts b/apps/web/app/robots.ts
index 1a05aae..1bce9a0 100644
--- a/apps/web/app/robots.ts
+++ b/apps/web/app/robots.ts
@@ -16,13 +16,13 @@ export default async function robots(): Promise<MetadataRoute.Robots> {
   return {
     rules: [
       { userAgent: "Googlebot", allow: "/" },
-      { userAgent: "Google-Extended", disallow: "/" },
+      { userAgent: "Google-Extended", allow: "/" },
       { userAgent: "OAI-SearchBot", allow: "/" },
       { userAgent: "ChatGPT-User", allow: "/" },
-      { userAgent: "GPTBot", allow: "/" },
+      { userAgent: "GPTBot", disallow: "/" },
       { userAgent: "Claude-SearchBot", allow: "/" },
       { userAgent: "Claude-User", allow: "/" },
-      { userAgent: "ClaudeBot", disallow: "/" },
+      { userAgent: "ClaudeBot", allow: "/" },
       { userAgent: "CCBot", disallow: "/" },
       { userAgent: "Bytespider", disallow: "/" },
       { userAgent: "*", allow: "/" },