diff options
| author | Leitwolf | 2026-10-03 17:35:35 +0300 |
|---|---|---|
| committer | GitHub | 2026-10-03 17:35:35 +0300 |
| commit | 710dc90b0503cd1de1ae6159e01fb22d1b336aba (patch) | |
| tree | 654c64a225f57867b93123a816ddd5b0de77b928 | |
| parent | 721049419bea10c07ca8a4f3b206908c6b673d49 (diff) | |
| parent | befe7555f6b39b384f5d31b502d7f43760338228 (diff) | |
| download | website-710dc90b0503cd1de1ae6159e01fb22d1b336aba.tar.gz | |
Merge pull request #13 from Progmasoft/chore/ai-crawler-rules
Change which AI crawlers may read the site
| -rw-r--r-- | frontend/public/robots.txt | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/frontend/public/robots.txt b/frontend/public/robots.txt index 9bde1d2..8a9b365 100644 --- a/frontend/public/robots.txt +++ b/frontend/public/robots.txt @@ -5,7 +5,7 @@ User-agent: Googlebot Allow: / User-agent: Google-Extended -Disallow: / +Allow: / User-agent: OAI-SearchBot Allow: / @@ -14,7 +14,7 @@ User-agent: ChatGPT-User Allow: / User-agent: GPTBot -Allow: / +Disallow: / User-agent: Claude-SearchBot Allow: / @@ -23,7 +23,7 @@ User-agent: Claude-User Allow: / User-agent: ClaudeBot -Disallow: / +Allow: / User-agent: PerplexityBot Disallow: / |