diff options
| author | Leitwolf | 2026-11-05 16:35:01 +0300 |
|---|---|---|
| committer | GitHub | 2026-11-05 16:35:01 +0300 |
| commit | 3da7af66f343e532a9ea2aa7236152bf2225499b (patch) | |
| tree | 109b02492775d35a132d7495a7a6af2713993906 | |
| parent | 81359cfe2d60dfa4a18551d72ead860082598433 (diff) | |
| parent | 5c6d9ea719fc2871b58d10165def473fd9d194a4 (diff) | |
| download | tazel.net-3da7af66f343e532a9ea2aa7236152bf2225499b.tar.gz | |
Merge pull request #2 from Progmasoft/seo/robots-txt
Name the crawlers robots.txt lets in
| -rw-r--r-- | public/robots.txt | 32 |
1 files changed, 31 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt index 80d0e4e..7853582 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,4 +1,34 @@ -User-agent: * +User-Agent: Googlebot +Allow: / + +User-Agent: Google-Extended +Allow: / + +User-Agent: OAI-SearchBot +Allow: / + +User-Agent: ChatGPT-User +Allow: / + +User-Agent: GPTBot +Disallow: / + +User-Agent: Claude-SearchBot +Allow: / + +User-Agent: Claude-User +Allow: / + +User-Agent: ClaudeBot +Allow: / + +User-Agent: CCBot +Disallow: / + +User-Agent: Bytespider +Disallow: / + +User-Agent: * Allow: / Sitemap: https://tazel.net/sitemap.xml |