diff options
| author | Leitwolf11 | 2026-08-28 18:26:30 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-08-28 18:26:30 +0300 |
| commit | 319c9800ba6cb4e1311846ed538cf0e5c8df3288 (patch) | |
| tree | c42f36af444835b693b64d61087626c0a30989a2 | |
| parent | 5c479373db96fcd2822554fceb23b22c4e9f12b6 (diff) | |
| download | website-319c9800ba6cb4e1311846ed538cf0e5c8df3288.tar.gz | |
Add crawler policy for the Visual X# website
| -rw-r--r-- | frontend/public/robots.txt | 61 |
1 files changed, 61 insertions, 0 deletions
diff --git a/frontend/public/robots.txt b/frontend/public/robots.txt new file mode 100644 index 0000000..801efe3 --- /dev/null +++ b/frontend/public/robots.txt @@ -0,0 +1,61 @@ +User-agent: Googlebot +Allow: / + +User-agent: Google-Extended +Disallow: / + +User-agent: OAI-SearchBot +Allow: / + +User-agent: ChatGPT-User +Allow: / + +User-agent: GPTBot +Allow: / + +User-agent: Claude-SearchBot +Allow: / + +User-agent: Claude-User +Allow: / + +User-agent: ClaudeBot +Disallow: / + +User-agent: PerplexityBot +Disallow: / + +User-agent: Perplexity-User +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: CCBot +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +User-agent: meta-externalfetcher +Disallow: / + +User-agent: FacebookBot +Disallow: / + +User-agent: DuckAssistBot +Allow: / + +User-agent: Amazonbot +Disallow: / + +User-agent: Google-CloudVertexBot +Disallow: / + +User-agent: MistralAI-User +Disallow: / + +User-agent: * +Allow: / + +Sitemap: https://xsharp-lang.xyz/sitemap.xml |