diff options
| author | Leitwolf11 | 2026-11-05 06:50:56 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-05 06:50:56 +0300 |
| commit | 67423dfb0d867326c4e6f1f1f0d4ee3495837258 (patch) | |
| tree | 2ab2db624c0bc400600fd1b5790ba21904889ba1 /frontend/src/styles | |
| parent | 9b4cc75dade5d9ab7e03284d52916bb2a6fddf36 (diff) | |
| download | website-seo/robots-txt.tar.gz | |
Use the crawler rules of progmasoft.com on tala.runseo/robots-txt
tala.run and its talac, tazel, talafmt and talalint hosts take the
rules progmasoft.com and tazel.net use: Googlebot, Google-Extended,
OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and
ClaudeBot are allowed; GPTBot, CCBot and Bytespider are turned away;
every other crawler is allowed. The rules for PerplexityBot,
Perplexity-User, meta-externalagent, meta-externalfetcher, FacebookBot,
DuckAssistBot, Amazonbot, Google-CloudVertexBot and MistralAI-User go,
so they fall under the rule for every crawler. Each host keeps its own
sitemap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffstat (limited to 'frontend/src/styles')
0 files changed, 0 insertions, 0 deletions