Ihor_Zhekov/Superfice_Website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

robots symbols fix

Browse Source
This commit is contained in:
Ihor_Zhekov
2026-05-04 11:22:51 +02:00
parent b0ab44199d
commit 51b29aaa1c
3 changed files with 65 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
public/robots.txt
View File

@@ -1,12 +1,12 @@
# ── Default: all bots ────────────────────────────────────────────
# Default: all bots
User-agent: *
Allow: /
Disallow: /impressum
Disallow: /datenschutz
Disallow: /api/
# ── AI crawlers: same restrictions ───────────────────────────────
# Personal data (Impressum, Datenschutz) must not be indexed — DSGVO.
# AI crawlers: same restrictions
# Personal data (Impressum, Datenschutz) must not be indexed - DSGVO.
# Business content and llms.txt are explicitly available.
User-agent: GPTBot
@@ -90,7 +90,7 @@ Disallow: /impressum
Disallow: /datenschutz
Disallow: /api/
# Bing (explicit — used by Copilot / ChatGPT via Bing index)
# Bing (explicit - used by Copilot / ChatGPT via Bing index)
User-agent: Bingbot
Allow: /
Allow: /llms.txt