Validar robots txt e cache do Google

• Cache no Google Search Console | robots.txt

verifique a data da ultima indexação realizada pelo google e solicite um novo rastreamento do arquivo robots.txt

---

---

• ferramentas p/ verificação do robots.txt online:

Robots.txt Tester: Check Your Robots.txt File for FreeSE Ranking

robots.txt Testing ToolTame The Bots

---

• exemplos robots.txt padrão: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt?hl=pt-br

User-agent: Applebot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Bingbot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Googlebot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Googlebot-Image
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Mediapartners-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Adsbot-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: AdsBot-Google-Mobile
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Storebot-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Google-InspectionTool
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: GoogleOther-Image
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: msnbot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account


User-agent: *
Disallow: /

---

• Veja tambem: Site-map e Google Search Console.