Validar robots txt e cache do Google

Cache no Google Search Console | robots.txt

ferramentas p/ verificação do robots.txt online:

exemplos robots.txt padrão: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt?hl=pt-br


User-agent: Applebot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Bingbot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Googlebot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Googlebot-Image
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Mediapartners-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Adsbot-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: AdsBot-Google-Mobile
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Storebot-Google
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: Google-InspectionTool
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: GoogleOther-Image
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account

User-agent: msnbot
Allow: /
Disallow: /cart
Disallow: /index.php?spsr=checkout
Disallow: /index.php?spsr=account
Disallow: /checkout
Disallow: /account


User-agent: *
Disallow: /

Veja tambem: Site-map e Google Search Console.

PreviousValidação de domínio por DNS - Inserção de registro TXT via Registro.BR NextAdicionar Selos e Certificados SSL no site

Last updated 4 months ago

User-agent: Applebot Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Bingbot Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Googlebot Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Googlebot-Image Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Mediapartners-Google Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Adsbot-Google Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: AdsBot-Google-Mobile Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Storebot-Google Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: Google-InspectionTool Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: GoogleOther-Image Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: msnbot Allow: / Disallow: /cart Disallow: /index.php?spsr=checkout Disallow: /index.php?spsr=account Disallow: /checkout Disallow: /account User-agent: * Disallow: /