Google-Extended
Google-Extended is a robots.txt product token that controls whether Google uses crawled content for Gemini and Vertex grounding. It is not a distinct crawler that shows up in access logs as Google-Extended.
- Operator
- Google
- Motif family
- google_ai
- Motif class
- robots.txt product token
- In Motif classifier
- No
- robots.txt
- This token exists for robots.txt control, not as a crawl User-Agent
- Executes JavaScript
- Not applicable
- Verified in Motif
- Not an HTTP User-Agent Motif can match
What Motif will and will not claim
Motif's classifier intentionally omits Google-Extended. Robots policy-only tokens are not HTTP User-Agents, so a tracker can never observe them. Directories that list Google-Extended beside GPTBot are mixing two different objects.
Revenue
Changing Google-Extended does not create or remove gemini.google.com visitors. Gemini first touch is still a person.
Vendor documentation
Start with the vendor page at https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers. Motif stores the controlled token from a self-reported User-Agent.
Google-Extended User-Agent example
Not a User-Agent. Place User-agent: Google-Extended in robots.txt.
Allow Google-Extended in robots.txt
User-agent: Google-Extended
Allow: /
Block Google-Extended in robots.txt
User-agent: Google-Extended
Disallow: /
Does Google-Extended count as a visit in Motif?
Google-Extended is not an HTTP User-Agent, so Motif never records it as a crawler hit or a visit.
Will Motif show Google-Extended on my HTML pages?
Only if that client requested /m.js or POST /v1/events. HTML-only crawls of the origin never reach Motif. Saying otherwise would be a product lie.
Should I block Google-Extended?
This is a robots.txt product token. Allow or disallow it according to whether you want that vendor to use Applebot or Googlebot-crawled content for the named product. Motif will not observe the token either way.
Related