Meta-WebIndexer
Meta documents Meta-WebIndexer as search indexing. Motif keeps it distinct from Meta-ExternalAgent and Meta-ExternalFetcher.
- Operator
- Meta
- Motif family
- meta
- Motif class
- Search indexing
- In Motif classifier
- Yes
- robots.txt
- Check Meta's current crawler documentation
- Executes JavaScript
- No
- Verified in Motif
- No public IP list Motif relies on
What Motif will and will not claim
Motif classifies Meta-WebIndexer as family meta, class search. Many directories omit this token; Motif does not.
Revenue
Indexing is not a Stripe customer. Crawler hits stay off the ranking.
Vendor documentation
Start with the vendor page at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. Motif stores the controlled token from a self-reported User-Agent.
Meta-WebIndexer User-Agent example
meta-webindexer
Allow Meta-WebIndexer in robots.txt
User-agent: Meta-WebIndexer
Allow: /
Block Meta-WebIndexer in robots.txt
User-agent: Meta-WebIndexer
Disallow: /
Does Meta-WebIndexer count as a visit in Motif?
No. If Meta-WebIndexer hits the Motif tracker or collector it increments AI crawlers. It does not create a visitor and cannot win a recommendation.
Will Motif show Meta-WebIndexer on my HTML pages?
Only if that client requested /m.js or POST /v1/events. HTML-only crawls of the origin never reach Motif. Saying otherwise would be a product lie.
Should I block Meta-WebIndexer?
Search-index bots are how some assistants find pages to cite. Blocking them is a discovery policy, not a ranking in Motif. User-requested fetchers and model crawlers are separate decisions.
Related