Meta-WebIndexer

Meta documents Meta-WebIndexer as search indexing. Motif keeps it distinct from Meta-ExternalAgent and Meta-ExternalFetcher.

Operator
Meta
Motif family
meta
Motif class
Search indexing
In Motif classifier
Yes
robots.txt
Check Meta's current crawler documentation
Executes JavaScript
No
Verified in Motif
No public IP list Motif relies on

What Motif will and will not claim

Motif classifies Meta-WebIndexer as family meta, class search. Many directories omit this token; Motif does not.

Revenue

Indexing is not a Stripe customer. Crawler hits stay off the ranking.

Vendor documentation

Start with the vendor page at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. Motif stores the controlled token from a self-reported User-Agent.

Meta-WebIndexer User-Agent example

meta-webindexer

Allow Meta-WebIndexer in robots.txt

User-agent: Meta-WebIndexer
Allow: /

Block Meta-WebIndexer in robots.txt

User-agent: Meta-WebIndexer
Disallow: /

Does Meta-WebIndexer count as a visit in Motif?

No. If Meta-WebIndexer hits the Motif tracker or collector it increments AI crawlers. It does not create a visitor and cannot win a recommendation.

Will Motif show Meta-WebIndexer on my HTML pages?

Only if that client requested /m.js or POST /v1/events. HTML-only crawls of the origin never reach Motif. Saying otherwise would be a product lie.

Should I block Meta-WebIndexer?

Search-index bots are how some assistants find pages to cite. Blocking them is a discovery policy, not a ranking in Motif. User-requested fetchers and model crawlers are separate decisions.

Related