llms.txt vs robots.txt vs sitemap.xml
Three files, three jobs. Motif ships all three on motiftrace.com. Only robots.txt has a vendor-documented effect on named crawlers. llms.txt remains a contested signal.
robots.txt
This is the policy file GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and CCBot document. Search-index bots and model crawlers are different decisions. User-requested fetchers often ignore it. Motif's marketing robots.txt allows public pages and disallows app routes.
sitemap.xml
A sitemap lists canonical URLs. Motif's sitemap is generated from the same catalog as the footer, so a page cannot be indexable in one list and missing in the other.
llms.txt
An emerging convention: a markdown brief at /llms.txt summarizing the product for language models. Motif publishes one with operator, audience, money rules, and every indexable page. We do not claim it changed citation rates. Anyone who tells you it did, without a measurement loop, is selling a file.
How they conflict
Allowing a bot in robots.txt while omitting the URL from the sitemap is allowed; the bot may still find the page. Disallowing a bot while advertising the page in llms.txt is a mixed signal. Motif will not resolve vendor crawler policy for you. The robots.txt checker shows which of Motif's classified tokens you named.
Should I publish llms.txt?
It is cheap and honest if the file matches the site. It is not a substitute for first-touch measurement. Motif publishes one because the product facts belong in a stable, citeable form.
Does Motif read my llms.txt?
No. Motif's tracker does not fetch llms.txt from customer sites.
Related