Does OneFront let AI crawlers read my page?
Updated July 20, 2026
The short answer
Yes — OneFront's robots.txt explicitly allows the major named AI crawlers, including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, to read public pages, while keeping private areas like the dashboard and API disallowed. Combined with server-rendered content, that means these crawlers can actually reach and read your page's text and structured data. robots.txt is a voluntary standard, so it works for crawlers that choose to honor it — which the major named ones document that they do.
There's no point having a machine-readable page if the machines are turned away at the door. OneFront's robots.txt explicitly invites the major AI crawlers in — while keeping your private areas closed.
What the allowlist says
Visit /robots.txt and you'll see named rules allowing GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended, alongside standard search crawlers, to read public pages. Private areas — the dashboard, onboarding, the API — are disallowed. So the parts of OneFront meant to be public are open to the crawlers that increasingly power AI answers, and the parts that aren't stay shut.
Readable, not guaranteed
Two honest caveats. First, robots.txt is a voluntary standard — it only governs crawlers that choose to honor it, which the major named ones document that they do. Second, being crawlable is about eligibility, not outcomes: allowing crawlers plus server-rendered content makes your page genuinely readable and usable, but no site can promise it'll be cited in a given AI answer or search result. What OneFront controls is the readability. It gets that part right by default.
How to allow ai crawlers, step by step
- 1See the allowlist. Visit onefront.org/robots.txt to see the rules — the named AI crawlers are explicitly allowed to read public pages.
- 2Note what's blocked. Private areas (the dashboard, onboarding, and API) are disallowed, so only your public content is crawlable.
- 3Publish to be readable. Because pages are server-rendered, an allowed crawler that fetches your page gets your real content, not an empty shell.
Frequently asked
- Which AI crawlers are allowed?
- The robots.txt names GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended, alongside standard search crawlers, and allows them to read public pages.
- Does allowing crawlers guarantee I'll be cited?
- No. Allowing crawlers plus server-rendering makes your page readable and eligible to be used — but no site can guarantee inclusion in any AI answer or search result.
One clean page for you and your agents — real links, readable by people and AI alike.
See how it works