> For the complete documentation index, see [llms.txt](https://docs.usepylon.com/pylon-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.usepylon.com/pylon-docs/knowledge-base/seo-and-ai-crawlers.md). # SEO & AI Crawlers How Pylon makes your public knowledge base discoverable by search engines and AI tools. Pylon publishes the files search engines and AI tools need to find and read your public articles. There's nothing to set up. ## What gets indexed Only published, **Public** articles and collections are included. Customer, Internal, and **Direct link only** articles are left out. If your knowledge base's default visibility isn't **Public**, the homepage is left out too. Once your [custom domain](/pylon-docs/knowledge-base/styling-and-customization.md#custom-domain) is verified, the default Pylon-hosted address tells search engines not to index it, so your custom domain gets the search ranking. ## Sitemap Your sitemap is at `/sitemap.xml`. It lists your public homepage, collections, and articles, with the last-updated date for each. For each translated page, it also lists the other languages that page is published in, so search engines can show visitors the right [translation](/pylon-docs/knowledge-base/translations.md). Large knowledge bases are split into several sitemap files automatically. ## robots.txt `/robots.txt` points crawlers to your sitemap. It allows major search engines (such as Google, Bing, and DuckDuckGo) and AI crawlers (such as GPTBot, ClaudeBot, and PerplexityBot), and blocks other crawlers. ## llms.txt `/llms.txt` gives AI tools a Markdown index of your public articles: your knowledge base title, then each collection with links to its articles. It covers your default language. ## Pages for crawlers Search engines, social link previews, and AI crawlers receive a fully rendered HTML version of each article, collection, and homepage, so they can read your content without running JavaScript. ## Social previews Upload an **Open Graph Image** in **General** settings to use in link previews on social media and chat apps. If you don't set one, your knowledge base logo is used. To see how much of your traffic comes from bots and AI crawlers, see [Analytics](/pylon-docs/knowledge-base/analytics.md#human-bot-and-ai-crawler-traffic). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.usepylon.com/pylon-docs/knowledge-base/seo-and-ai-crawlers.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.