The short answer: AI answer engines can only cite what they can reach and understand. So let their crawlers in, put the answer in the page's HTML, add llms.txt and structured data, and write each section so it still makes sense when it's quoted on its own. Ranking well in normal search still matters, because the AI tools lean on the same pages.
This is called answer engine optimization (AEO), or GEO. Here is the checklist I just ran on theaikhan.com, and why each step matters.
1. Are the AI crawlers allowed in?
Check this first, because nothing else matters if it fails. Your robots.txt should allow the crawlers that feed answer engines: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended.
On this site, robots.txt names each one explicitly and only blocks /dev/ and /api/. Several 2026 guides make the same point: crawler access is pass-or-fail, and no amount of good writing makes up for a blocked bot (Techmagnate's guide is a good overview).
2. Is the answer in the HTML?
If your key text appears only after JavaScript runs, many crawlers never see it. Every page on this site is rendered on the server, so the FAQ answers, the case files and the posts are all in the HTML that arrives first. Anything hidden behind a click-to-expand that's built in JavaScript is a risk.
3. Does every page open with the answer?
Write the first sentence under each heading as the answer itself. The FAQ on this site uses question-shaped headings ("Who is Areeb Ahmed Khan?") followed straight away by a 40–60 word answer. Each post opens with "The short answer:". An AI tool that quotes one paragraph should get a complete thought, with the subject named, not "as mentioned above".
4. Add structured data
Structured data (JSON-LD) tells machines what a page is. On this site:
- a site-wide
PersonandWebSite, so every page points back to one identity FAQPageon the FAQ,ProfilePageon About,BlogPostingandTechArticleon postsBreadcrumbListon every inner page, matching the visible breadcrumb
5. Add llms.txt
llms.txt is a plain-text map of your site for language models: a one-paragraph summary and a list of your important pages, each with one line on what's there. Mine lists every case file, project, guide and the contact page. It's a small file and easy to keep current.
6. Keep facts consistent everywhere
This is the step people miss. While doing this pass I found my llms.txt still credited a certificate system to the wrong project. If your own pages disagree, an AI tool will repeat whichever version it saw. Fix facts in one place and make every page read from it.
7. Get mentioned elsewhere
Your own site is only part of the picture. Answer engines also trust what others say about you: interviews, directories, social profiles, other people's posts. Use the same name, title and description everywhere so they all point to one person.
The checklist
- AI crawlers allowed in
robots.txt - Key answers rendered in the HTML
- Answer-first sections with question-shaped headings
- JSON-LD on every page, with breadcrumbs
-
llms.txtwith your important pages - Every page in the sitemap
- One set of facts, used everywhere
Want this done for your site? Open a case file.