How to Optimize Your Website and Blog for AI

For twenty years we optimized sites for Google. Now we need to explain them to machines that read the site instead of a human.

A search engine showed a person ten blue links — then the person figured things out on their own. An AI agent doesn’t work that way: it reads the page, understands it, and immediately answers the user or makes a decision — who to recommend, who to contact, what to buy. If the agent didn’t understand you, you’re not «lower in the rankings.» You’re simply not in the answer.

Good news: you don’t need a new site. You need to tidy up a few layers — and they only work together.

Three layers: gatekeeper, guidebook, passport

In short:

• robots.txt — decides who gets in at all, and points bots to llms.txt.
• llms.txt — a human-readable summary at the site root: who you are, services, contacts, key pages. A search engine crawls the whole site and guesses. For an agent, you provide a ready-made map.
• Schema.org / JSON-LD markup — a machine-readable passport for the page: not «nice text about the company,» but structured data — organization, contacts, services, and the relationships between them. Without it, the agent guesses. With it, the agent knows.

The gatekeeper lets you in, the guidebook explains, the passport confirms. Remove any layer — the chain breaks.

Honest caveat: llms.txt is a young standard — more of an emerging convention than an approved norm — and not all bots read it the same way. But it takes about half an hour, and it doesn’t hurt — so it’s on my checklist.

1. Core: make content easy to understand

No robots.txt will save a page that neither humans nor machines can parse.

• One page, one main topic.
• Clear H1, logical H2/H3, short paragraphs.
• Direct answers, definitions, lists, FAQs.
• Author, publish date, and update date.
• Sources, contacts, an About page, and an editorial policy.

2. Site structure

• Clean URLs, breadcrumbs, canonical tags, sitemap.xml.
• Semantic HTML: <header>, <main>, <article>, <nav>, <footer>.
• Related content and internal linking.
• Don’t hide important text inside JavaScript only.
• Mobile version and fast load times are mandatory.

3. robots.txt — crawling rules

The root file controls bot access to sections and points to the sitemap:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

# Link to site info for LLMs
# https://example.com/llms.txt

Linking to llms.txt in a comment is a useful practice, though not yet an official universal standard.

4. llms.txt — a short site map for LLMs

A root file (/llms.txt) briefly explains the site’s purpose, key sections, documentation, contacts, and top resources.

Example:

# Example Site
Description: Blog and knowledge base about AI and SEO.
Sections: /blog/, /docs/, /faq/
Key resources: /guide/ai-seo
Contact: /contact/
Updated: 2026-06-02

Include: a short project description, target audience, main sections, links to best materials, contacts, policy, and last update date.

5. Structured data / Schema.org

JSON-LD is the preferred format. Recommended types:

Minimal article template:

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "...",
  "description": "...",
  "author": { "@type": "Person", "name": "..." },
  "datePublished": "YYYY-MM-DD",
  "dateModified": "YYYY-MM-DD",
  "mainEntityOfPage": "URL"
}

6. What AI systems especially like

• Clear summary blocks at the start of the content.
• Term → definition → example.
• Lists, tables, checklists.
• Q&A and «Quick take» / TL;DR blocks.
• Alt text on images and captions on illustrations.
• Transcripts for video and audio.

7. Managing AI bots

Review rules for specific user-agents. Decide what to index and what not to. Block private, test, and duplicate sections. Revisit robots.txt regularly — the bot list keeps growing.

User-agents worth managing: GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot.

8. Pre-publication checklist

• [ ] Clear title and H1
• [ ] Summary and FAQ included
• [ ] Author and update date present
• [ ] Schema.org / JSON-LD implemented
• [ ] sitemap.xml updated
• [ ] robots.txt configured
• [ ] llms.txt prepared
• [ ] Internal linking in place

Takeaway

The site stays the same for people. Now machines understand it too. In a world where an agent — not a person — increasingly picks the vendor, this isn’t «nice to have.» It’s hygiene.

The formula: strong content + clear structure + robots.txt + llms.txt + schema.org = a site that both people and AI understand better.

Works for blogs, corporate sites, documentation, e-commerce, and knowledge bases.

Agent artifact: generate llms.txt in 15 minutes

Theory above — now practice. This open repository lets readers hand their AI agent (Cursor,
Claude, ChatGPT, etc.) enough context to draft llms.txt plus a robots.txt
hint for their site.

Open the repository: ai-llms-generator on GitHub

Repository contents:

Cursor

1. Open the repository and copy SKILL.md (and templates) into .cursor/skills/generate-llms-txt/ in your project.
2. In chat: /generate-llms-txt create llms.txt for https://my-site.com

Claude / ChatGPT / other agents

1. Open PROMPT.md in the repository.
2. Fill in URL, site type, language, and audience.
3. Paste into chat → save output as llms.txt → deploy to the site root.

The agent will inspect the sitemap and navigation, pick key pages, and return a
robots.txt snippet. Verify URLs yourself before going live — the agent does
not replace judgment.

Originally published on LinkedIn.