Writing HTML For A World Where AI Reads It Too

A proposed convention places a plain markdown file named llms.txt at a site's root — directly modeled on robots.txt's well-established location convention — containing a concise summary of the site's purpose and a deliberately curated (not exhaustive) list of links to its most important content, organized under simple markdown headings. This gives an LLM-based tool a fast, human-and-machine-readable way to orient around a site's actual purpose and priority content, without needing to first crawl and interpret the entire site structure from scratch.

This is explicitly complementary to, not a replacement for, sitemap.xml — a sitemap aims for completeness (every crawlable URL, for traditional search indexing), while llms.txt aims for curated relevance (the handful of pages that actually matter most for an AI tool trying to quickly understand what a site is and does).

The genuinely good news: correctly-used heading hierarchy, <main>/<article>/<section> used for their real structural meaning rather than as generic containers, and machine-readable dates via <time datetime="..."> are exactly the signals that let an AI-based tool cleanly extract a page's actual structure and meaning — the same discipline that has already benefited screen reader users and search crawlers throughout this course requires no separate, AI-specific markup vocabulary to also benefit this third audience.

Structured data — JSON-LD, covered in this course's SEO module — extends this further by stating certain facts (an article's author, publish date, a product's price) explicitly and unambiguously, rather than requiring any reader, human or machine, to correctly infer them from surrounding prose alone.

robots.txt supports naming specific AI crawler user-agents (such as GPTBot or other named AI-training crawlers) independently from traditional search engine bots — a site can, as a deliberate policy decision, allow indexing by search engines while disallowing a specific AI training crawler, or configure any other combination that matches its actual content and licensing intentions.

Beyond markup and crawler policy, how content itself is *written* affects extraction reliability: placing a section's direct, core answer near its beginning (rather than after several paragraphs of context-setting) and keeping sections genuinely self-contained increases the odds that a partial extraction or summary by an AI system still captures the actually important information, rather than only introductory framing.

Your Markup Now Has A Second Audience. Search crawlers were HTML's first non-human audience. AI agents and LLM-based assistants summarizing, citing, and acting on your pages are the second — and they reward exactly the same disciplined semantic HTML this course has taught from the start, plus a few new, deliberate signals.

llms.txt Gives AI Crawlers A Curated Entry Point. A proposed convention (llms.txt, at a site's root, modeled on robots.txt) offers a concise, markdown-formatted summary of a site's purpose and links to its most important content — a curated map specifically for LLM-based tools, distinct from the exhaustive, unranked list a traditional sitemap.xml provides.

llms.txt's Purpose. How does llms.txt differ from a traditional sitemap.xml?

• →llms.txt is a curated, prioritized summary; sitemap.xml is an exhaustive, unranked URL list
• →They serve an identical purpose with different file formats only
• →llms.txt fully replaces the need for sitemap.xml

Semantic HTML Is Already The Best AI-Extraction Format. Every discipline this course has taught — real headings in hierarchical order, <main> for primary content, <article> for self-contained pieces, <time datetime> for dates — directly determines how cleanly an LLM-based tool can extract a page's actual structure and meaning, exactly the same signal that already benefited screen readers and search crawlers.

Semantic HTML's AI Payoff. Does correctly using <article>, real headings, and <time datetime> require any AI-specific markup beyond standard semantic HTML practice?

• →No — the same semantic HTML discipline already taught for accessibility and SEO directly benefits AI extraction too
• →Yes, entirely separate AI-specific tags are required
• →Only JSON-LD structured data matters; regular HTML semantics are irrelevant to AI tools

Structured Data (JSON-LD) Gives AI Systems Unambiguous Facts. Beyond well-structured prose, JSON-LD structured data (already covered in this course's SEO module) gives AI systems explicit, unambiguous facts — a product's exact price, an article's precise author and publish date — reducing reliance on the system correctly inferring those facts from surrounding text alone.

Structured Data For AI. What advantage does JSON-LD structured data give an AI system over relying purely on inferring facts from a page's visible prose?

• →Explicit, unambiguous facts, reducing reliance on inference from surrounding text
• →It makes the page load measurably faster
• →None — AI systems ignore structured data entirely

robots.txt And Meta Robots Now Have AI-Specific Directives. Beyond the traditional Googlebot/Bingbot entries, robots.txt can now name specific AI crawler user-agents (like GPTBot or specific AI-training crawlers) to allow or disallow independently from traditional search indexing — letting a site permit search indexing while opting out of AI model training crawling, or vice versa, as a deliberate policy choice.

AI-Specific robots.txt Entries. Can a site's robots.txt allow indexing by traditional search crawlers while disallowing a specific AI training crawler?

• →Yes — each user-agent entry is controlled independently
• →No, robots.txt only supports one universal allow/disallow policy for every bot
• →Only via an entirely separate file format, not robots.txt itself

Write For Extraction: Answer-First Structure And Self-Contained Sections. Beyond markup mechanics, content structured with a clear, direct answer near the top of a section (rather than requiring several paragraphs of buildup first) and genuinely self-contained sections (each meaningful without necessarily reading everything before it) extract more reliably and accurately when an AI system summarizes or quotes a specific portion of a page.

Answer-First Content Structure. Why does placing a section's direct answer near the top, before extended elaboration, help AI-based extraction and summarization?

• →A system extracting or quoting only part of a section is more likely to capture the actual core answer
• →It makes the page crawl noticeably faster for bots
• →It's a formal requirement of the HTML specification

HTML For AI & LLMs Mastered. You now understand llms.txt as a curated AI entry point, why the semantic HTML discipline this course has taught all along already serves AI extraction, how structured data gives AI systems explicit facts, and how to set independent crawl policy for AI-specific bots.

Make Your Page's Meaning Explicit For AI Crawlers. Structured data via JSON-LD gives AI systems and search engines an explicit, unambiguous meaning for your content.