Insights · Advisory

Technical SEO in the AI era: what still matters, what stopped mattering.

The honest technical SEO split: what still matters mechanically in the AI era, what stopped deserving budget, and the new items the old checklists miss.

01The article

The technical SEO checklist has been growing for twenty years. Every algorithm update added items and almost nothing was ever removed. Most organisations now hold a checklist of a couple of hundred items. A handful genuinely matter, a few dozen mattered once, and the rest are there because a tool flags them and nobody has the authority to call them noise.

AI search is a good moment to say so. The engines that now answer buying questions (AI Overviews, ChatGPT, Perplexity, Copilot) did not add a new technical discipline. They raised the price of failing the old one, and they made most of the rest irrelevant. Technical SEO in the AI era is a shorter list held to a higher standard.

This article makes the split honestly: what still matters and why, mechanically; what stopped mattering or never did; and the small set of new items the old checklists do not contain.

Why the split exists

The mechanics decide the list, so they are worth stating once. AI answer engines do not maintain a private map of the web. They retrieve from a search index, a crawl store or a live fetch, then generate an answer grounded in what came back. Google’s AI Overviews draw on Google’s index. ChatGPT’s search features depend on OpenAI’s own crawling. Perplexity retrieves and reads pages in close to real time.

That retrieval step is the entire reason technical SEO survives. A page that cannot be crawled cannot be retrieved, and a page that cannot be retrieved cannot be cited. The infrastructure questions matter exactly as much as they did in 2010, and for the same reason: can a machine reach this content, read it, and understand what it is. What changed is everything downstream of retrieval. Ranking signals that decided position four versus position seven decide very little in an answer that names three brands and ignores the rest. How engines choose those brands is its own subject. The technical layer is now pass/fail. It cannot win you the citation, and it can quietly disqualify you from one.

Everything below follows from that.

Still matters: crawlability and indexation

This is the most important item on the list. If crawlers cannot reach the page, it does not exist to any engine, traditional or generative. The usual causes are robots.txt rules written years ago for a site that no longer exists, authentication walls, orphaned pages with no internal links, and redirect chains that expire the crawler’s patience. Indexation is the same gate one step later. A page crawled but not indexed, because it is duplicated, thin or accidentally noindexed, is equally absent from the answer.

This deserves standing attention because it regresses. Every replatform, every staging deployment, and every well-meaning developer copying a robots.txt across environments can quietly de-index commercially important pages, and nothing visible breaks. The site works, the traffic erodes, and the citations never appear. Crawl and indexation status of the pages that answer buying questions is one of the few things worth checking on a schedule, because the failure is silent and total.

Still matters: rendering

AI-era retrieval sharpened an old problem. Some content exists only after JavaScript executes: client-rendered product details, specifications loaded into tabs, answers assembled in the browser. Reading it depends on the crawler being willing to render the page. In practice Google largely will, at a delay. Many AI crawlers and live-fetch systems largely will not. They take the server’s HTML response and move on.

The practical rule is blunt. Any content a brand wants quoted in an answer should be present in the server-rendered HTML, visible in view-source. Server-side rendering or static generation for commercially important pages decides whether every retrieval system can read the content or only the most patient one can. Sites built as JavaScript applications with content injected at runtime are invisible to a meaningful share of the systems now answering their buyers’ questions. Content quality cannot compensate for markup the reader never receives.

Still matters: speed, at the extremes

Site speed remains real, and the honest version is unfashionable. It matters at the extremes and barely anywhere else. A page that takes eight seconds to respond wastes crawl budget, fails live fetches, and loses human readers before the first paragraph. Fixing that is high-value work.

Shaving the last hundred milliseconds off an already-fast page is not. The industry built a cottage economy around chasing perfect Core Web Vitals scores, and the returns diminish sharply once a site is ordinarily fast. A slow site is a technical finding worth budget. Making a fast site marginally faster mostly produces an invoice.

Still matters: structured data, architecture, canonical hygiene

Three items share a mechanism, which is disambiguation: helping a machine be certain what a page is and which version of it counts.

Structured data earns its place because generative systems reason over entities as well as text. Schema that declares the organisation, its services, its people and its location in machine-readable terms gives every retrieval system a version of the facts that cannot be misread. The implementation detail is in a companion article on the machine-readable brand layer, and the broader discipline of being a clearly-defined entity is covered under entity SEO. Schema has moved from rich-snippet decoration to entity infrastructure.

Clean information architecture matters because retrieval happens at passage level and depends on context. A site where URL structure, internal links and headings agree about what each page is for gives an engine confidence about which page answers which question. A site where five pages half-answer the same question gives it a reason to cite none of them.

Canonical hygiene is the quiet one. Parameter duplicates, http/https splits, trailing-slash variants and syndicated copies without canonicals fragment one page’s standing across several URLs. Engines that cross-check sources treat unresolved duplication as an inconsistency, and an inconsistency is a reason to prefer a cleaner competitor.

What stopped mattering, or never did

The other half of the ledger is longer.

Keyword-density rituals are dead, and generative retrieval killed them twice over. The systems reading pages today are language models, and they read meaning. The meta keywords tag has been ignored by Google for well over a decade and still appears in audits as a finding. Chasing 100/100 scores in auditing tools is a category error: the tools measure what is measurable, which is a different thing from what is commercial. A 96 on a fast, crawlable, well-structured site does not need fixing.

The larger, less discussed category is the one-time item billed as ongoing work. Most entries on a typical technical SEO checklist are hygiene: XML sitemaps, canonical tags, redirect maps, robots.txt, hreflang and HTTPS. Configured correctly once, they stay correct until the site materially changes. They justify a thorough initial fix and a periodic check. They do not justify a permanent monthly line item. A large share of technical SEO retainers are one-time hygiene re-verified monthly and re-billed monthly, because nobody asked the difference between monitoring and doing.

The items are worth doing. They are also finished, and finished work should stop being paid for.

The new entrants

Three items belong on the list that the old checklists do not contain.

The first is the AI crawler access decision. Sites now receive distinct crawlers with distinct purposes, and the controls are independent of each other. OpenAI’s crawler documentation distinguishes, among others: OAI-SearchBot, which surfaces sites in ChatGPT’s search answers; GPTBot, which gathers training data; and ChatGPT-User, which fetches pages on a user’s behalf. Search inclusion is controlled via OAI-SearchBot, not GPTBot, and OpenAI directs publishers to use OAI-SearchBot in robots.txt for search opt-outs. Google’s equivalent control, Google-Extended, governs whether content trains or grounds Gemini models and, per Google’s documentation, does not affect inclusion or ranking in Google Search. These are distribution decisions with commercial consequences: who may read the content, and who may answer with it. Most of them are being made by default, in robots.txt files nobody has reviewed since the crawlers appeared. Deciding them deliberately is new technical work, and the decision belongs with strategy.

The second is extractable content structure: whether the pages that answer buying questions contain passages that survive being quoted alone. That means the answer stated declaratively, near the top, under a heading that names the question. This is on the border of technical and editorial, and it is examined properly in the citation framework. It earns a mention here because templates decide it as much as writers do.

The third is entity markup treated as core infrastructure. It is covered above, and it is listed again because its priority changed more than any other item on the list.

A shorter list, and what to do with it

The AI-era technical list is short: crawlability and indexation of the pages that matter, server-rendered content, absence of genuine speed failures, accurate structured data, coherent architecture, resolved duplication, and deliberate AI-crawler decisions. That is the whole list. Everything on it can be verified, most of it is fixable once, and very little of it justifies recurring spend once fixed.

That makes it a governance question. The commercial questions for a marketing leader are simple: is the short list right on our site, and are we still paying for the long list? Both are audit findings. The first is checked against the site itself in days, and the second is checked against the invoices. An independent marketing audit covers technical health on those terms, what is broken and what is finished, alongside the paid accounts and the measurement the technical layer feeds.

AI search clarified technical SEO. A machine either reaches, reads and understands your pages or it does not, and almost everything else on the checklist can be safely crossed off.

03Contact

Let’s talk about what’s next.

For executive advisory, fractional CMO, AI search strategy or speaking enquiries.

sam@sampark.com.au
Brisbane, Australia
Enquiry form