Technical SEO: What Search Engines Read Before Visitors Do
Technical SEO is the work of making sure a search engine can fetch your page, read its code, understand what it is about and judge how well it loads, all before a visitor arrives. A crawler (the program a search engine uses to fetch pages) reads your raw code first, decides whether to run it, then reads the tags in the head of the page and the structured data inside it. Each step is a place where a page is understood correctly or misread. This guide follows that pipeline in order and explains what each piece of backend code does, why it exists and what goes wrong when it is missing.
Key takeaways
- Search engines process a page in phases (crawling, rendering and indexing), and the raw code a server sends is read first, so it must carry the page's real content and tags.
- None of the major AI crawlers measured by Vercel and MERJ rendered JavaScript, so text that appears only after scripts run can be invisible to them.
- A "not found" page that answers with a success code can be indexed as if it were a real page.
- Every page needs exactly one canonical tag pointing at its own preferred address, set in the HTML as a fixed value.
- Structured data describes a business to machines in JSON-LD and must match what visitors see. Google says no special markup is needed for its AI features.
- The good Core Web Vitals thresholds are 2.5 seconds for Largest Contentful Paint, under 200 milliseconds for Interaction to Next Paint and under 0.1 for Cumulative Layout Shift.
- Free tools let a non-developer check every layer described here.
In this guide
The journey of a page: crawl, render, index, serve
A page reaches a searcher in four stages: it is crawled, rendered, indexed and served. Google's documentation on JavaScript SEO basics describes the first three as the phases it uses to process JavaScript pages.
Crawling starts when Google fetches a URL. It checks robots.txt (a plain text file at the root of a site that tells crawlers which paths they may fetch), parses the response for links in href attributes and queues pages for rendering. Rendering means a headless Chromium (a browser with no visible window) runs the page's JavaScript and produces the finished HTML. Google then indexes that rendered HTML, meaning it stores and organizes the content so it can be retrieved for a search.
The queue before rendering matters. Google's documentation says a page "may stay on this queue for a few seconds, but it can take longer than that." Anything that appears only after JavaScript runs waits its turn.
| Stage | What happens | What it reads |
|---|---|---|
| Crawl | The URL is fetched after a robots.txt check; links are collected | Raw HTML and the status code |
| Render | A headless Chromium runs the JavaScript | The finished page |
| Index | The rendered HTML is stored | Text, tags, links, structured data |
| Serve | The page is ranked for a query | Everything stored at indexing |
HTML first: why the raw code still matters
The raw HTML is the document a server sends before any script runs, and it is the most dependable thing a crawler reads. A page can look complete to a visitor because JavaScript fills it in, while the code underneath is nearly empty.
This is common on single-page applications (sites that load one shell and draw each page with JavaScript). Google can render them, but it is not the only reader. Its guidance says server-side or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript." Server-side rendering means the server builds the finished HTML before sending it.
Which crawlers never run JavaScript
AI crawlers are the clearest case. In The rise of the AI crawler (December 2024), Vercel and MERJ found that none of the major AI crawlers measured rendered JavaScript. That covered OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. In one month on Vercel's network, GPTBot made 569 million requests and ClaudeBot 370 million.
If the text describing your services appears only after scripts run, a crawler that does not run scripts reads an empty page. Each address should serve its own content in the raw code.
Status codes: the answer before any content
A status code is a three-digit number a server returns before it sends any content, telling the crawler what happened to the request. A 200 means the page exists and was delivered. A 301 means the page moved permanently and the crawler should follow the redirect. A 404 means nothing exists at that address.
The soft 404 trap
A soft 404 is a page that shows "not found" to a visitor but answers with a success code. Because the server said 200, the crawler treats the page as real and can index it.
Google's documentation gives two ways to avoid soft 404 errors in single-page apps: redirect to a URL where the server answers with a 404 status code, or add a robots noindex tag (a tag telling search engines not to index a page) to error pages. A raw status check cannot catch the problem, because it reads 200 and moves on. Only looking at what renders will.
The canonical tag: which address is the real one
A canonical tag is a line in the head of a page that names the preferred address for that page. It exists because one page can be reachable at several addresses: http or https, upper-case or lower-case letters, with or without a trailing slash. Without it, search engines guess which version to credit, and clicks can be divided between versions.
Google's documentation says injecting a canonical tag with JavaScript is possible but not recommended, and that you shouldn't use JavaScript to change the canonical URL to something else. The HTML is the best place for it. A page should carry exactly one, because two conflicting tags leave the crawler to choose.
Where the domain in the tag comes from also matters. A canonical read from the browser's current address can point search engines at a preview copy of the site, because the preview has its own address. Writing the domain as a fixed value avoids that.
The head of the page
The head is the section of the HTML that holds information about the page rather than content shown on it. Several tags in it shape how the page appears in results.
Title tag.
The headline shown for the page in results. Lead with the keyword and the place, and keep it short enough to display in full.
Meta description.
The summary shown under the title. It is the pitch a searcher reads before deciding to click.
Robots meta tag.
Tells crawlers whether to index the page and follow its links. A stray noindex removes a good page from results.
Open Graph tags.
Tags that control how a page appears when shared and read by other services. The og:site_name tag names the site.
Keywords meta tag.
Google's 2009 announcement says Google does not use it in web ranking, so it can be skipped.
Why the description is a real lever
The site name line
Google's site name documentation says its site-name line draws on the WebSite structured data name, the og:site_name tag and the homepage title. When the three disagree, the line can show a name the owner did not choose. We publish all three from one block so they always agree, and the homepage title is the business name only.
Structured data: describing a business to machines
Structured data is code that states facts about a page in a fixed vocabulary, so a machine does not have to infer them from prose. Google's introduction to structured data recommends JSON-LD, a block of data placed in the page's code, separate from the visible layout. Google says it works hard to understand the content of a page and that structured data gives it explicit clues about the page's meaning, which can enable more engaging results known as rich results (enhanced listings with extra details). Google also says not to add structured data about information that is not visible to the user.
For a local business the vocabulary comes from schema.org, a shared set of type names.
| Type | What it states | Why it matters |
|---|---|---|
| WebSite | The site's name and address | Feeds the site-name line |
| LocalBusiness (or a more exact subtype) | Name, hours, service area, map link | Describes the business as an entity |
| Service | Each service and the areas served | Ties services to places |
| Review and AggregateRating | Real ratings and quoted reviews | States reputation in machine-readable form |
| BreadcrumbList | The page's position in the site | Shows how pages relate |
| FAQPage | Questions and answers on the page | Describes the content to machines |
What structured data does and does not do
It helps machines understand a page and can enable rich results. It does not promise a ranking. Rich results also change: a June 2025 post on Google's blog says Google would phase out support for several structured data features, and the Search Central updates log records that the FAQ rich result stopped appearing in Google Search starting May 7, 2026. FAQ markup still describes the content to machines. It no longer produces the expanded listing.
On AI features, Google's documentation on AI features and your website says there are no additional requirements to appear in AI Overviews or AI Mode, and no special markup or AI text files are needed. Structured data is best treated as clear labeling of a page, not a switch for AI visibility.
Hidden content: accordions, tabs and what a crawler can see
An accordion is a block that expands when clicked, and a tab is a panel that shows when its label is selected. Both hide content until asked. The risk is in how the hiding is done.
Some accordion components remove the answer text from the page code while collapsed and add it back only on a click. A crawler does not click, so it reads the questions with no answers beneath them.
Google's spam policies address a different matter: content placed on a page solely to manipulate search engines and not readily visible to people is a violation. An answer a visitor can open is content for people. The aim is to keep it in the code so a crawler reads what a visitor can open.
Speed and stability: Core Web Vitals
Core Web Vitals are three measurements of how a page feels to load and use. Largest Contentful Paint (LCP) measures how long the biggest visible element takes to appear. Interaction to Next Paint (INP) measures how quickly the page responds to a tap or click. Cumulative Layout Shift (CLS) measures how much the layout jumps while loading.
Google Search Central sets the good scores. The web.dev pages on LCP, INP and CLS give the poor scores, and web.dev's Web Vitals page notes that INP became a stable Core Web Vital in 2024, replacing First Input Delay.
| Metric | Good | Poor |
|---|---|---|
| Largest Contentful Paint | Within 2.5 seconds | Over 4 seconds |
| Interaction to Next Paint | Under 200 milliseconds | Over 500 milliseconds |
| Cumulative Layout Shift | Under 0.1 | Over 0.25 |
For context, the HTTP Archive Web Almanac 2025 Performance chapter found that 48% of websites passed all three on mobile (44% in 2024) and 56% on desktop.
Getting found: sitemaps, IndexNow and crawl budget
A sitemap is a file listing the addresses a site wants crawled. Sitemap generators that read route patterns count one dynamic route (a pattern such as a city page that fills in a place name) as one URL, so a site with hundreds of real pages can publish a sitemap listing a handful. The fix is to list every real address.
IndexNow lets a site notify participating search engines, including Bing, as soon as a page is added, updated or deleted. It supplements a sitemap rather than replacing it. We submit all client domains in bulk (1,896 URLs in one push in July 2026), resubmit sitemaps weekly and keep every domain verified in Bing Webmaster Tools.
Crawl budget is the set of URLs Google "can and wants to crawl," according to Google's crawl budget guide. That guide is aimed at large sites (1 million or more unique pages whose content changes about once a week) and medium or larger sites (10,000 or more unique pages with content that changes daily). Smaller sites rarely need to manage it, so for a local service business the better use of time is correct status codes, one canonical per page and a complete sitemap.
How to check your own site
Each step uses a free tool and needs no coding.
View the raw code.Right-click a page in your browser and choose View Page Source. Search the code for a sentence from the page. If it is missing, JavaScript is adding the text after load.
Check the head tags.In the same view, search for
<title,rel="canonical"andname="description". Confirm the title and description are unique and that exactly one canonical tag holds your https address.Inspect the URL.Paste the address into URL Inspection in Google Search Console. It shows whether the page is indexed, which canonical Google chose and how the page renders.
Search the rendered HTML.Use the live test in URL Inspection to view the rendered HTML, and look for a sentence from inside an accordion. If the answer is there, a crawler can read it.
Test a missing page.Type a made-up address on your own domain. A correct site shows a not-found page and returns a 404, which URL Inspection will confirm.
Run the Rich Results Test.Paste a page address into Google's Rich Results Test to see which structured data types it finds and whether any are invalid.
Measure speed.Enter the address into PageSpeed Insights and compare its LCP, INP and CLS results with the thresholds above.
Review the sitemap.Open yoursite.com/sitemap.xml and compare its URL count with your real page count.
What clean data says about the business
Technical work is what makes the numbers trustworthy. When pages are indexed correctly and call taps, text taps and form submissions are recorded properly, Search Console and Analytics show which services people want, where the calls come from and at what time of day. Without that, an owner is guessing.
Those numbers inform decisions that have nothing to do with code: how to price a service that draws steady calls, which hours to keep, which towns to serve, and when the call volume is high enough to justify another technician. Working with us includes this kind of business advice, built on what we see each month in the search demand, the call and message volume and the competitors in each market. Our business growth advice page lays out the topics, and hiring and growing a team shows how call data feeds that decision.
How to apply this
Make each URL serve its own content in the raw HTML, and confirm it with View Page Source.
Give every page one self-referencing canonical in lowercase on the https domain, written as a fixed value.
Confirm missing pages return a true 404 and no error page answers with a 200.
Write a unique title and a 140 to 160 character description for each page, with a homepage title that is the business name only.
Publish structured data that matches what visitors see, and keep every collapsed answer in the code.
Preload the main image, give images fixed dimensions and aim for the good Core Web Vitals thresholds.
List every real address in the sitemap, resubmit it and verify the domain in Bing Webmaster Tools.
Keep a permanent Search Console verification tag in the head, and judge indexing with URL Inspection rather than the
site:search operator.Verify tracking for call taps, text taps and form submissions from Analytics reporting data rather than from the browser.
FAQ
No. The checks in this guide use tools that report in plain language. Fixing what they find is usually a task for whoever builds the site, but an owner can find the problems and describe them precisely.
Not by itself. Google renders JavaScript pages. The work is making sure each address also serves its content and tags in the raw code, which protects the page with crawlers that do not run scripts.
That usually points to missing or inconsistent canonical tags. When http and https, or upper-case and lower-case paths, are both reachable and neither is named as preferred, search engines can credit them separately.
Google describes it as a way to help it understand a page and make the page eligible for rich results, not as a ranking boost. Treat it as accurate labeling of content already on the page.
Google's updates log records that the FAQ rich result stopped appearing in Google Search starting May 7, 2026. The markup still describes the question and answer content to machines.
Yes, once tracking and indexing are correct. Reports on calls, texts and page visits show which services are in demand and when customers reach out, which informs pricing, hours and hiring. We include that kind of advice when we work with a business.
How GMScalez applies this
For the mobile and service-area businesses we work with, the technical layer is built page by page. Each address serves its own content, carries its own canonical and written head tags, and describes the business in structured data that matches the visible page. The setup is tailored to the trade, the service areas and the way customers search, and each month we adjust it using Search Console and Analytics data. The conversation starts with a free one on one marketing consultation.
Related guides and pages
- Services: tracking and reporting and local SEO websites
- Guides: website structure and depth and when to hire your first technician
- Pricing and the Plano mobile mechanic case study
Call or text us at (712) 251-1967. Email us at garinfroehlich37@gmail.com.
Last updated: November 2026
Sources
- Google Search Central, Understand JavaScript SEO basics
- Vercel and MERJ, The rise of the AI crawler (December 2024)
- Google Search Central, Core Web Vitals
- web.dev, Web Vitals, Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift
- Google Search Central, Site names
- HTTP Archive, Web Almanac 2025, Performance chapter
- Google, Crawl budget guide
- Google Search Central, Spam policies
- Google Search Central, Introduction to structured data
- Google Search Central, Updates log
- Google Search Central blog, Simplifying the search results page (June 2025)
- Google Search Central blog, Google does not use the keywords meta tag in web ranking (2009)
- IndexNow, FAQ
- Google Search Central, AI features and your website

