Crawlability is whether a bot (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot) can reach your pages by following links and requesting URLs without being blocked. Blockers include robots.txt disallows, login walls, links hidden inside JavaScript click events, infinite redirect loops, and firewalls or bot-protection services that challenge crawlers.
Analogy: your site is a library and crawlers are inspectors. If the doors are locked or the hallways dead-end, the best books inside are never catalogued. A common real-world failure is a Cloudflare or WAF rule that blocks AI crawlers by default, which silently removes a brand from ChatGPT and Perplexity answers.
Why 10: nothing else matters if bots cannot reach the page. This applies doubly to LLMs, since many AI crawlers do not render JavaScript and give up faster than Googlebot.
#Indexability is whether a crawled page is allowed into, and accepted by, the search index. Pages can be crawlable but not indexable because of a noindex meta tag, an X-Robots-Tag header, a canonical pointing elsewhere, thin or duplicate content, or a quality judgment by Google ("Crawled, currently not indexed" in Search Console).
Example: a staging-site noindex tag accidentally pushed to production can deindex an entire site within days. That single line of code is one of the most common causes of catastrophic traffic loss.
Why 10: a page that is not indexed cannot rank in Google, and AI Overviews and Gemini draw from Google's index. Bing's index similarly feeds ChatGPT search and Copilot.
#An XML sitemap is a file (usually /sitemap.xml) listing the URLs you want indexed, often with last-modified dates. It is a discovery aid, not a ranking factor. Think of it as a table of contents handed to the crawler, useful when the internal link structure alone might miss pages.
Example: a news site with 50,000 articles uses sitemaps segmented by type and date so new articles are found within minutes. Sitemaps also give clean per-section index coverage reports in Search Console.
Why 5: essential for large, new, or poorly linked sites, but a small site with solid internal linking gains little. Accurate lastmod dates do help fresh content get recrawled, which matters for AI search freshness.
#Robots.txt is a plain-text file at your domain root that tells crawlers which paths they may or may not request. It controls crawling, not indexing: a disallowed URL can still appear in results if other sites link to it. In the AI era it also governs which AI bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) can access your content.
Analogy: it is the "Staff Only" sign on doors. Polite visitors obey it, and it is the main lever for deciding whether AI systems can read and cite you. Blocking OAI-SearchBot, for example, removes you from ChatGPT search citations.
Why 7: misconfiguration is catastrophic and the AI-bot decisions are strategic. Once set correctly, though, it needs little ongoing effort.
#A canonical tag (<link rel="canonical" href="...">) tells search engines which URL is the master version when several URLs show the same or very similar content. Common sources of duplicates: tracking parameters, sort orders, print versions, HTTP vs HTTPS, and trailing slashes.
Example: /shoes?color=red&sort=price and /shoes?sort=price&color=red are the same page. A canonical pointing both to /shoes?color=red consolidates link signals onto one URL instead of splitting them. Note that canonicals are hints; Google may override them if other signals disagree.
Why 7: central for e-commerce and large CMS sites, where parameter duplication can dilute rankings badly. Lower stakes for a 30-page brochure site.
#Site architecture / URL structure
8/10Site architecture is how pages are organized and linked: the hierarchy of categories, subcategories, and detail pages, and how many clicks each page sits from the homepage. URL structure is the visible reflection of that hierarchy (/services/roofing/metal-roof-repair/).
Analogy: a well-organized department store puts related items in clearly labeled sections a short walk from the entrance. A flat, logical structure (important pages within three clicks) concentrates authority on key pages and tells search engines and LLMs what your site is about.
Why 8: architecture shapes how authority flows, how topics are understood, and how efficiently bots crawl. It is hard to fix later, so getting it right early compounds.
#HTTPS encrypts traffic between browser and server using an SSL/TLS certificate. Google confirmed it as a lightweight ranking signal in 2014, and browsers label HTTP sites "Not secure."
Example: an HTTP checkout page triggers browser warnings that kill conversions and trust. Mixed content (HTTPS pages loading HTTP images or scripts) can also break features.
Why 6: today it is table stakes. Nearly every competitor already has it, so it gives no edge, but lacking it is a visible trust and ranking penalty.
#Core Web Vitals (LCP, INP, CLS)
5/10Core Web Vitals are Google's user-experience metrics measured from real Chrome users. LCP (Largest Contentful Paint) is how fast the main content appears; good is under 2.5 seconds. INP (Interaction to Next Paint, which replaced FID in March 2024) is how fast the page responds to clicks and taps; good is under 200 ms. CLS (Cumulative Layout Shift) is how much the layout jumps while loading; good is under 0.1.
Analogy: a restaurant where food arrives fast (LCP), the waiter responds when you wave (INP), and the table does not move while you eat (CLS). Example: an ad slot that loads late and pushes the article down causes high CLS and accidental clicks.
Why 5: Google calls page experience a tie-breaker. Relevance and authority beat speed, but in competitive SERPs where content is comparable, good vitals help, and they lift conversions regardless.
#Page speed optimization
6/10Page speed optimization is the broader engineering work behind fast pages: reducing server response time (TTFB), minifying and deferring JavaScript and CSS, caching, compressing assets, and cutting third-party scripts. Core Web Vitals measure the outcome; page speed work is how you get there.
Example: removing three unused marketing tags and deferring a chat widget can cut load time by a second or more. Speed also affects crawl efficiency; a faster server lets Googlebot fetch more pages per visit, and AI fetchers that retrieve pages in real time may time out on slow ones.
Why 6: slightly above CWV because it also affects crawl rate, AI retrieval reliability, bounce rate, and conversions.
#Mobile-friendliness / responsive design
8/10Responsive design means one site adapts its layout to any screen size. Google uses mobile-first indexing, meaning it crawls and ranks based on the mobile version of your pages. If content, links, or structured data are missing on mobile, Google effectively does not see them.
Example: a site that hides its FAQ section and half its internal links behind a desktop-only menu loses those signals entirely under mobile-first indexing.
Why 8: most searches happen on mobile and Google ranks the mobile view. Nearly all modern themes are responsive, so the main risk is content parity, not layout.
#Structured data / schema markup (JSON-LD)
7/10Structured data is code (usually JSON-LD in the page head) that labels your content with a shared vocabulary from Schema.org: this is a Product with a price of $49, this is an Organization with these social profiles, this Person wrote this Article. It powers rich results (review stars, product prices, breadcrumbs, events) and helps machines understand entities and relationships.
Analogy: your page is a letter; schema is the clearly printed address label. The postal worker could read the letter to figure out where it goes, but the label removes the guesswork.
Why 7: it is not a direct ranking factor, but it unlocks rich results that lift click-through rate and clarifies entities for Google's Knowledge Graph. The LLM benefit is real but indirect. Note that Google retired FAQ rich results on May 7, 2026, after retiring HowTo in 2023, so markup choices should follow what Google still displays.
#Hreflang / international targeting
4/10Hreflang attributes tell Google which language and regional version of a page to show to which users (for example en-us, en-gb, es-mx). They prevent the UK version of a page from ranking in the US and stop near-identical English versions from being treated as duplicates.
Example: a SaaS company with /us/pricing in dollars and /uk/pricing in pounds uses hreflang so British searchers see pound prices. Implementations must be reciprocal (each page references all others), which is where most errors happen.
Why 4: it scores 0 for single-country sites and 9 for multinational ones. The average is low because most sites do not need it.
#Pagination handling is how you structure content split across multiple pages (/blog/page/2, /category?page=3). Google dropped support for rel=next/prev in 2019, so best practice now is plain crawlable links to each page, self-referencing canonicals on each page, and not noindexing deep pages that hold unique products or articles.
Example: a category page with 400 products across 20 pages; if pages 2 to 20 are canonicalized to page 1, the products listed only there may never get discovered.
Why 3: matters mainly for large e-commerce, forums, and blogs. A secondary concern for most sites.
#JavaScript rendering / SSR vs CSR
8/10Client-side rendering (CSR) builds the page in the visitor's browser with JavaScript; the raw HTML is nearly empty. Server-side rendering (SSR) or static generation sends fully formed HTML. Googlebot can render JavaScript, but in a deferred second wave, and most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript at all.
Analogy: CSR sends a flat-pack box and assembly instructions; SSR sends the finished chair. Google will assemble it eventually. Most AI bots look at the box, see no chair, and leave.
Example: a React single-page app with product descriptions loaded via API calls may be invisible to ChatGPT and Perplexity even while ranking fine in Google.
Why 8: for JavaScript-heavy sites, this is the difference between existing and not existing in AI search. It is the most under-appreciated technical issue for LLM visibility.
#Duplicate content management
6/10Duplicate content is substantially identical text appearing at more than one URL, on your site or across sites. Google does not penalize ordinary duplication, but it picks one version to show and splits link signals among the copies. Management tools include canonicals, 301 redirects, parameter handling, noindex, and writing unique content.
Example: a franchise with 40 location pages that differ only in the city name creates near-duplicates; Google may index a few and ignore the rest.
Why 6: rarely dramatic on small sites, but scaled duplication (location pages, product variants, syndicated articles) wastes crawl budget and dilutes rankings.
#Redirect management (301, 302, chains)
6/10Redirects send users and bots from one URL to another. A 301 is permanent and passes ranking signals to the destination; a 302 is temporary and tells Google to keep the original indexed. Chains (A to B to C) and loops slow crawling and can leak signal.
Analogy: a forwarding address after you move. One forward works fine; a chain of five forwarding addresses means some mail gets lost.
Example: during a site migration, failing to 301 old URLs to their new equivalents is the most common cause of a 30 to 70 percent traffic drop.
Why 6: low day-to-day, but critical during migrations, redesigns, and URL changes, when mistakes can erase years of accumulated authority.
#HTTP status codes / error handling (404, 5xx)
6/10Status codes are the server's response type. 200 means OK, 301/302 mean redirect, 404 means not found, 410 means gone permanently, 5xx means server error. "Soft 404s" are empty or error pages that wrongly return 200.
Example: 404s on deleted pages are normal and harmless. Recurring 5xx errors are not; if Googlebot repeatedly hits server errors it slows crawling and can drop pages from the index.
Why 6: occasional 404s do not hurt rankings. Persistent 5xx errors, broken internal links, and 404s on pages with valuable backlinks do.
#Log file analysis means reading your server's raw access logs to see exactly which bots requested which URLs, how often, and what status codes they got. It is the only ground truth for crawler behavior, and it now shows which AI crawlers visit and which pages they fetch.
Example: logs reveal Googlebot spends 60 percent of its visits on filter URLs with no search value, while new product pages are crawled once a month.
Why 3: a diagnostic, not a lever. Invaluable on large sites, overkill for small ones.
#Crawl budget optimization
3/10Crawl budget is the number of URLs a search engine will crawl on your site in a given period, shaped by server capacity and how much Google wants your content. Optimization means cutting waste (duplicate, parameter, and low-value URLs) so crawlers spend time on pages that matter.
Analogy: an inspector with eight hours a day. If half the day goes to identical storage closets, the new wing never gets inspected.
Why 3: Google says crawl budget matters mainly for sites with more than roughly a million pages, or tens of thousands that change daily. Most sites never hit the limit.
#Faceted navigation control
4/10Faceted navigation is the filter system on category pages (size, color, brand, price). Each combination can create a unique URL, producing millions of near-duplicate pages. Control means deciding which filter pages deserve indexing ("red running shoes" has search demand) and blocking or canonicalizing the rest.
Example: a site with 10 filters of 10 values each can theoretically generate billions of URL combinations, swamping crawlers.
Why 4: a site-killer for large e-commerce if ignored, irrelevant elsewhere. The e-commerce section scores its opportunity side separately.
#CDN / hosting performance
5/10A CDN (content delivery network) stores copies of your site on servers around the world so users load from a nearby location. Hosting quality determines server response time, uptime, and capacity to handle crawler and user traffic.
Example: a site on overloaded shared hosting with 1.5-second server response will struggle with Core Web Vitals no matter how optimized the front end is. CDN bot-protection settings can also accidentally block AI crawlers, which ties back to crawlability.
Why 5: uptime and speed are foundations, and cheap hosting creates a ceiling. Beyond "fast and reliable," further investment shows diminishing returns.
#Image optimization (compression, WebP, lazy loading)
4/10Image optimization means serving images at the right dimensions, compressed in modern formats (WebP, AVIF), with lazy loading for images below the fold and eager loading for the main hero image. Images are usually the heaviest part of a page.
Example: a 4 MB hero photo converted to a 180 KB WebP can cut LCP by over a second. A common mistake is lazy-loading the hero image, which delays LCP.
Why 4: mostly a page-speed contributor. It also enables Google Images traffic, which matters a lot for e-commerce, recipes, and design niches.
#Orphan page detection
4/10An orphan page has no internal links pointing to it. Bots can only find it via sitemaps or external links, and it receives no internal authority. Detection means comparing your crawl against sitemaps, analytics, and logs to find pages that exist but are unlinked.
Analogy: a room in your house with no door from any hallway. It exists, but nobody wanders into it, and visitors assume it is unimportant.
Example: landing pages built for paid campaigns, or old blog posts dropped from pagination, often become orphans.
Why 4: fixing orphans is quick, cheap, and reliably helps the affected pages. But it rarely moves sitewide performance unless many important pages are orphaned.
#