Monograph III · Serving tiers
Crawl Budget, Serving Tiers and the Physics of “Crawled, Currently Not Indexed”
“Crawled, currently not indexed” is not a quality judgement in the editorial sense. It is an accounting decision: the URL did not justify the cost of a slot in fast storage.
| Sources | DOJ v. Google trial testimony (2023); Content Warehouse API documentation (2024); Google Search Central crawl-budget guidance |
|---|---|
| Content Warehouse attributes | scaledSelectionTierRank, crawlPagerank, pagerank2, crawlerStatus, navboost, chromeInTotal |
| Tier vocabulary | Flash (fastest), Base (SSD-class), secondary / pruned (HDD-class) |
During the 2023 antitrust trial, Google engineers described an index that is not one store but three, distinguished by the physical medium that serves them. The freshest and most frequently requested documents live in flash memory. The main body of the index, the tier the documentation calls Base, lives on solid-state storage. Everything else sits on spinning disk, consulted rarely and crawled less. The Content Warehouse encodes a document’s position in this hierarchy as scaledSelectionTierRank, a value normalised to the range 0–32767, and the attribute name tells you what the tiers are for: selection. The index does not keep everything it crawls. It selects.1Flash storage costs roughly an order of magnitude more per gigabyte than disk. A URL promoted to the fast tier is therefore a URL the system expects to serve often enough to repay that cost. “Currently not indexed” frequently means “currently not worth a slot.”
Crawl is downstream of tier
Practitioners treat crawl budget as the cause and indexing as the effect. The mechanics run the other way. A document’s tier sets the ceiling on how often the scheduler revisits it, and the scheduler’s priority is set by a crawl-specific PageRank, exposed as crawlPagerank, that is computed over the link graph the crawler already knows about. A URL that is poorly linked internally has a low crawl PageRank, is scheduled infrequently, arrives in the index with stale signals, and competes for a Base slot against documents whose freshness and demand are better evidenced. It is crawled, it is evaluated, and it is parked.
tier(u) = select( demand(u), cost(u) )
demand(u) ∝ α · navboost(u) + β · crawlPagerank(u) + γ · freshness(u)A working model. Demand is dominated by observed click flux and internal link equity; cost is serving size and refresh frequency. Coefficients are unknown and almost certainly query-class dependent.
NavBoost: click flux as a promotion signal
NavBoost is the system, confirmed at trial, that aggregates thirteen months of click behaviour per query and per document: good clicks, bad clicks, last-longest clicks and the slices by device and locale. Its presence in the Content Warehouse as a per-document module settles an old argument. Clicks are not a ranking factor in the naïve sense, but click flux, the rate at which a document is chosen and satisfies, is the clearest demand signal the system has, and demand is what tier selection optimises for. A page that nobody clicks is a page the system has little reason to keep in fast storage, regardless of how well written it is.2Chrome visit counts, exposed as chromeInTotal, give the system demand evidence even for pages that have never ranked, which is how a new URL with genuine direct traffic can be indexed quickly while a better-written orphan page waits for weeks.
Internal PageRank funnelling
If click flux is the demand signal you cannot manufacture, internal PageRank is the one you can. The homepage of any established domain carries the largest reservoir of equity on the site, documented as homepagePagerankNs, and every URL’s crawl priority is a function of how many damped hops separate it from that reservoir. Each hop multiplies by 0.85 and divides by the outdegree of the intermediate page. A URL four clicks deep, reached through pages that each carry eighty links, inherits a vanishing fraction of root equity, and its crawl PageRank reflects that.
PR(u) ≈ PR(home) · Π_{k=1..depth} ( 0.85 / OBL_k )Through three hops of 60-link pages the factor is (0.85/60)³ ≈ 2.8 × 10⁻⁶. Through three hops of 8-link pages it is (0.85/8)³ ≈ 1.2 × 10⁻³, more than 400 times larger.
The remedy is architectural rather than editorial. Reduce outdegree on the pages that stand between the root and the URLs you need indexed. Link to new sections from the homepage during their first weeks. Remove the thousand-link footer and the tag archives that split the root reservoir into dust. Every one of these moves raises crawl PageRank, raises scheduled frequency, and gives the document a fresher, better-evidenced case when tier selection runs.
Reading the Search Console verdicts
- Discovered, currently not indexed. The crawler knows the URL exists but has not scheduled it. This is a crawl PageRank problem: the URL is too far from the root or linked from pages with too much outdegree.
- Crawled, currently not indexed. The document was fetched and evaluated but did not win a Base slot. This is a demand problem: no click flux, no external links, and nothing on the page that the system has not already seen on a better-served URL.
- Indexed, not submitted in sitemap. Usually harmless, but a warning that the crawler is finding URLs through paths you did not intend, and spending budget on them.
None of these verdicts is permanent. Tier selection is re-evaluated continuously, and a document that acquires demand, whether through an external link, a surge of direct visits or a tighter internal path from the root, is promoted on the next pass. The system is not hostile to your page. It is merely refusing to pay for it until you give it a reason.