A page you can open in a browser is not a page anything has found. Between the moment a URL exists and the moment it earns a click sit several separate events, and a Front Range service-area site usually fails at the first of them.
The failure is quiet, which is why it survives so long. Nothing errors. The menu works, the sitemap validates, the pages load in under a second. And a large fraction of the addresses on the site have never been fetched by any crawler, so no amount of rewriting the copy on them will change a number.
Four gates between publishing and traffic
Publishing puts a file where a server can serve it. That is all it does. Google has to learn the address exists, decide the address is worth fetching, fetch it, evaluate what came back, and only then choose whether to hold it in the index. Each of those is a separate decision made for separate reasons, and a URL can sit for months at any one of them. It is because these are distinct failures that indexing gets a section of its own rather than being folded into the analytics screens, which report on pages that already cleared every gate.
- Existence. The file is reachable. Nothing about that is visible to anyone until something else points at it — a link, a sitemap entry, a submission.
- Discovery. The address is now known. Known is not scheduled: a crawler holds far more addresses than it intends to fetch this month.
- Crawling. Something fetched the page and received a response. A slow response, a redirect chain or a soft error all count as a fetch and all waste it.
- Indexing. The content was judged worth keeping. Thin, duplicated or templated pages are routinely fetched, evaluated and then dropped without a trace in your analytics.
What a crawler spends its visit on instead
Every site receives a rough allowance of crawler attention, set by how much the host can take and how much the content appears to justify. Small sites rarely exhaust it. Sites that generate addresses from a template exhaust it constantly, and generation is exactly what a corridor service business does.
The allowance goes on whatever is reachable, not on whatever matters. A crawler following links has no notion that one address is your flagship and another is a sorting parameter appended by a plugin nobody configured.
Addresses that should not exist
Parameters, session identifiers, print variants, calendars that generate a page for every future month.
- Filter combinations multiply fast
- Tag archives on a blog
- Paginated lists with no end
Fetches that return nothing useful
Redirect chains, slow responses, and pages that answer 200 while showing an apology.
- Old town URLs redirected twice
- Empty results pages
- Staging copies left reachable
Soft errors deserve their own sentence. A page that says "no technicians currently serving Evans" while returning a success code is, to a crawler, a valid page with almost no content. Generate ninety of those and you have taught something that your site produces empty pages at scale, which is a lesson that costs you on the pages that are not empty.
One page per town, and the list nobody audits
The Front Range invites list-building more than almost any other market. It is a line, not a ring, and the line is long: Fort Collins, Windsor, Loveland, Berthoud, Longmont, Frederick, Erie, Lafayette, Louisville, Broomfield, Westminster, Thornton, Northglenn, Arvada, Wheat Ridge, Golden, Lakewood, Denver, Aurora, Englewood, Littleton, Centennial, Highlands Ranch, Parker, Lone Tree, Castle Rock, Monument, Colorado Springs. Nobody had to invent those names. They are simply what the corridor contains.
So the arithmetic happens on its own. Somebody lists the towns, somebody lists the services, and a content plan appears with no author: one page for each pair. Six services against thirty-four towns is 204 addresses, and a plugin will produce them in an afternoon, each with the town name substituted into four places in an otherwise identical paragraph.
What comes out reads, from outside, exactly like the spam it is often mistaken for. Not because the intent is dishonest — the business really would like work in Erie — but because the pages are indistinguishable from each other except for one proper noun, and there is no way for anything reading them to tell the difference between a town you cover daily and a town you added because it fell between two towns you already had.
That is where the crawl budget goes, and it is also where the pattern gets learned. A crawler that fetches twenty near-identical town pages slows down on the twenty-first, and the twenty-first may be Longmont, where you actually have three crews.
Which of these towns can you actually staff?
The useful question is not an SEO question at all, which is why it is rarely asked in an SEO meeting. Take the town list to whoever builds the schedule and ask, town by town, whether a job there next Tuesday gets accepted, quoted high to discourage it, or declined. That conversation produces a shorter list than the website has, every single time.
| Tier | Test it has to pass | Typical share of the list | What the page should be |
|---|---|---|---|
| Core | Crews based there or work there weekly | 3 to 8 towns | A real page, written separately, with local proof |
| Served | Jobs accepted without hesitation | 6 to 12 towns | A real page, thinner, honest about travel |
| Reachable | Accepted if the job is large enough | 8 to 15 towns | A line on a regional page, not a page of its own |
| Aspirational | Nobody has ever worked there | Whatever remains | Nothing. It should not be an address. |
The tiers matter because they map onto crawl economics directly. Core and served pages deserve links, submission and attention. Reachable towns belong in a paragraph on one page covering a stretch of the corridor, where the honest sentence about travel time does more for a reader than a dedicated page ever did. Aspirational towns produce fetches that teach a crawler your site generates emptiness.
There is a second test worth running beside the scheduling one. Open the term list for an existing town page and look for the town's name in it. If a page has been live for six months and has never once surfaced for a query containing that name, the page is not competing — it is consuming.
What to do with everything that fails the test
Deleting pages feels like retreat, so it usually does not happen and the list keeps growing. But the alternatives are not delete-or-keep; there are four dispositions, and choosing among them is the whole exercise.
Into a corridor page
One page covering a stretch — the northern towns, the south metro — with a paragraph each and honest travel notes.
- Old URLs redirect once, not twice
- Ranks better than either page did
Gone, and returning 410
For pages that never earned an impression and cover work you would decline. A definite gone beats a soft one.
- Drops out of the crawl queue faster
- Nothing left to redirect wrongly
Reachable but not offered
Pages a visitor may legitimately land on that you do not want competing — old campaign copies, near-duplicates.
- Excluded from every sitemap
- Never submitted for indexing
The short list that survives
Core and served towns get separate writing: crew names, job photographs, travel times, the neighborhoods people actually name.
- Linked from the main navigation
- First into every submission batch
The redirect discipline matters more than it sounds. Merging forty town pages into five regional ones creates forty redirects, and if the targets themselves were moved a year earlier, every one of those is now a chain. Chains are fetched, followed and charged to your allowance, and a chain three hops long is three fetches to deliver one page.
The sitemap is where you say what counts
A sitemap is often treated as an export — everything the CMS knows about, dumped to XML on a schedule. Read as a communication instead, it becomes useful: this is the set of addresses I am asserting are worth your time. An export asserts nothing, because it contains the aspirational towns too.
Structure is the second half of it. A flat file listing 900 addresses tells you nothing when 300 of them stop being fetched. Split by purpose — services, core towns, regional pages, editorial — and the coverage numbers become diagnostic, because a gap now has a name attached to it. Splitting also survives growth: adding a corridor segment means adding one child file, not regenerating a monolith and hoping the diff was clean.
Submitting a tree rather than a file
For sites whose address list is generated and therefore larger than anybody remembers.
- Nested files are followed automatically. An index pointing at other indexes is parsed recursively to three levels, so a split structure does not have to be flattened before submission.
- The ceiling is high enough to stop being a constraint. A single job accepts up to 1,000 sitemaps, which is past the point where any local service site could reach it.
- Work is queued, not raced. Two jobs run at once and up to 20 wait behind them, so an agency can start a portfolio in the morning and read results in the afternoon.
Three levels is more room than most sites use and exactly enough for a corridor structure: an index at the root, one child per category, and the address lists beneath them. Build it that way and the question "are the northern town pages being crawled?" has an answer you can read off a screen instead of inferring. Files can be handed over as an upload or as a URL, so a tree the CMS already publishes needs no export step — point the job at the existing index and let the parser walk it.
Direct submission, and what a batch tells you
Submission is the second instrument, and it works on addresses rather than files. Up to 10,000 URLs go in as a single batch, while the account draws down a daily allowance of 1,000 URLs. Those two numbers describe different things and are frequently confused: the batch is what you hand over, the daily figure is the rate at which it is worked through.
Transmission goes through the IndexNow API, which notifies GoogleBot and BingBot that an address has appeared or changed. What comes back is a log kept per URL rather than a single status for the whole job: which bot arrived, when it arrived, what status it received, and the detail of any error. Alongside it run live counters for submitted, found and failed, so a batch can be watched while it drains rather than reviewed once it is over.
The log is more useful read as a pattern than as rows. Three shapes come up constantly, and each points somewhere different.
| What the batch shows | What it means | Where to go next |
|---|---|---|
| Submitted, never visited | The address is queued and unattractive | Internal links and sitemap placement |
| Visited, then nothing follows | Fetched and judged not worth keeping | The page itself: thin or duplicated |
| Errors clustered in one branch | A structural fault, not page-level | Redirects, server responses, templates |
| Visited and appearing in reports | The gate is passed | Nothing here; go and read the queries |
AutoSEO — discovery as a standing job
For a site that keeps generating addresses and has nobody watching whether they land.
- Submission stops being a task somebody forgets. Indexing runs beside the campaign rather than as a manual chore after each publication.
- Authority arrives from outside as well. Links are placed across a partner network of more than 230,000 websites, and an external link is a discovery route in its own right.
- Expect weeks, not days. First measurable movement typically lands 4 to 8 weeks out, which is the honest planning horizon for any of this.
Address lists can also be handed to Stream in bulk, which suits a corridor site producing one list per segment of the line, and the same feed carries the resulting reports and open tasks in one chronological place. Further walkthroughs sit on our blog, and the campaign side is described under the services we run.
Questions that come up while a batch runs
We submitted 600 town pages and traffic did not move. Why?
Because submission addresses discovery and the problem was almost certainly elsewhere. If the pages differ only by a proper noun, they will be fetched and declined, and repeating the submission changes the speed of the decline rather than its direction. Cut the list to the towns you serve and rewrite those.
Is it worth submitting the same URLs again after an edit?
Yes, for genuine changes — a rewritten page, a merged page, a new address. No, on a schedule for pages nothing has changed on. The daily allowance is finite, and spending it on unchanged pages is spending it instead of the ones you just published.
Should removed town pages be redirected or deleted outright?
Redirect where a genuinely equivalent page exists, which for a merged town usually means the regional page that now covers it. Where nothing equivalent exists, return a definite gone rather than sending everything to the homepage. A pile of unrelated redirects into one address is read as exactly what it is.
How long before a submitted address shows up anywhere?
The bot visit and its status appear in the per-URL log quickly. Anything about performance is slower: analytics data lags roughly two days, and appearing at all depends on the page passing an evaluation nobody publishes the criteria for. Judge submission by the log, and judge the page by the query reports weeks later.
Do we need every page in the sitemap?
No, and including every page is the common mistake. A sitemap is an assertion about which addresses deserve attention. Anything you would not defend in a meeting — filtered variants, aspirational towns, archives — belongs outside it, and leaving it out is a decision rather than an oversight.
The calculation to run before building anything
Take the equipment supplier with a catalog. Five service categories across thirty-eight corridor towns is 190 pages. Add four filter combinations reachable by link on each and the number is 760. Add three hundred product and editorial addresses and the site is asking for roughly 1,060 fetches to be seen once.
Against a daily allowance of 1,000 URLs, that whole site clears in two days, and a single batch holds ten times it. The ceiling was never the constraint. The constraint is that when the schedule test is run honestly, fourteen towns survive, which turns 190 pages into 70 and the filter variants into zero, because none of them should have been linked in the first place.
Seventy pages worth defending, submitted and crawled properly, will beat 760 that dilute each other. That is not a moral position about content quality; it is arithmetic about a finite allowance and a crawler that learns what your site tends to contain.
To see which gate your own addresses are stuck at, connect the domain, submit the tree as it stands, and read the per-URL log before changing a single word of copy. Open the Indexing Hub and start with a sitemap job. The first thing most owners find is not a ranking problem at all: it is a third of the town list that nothing has ever fetched, sitting there since the plugin generated it, quietly consuming the attention meant for Longmont.
