Skip to content

Why we crawl a site before we look at a single keyword.

SEO 5 min read Updated

Keyword research tells you what people want. A crawl tells you whether your site can be found for any of it. We always do the crawl first, and here is what it usually turns up.

A field of small points, one per page of a site. A light walks the links from the home page outward; pages it cannot reach stay dark, broken ones flash red and turn blue as they are fixed.

Most SEO proposals open with a keyword spreadsheet. It looks like progress: hundreds of rows, search volumes, a difficulty score in a coloured column. We understand the appeal. But a keyword list answers one question, which is what people are searching for, and skips an earlier one: can search engines actually reach, read and index the pages you want to rank?

If the answer is no, the spreadsheet is a wish list. So the first thing we do on any SEO engagement is crawl the site.

What a crawl actually is

A crawler, such as Screaming Frog or Sitebulb, starts at your home page and follows every link it can find, the way a search engine bot would. For each URL it records the status code, the title, the canonical tag, the indexing directives, the internal links pointing in and out, the response time and a great deal more.

The point is not the tool. The point is seeing your site the way Google sees it, rather than the way your menu presents it. Those two views are often very different.

We then set the crawl beside two other sources: the Page indexing report in Google Search Console, which shows which URLs Google has chosen not to index and why, and your XML sitemap, which shows what you have told Google matters. Where the three disagree is where the work is.

The five things it most often finds

These come up on site after site, whatever the platform.

Pages that exist but cannot be found. A service page linked only from a dropdown that renders in JavaScript, or a location page nobody linked to after launch. If no crawlable link points at a page, it relies on the sitemap alone, and pages in that position tend to be crawled rarely and ranked poorly.

Several URLs for one page. The same product reachable with and without a trailing slash, with tracking parameters, through two category paths. Each variant splits the signals that should be concentrating on one address. Canonical tags help, but only when they are consistent, and a crawl is how you find out whether they are.

Indexing instructions nobody meant to leave. A noindex tag copied over from the staging site. A robots.txt rule written for a folder that has since been reused. These are one-line problems with site-wide consequences, and they are invisible unless you look.

Redirect chains. Page A redirects to B, which redirects to C. Each hop costs crawl time and some signal, and chains accumulate quietly every time a site is restructured without the old redirects being tidied up.

Thin and duplicate templates. Tag archives, empty category pages and near-identical location pages that give a search engine many weak pages to choose between instead of a few strong ones.

None of this is glamorous, and none of it shows up in a keyword tool. All of it limits what any amount of content can achieve.

Why the order matters

Imagine you commission forty new articles targeting carefully researched terms. They go into a blog whose category pages carry a stray noindex, whose internal linking buries anything older than a month, and whose article template loads slowly on phones. The articles are good. They underperform anyway, and the natural conclusion is that the keywords were wrong or the writing was weak. You spend the next quarter fixing the wrong thing.

Crawl first and that argument never happens. You know the foundations can carry weight before you put anything on them.

There is a commercial reason too. Technical fixes are usually the cheapest work in an SEO engagement relative to their effect, because each one applies across every page that shares the template. One corrected canonical rule can clean up thousands of URLs. That is why, in our process, the first month is spent on the audit and on making the numbers trustworthy, and new content comes after.

What you get from the crawl

We do not hand over the raw crawl export. It is tens of thousands of rows and it hides the three things that matter among the three hundred that do not. Instead you receive:

  • A written list of problems, each with what it affects, how many URLs are involved, and why it matters in plain English.
  • A priority order based on effect and effort, so a developer can start at the top and stop anywhere without having wasted time.
  • The exact fix for each, written so whoever maintains the site can implement it, whether that is us or your own team.
  • A baseline from Search Console, so that when the fixes go live, everyone can see what changed.

Then the keywords

Once the site can be crawled and indexed properly, keyword research becomes useful, and it is better research, because it starts from what the site can already rank for rather than from a blank page. You can see which pages are sitting on page two, which queries they are nearly winning, and where a missing page would fill a gap.

That is when the spreadsheet earns its place. Just not first.

If you would like us to run a first-pass crawl on your site before we speak, send the domain. The conversation will start with findings rather than a pitch.

Filed under SEO. Related service: SEO and search strategy.

Talk to the people who wrote this.

If this sounds like your account or your site, tell us. A person replies within one working day.