Internal Linking Site Audit: Ahrefs vs. Screaming Frog

Having decided to audit one of my sites for internal linking patterns, I decided to give two different tools a test –

  1. Screaming Frog‘s free software (downloadable and free)
  2. The Site Audit tool by Ahrefs (online and paid).

What I needed was a clear outline of all of the pages the site had, with a clear understanding of which pages are getting enough link juice from related pages – and which are not.

The site in question doesn’t have a ton of pages. This is a travel blog that is just over two years old and currently has 106 published posts. Obviously, posts aren’t the only pages in a blog. It also has –

  • 3 published WordPress pages
  • 7 categories
  • 84 tags (only 55 of them used)
  • The main page

That should come to a total of 201 pages, right? And that’s assuming the spiders can somehow get to tag pages of unused tags.

The site in question actually used to have more posts or at least different ones. About half a year ago, I changed the structure of this blog, and pruned it a little, deleting over a dozen posts which were no good (and had no traffic).

I started this Internal Linking Project with two goals in mind –

  • Find and fix broken links to posts/categories that no longer exist.
  • Adjust linking patterns so that older strong posts send visitors to newer high-quality posts.

The second item is something I should be doing more of. When publishing a new post, I usually link back to relevant posts. That means the older a post is, the more incoming internal links it’s likely to have.

Which isn’t a terrible thing, it’s just that these older posts get a lot more Google traffic and I’d like to send some of that traffic to my newer posts on related topics.

Why these two tools?

Ahrefs seems to be the current industry standard. It’s not cheap though, so at this point in time, I don’t have an ongoing monthly subscription with them. I had it in the past for a while, and now it’s more on a “need to use” basis. Paying for Ahrefs for 1-2 months a year makes sense for my business model. More than that would just be a waste of money.

So, it made sense to renew my Ahrefs paid plan for the sake of this project. I wanted to give their new(ish) Site Audit tool a try and compare it to a free alternative.

(I’ll also take the chance to do more keyword research this month, since I already have the plan activated.)

As for Screaming Frog. I’ve read about it for a while now and wanted to give it a go. It’s an old-school kind of tool, at least in the sense that it’s an actual piece of software that you need to download and operate.

I actually like that. Plus, it’s free, so I figured, why not. Before I begin to describe what I did with each one – and what the results look like, here’s a quick comparative overview:

Ahrefs Site Audit Tool

  • Online tool
  • Not included with Ahrefs free account.
  • Paid version stats at $99 a month but gives you access to all of Ahrefs tools (plus, there’s the famous $7 for 7 days trial).
  • You need to verify the site

(And hey, I got to use the new columns block in Gutenberg. And it’s not very good… Can’t lose this additional block I’m typing in right now, no matter what.)

Screaming Frog

  • Downloadable
  • Free for up to 500 URL’s
  • Paid version (£149.00 Per Year) offers unlimited URL crawl and better configuration options.
  • You don’t need to verify the site (you can crawl a competitor’s site too)

So, time to dive into the actual crawls and their results.

Internal Linking Analysis with Screaming Frog

It’s worth mentioning that I am not an SEO expert and I’m most definitely not a Screaming Frog expert. The software looks extremely complicated and has TONS of features.

Fortunately, carrying out a spider crawl seemed straightforward enough. You just enter the URL and click “Start”. The site I’m auditing is relatively small, so Screaming Frog had the results in by the time I was back with my coffee.

It found a total of 270 URL’s.

Filtering the results comes next. So, I chose “HTML” only, as I didn’t want to follow up on images and javascripts at this point.

That brought the results down to… 70. Which is less than what I had anticipated and doesn’t even cover 106 posts. A quick scan revealed that SF only found all of the category pages, but only 2 tag pages and, what’s worse, only 60 posts (out of 106).

The free version of Screaming Frog does not allow you to control the settings, but you can still see them:

So, no limit on the depth of the crawl. The only limit is on the total number of URL’s crawled but the spider never hit that limit.

The blog doesn’t actually have 46 orphaned posts… So, I wasn’t very happy with the results. I checked the URL’s for depth of linking, from the main page, and indeed, the highest generated number was 2.

I think this means Screaming Frog didn’t scan anything beyond that. I’m not sure why, as the configuration suggests it should but I have no way to play around with that in the free version.

The report did provide me with two important elements (among many others) –

  • The number of incoming internal links
  • A list of those incoming links for each page that actually was scanned

Which was basically what I needed in order to figure out which pages were getting a lot of incoming internal links and which were not (had it covered all site pages).

Once I started digging in, I didn’t like what I see. The report didn’t really make sense.

For example, the homepage was showing up having fewer incoming internal links than many other pages. Which doesn’t make sense when the logo links back to the homepage from every single page on the site.

It wasn’t just the homepage. As soon as I sorted the table by the number of incoming links, I could easily see that some pages were showing more internal links compared to others – where I knew it should have been the other way around.

So, I wasn’t happy with the results. Mind you, this is the free version – but it’s still supposed to crawl deeper than it did. I don’t mind paying for useful tools but in this case, the free version just didn’t give any indication that paying for the full one would be worth it.

Moving over to Ahrefs.

Ahrefs is an online service, so there is no software to download and install. Also, I could set up the configuration for the crawl.

I chose not to include images, css and javascripts.

Even with that setting applied, Ahrefs fetched a whopping 361 URL’s! Which baffled me at first.

Looking into the data in more depth, it turned out to be jpeg files. I’m not sure why, but despite switching off image resources, quite a few jpeg files made it into the list.

Once I filtered out the image files and the 301 redirects, I had a final list of 183 URL’s. Which is in the range of what I had expected.

Ahrefs also reported zero 4xx links which basically means no broken internal links on the site – woohoo! There were however 125 links redirected to other pages.

Finally, I checked where my inbound links are actually heading. Naturally, the main page is at the top of the list, followed by the category pages.

Next, came the list of posts in the “latest post” widget which I have in the sidebar.

Which got me thinking.

On the one hand, this puts the freshest content in front of every site visitor – in every landing page. On the other hand, that isn’t always the best content that I have, especially if you take into account the topic.

Remember, this is a travel blog. The categories are basically destinations. When a visitor comes looking for information on destination A, it would make more sense to showcase the best content I have in that area. Not the last 10 posts on totally non-related topics.

So, my first take of the Ahref report is to look into creating different sidebars for different categories.

Also, as I had anticipated, newer content isn’t getting a lot of love from older pieces. Now I can use the data collected to start digging in and creating a better link structure – putting my best content forward (even when it’s not the oldest).

In Summary

While Ahrefs was the more expensive option, in this case, it delivered far better results. Screaming Frog was quick, relatively easy to use and generated a lot of data, however it failed to find all of the site’s pages.

I would say that if you can’t afford Ahrefs, running Screaming Frog will probably be better than nothing. It can help you get more information about your site which is always nice.

And again, remember that this was the free version of Screaming Frog. I can’t tell whether or not the paid version will be inherently better. I just expected the free version to do as it says and actually crawl deeper into the site. It didn’t.

It’s definitely thumbs up for Ahrefs. They crawled the entire website, as far as I can tell, in a very accurate way. Just what I was looking for.

And now, if you’ll excuse me, I have internal links to restructure!

2 Comments

    • I don’t audit internal links anymore. Not saying that it isn’t important, but just that I have more important things to do right now.

Leave a Reply

Your email address will not be published. Required fields are marked *