How to use Screaming Frog to find all of your old guest blog posts

  • 1
  • January 22, 2014
Jack Cornwall

Jack Cornwall

Search Strategist

On Monday, the Twitter feeds of the SEO world were full of talk about Matt Cutts’ latest rant. His article entitled ‘The decay and fall of guest blogging for SEO’ announces outright that yet another harmless internet phenomena has been abused by SEO’s to the point where none of it can be trusted by Google at all.

He states:

“If you’re using guest blogging as a way to gain links in 2014, you should probably stop.”

He then goes on to give an example of a generic email which, if you have ever owned a blog, you’ll have seen far too often.

Branded3 has used guest posting before for clients and for ourselves and absolutely will continue to do so because for us, it’s not a question of producing thousands of low quality posts that can be posted on every site out there, but rather, it’s a great opportunity for us to promote the authority of our clients’ expertise, and our own.

Our clients know their industries better than anyone so if we can help connect one blogger, journalist or webmaster with them, then our client can help educate, inform or entertain the readers of that website. This is the best way to use a guest post, don’t look at it as a link, but an opportunity to connect with a new audience.

Clearly, Google has been trying to monitor and police the guest posting and authorship sphere with its attempts to encourage Google+ authorship, and there are indeed some great authorities out there with fantastic Google+ profiles, but it looks like it may be some time before Google can fully trust an author through this method.

The latest advice from Matt Cutts resembles what Google has been saying about followed links in advertorial content. These links should not pass Page Rank if they are sponsored, even if they are disclosed as such, and should therefore be no-followed. There were quite a few websites affected as a result of advertorials and we imagine that any site using low quality guest posting as an SEO technique, need to re-think their strategy on the whole.

In the below video released at the end of May 2013 on the Google Webmaster Help YouTube channel, Matt Cutts explains what he considers advertorial and editorial content and how they differ.

Just before that, he tried to explain to Webmasters on the Webmaster Central Blog that links that have to be paid for should be no-followed, a warning that was not heeded by some. Now, we are getting a similar warning about the use of guest blogging.

Should we panic? Well, if you have been approaching bloggers in a similar way to the examples Matt Cutts has given, you probably should. But this isn’t new news.

If you do want to follow Google’s guidelines to the letter (and you should) without managing to annoy bloggers or webmasters you may want to work with again in the future; don’t just start removing links on mass or cramming your disavow files.

At this stage, getting the links no-followed is probably the best course of action. If the link shouldn’t be there, it shouldn’t be there regardless of whether it is labelled as a guest post. If the link is natural and driving traffic but labelled as a guest post, there are a few easy ways to find it in your client’s link profile with relatively little effort before deciding whether or not you need to get in touch with the website to get it no-followed.

Determining the extent of the problem

Firstly, get your link profile together. Use the standard tools such as Open Site Explorer, Ahrefs and Webmaster Tools. Create a .txt file with a list of all of the linking page URLs, of course you can filter with a spreadsheet so that you are only taking the followed links into account.

Next, we need Screaming Frog SEO Spider – if you don’t know what this is, it’s a web crawler, usually used to crawl websites in order to find issues which need fixing such as broken links or duplicated meta data, however it has some configurations and settings that make it perfect for this job.

Which links might be guest posts?

Set the tool to List Mode; rather than crawl all of the links from a starting URL, this will allow you to upload a specific list of URLs to crawl.

You can then simply upload the list of URLs you want to crawl from the .txt file we created. Next, you’ll need to change the filters in order to find what you’re looking for.

Screaming Frog

There are 10 filters in Screaming Frog so you can simply filter for anything you want to look for in the source code. This will find words in the source code which exactly match the terms we search for. It should be noted at this point that Screaming Frog only crawls the source code and not the URLs, these should still be manually filtered in Excel to find URLs containing suitable terms.

Your custom filters should look something like this:

Custom Filter Configuration

It should also be noted that the filters are case sensitive, so if you can think of more than 10 terms that you want to search for, such as variations of the word ‘article’ (sponsored article, featured article etc.) you may have to run the crawl more than once.

This will bring back some irrelevant stuff, for example, pages may have site-wide footer links to a guest posting sections without actually marking the section as a guest post, but this narrows down all of the sites which may be guest posting on mass.

Now you will have a list of all of your links that could potentially be guest posts and you can then manually determine which to keep or which you may wish to no-follow or disavow.

The future of guest posting

When it comes to acquiring links, should the whole practice of guest posting be pushed to one side? In my opinion, absolutely not. You still have to look at each blog you want to work with, assess whether your client has something real and valuable to offer it and approach the site with the intention to give them a story.

If you get a link out of the whole thing, then great, otherwise, you’ve still secured some coverage for your client in a publication that you thought was worth contacting in the first place. You’ve also been able to get the client’s message out there to help build them as an authority on that subject, which is exactly what guest posting should be about.

If you do work in Outreach and bulk guest posting, you don’t want to contact websites with URLs such as:

Chances are, this person is just trying to match as many “inurl:” searches as possible.

Another indicator to watch out for is a contact page which is either just a form, an email address or worse, a WordPress default ‘hello world’ style post. How much can the owner of that blog actually care about their website?

There is some useful guest posting advice here. New York Times bestselling author, Ramit Sethi, challenges you to write on his blog. I doubt he replies to the generic email that Matt Cutts showed us an example of.

He goes into detail about what he is looking for; giving examples of guest posts he has written and tells you how to go about contacting him and what sort of collaboration you can expect from him.

Quite frankly, he cares about his blog and I don’t think whether or not he labels it as a ‘guest post’ is going to make much difference.

In December 2013, Matt Cutts looked into the ever-increasing scepticism on guest blogging, so this has been on the cards for some time. Although he didn’t condemn it then; he did say that action will be taken on spammy behaviour, and this is really the key to all of this.

Do guest posting because it’s great for your site, your brand and your audience; not because you want to get more links to your site.