Author: Sam Partland

Stripping and Rebuilding a URL Structure for Migration Testing

Stripping and Rebuilding a URL Structure for Migration Testing

Some clients have recently required a migration mapped out for 10’s to 100’s of thousands of URLs.

To do this, we needed to capture all the different URLs, build out the migration rules, and then test the redirects.

Mapping out and building the rules is one thing, but then how do we bulk generate what the URLs should look post-migration so that we can actually test the redirects?

How can we test the implemented redirect rules, to ensure they match what we originally mapped out?

No one wants to manually create and map 1,000+ URLs to compare the redirect path too, and I personally prefer to not just test the redirects then give them ‘after’ the once over.

I’d like something to compare the implementation to.

Using Google sheets (or Excel) you can strip down the existing URL, extract the parts the redirection rules will use, and then rebuild the existing URL into their new formats.

This will allow you to do a bulk migration test against the implemented redirect rules, and confirm everything redirects where it should.

There are a couple of ways of achieving this, and I will just go through the different methods I use below.

A quick note though, this has nothing to do with building the actual redirection rules themselves. Whilst it might help you build better examples of what you need to redirect, this post is purely about building the expected URLs so you can confirm the redirects match.

 

Stripping a URL down

There are a few main types of URLs we can account for here.

The standard hierarchical folder-based structure, the flat URL structure with everything to load the page in the base URL, and the ugly query parameter based structure.

They can all be stripped down relatively easily, but firstly we can strip out the URL path from the domain to simplify issue patching later.

 

Removing the domain from a URL Path

A handful of different ways to strip a domain from a URL are available, but this formula is possibly the easiest;

=RIGHT(A2,LEN(A2)FIND(“domain.com”,A2)LEN(“domain.com”))

This will strip everything up to and including ‘domain.com’ allowing us to account for all https, http, www, and non-www variations of a URL leaving us with just the URL path.

Stripping the domain from a URL

It will also cull subdomains, so just keep that in mind. If you need subdomains left, you can still use the same formula, but each subdomain will need it’s own.

I’d recommend extracting the subdomain in a separate column, and then referencing the subdomain’s cell, rather than plain text domain. That should save you a bit of time!

 

Extracting query parameter values from a URL

Let’s go over this one first, as it is slightly simpler than breaking down a ‘pretty’ URL.

There’s just one formula you need, and that’s regex extraction.

=iferror(regexextract(B2, “[?&]parent=([^&]*)”),“”)

That will extract the value for the ‘parent’ query parameter, and leave the cell blank if the parameter doesn’t exist, or is blank.

Extracting query parameter value from a URL

You then just need to repeat that formula, for each query parameter value you need to extract.

 

Stripping a hierarchical ‘pretty’ URL structure

Stripping down a folder-based, or hierarchical is the next structure that could be at play here – and generally my favourite to use.

The absolute simplest way is to use the “split text to columns” method.

Splitting url path text to columns

Highlight your text, click ‘tools’, and then click on ‘Split text to columns’.

You’ll then get asked for a separator. In this case, the separator is a ‘/’, so you’ll need to click ‘custom’ and enter the ‘/’.

After which, you’ll immediately get the following;

Select the URL separator

Each folder split across the different cells, giving you all the different folder levels all accounted for.

Works perfectly! If you don’t mind splitting any new URLs like this going forward any time you add new ones.

I personally prefer to try and work with formulas where possible, so that we can keep expanding the URL list, along with tweaking it as we go if modifications are required.

Provided you did the URL path column above, the following formula is what you’ll need;

=iferror(LEFT(B6,FIND(“/”,B6)1),“”)

Parent folder extraction from URL

That will just strip off everything after the first slash, leaving you with the parent folder.

Next, you’ll do a similar thing to get the child folder, but first, you’ll just subtract the parent folder from the URL path.

=iferror(LEFT(substitute(B6,C6&“/”,“”),FIND(“/”,substitute(B6,C6&“/”,“”))1),“”)

This will just remove the parent folder from the URL path, and leave you with the child folder, which will be the second folder in the path.

Extracting a child folder from a URL

And if you’re playing with 3 or more levels, here’s the formula you can modify;

=iferror(LEFT(substitute(B6,C6&“/”&D6&“/”,“”),FIND(“/”,substitute(B6,C6&“/”&D6&“/”,“”))1),“”)

You’ll just need to keep adding the new cells in there each time you need a new level

 

Breaking down a flat URL structure

These are extremely difficult to break down.

Unless there is consistency in how they’re formatted, this could be a very manual process.

However, I am just going to assume you’ve gotten lucky with something that resembles a consistent format for you to pull apart.

=LEFT(B15,FIND(“-for”,B15)1)

Stripping a flat URL structure

For the child folder, there are two formulas I have that basically just do the same thing.

They both look at a break point and just trim everything including and after it. For my example, my breakpoint is “-in”.

=substitute(REGEXREPLACE(B15,“(.*)-in.*”,“$1”),C15&“-“,“”)

Extract a part of the URL

=substitute(LEFT(B16,FIND(“-in”,B16)1),C16&“-“,“”)

Google sheets URL modification formula

Same same but different.

I just don’t remember how they’re different lol.

The last part here is to extract the location.

If your location is like mine, you don’t need anything fancy.

You just need to substitute all the previous folders from the structure, then remove the slash, and you’ve got your location.

=substitute(SUBSTITUTE(B15,C15&“-“&D15&“-in-“,“”),“/”,“”)

Extraction a location from a URL for SEO

You essentially rebuild the URL that came before it, based on what you had stripped out, and then you’re left with the ‘location’ that is on the end.

If your location isn’t on the end or doesn’t have something that can be clearly substituted out after it, you’ll need to modify the formula used for the child folder.

 

Rebuilding the New Post-Migration URL Structures

Once you’ve stripped down the old URL to its main elements, you can start building the new one!

If you’re merely converting query parameters to a clean structure, with no folder changes, you’re super lucky.

Many of us though, are not that lucky.

 

Converting query parameters to pretty SEO URLs in Google Sheets

Okay you lucky people, this is super easy.

=”https://www.domain.com/”&C2&”/”&D2&”/”

Converting query parameter to pretty SEO URL in Google Sheets

You just piece the URL together using the ampersand, ‘&’.

Include your domain with a slash, then the parent folder and a slash after it, and then the child folder and another slash.

Some people like to use concatenate, but you complicate things. Complications through concatenations.

Annnnnd you’re done! You’ve now turned your query parameters into a pretty URL structure, in bulk, so that you can test these out.

It’s rarely that easy though.

 

Rebuilding a URL structure where folders change names or get merged

Whilst it’s easier when things don’t change, it really isn’t that hard when they do now that you’ve stripped the URL down.

The first step is to paste all your parent, child, and any other folders into a separate column, and then remove all their duplicates.

Removing duplicates from a list of URL paths

This will leave you with a unique set of folder values.

You then just need to add their new value next to them.

If the value doesn’t change, just paste the same value in.

If the folders are getting merged, just paste the folder you’re merging into here.

If it is a completely new value, type out the replacement.

Old to new URL folder mappings

The mapping of the values might take time, as some websites could have hundreds and hundreds.

The benefit of doing it this way is that you aren’t dealing with all the possible variations & combinations.

Separating the child folders from the parent folders lets us independently map them to their new values.

Now we need to bring these new values into the mix, with a good old vlookup (if you don’t know how to do it, check out my SEO Excel formulas post).

=VLOOKUP(C2,M:N,2,0)

Vlookup the old to new parent folders

Repeat the process for the child folders, and you’ll get all the shiny new folder values.

You’ll just need a slight tweak to the child formula, due to the possibility of there being no child.

=iferror(VLOOKUP(D2,P:Q,2,0),“”)

If no child exists, the cell will remain blank.

Vlookup the new child URL folders

You should now have completely mapped the old to new folder values, so we can move on to constructing the URL.

Now you just piece it all together!

=“https://www.domain.com/”&E2&“/”&IF(len(F2)>1,F2&“/”,“”)

Rebuilding a new SEO URL for migration testing

This will add one folder to the other, and rebuild your new URL structure for you!

If there is no child available, it will ensure there is no extra slash added.

If you prefer the old flat structure, tweak the formula to be;

=“https://www.domain.com/”&E2&IF(len(F2)>1,“-“&F2&“/”,“/”)

Which will ensure there is no extra dash added, unless the child exists.

And if you need a stop word, or two, in the middle to clean her up a bit, just do the following;

=“https://www.domain.com/”&”theres-a-“&E2&“-before-the-“&F2&“/”

You’ll have to account for your own variations of the template, and whether a child may or may not exist.

 

Rebuilding a Structure that Previously Used ID’s

What if you’re just stuck with a heap of parameter IDs, and no ‘pretty’ URL to start with?

The benefit to the above vlookup style URL rebuilding is that you can extract the query parameter values, and then just map the old values to their new pretty folder name.

A quick rebuild, and you’ve gone from the old ugly URLs, to what could be thousands of crawlable new versions of the URL!

Start by getting the path out of the URL, and then extract all the parameter values, like I showed you earlier.

Converting query parameters to a pretty URL

Add your new parameter values to the parent/child vlookup lists;

Query parameters to clean folders

Then just vlookup the parameter value, to the new parent & new child values, and build your new URL!

New pretty URLs for query parameter ugly URLs

Everything was covered earlier, so just grab all the formulas from there.

That covers it though!

You’ve gone from the ugly parameter version to a shiny new & testable pretty URL.

 

Bulk testing your new URL structure

You can now run the old URLs through screaming frog (or your preferred bulk testing tool) and copy out the final URL in the redirect path.

Match it up to your expected ‘new URL’ and confirm whether it is the same or whether they’re different.

If they’re different, is it the redirect rule that doesn’t match? Is there an error in the expected URL? Are the redirects just not implemented?

This is guaranteed to save you time bulk testing your migration.

 

Testing a URL migration on staging before going live

In an ideal world, for a migration this size, you should be able to test out your redirection rules on a staging environment.

Swap out the website’s domain in the formula, for the URL of the staging environment, and do everything else the same.

You’ll get thousands of staging URLs ready to load screaming frog up with!

 

Download the sample

You can download all my sample data from here if you want to jump right in.

I’m always looking at new ways to simplify bulk analysis, so if this worked for you, or you do something completely different, let me know!

I’d love to hear about it all ūüôā

 

Also, just a quick shout out to the guys at ExtendOffice.com though!

Plenty of awesome formulas, with this one definitely being one for the bookmarks: https://www.extendoffice.com/documents/excel/1783-excel-remove-text-before-character.html

Excel Formulas Every SEO Needs to Know

Excel Formulas Every SEO Needs to Know

As an SEO, you really can’t escape Excel. Almost every month a new SEO tool pops up, that can do so much, but will never seem to replace those little things you can do in Excel.

You find yourself using a tool for 75% of the job, but will then always jump back into Excel to finish it off.

Well, that’s me anyway. If it’s not you, then you probably just don’t know many of the things that Excel can do!

I run through VLOOKUPs, IF statements, LEN, TRIM, PROPER / UPPER / LOWER, COUNTIFs & SUMs and show you some basic usage examples. I’ve also added a few extras to the bottom that aren’t in the video!

Basic Excel Formulas for SEOs

Whether you’re starting out, or an experienced SEO, these are the basic building blocks to so many advanced Excel uses for SEO. You need to know these.

VLOOKUP – Find data for a value, within a range

The simplest summary of what a VLOOKUP will do is to match two sets of data where you have something to match them with.

So is long as one column in each data set has some sort of unique identifier, then you can VLOOKUP between the two datasets and bring the data together.

If you don’t have a unique identifier, will you have one if you match two columns¬†together?

You might have a date that isn’t unique, but if you add it to another column (like colour) you will get a unique value that you can use to match the data sets.

=VLOOKUP(*CELL of what you want to search for*,*RANGE of where you want to look for it*,*COLUMN NUMBER the data is in you want to extract*,*TRUE (1) or FALSE (0) but 99% will be FALSE*)

LEN РCount character length of a cell

Nice and simple formula. The LEN formula is great for counting the lengths of page titles and meta descriptions!

Plenty of other uses, but that’s what the majority of people will use it for.

=LEN(*CELL to count*)

TRIM – Remove spaces before & after a cell

Sometimes when trying to match data up, the cell might visually look the same as another cell, but you can’t seem to match it up!

The amount of times I have run into this is crazy, and the majority of times it happens there is a pesky space before or after the text.

The TRIM formula will clean these up for you and make sure there isn’t something hidden there wasting your time!

=TRIM(*CELL to trim*)

SUM –¬†Add all numbers in a range together

Just adds numbers. It’s that simple. Just select the range you’d like to add together and away it goes.

=SUM(*RANGE to add together*)

COUNT –¬†Display a count of cells with numbers in them

Out of your selected range, how many of the cells have a number in them?

=COUNT(*RANGE to count for numbers*)

COUNTA –¬†Display a count of cells¬†with numbers and letters in them (so not empty)

Out of your selected range, how many of the cells have a number of letters in them? This is what you need to use if you are trying to count cells with characters rather than numbers.

=COUNTA(*RANGE to count for numbers*)

COUNTIF –¬†Count all the cells when a specific criterion is met

Only count a cell that matches your criteria. This is great for being able to select cell as your criteria.

So if you have a column of ‘Colour’ and a cell that has the text of ‘Blue’, you could select that ‘Blue’ cell and then the range that has all the colours, and the formula will count how many times ‘Blue’ appears in the range.

=COUNTIF(*RANGE to count for numbers*,*CRITERIA for when to count*)

PROPER –¬†Capitalise the first letter of every word in a cell

Great for cleaning up a list of titles, or any words, where you’d like to capitalise the first letter of every word.

=PROPER(*CELL to capitalise first letters*)

UPPER –¬†Capitalise every letter of every word in a cell

This will format it AND MAKE IT LOOK LIKE YOU’RE SHOUTING. Just use the upper formula when you need to make sure everyone knows you’re SERIOUS.

=UPPER(*CELL to capitalise everything*)

LOWER РMake every letter of every word in a cell lowercase

The lower formula will make everything lower case. Great to run through your keywords with to ensure there isn’t any weird capitalisation going on.

=LOWER(*CELL to lowercase*)

LEFT & RIGHT – Extract x characters from the left or right of a cell

This is a simple way of extracting a certain amount of characters from the left or the right of some text.

Great if you want to cull the end of a title, remove a domain, or trim some trailing slashes from a URL.

=LEFT(*cell reference for text to look in*, *number of characters*)

=RIGHT(*cell reference for text to look in*, *number of characters*)

However, if you are looking trim everything before or after a certain character, instead of a number of characters, check below for a more advanced formula!

IF –¬†Only do something when a certain condition is met

The IF formula lets you create rules for your cells. Maybe you need to filter out titles that are too long.

You could do an IF formula in a new column that says =IF(length>70,”LONG”,”OKAY”). This will give you back a cell that says LONG if the title is too long, or OKAY if everything is okay.

Probably not the best use case, but gives you the idea!

=IF(*The condition to run*,*What to do if it’s true,*What to do if it’s false*)

Does Cell Contains Text – Run condition if the cell contains specific text

A quick formula to use if you want to do something off the back of a cell containing a specific piece of text.

This formula will return TRUE or FALSE, which is what I normally use with this one, but you can expand it do use IF. So IF bla = TRUE, do this.

Great way to quickly segregate data by whether something is in a URL, or keyword.

=ISNUMBER(SEARCH(“text to look for”,*Cell to look in*))

& – Yeap, the ampersand bad boy is rather useful

If you’re trying to merge stuff together in Excel, you need the ampersand. This thing will let you attach anything, to anything else.

It will let you run multiple formulas, then just stick them all together to form what is essentially a sentence.

=”Stick this “&”,with this “&”, and merge all “&2+2&” sentences together”

Which will output:

Stick this, with this, and merge all 4 elements together

SUBSTITUTE – Swap out any text, with any other text

This formula is invaluable if you are doing any sort of text editing or manipulation.

You select the cell you’d like to manipulate, enter the text that currently exists, then enter the text you wish to change it to.

=SUBSTITUTE(*text to look in*,”old text here”,”new text here”)

Trim URL to Domain / Subdomain

This is something that you will find incredibly useful if you’ve never done it before. Some extra tools do it, but you can achieve almost perfect trimming with just a formula – SOOOO much easier!

Quick and Dirty Trim Domain Prefixes Method

If you are just looking to trim the HTTP part, here is a hacky formula that will just substitute out the HTTP part. It’ll leave subdomains / domains, and all their folders intact however.

=SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(substitute(SUBSTITUTE(A2 ,”https://www.”,””),”https://”,””),”http://www.”,””),”http://”,””),”www.”,””)

Trim URL to Subdomain

Spend a bit more time playing with a formula, and you can get something that will do a much better job though!

This formula will trim any URL back to its subdomain. It will remove all prefixes, or any suffixes (aka folders) from a URL so that you are just left with the subdomain & domain combo.

=SUBSTITUTE(REPLACE(REPLACE(A2, 1, IFERROR(FIND(“//”, A2)+1, 0), TEXT(,))&”/”, FIND(“/”, REPLACE(A2, 1, IFERROR(FIND(“//”, A2)+1, 0), TEXT(,))&”/”), LEN(A2), TEXT(,)), “www.”, TEXT(,))

And another formula is;

=REGEXREPLACE(REGEXREPLACE(A2,”(http(s)?://)?(www\.)?”,””),”/.*”,””)

With A2 being the cell reference to where you have a URL to trim.

Trim URL to Root Domain

If you are wanting to get rid of the subdomain too, so you are just left with the root domain, you can use this formula. It will remove folders, subdomains, prefixes, and anything else so that you are just left with the root domain stripped out.

=SUBSTITUTE(TRIM(RIGHT(SUBSTITUTE(REPLACE(REPLACE(A1, 1, IFERROR(FIND(“//”, A1)+1, 0), TEXT(,))&”/”, FIND(“/”, REPLACE(A1, 1, IFERROR(FIND(“//”, A1)+1, 0), TEXT(,))&”/”), LEN(A1), TEXT(,)), CHAR(46), REPT(CHAR(32), LEN(A1))), LEN(A1)*2)), CHAR(32), CHAR(46))

Stripping Text Before/After Characters

This is something I have only picked up in the last couple of years, and these formulas allow you to strip out the text before or after a character or a set of characters.

I will leave this up to your imagination with how they work, but a great usage is stripped title tags of anything after a | (pipe).

Yes, you could find/replace this stuff out, or substitute it, but maybe there are 50 different items to swap out. That will take ages!

If they all have a pipe right before them, you can strip them all out with one formula.

Found these formulas here, thanks guys!

Strip Text Before a Character

This formula will remove everything before the pipe. Just edit the pipe out with any character, word, phrase, anything, and it will remove all the text before it.

=RIGHT(A1,LEN(A1)-FIND(“|”,A1))

Strip Text After a Character

Again, just replace the pipe with anything you want to have this formula strip all the text that is after it.

=LEFT(A1,FIND(“|”,A1)-1)

Finding Your Own Excel Formulas

The biggest thing to remember with Excel formulas is that chances are someone else has tried what you’re doing before.

Just break down what you’re doing, and Google it piece by piece. You might end up putting something together yourself, through what you find from others.

Building a Keyword Research Dashboard in Excel

Building a Keyword Research Dashboard in Excel

An extremely large table of all their keywords, isn’t going to give the insight some clients would want. Some would like more a bit more insight into the data they’ve paid thousands for.

Building a keyword research dashboard in Excel is a great way to allow a client to interact with their keyword data.

In this short video, I will show you how you can take a categorised keyword list and turn it into a client-friendly dashboard!

The video is a couple years old now, and I may eventually update it, however, for the most part, it still holds true today.

Now, this won’t be great for some clients. You could end up giving them way too much information, and have them asking way too many questions.

However, I personally don’t mind that. I love when clients have an interest in SEO as it gets them more involved in the project, and more excited when the results kick in.

How to build a keyword research dashboard in Excel

  1. Categorise up your keyword research data
  2. Create pivot tables for each category level
  3. Attach slicers to the categories so you can filter the data
  4. Design the tables & slicers to match your client’s branding

And there you have it, you will have an interactive Excel keyword research dashboard to send to your clients.

Creating an online keyword research dashboard

You can save this Excel dashboard to Onedrive and then share with your clients via a link instead of sending a file.

For dashboards under a certain size this means they will be able to view the dashboard online instead of having to download the file. You can then simply update the keyword research file, and they will be able to see the update when they refresh their browser.

This becomes a massive help when trying to send updated data to a client. No more v53 tacked onto the end of a file because you’ve updated it 53 times!

Download the Excel Keyword Research File

If you watched the video and can’t be bothered doing the work yourself, you can just have my file!

Or if you’re doing keyword research for car hire you can also just steal my keywords – chances of that have to be slim.

Download the Excel Keyword Research Template
  You will receive a download link via email. We hate spam and never share your details.

 

Don’t forget to add some proper styling to the file though! She ain’t exactly pretty at the moment.

If you have any issues with the Excel file, or just need a hand with anything from the video, just leave me a comment below and I can help you out.

10 Years of Keyword Research and What You Can Learn From It

10 Years of Keyword Research and What You Can Learn From It

I’ve been doing SEO a little while now on and off over the years (but mostly on) and finally getting back into it full time now.

Over time, I have learned quite a few little handy tricks doing keyword research that has helped me save countless hours… and many weeks!

I figured it was about time to share my little keyword journey and hopefully, you can leverage my wasted time and speed up your research.

Now, I am not going to spend time explaining some stuff, so if you haven’t done keyword research before you should definitely read the¬†Backlinko guide, and then check out the¬†ahrefs tools guide. Some of the post contains basic information, but much of it will only slightly touch on the basics so it’s best you read those guides beforehand.

 

Niche/topic planning

This is where I started back then, and it is where I start now.

The initial topic planning has barely changed for me over the years, as I am sure it hasn’t for many other people.

It does not matter how smart an automated tool is, it will never be as good as you just jotting down some ideas on a spreadsheet.

Back then, I used Excel. Now I use Google Sheets where possible. That’s pretty much the main difference for topic planning!

 

Google Sheets for niche & topic planning

The first part here is just jotting down all the ideas you can think of off the top of your head.

Each column should be separate subtopics, all sitting under a parent topic.

Using the video game niche as an example, this is what my planning would look like;

Google sheets niche research

This would be duplicated out based on every topic and sub-topic I can think of at the time.

The idea here isn’t to cover all the topics within the niche, it’s just to come up with what you can think of so you can build it out later.

So get your research started cleanly.

 

Finding additional topics by analysing competitors

I then try and find new topics along with variations of the existing topics that you think are worth finding sub keywords for.

Just start Googling all your topics and see if anything stands out, then either add it to an existing topic or create a new one.

You should be able to get a few variations just by looking at the SERPs, but jump into the competitor websites and check them too.

Pick up any similar keywords/namings to the topics you have, along with any topics you completely missed.

It doesn’t matter how well you think you know a niche, unknown sub-niches are just a few searches away!

 

What you’re looking for

The aim is to find the gems that an automated tool might not be able to pick up for you. With a secondary goal of starting to build out some top keywords.

Continuing with the gaming niche research as an example, we can take “ps4” and immediately see it’s also known as “playstation 4”. Whilst Google, and many tools, will see this as the same it is still worth jotting it down as the acronym and full spelling might not always be picked up as being the same.

Breaking out the ‘PS4 Consoles’ topic, I quickly come up with;

PS4 Consoles
Playstation 4 Consoles
Playstation 4 Slim
Playstation 4 Pro
Playstation 4 Bundle
Playstation 4 Deals
Preowned Playstation 4

It used to be a lot harder to get keywords like ‘playstation 4 pro’ out of ‘ps4 consoles’, but with the tools available these days it’s so much simpler.

The improvements Google has done on related searches, you can just search for ‘ps4 consoles’ and it’ll show ps4 pro right away;

Google related searches to keyword

But I will go into that in a bit.

Just make sure you get as many different variations of your topic as possible so that it is easier to leverage tools to expand on your seed keywords.

Expanding on topics and finding longer tail keywords

Once you have the initial topics mapped out, you need to start on expanding the keyword set for each of them. This will help you better understand the highest priority keywords so you know what to target on your website.

On top of this, you’ll be able to better understand the entire niche, rather than just a tiny keyword set, which will help with performance analysis and understand where you might be lacking.

 

Google Keyword Tool (now planner)

Finding keywords with Google Keyword Tool

In simpler times, the Google keyword tool was a real goto for my research.

For one, it gave great search volume data.

But its strength was the little “Match Types” selection on the left-hand side. You could actually get Google to come back with suggestions based on Broad / Phrase match, or just get the data for keywords you entered into the list with the exact match.

I would go through topic by topic, and add related keywords to a list, which could then be bulk exported keeping each keyword assigned to the list.

None of this range data, but especially none of the volumes for similar keywords being merged as they do now.

The newer Keyword planner is certainly not at the top of my list as a go-to anymore.

 

Google Autocomplete

Keyword research with Google autocomplete

Originally launched in 2004, Google autocomplete has been an invaluable resource for many SEOs for years.

Google literally gives you the longtail keywords that expand upon the keyword you’re searching for. Use them!

At first, just manually grab them. Do a few manual searches so you can see how it all works, then move into the automated tools to scrape it.

 

Google related searches

Just like Google autocomplete, the related searches offers so much value to SEOs.

Related searches used to be very similar to the autocomplete, and didn’t provide too much value.

Old google related keywords search

However, in 2009, Google added some extra intelligence to them and updated their design.

 

Updated google related searches

This was the first version of intelligence beyond the autocomplete and would be the start of many updates they have since made to it.

Took me a little while to get onto it, but they’ve proved invaluable since.

 

Keyword research tools to assist keyword generation

Over the years I have used quite a few tools to grab me the keywords.

Initially, I was rather cheap, so just stuck to the Google keyword tool. Took a bit longer, but at least I had somewhat quality keywords in there.

After a year or so I finally threw some money at a few tools to help with my research.

 

Scrapebox

Keyword scraper with Scrapebox

Scrapebox was the first research tool I grabbed back in 2009 with a nice little discount off BHW (Blackhatworld, back when it was cool… I think?).

Not initially for the keyword tool, but it soon proved rather useful at building out a nice list.

You can basically just plug a few seed keywords in there, select how many levels deep (awesome for super long keywords), and then click start!

The tool will use a variety of sources and grab you all the long tail keywords you ever need.

I used to just plug all my topics in one by one, along with any keywords I had already found for them, and then let scrapebox do its thing and generate my keywords.

However, there is one catch. You get A LOT of data. What you will need to do is seriously filter this stuff down.

Still a viable tool, and you can grab it here.

 

Market Samurai

Using market samurai for keyword discovery

 

I think Market Samurai was launched in 2008, but I grabbed it in 2010. This tool started to change how I looked at SEO.

It was the first tool I used that started to make me look at SEO a little bit deeper than just a few keywords and some automated link building (2010 SEO was fun!).

Market Samurai had me starting to spend more time on the keyword research, and I was looking at competition in a niche, rather than just trying to rank for anything.

The site still works, and I apparently still have access to the tool, not sure if it still works though. Better tools around these days.

 

Uber Suggest

 

Researching keywords with Ubersuggest

 

Uber suggest launched around late 2009 / early 2010, and quickly became rather useful for discovering new keywords.

Just like the other tools, you throw in a seed keyword and the tool will quickly spit out hundreds of longtail keywords for you to use.

Neil Patel recently-ish acquired Uber Suggest and has been making some updates that have only made the tool better, and at no cost.

I haven’t used the newer version too much, but can safely say I got plenty of value out of the old one!

Give the new tool a try here.

 

Longtail Pro

Longtail Pro was released back in 2011, and slotted itself in as a “new shiny” for me for a while.

I didn’t get massive amounts of value out of it just because I had a handful of other tools that were giving me what I needed at the time.

But who doesn’t like shiny new tech?

It has since been improved upon, grown in features, and can still help many people out.

The original creator sold the software a couple years ago but you can still pick up the software here.

 

SEMrush

SEMrush keyword research

I used SEMrush for years. It became a staple for me… until they started messing with their pricing and removing my API access.

I had a little Google Sheets setup that would hook in with the API and scrape me heaps of keywords & rankings. This let me do heaps of research fast and was within the credit counts that came with my basic subscription. They then removed the API access for this level and jacked up the prices.

Completely understand the business move, and I am sure it worked well for them, but just meant I needed an alternative.

Still worth checking out, as you may enjoy it more than alternatives. They’re still releasing new features you can jump on and you can still get some great value out of downloading the keywords for a domain / URL.

 

keywordtool.io

longtail keywords with keywordtool.io

Once SEMrush didn’t prove valuable for me any more, I turned to KeywordTool.io.

It quickly became my go-to research tool for a couple years due to its speed, simplicity, and filtering rules.

I used the pro version from around 2016 to early 2018, and just loved the fact I could have separate users & share access with outsourcers/mates.

Jump in here and let me know what you think if you haven’t used it before.

 

Ahrefs

It’s safe to say ahrefs is pretty epic. Epic in what you can get out of it, and epic in pricing too. Not recommended for early SEOs, but certainly should be on the list once you start getting work or making some money!

How do I get the value out of it for keyword research?

By stealing competitors keywords!

Keyword download with ahrefs

Just throw in a competitors domain, or the URL path directly related to your niche, and you get a massive list of keywords that you can start to filter through! Do it for the top 10-20 sites, and it’s pretty safe to say you’ll cover a significant chunk of the niche.

Worth a buy for just this, but only if you can actually afford it! Don’t raid your rent money for it. You can get by without it.

 

Keywords Everywhere

Related searches & volume with Keywords Everywhere

I think every SEO under the sun uses this free chrome extension now.

It isn’t a tool that I used very early on in it’s release, and have only just started using it properly, but whenever you Google something the tool will show you related searches on the right.

This significantly helps when expanding out your topics, and you can quickly export all the keywords and their volumes pretty easily.

Give it a crack here.

It does get a bit annoying through so I tend to turn it off regularly.

 

Keysearch

 

 

I have recently switched over to Keysearch for my research (grabbed a Black Friday special… so I am set for a year!).

Friendly interface, and fast keyword selection. I can just tick on the related keywords I like, and then save them to a little list.

A drawback here is each list needs to be exported separately. Takes a couple extra minutes to do that, then a couple minutes to re-merge, but I am happy with that for now.

This is ideal for generating an audited list of related keywords pretty fast, saving you on the cleanup later on.

Definitely worth it for newbies and more advanced users alike.

Give it a test run and jump into the tool here.

 

KeywordKeg

Using keysearch for keyword research

I only played with Keyword Keg for a little bit, since I grabbed Keysearch instead. Definitely one I will be keeping an eye on, as I could switch over at some time in the future.

For now, I am pretty happy using my current toolset.

Still worth having a play with Keyword Keg though, and you can check it out here.

It’s made by the guys that did Keywords Everywhere.

 

Getting the search volume

I have always used Google for my search volume.

The fact they’d allow an upload of 3,000 keywords via csv at a time and then offered a nice CSV export just had me sold from day one.

However, as many of you would know Google did this stupid thing of combining all the volumes for similar keywords.

keywoord search volumes for melbourne plumbers

So even though each of those 3 keywords would get searched a completely different amount of times, Google will now aggregate their volumes into a single one.

Which is an absolute pain if you use search volume for anything. You either need to use an alternate source or leverage one of the third party tools and their estimations if you want to separate out the volume for these keywords.

I personally prefer to try and remove all these similar keywords and just focus on the primary one, as that allows me to continue to use Google for the volumes.

Why? I use the search volume for a heap of different analysis & performance monitoring. You could still get away using this, but you will get inflated “estimated traffic” values, and some other metrics that might be more off than normal.

Google keyword planner upload

Unfortunately, Google killed off the 3,000 keyword limit upload via CSV in the latest planner, and also made it slightly easier uploading in that you can just paste a bulk keyword set in (used to be an 800 limit via pasting).

But there is always a catch!

The new keyword limit appears to currently be 2,500.

Still workable, just a pain.

Oh, and you need an active Adwords account to even get a somewhat accurate volume these days. So if you don’t have one of them, or can’t access one through a client, it’s probably best you look elsewhere for volumes.

 

Playing with large location-based keyword sets

Things really changed for me process wise when I jumped into agency life in 2013. Moving interstate and re-settling was nothing compared to changing from small business clients to some rather large businesses!

I went from helping a plumber to working with large international hotel chains.

I went from a home-based cupcake business to ranking a national car rental agency.

But most importantly, I went from a few hundred keywords, to keyword sets of 100,000+.

This forced me to rethink how I did SEO… and this is where the real time savers kick in!

 

Bulk generating keywords

When I was starting agency side, I could use Excel but I certainly didn’t have any sort of advanced skills.

I quickly saw how much time I could save with some of the processes, and began learning more about it.

The biggest thing I was using it for at first, was bulk generating keywords.

I had a handful of keywords, that I need to replicate across multiple locations.

 

How I started generating

So, before I played with Excel too much, I did a pretty manual thing to swap out locations in keywords.

I used to find/replace a location out of the keyword.

Find and replace location keywords

Hey, it worked and got the job done! Once I started playing with 10+ locations and larger keyword sets this got really annoying.

I needed a different way of doing this.

 

Along came concatenation (my version of it anyway)

Concatenation is basically just joining things together, but I started doing concatenation before I even knew what that meant.

Within Excel, you can use the ampersand (&) as a magical way to join things together.

You can also use a space between two quotations to insert a blank space into your formula.

By leveraging these two things, I was able to merge together different cells to generate my keywords.

Concatenating keywords in Excel

The formula for my above example is;

=$A3&” “&C$2

It allows me to tack on the location to the end of the keyword template, and create me the keywords I need.

The $ symbol locks in a cell reference. So by locking in the first column (the keyword ‘template’) and the 2nd row (the location), I am able to drag the formula down and across.

Excel actually has a formula to do this;

=CONCATENATE(text1, [text2], …)

To this day, I don’t know why you would use this over the ampersand though. The ampersand version is just so much more flexible, and easier to maintain as you build out your formulas.

Would love to hear of a usecase if you have one though!

 

Substitute yourself to local keywords

You might notice in my concatenation example I had “car hire” and “car rental” templates.

This is because I would like both ‘car hire sydney’ and ‘sydney¬†car hire’, but to do this with concatenation you would need to modify the formula.

This is where some substitution magic comes in!

=SUBSTITUTE(text, old_text, new_text)

This formula quickly became how I generated location keywords and is still used in one form or another today!

It looks inside a piece of text, for specific text, then replaces that specific text with whatever you tell it to.

Instead of adding a template and location together, I used;

Car hire LOCATION

LOCATION car hire

And just swapped out the location for the location I wanted to use.

Substitution formula for keyword generation

The formula to swap the text LOCATION for the location is;

=SUBSTITUTE($A3,”LOCATION”,C$2)

And to take it one step further, you can even replace ‘car hire’ with SERVICE, and then add that to the substitution formula.

This could also be modified to suit any sort of keyword, so you could easily tack on modifiers if you wanted to. No reason you can’t substitute out 3,4, or 5+ words.

I made a video about it a couple years ago, so this might be a bit more useful ūüôā

The issue when you start to do that is that you end up with too many different variations, and are just copying and pasting your formula all over the place.

Which is where I started learning about macros and how to repetitively do something I hated doing.

 

My excel keyword builder (magic macro)

Keyword builder in Excel

Created in 2014, this magical thing would cycle through keyword types, keywords, locations & even secondary locations, and bulk generated me a tonne of keyword variations.

The macro died off a little while ago, well something broke and I can’t be bothered fixing it, but saved me many hours.

Back when Google gave a good volume value, this worked best, as it let me easily identify the top search terms to target.

It sometimes took a little while to generate, and it was hard to get setup initially due to the macro.

So I needed a new plan.

 

IMforSMB Keyword Tool

I’d always wanted to learn a bit more about coding, and so in 2015, I decided to take my Excel tool and build out my own web tool.

Years of playing with WordPress gave me a decent enough understanding of HTML & PHP, so that was my obvious choice for this tool.

After about a week of hacking, I finally pulled something together.

IMforSMB Bulk Keyword Generator

This completely replaced the Excel generator for me.

Using the DIY Generator of the tool, I can now just paste in services & locations and click generate and the tool will spit out all my keywords.

IMforSMB search term generator

I threw in some templates services for people to use the builder if they’d rather, but my main focus here was just the simple generator so that I could let my Excel sheet die off.

You can check out the old bulk keyword generator here, however, this is going to get replaced in the future.

This tool started me thinking about a few things and got me working on my latest project.

 

SEOpp Keyword Generator

SEOpp keyword tool

After the IMforSMB tool started getting used, I figured it was time to build the tool out properly.

The first piece has just finally been launched with the SEOpp keyword generator now live.

The tool currently forces a business type selection, however, it will soon allow you to also enter custom services.

There is a big difference between this and the IMforSMB tool. SEOpp integrates with millions of locations, so you can generate a list of locations for your research!

You pick an initial location, a distance you service, and then it will spit out the locations for you to generate your keywords with.

Plenty more coming soon, so stay tuned!

 

Managing your keywords

It’s all well and good having a bajillion keywords laying around, but you need to be able to move them around manage them effectively.

I try and do as much of my work in a single Google Sheets or Excel File, depending on how large the data is and what I need to do.

 

Splitting CSV files of keywords

If you generated a list of 50,000 keywords, you’ll probably need to break it up if you want to run it through the Google keyword planner, or any other bulk upload tools.

Not many tools will accept an upload of 50k at once!

I started by manually doing this, 3,000 at a time for the planner. It is a tonne of copy/pasting and drove me mental.

Until I found the CSV splitter here.

Select a CSV, then enter how many rows for each file. You can tick on whether to have the header remain on each individual file or not as well.

This will spit out all the files in the same folder as where you have the large CSV, so if you’re splitting a heap of files just organise it accordingly beforehand.

 

Merging CSV files of keywords

If you’ve ever downloaded a heap of CSVs at once, you know how much of a pain it can be merging these all back together.

I too started by manually copy/pasting all the data one by one into a master set.

Mostly search volume exports, ranking data or competitor analysis, I had enough pretty quick and discovered a neat little CMD trick.

Mergine CSV files with CMD

How to merge CSV files together

  1. Open up the command prompt by typing CMD into windows search.
  2. Type ‘CD’ (change directory) and paste in the folder path for where all your CSV files are and hit enter
  3. Type ‘copy *.csv filename.csv’ and hit enter

If successful your merged file will be in the same directory where all your smaller files were stored.

You will be able to merge hundreds of files in seconds, so you can save quite a bit of time!

Not sure if this trick works on Mac, so feel free to give it a crack and let me know.

 

Bulk categorisation & classification of keywords

Categorising (or bucketing/classifying) keywords is a pretty big pain once you begin managing hundreds to thousands of keywords, let alone hundreds of thousands.

This is where Excel has really shined for me over the years.

I discovered a little magical formula that has allowed me to essentially create category ‘seed’ lists, and then bucket the keywords off the back of this.

=IFERROR(LOOKUP(9.99999999999999E+307,SEARCH(Category1[Find],$Keyword Cell Reference$),Category1[Category]),”Generic”)

Ignoring the super big number at the start (something to do with that being the largest row count Excel can do), the formula is actually really easy to use.

How to bulk classify keywords for SEO

  1. Create a table with two columns, ‘Find’, and ‘Category’, and name the table “Category1”
  2. Enter your ‘search’ words that you want to look for in the left column
  3. Enter your categories next to the search words, that will be given when a search word is found
  4. Enter the categorisation formula in a cell next to your keyword and update cell reference to point at it
  5. The formula should now categorise your keywords, based on what you entered

Bulk keyword categoriser & classifier

So essentially, it just looks for words (or parts of words) within your keyword, then categorises or classifies them appropriately off the back of that. If it doesn’t find something, it will just say ‘Generic’ instead, which you can update to something else in the formula. You can even use a cell reference, and customise it per category if you want.

No more match, lookup or vlookup formulas just looking for a single category at a time!

I don’t remember what it was originally used for, certainly wasn’t for SEO though! It was hidden away on some accounting forum, where SEOs will rarely go. Goes to show what you can find when you Google hard enough!

Another one I made a video for a couple years ago, which might explain the formula a little better.

I also created a keyword categorisation template you can download here.

It’s pretty useful for a few things, and I would love to hear what you do with it!

Among other things, I use it for dashboarding and keyword mapping to get them all assigned to the right URLs.

 

Leveraging categories for keyword to URL mapping

Once you’ve got your categories, you can throw them into a URL structure with a formula. There are quite a few variables here, and it sometimes comes down to personal choice as to how you’d like your URLs, but see below for a starting example.

Now, this isn’t how I would be grouping keywords & structure the URLs for this particular keyword set, but it still provides a good example.

Keyword to URL mapping in Excel example

=SUBSTITUTE(SUBSTITUTE(LOWER(“/”&[@[Category Type 1]]&”-“&[@[Category Type 2]]&”/”&[@[Category Type 3]]&”/”),” “,”-“),”generic/”,””)

The formula is taking the categories, and merging them all together, whilst following the standard practice of making everything lower case, and replacing spaces with dashes.

You’ll also see that I merged the first two category types together so they sat at the same level. Just for something different!

Then you just need to create separate pivot tables for each URL you’ve created, and you will have your URL > Keyword mapping.

Excel pivot tables with keyword mapping

You’ll then end up with a neatly mapped set of keywords, and some pretty tables that show the mapping.

You can event take it a step further and cross check this mapping against the actual rankings for the website, see what’s not ranking for the right page, then try to determine why.

But that’s for another time!

 

Building a keyword dashboard

By just leveraging pivot tables & slicers, you can create an interactive keyword dashboard. This could be for a client, or just to make it easier for you to see top keywords when you’re dealing with keyword lists of thousands.

For this one, probably best you just watch the video!

 

And that’s a wrap… for now.

I’d love to be able to break down the different bits here into mini-tutorials, so if there is anything, in particular, you want to find out more about, let me know in the comments below!