Some clients have recently required a migration mapped out for 10’s to 100’s of thousands of URLs.
To do this, we needed to capture all the different URLs, build out the migration rules, and then test the redirects.
Mapping out and building the rules is one thing, but then how do we bulk generate what the URLs should look post-migration so that we can actually test the redirects?
How can we test the implemented redirect rules, to ensure they match what we originally mapped out?
No one wants to manually create and map 1,000+ URLs to compare the redirect path too, and I personally prefer to not just test the redirects then give them ‘after’ the once over.
I’d like something to compare the implementation to.
Using Google sheets (or Excel) you can strip down the existing URL, extract the parts the redirection rules will use, and then rebuild the existing URL into their new formats.
This will allow you to do a bulk migration test against the implemented redirect rules, and confirm everything redirects where it should.
There are a couple of ways of achieving this, and I will just go through the different methods I use below.
A quick note though, this has nothing to do with building the actual redirection rules themselves. Whilst it might help you build better examples of what you need to redirect, this post is purely about building the expected URLs so you can confirm the redirects match.
Stripping a URL down
There are a few main types of URLs we can account for here.
The standard hierarchical folder-based structure, the flat URL structure with everything to load the page in the base URL, and the ugly query parameter based structure.
They can all be stripped down relatively easily, but firstly we can strip out the URL path from the domain to simplify issue patching later.
Removing the domain from a URL Path
A handful of different ways to strip a domain from a URL are available, but this formula is possibly the easiest;
As an SEO, you really can’t escape Excel. Almost every month a new SEO tool pops up, that can do so much, but will never seem to replace those little things you can do in Excel.
You find yourself using a tool for 75% of the job, but will then always jump back into Excel to finish it off.
Well, that’s me anyway. If it’s not you, then you probably just don’t know many of the things that Excel can do!
I run through VLOOKUPs, IF statements, LEN, TRIM, PROPER / UPPER / LOWER, COUNTIFs & SUMs and show you some basic usage examples. I’ve also added a few extras to the bottom that aren’t in the video!
Basic Excel Formulas for SEOs
Whether you’re starting out, or an experienced SEO, these are the basic building blocks to so many advanced Excel uses for SEO. You need to know these.
VLOOKUP – Find data for a value, within a range
The simplest summary of what a VLOOKUP will do is to match two sets of data where you have something to match them with.
So is long as one column in each data set has some sort of unique identifier, then you can VLOOKUP between the two datasets and bring the data together.
If you don’t have a unique identifier, will you have one if you match two columns together?
You might have a date that isn’t unique, but if you add it to another column (like colour) you will get a unique value that you can use to match the data sets.
=VLOOKUP(*CELL of what you want to search for*,*RANGE of where you want to look for it*,*COLUMN NUMBER the data is in you want to extract*,*TRUE (1) or FALSE (0) but 99% will be FALSE*)
LEN – Count character length of a cell
Nice and simple formula. The LEN formula is great for counting the lengths of page titles and meta descriptions!
Plenty of other uses, but that’s what the majority of people will use it for.
=LEN(*CELL to count*)
TRIM – Remove spaces before & after a cell
Sometimes when trying to match data up, the cell might visually look the same as another cell, but you can’t seem to match it up!
The amount of times I have run into this is crazy, and the majority of times it happens there is a pesky space before or after the text.
The TRIM formula will clean these up for you and make sure there isn’t something hidden there wasting your time!
=TRIM(*CELL to trim*)
SUM – Add all numbers in a range together
Just adds numbers. It’s that simple. Just select the range you’d like to add together and away it goes.
=SUM(*RANGE to add together*)
COUNT – Display a count of cells with numbers in them
Out of your selected range, how many of the cells have a number in them?
=COUNT(*RANGE to count for numbers*)
COUNTA – Display a count of cells with numbers and letters in them (so not empty)
Out of your selected range, how many of the cells have a number of letters in them? This is what you need to use if you are trying to count cells with characters rather than numbers.
=COUNTA(*RANGE to count for numbers*)
COUNTIF – Count all the cells when a specific criterion is met
Only count a cell that matches your criteria. This is great for being able to select cell as your criteria.
So if you have a column of ‘Colour’ and a cell that has the text of ‘Blue’, you could select that ‘Blue’ cell and then the range that has all the colours, and the formula will count how many times ‘Blue’ appears in the range.
=COUNTIF(*RANGE to count for numbers*,*CRITERIA for when to count*)
PROPER – Capitalise the first letter of every word in a cell
Great for cleaning up a list of titles, or any words, where you’d like to capitalise the first letter of every word.
=PROPER(*CELL to capitalise first letters*)
UPPER – Capitalise every letter of every word in a cell
This will format it AND MAKE IT LOOK LIKE YOU’RE SHOUTING. Just use the upper formula when you need to make sure everyone knows you’re SERIOUS.
=UPPER(*CELL to capitalise everything*)
LOWER – Make every letter of every word in a cell lowercase
The lower formula will make everything lower case. Great to run through your keywords with to ensure there isn’t any weird capitalisation going on.
=LOWER(*CELL to lowercase*)
LEFT & RIGHT – Extract x characters from the left or right of a cell
This is a simple way of extracting a certain amount of characters from the left or the right of some text.
Great if you want to cull the end of a title, remove a domain, or trim some trailing slashes from a URL.
=LEFT(*cell reference for text to look in*, *number of characters*)
=RIGHT(*cell reference for text to look in*, *number of characters*)
However, if you are looking trim everything before or after a certain character, instead of a number of characters, check below for a more advanced formula!
IF – Only do something when a certain condition is met
The IF formula lets you create rules for your cells. Maybe you need to filter out titles that are too long.
You could do an IF formula in a new column that says =IF(length>70,”LONG”,”OKAY”). This will give you back a cell that says LONG if the title is too long, or OKAY if everything is okay.
Probably not the best use case, but gives you the idea!
=IF(*The condition to run*,*What to do if it’s true,*What to do if it’s false*)
Does Cell Contains Text – Run condition if the cell contains specific text
A quick formula to use if you want to do something off the back of a cell containing a specific piece of text.
This formula will return TRUE or FALSE, which is what I normally use with this one, but you can expand it do use IF. So IF bla = TRUE, do this.
Great way to quickly segregate data by whether something is in a URL, or keyword.
=ISNUMBER(SEARCH(“text to look for”,*Cell to look in*))
& – Yeap, the ampersand bad boy is rather useful
If you’re trying to merge stuff together in Excel, you need the ampersand. This thing will let you attach anything, to anything else.
It will let you run multiple formulas, then just stick them all together to form what is essentially a sentence.
=”Stick this “&”,with this “&”, and merge all “&2+2&” sentences together”
Which will output:
Stick this, with this, and merge all 4 elements together
SUBSTITUTE – Swap out any text, with any other text
This formula is invaluable if you are doing any sort of text editing or manipulation.
You select the cell you’d like to manipulate, enter the text that currently exists, then enter the text you wish to change it to.
=SUBSTITUTE(*text to look in*,”old text here”,”new text here”)
Trim URL to Domain / Subdomain
This is something that you will find incredibly useful if you’ve never done it before. Some extra tools do it, but you can achieve almost perfect trimming with just a formula – SOOOO much easier!
Quick and Dirty Trim Domain Prefixes Method
If you are just looking to trim the HTTP part, here is a hacky formula that will just substitute out the HTTP part. It’ll leave subdomains / domains, and all their folders intact however.
With A2 being the cell reference to where you have a URL to trim.
Trim URL to Root Domain
If you are wanting to get rid of the subdomain too, so you are just left with the root domain, you can use this formula. It will remove folders, subdomains, prefixes, and anything else so that you are just left with the root domain stripped out.
An extremely large table of all their keywords, isn’t going to give the insight some clients would want. Some would like more a bit more insight into the data they’ve paid thousands for.
Building a keyword research dashboard in Excel is a great way to allow a client to interact with their keyword data.
In this short video, I will show you how you can take a categorised keyword list and turn it into a client-friendly dashboard!
The video is a couple years old now, and I may eventually update it, however, for the most part, it still holds true today.
Now, this won’t be great for some clients. You could end up giving them way too much information, and have them asking way too many questions.
However, I personally don’t mind that. I love when clients have an interest in SEO as it gets them more involved in the project, and more excited when the results kick in.
How to build a keyword research dashboard in Excel
Categorise up your keyword research data
Create pivot tables for each category level
Attach slicers to the categories so you can filter the data
Design the tables & slicers to match your client’s branding
And there you have it, you will have an interactive Excel keyword research dashboard to send to your clients.
Creating an online keyword research dashboard
You can save this Excel dashboard to Onedrive and then share with your clients via a link instead of sending a file.
For dashboards under a certain size this means they will be able to view the dashboard online instead of having to download the file. You can then simply update the keyword research file, and they will be able to see the update when they refresh their browser.
This becomes a massive help when trying to send updated data to a client. No more v53 tacked onto the end of a file because you’ve updated it 53 times!
Download the Excel Keyword Research File
If you watched the video and can’t be bothered doing the work yourself, you can just have my file!
Or if you’re doing keyword research for car hire you can also just steal my keywords – chances of that have to be slim.
I’ve been doing SEO a little while now on and off over the years (but mostly on) and finally getting back into it full time now.
Over time, I have learned quite a few little handy tricks doing keyword research that has helped me save countless hours… and many weeks!
I figured it was about time to share my little keyword journey and hopefully, you can leverage my wasted time and speed up your research.
Now, I am not going to spend time explaining some stuff, so if you haven’t done keyword research before you should definitely read the Backlinko guide, and then check out the ahrefs tools guide. Some of the post contains basic information, but much of it will only slightly touch on the basics so it’s best you read those guides beforehand.
This is where I started back then, and it is where I start now.
The initial topic planning has barely changed for me over the years, as I am sure it hasn’t for many other people.
It does not matter how smart an automated tool is, it will never be as good as you just jotting down some ideas on a spreadsheet.
Back then, I used Excel. Now I use Google Sheets where possible. That’s pretty much the main difference for topic planning!
Google Sheets for niche & topic planning
The first part here is just jotting down all the ideas you can think of off the top of your head.
Each column should be separate subtopics, all sitting under a parent topic.
Using the video game niche as an example, this is what my planning would look like;
This would be duplicated out based on every topic and sub-topic I can think of at the time.
The idea here isn’t to cover all the topics within the niche, it’s just to come up with what you can think of so you can build it out later.
So get your research started cleanly.
Finding additional topics by analysing competitors
I then try and find new topics along with variations of the existing topics that you think are worth finding sub keywords for.
Just start Googling all your topics and see if anything stands out, then either add it to an existing topic or create a new one.
You should be able to get a few variations just by looking at the SERPs, but jump into the competitor websites and check them too.
Pick up any similar keywords/namings to the topics you have, along with any topics you completely missed.
It doesn’t matter how well you think you know a niche, unknown sub-niches are just a few searches away!
What you’re looking for
The aim is to find the gems that an automated tool might not be able to pick up for you. With a secondary goal of starting to build out some top keywords.
Continuing with the gaming niche research as an example, we can take “ps4” and immediately see it’s also known as “playstation 4”. Whilst Google, and many tools, will see this as the same it is still worth jotting it down as the acronym and full spelling might not always be picked up as being the same.
Breaking out the ‘PS4 Consoles’ topic, I quickly come up with;
Playstation 4 Consoles
Playstation 4 Slim
Playstation 4 Pro
Playstation 4 Bundle
Playstation 4 Deals
Preowned Playstation 4
It used to be a lot harder to get keywords like ‘playstation 4 pro’ out of ‘ps4 consoles’, but with the tools available these days it’s so much simpler.
The improvements Google has done on related searches, you can just search for ‘ps4 consoles’ and it’ll show ps4 pro right away;
But I will go into that in a bit.
Just make sure you get as many different variations of your topic as possible so that it is easier to leverage tools to expand on your seed keywords.
Expanding on topics and finding longer tail keywords
Once you have the initial topics mapped out, you need to start on expanding the keyword set for each of them. This will help you better understand the highest priority keywords so you know what to target on your website.
On top of this, you’ll be able to better understand the entire niche, rather than just a tiny keyword set, which will help with performance analysis and understand where you might be lacking.
Google Keyword Tool (now planner)
In simpler times, the Google keyword tool was a real goto for my research.
For one, it gave great search volume data.
But its strength was the little “Match Types” selection on the left-hand side. You could actually get Google to come back with suggestions based on Broad / Phrase match, or just get the data for keywords you entered into the list with the exact match.
I would go through topic by topic, and add related keywords to a list, which could then be bulk exported keeping each keyword assigned to the list.
None of this range data, but especially none of the volumes for similar keywords being merged as they do now.
The newer Keyword planner is certainly not at the top of my list as a go-to anymore.
Originally launched in 2004, Google autocomplete has been an invaluable resource for many SEOs for years.
Google literally gives you the longtail keywords that expand upon the keyword you’re searching for. Use them!
At first, just manually grab them. Do a few manual searches so you can see how it all works, then move into the automated tools to scrape it.
Google related searches
Just like Google autocomplete, the related searches offers so much value to SEOs.
I used SEMrush for years. It became a staple for me… until they started messing with their pricing and removing my API access.
I had a little Google Sheets setup that would hook in with the API and scrape me heaps of keywords & rankings. This let me do heaps of research fast and was within the credit counts that came with my basic subscription. They then removed the API access for this level and jacked up the prices.
Completely understand the business move, and I am sure it worked well for them, but just meant I needed an alternative.
Still worth checking out, as you may enjoy it more than alternatives. They’re still releasing new features you can jump on and you can still get some great value out of downloading the keywords for a domain / URL.
Once SEMrush didn’t prove valuable for me any more, I turned to KeywordTool.io.
It quickly became my go-to research tool for a couple years due to its speed, simplicity, and filtering rules.
I used the pro version from around 2016 to early 2018, and just loved the fact I could have separate users & share access with outsourcers/mates.
Jump in here and let me know what you think if you haven’t used it before.
It’s safe to say ahrefs is pretty epic. Epic in what you can get out of it, and epic in pricing too. Not recommended for early SEOs, but certainly should be on the list once you start getting work or making some money!
How do I get the value out of it for keyword research?
By stealing competitors keywords!
Just throw in a competitors domain, or the URL path directly related to your niche, and you get a massive list of keywords that you can start to filter through! Do it for the top 10-20 sites, and it’s pretty safe to say you’ll cover a significant chunk of the niche.
Worth a buy for just this, but only if you can actually afford it! Don’t raid your rent money for it. You can get by without it.
I think every SEO under the sun uses this free chrome extension now.
It isn’t a tool that I used very early on in it’s release, and have only just started using it properly, but whenever you Google something the tool will show you related searches on the right.
This significantly helps when expanding out your topics, and you can quickly export all the keywords and their volumes pretty easily.
It’s made by the guys that did Keywords Everywhere.
Getting the search volume
I have always used Google for my search volume.
The fact they’d allow an upload of 3,000 keywords via csv at a time and then offered a nice CSV export just had me sold from day one.
However, as many of you would know Google did this stupid thing of combining all the volumes for similar keywords.
So even though each of those 3 keywords would get searched a completely different amount of times, Google will now aggregate their volumes into a single one.
Which is an absolute pain if you use search volume for anything. You either need to use an alternate source or leverage one of the third party tools and their estimations if you want to separate out the volume for these keywords.
I personally prefer to try and remove all these similar keywords and just focus on the primary one, as that allows me to continue to use Google for the volumes.
Why? I use the search volume for a heap of different analysis & performance monitoring. You could still get away using this, but you will get inflated “estimated traffic” values, and some other metrics that might be more off than normal.
Unfortunately, Google killed off the 3,000 keyword limit upload via CSV in the latest planner, and also made it slightly easier uploading in that you can just paste a bulk keyword set in (used to be an 800 limit via pasting).
But there is always a catch!
The new keyword limit appears to currently be 2,500.
Still workable, just a pain.
Oh, and you need an active Adwords account to even get a somewhat accurate volume these days. So if you don’t have one of them, or can’t access one through a client, it’s probably best you look elsewhere for volumes.
Playing with large location-based keyword sets
Things really changed for me process wise when I jumped into agency life in 2013. Moving interstate and re-settling was nothing compared to changing from small business clients to some rather large businesses!
I went from helping a plumber to working with large international hotel chains.
I went from a home-based cupcake business to ranking a national car rental agency.
But most importantly, I went from a few hundred keywords, to keyword sets of 100,000+.
This forced me to rethink how I did SEO… and this is where the real time savers kick in!
Bulk generating keywords
When I was starting agency side, I could use Excel but I certainly didn’t have any sort of advanced skills.
I quickly saw how much time I could save with some of the processes, and began learning more about it.
The biggest thing I was using it for at first, was bulk generating keywords.
I had a handful of keywords, that I need to replicate across multiple locations.
How I started generating
So, before I played with Excel too much, I did a pretty manual thing to swap out locations in keywords.
I used to find/replace a location out of the keyword.
Hey, it worked and got the job done! Once I started playing with 10+ locations and larger keyword sets this got really annoying.
I needed a different way of doing this.
Along came concatenation (my version of it anyway)
Concatenation is basically just joining things together, but I started doing concatenation before I even knew what that meant.
Within Excel, you can use the ampersand (&) as a magical way to join things together.
You can also use a space between two quotations to insert a blank space into your formula.
By leveraging these two things, I was able to merge together different cells to generate my keywords.
The formula for my above example is;
It allows me to tack on the location to the end of the keyword template, and create me the keywords I need.
The $ symbol locks in a cell reference. So by locking in the first column (the keyword ‘template’) and the 2nd row (the location), I am able to drag the formula down and across.
Excel actually has a formula to do this;
=CONCATENATE(text1, [text2], …)
To this day, I don’t know why you would use this over the ampersand though. The ampersand version is just so much more flexible, and easier to maintain as you build out your formulas.
Would love to hear of a usecase if you have one though!
Substitute yourself to local keywords
You might notice in my concatenation example I had “car hire” and “car rental” templates.
This is because I would like both ‘car hire sydney’ and ‘sydney car hire’, but to do this with concatenation you would need to modify the formula.
This is where some substitution magic comes in!
=SUBSTITUTE(text, old_text, new_text)
This formula quickly became how I generated location keywords and is still used in one form or another today!
It looks inside a piece of text, for specific text, then replaces that specific text with whatever you tell it to.
Instead of adding a template and location together, I used;
Car hire LOCATION
LOCATION car hire
And just swapped out the location for the location I wanted to use.
The formula to swap the text LOCATION for the location is;
And to take it one step further, you can even replace ‘car hire’ with SERVICE, and then add that to the substitution formula.
This could also be modified to suit any sort of keyword, so you could easily tack on modifiers if you wanted to. No reason you can’t substitute out 3,4, or 5+ words.
I made a video about it a couple years ago, so this might be a bit more useful 🙂
The issue when you start to do that is that you end up with too many different variations, and are just copying and pasting your formula all over the place.
Which is where I started learning about macros and how to repetitively do something I hated doing.
My excel keyword builder (magic macro)
Created in 2014, this magical thing would cycle through keyword types, keywords, locations & even secondary locations, and bulk generated me a tonne of keyword variations.
The macro died off a little while ago, well something broke and I can’t be bothered fixing it, but saved me many hours.
Back when Google gave a good volume value, this worked best, as it let me easily identify the top search terms to target.
It sometimes took a little while to generate, and it was hard to get setup initially due to the macro.
So I needed a new plan.
IMforSMB Keyword Tool
I’d always wanted to learn a bit more about coding, and so in 2015, I decided to take my Excel tool and build out my own web tool.
Years of playing with WordPress gave me a decent enough understanding of HTML & PHP, so that was my obvious choice for this tool.
After about a week of hacking, I finally pulled something together.
This completely replaced the Excel generator for me.
Using the DIY Generator of the tool, I can now just paste in services & locations and click generate and the tool will spit out all my keywords.
I threw in some templates services for people to use the builder if they’d rather, but my main focus here was just the simple generator so that I could let my Excel sheet die off.
Ignoring the super big number at the start (something to do with that being the largest row count Excel can do), the formula is actually really easy to use.
How to bulk classify keywords for SEO
Create a table with two columns, ‘Find’, and ‘Category’, and name the table “Category1”
Enter your ‘search’ words that you want to look for in the left column
Enter your categories next to the search words, that will be given when a search word is found
Enter the categorisation formula in a cell next to your keyword and update cell reference to point at it
The formula should now categorise your keywords, based on what you entered
So essentially, it just looks for words (or parts of words) within your keyword, then categorises or classifies them appropriately off the back of that. If it doesn’t find something, it will just say ‘Generic’ instead, which you can update to something else in the formula. You can even use a cell reference, and customise it per category if you want.
No more match, lookup or vlookup formulas just looking for a single category at a time!
I don’t remember what it was originally used for, certainly wasn’t for SEO though! It was hidden away on some accounting forum, where SEOs will rarely go. Goes to show what you can find when you Google hard enough!
Another one I made a video for a couple years ago, which might explain the formula a little better.
It’s pretty useful for a few things, and I would love to hear what you do with it!
Among other things, I use it for dashboarding and keyword mapping to get them all assigned to the right URLs.
Leveraging categories for keyword to URL mapping
Once you’ve got your categories, you can throw them into a URL structure with a formula. There are quite a few variables here, and it sometimes comes down to personal choice as to how you’d like your URLs, but see below for a starting example.
Now, this isn’t how I would be grouping keywords & structure the URLs for this particular keyword set, but it still provides a good example.
=SUBSTITUTE(SUBSTITUTE(LOWER(“/”&[@[Category Type 1]]&”-“&[@[Category Type 2]]&”/”&[@[Category Type 3]]&”/”),” “,”-“),”generic/”,””)
The formula is taking the categories, and merging them all together, whilst following the standard practice of making everything lower case, and replacing spaces with dashes.
You’ll also see that I merged the first two category types together so they sat at the same level. Just for something different!
Then you just need to create separate pivot tables for each URL you’ve created, and you will have your URL > Keyword mapping.
You’ll then end up with a neatly mapped set of keywords, and some pretty tables that show the mapping.
You can event take it a step further and cross check this mapping against the actual rankings for the website, see what’s not ranking for the right page, then try to determine why.
But that’s for another time!
Building a keyword dashboard
By just leveraging pivot tables & slicers, you can create an interactive keyword dashboard. This could be for a client, or just to make it easier for you to see top keywords when you’re dealing with keyword lists of thousands.
For this one, probably best you just watch the video!
And that’s a wrap… for now.
I’d love to be able to break down the different bits here into mini-tutorials, so if there is anything, in particular, you want to find out more about, let me know in the comments below!