Leveraging Screaming Frog’s Crawl Comparison Feature for Enhanced SEO Audits. Google continuously updates its algorithms, incorporating Rankbrain, BERT, and MUM to improve search results. But how does Screaming Frog stay ahead of the curve? With each Google update, this SEO spider tool introduces new features that are worth exploring.
One of the exciting feature updates I’m looking forward to trying out for our SEO Hacker team’s future website audits is Screaming Frog’s crawl comparison feature. This tool provides a way to compare crawls and analyze how crawl data has evolved over time. In this article, I’ll share some handy and time-saving “tricks” I’ve discovered while exploring this feature.
SEO Spider Tool’s Crawl Comparison Feature Update
Screaming Frog’s crawl comparison feature has the potential to revolutionize the way we conduct website audits. Over the years, my team and I have followed a relatively straightforward process for our clients’ website audits. We input the client’s website URL into the Screaming Frog software, let it crawl the site, save and export the data to a spreadsheet, and then examine it for errors or optimization opportunities. Subsequently, we create a user-friendly PDF report outlining the issues and recommended improvements. That’s been our standard procedure.
The crawl comparison feature aims to help monitor a website’s SEO progress and issues. It allows you to compare crawls over time, providing bite-sized information on data changes, issues, and opportunities.
To initiate a crawl comparison using Screaming Frog, follow these steps:
- Open the tool, and by default, you’ll find the “Mode” tab set to “Spider.” Change it from “Spider” to “Compare” mode to begin your crawl comparison assessment.
- Select the crawl files you want to compare side by side. Click the “compare” button to start the crawl comparison analysis.
- Once the analysis is complete, you’ll see the “Overview” tab on the right side of the interface, displaying your current and previous crawled data and highlighting any changes. For demonstration purposes, I used saved crawled data from a recent project.
Since both crawled files were from the same month, there weren’t significant differences to compare. However, after exploring this feature, I believe it will be highly beneficial for our future website audits.
I also noticed that, before switching to the database storage option, I was using RAM, which was causing my computer’s performance to lag. However, after changing the storage mode to Database, the tool’s performance significantly improved while running the crawl comparison analysis simultaneously.
In essence, the SEO spider tool defaults to Memory storage, meaning that all crawled data is automatically stored in your computer’s temporary memory (RAM). This can slow down your computer’s performance when dealing with extensive data. This is why Screaming Frog strongly recommends configuring your storage mode to Database. When I switched to Database storage before conducting the analysis, my computer’s performance improved notably.
Benefits of Screaming Frog’s Crawl Comparison Feature
To be honest, my team and I have been manually identifying technical SEO issues and tracking optimization progress since day one.
After experimenting with Screaming Frog’s crawl comparison feature, I find it efficient in generating the data I need for my technical SEO checks. Although its basic crawling functionality is already robust, I recommend investing in the licensed version of the tool to access and utilize this feature fully.
So, what led me to discover this lesser-known feature in the first place? Here are some reasons why I believe this feature might be valuable for you too:
1. Time-Saving
Screaming Frog’s crawl comparison feature simplifies data analysis, eliminating the need to manually transfer or export data from the software to a spreadsheet. With this feature, the software itself becomes the sole resource for examining your website’s technical issues and monitoring its progress.
Moreover, this feature streamlines the monitoring of your optimization efforts. Since your crawled data is imported directly into the tool, you can quickly access the crawl data you want to compare through the interface’s File > Data tab.
2. Memory Storage Optimization
Every time you perform a crawl comparison analysis, the tool automatically switches its processing from Memory to Database mode.
Screaming Frog prioritizes efficient memory allocation, especially since the free version has limitations. By switching to the licensed version of the tool, you gain access to Database storage. Database storage not only allows you to crawl beyond 500 URLs (essential for larger audits) but also prevents your computer from slowing down while the tool runs in the background.
3. Detecting Key On-Page Elements
The “Overview” tab on the right side of the interface provides default key metrics, such as the total number of URLs, internal and external subdomains, response codes, and crawl depth differences. However, it doesn’t show changes in other crucial elements, like title tags, meta descriptions, internal links, and more.
You can easily configure this by adjusting the comparison settings for change detection within the Screaming Frog software. This configuration allows the tool to analyze the key metrics you select, such as H1 tags, page titles, crawl depth, meta descriptions, internal links, and more. Optimizing these elements is vital for improving your site’s search engine rankings.
Click the cogwheel icon in “Compare mode” to choose which items you’d like to detect for changes.
This setting is especially useful for checking whether title tags, meta descriptions, or other SEO elements have been updated.
Key Takeaway
For years, my team and I have relied on Screaming Frog SEO spider to crawl and analyze websites to enhance our clients’ search rankings. To me, this is one of the most valuable SEO tools available. Now, it’s even more essential due to its crawl comparison feature update.
Exploring the crawl comparison feature is just one of the many lesser-known aspects of this tool that I’ve recently immersed myself in. If you have a licensed version of the tool, I encourage you to give it a try and experiment with how you can integrate it into your standard website health checks and audits.
Have you used Screaming Frog’s crawl comparison feature in your website audits? How did it go? Share your experiences by leaving a comment below!