I think there is nothing worse than a 404 error. For example, when you're looking for some content on the internet, you find an interesting site (or worse – a product) and bah: “error 404 page not found”. You must know that feeling. Frustrating, isn't it? And yet so common. The topic of 404 errors and how to handle them is popular in technical SEO. However, I believe that this topic is so important that it’s worth devoting a few words to it on our blog.
We don't like errors, especially 404
Error 404 (HTTP 404, 404 not found, 404, 404 error, page not found or file not found) is a wrong response code. Refers to a subpage that wasn’t found during the last visit of the crawler. It informs us that the user accessing the given subpage refers to a location or file that doesn’t exist. The "page not found" error is often confused with the "server not found" error.
So, in summary, error code 404 happens when the client (e.g. a web browser) is able to communicate with the server, but the server cannot find the requested file, or it's configured to not fulfill the request and not display why.
Common causes of 404 errors
If you notice 404 errors on your site, the most common causes are:
- Change in the structure of website URLs – it may happen due to the start of using a different CMS system. Then you should use the 301 redirect to redirect the old URLs to the new ones. If you don't do this, users may end up on subpages that don't exist.
- Modifying the URL, e.g. due to the change of the product name – in this case, you should also create a 301 redirect so that the user immediately finds a new subpage, and not a 404 error.
- Removal of an article, product page or any other subpage on the site. Remember that even if you delete a resource, it may still remain in the Google index for a while, and users will end up on non-existent pages until Google removes them from the index. However, it happens that users save the addresses of the subpages they are interested in and find them directly, not from the search engine.
- Changes to the .htaccess file.
- Uncleaned cookies and cache files – this may result in displaying an outdated version of the page that returns an error. This could be the case if the page is not loading properly on only one device.
- It happens that the 404 error is the result of a mistake of the user who made a typo in the website address. This often happens when trying to access a website by entering an address from memory.
Whether it's a user's error or yours – by not setting a redirect, the effect will be the same – the user won't see the resources they expected.
Hunting for errors. How to find a 404 error on your site?
There are some good tools out there to help you find 404 errors on your site. I personally use three of them on a daily basis:
- Google Search Console is a completely free treasury of knowledge about your site and indexation. With this tool, you can easily check the health of your website, not only for 404 errors.
source: Google Search Console
- Screaming Frog, undoubtedly one of the favorite tools of SEO specialists. The free version allows you to crawl up to 500 subpages of your website. After running the scan, you can see the server's response code in the "Status code" column and thus find pages with 404 error. Another solution that can help is Sitebulb, but it is a fully paid option. However, you can check its possibilities thanks to the 14 days of free use.
source: Screaming Frog
- Ahrefs – in my opinion it is a very simple and helpful tool to analyze pages in terms of SEO, especially in the case of link building, but I’ll write more about it someday. :) Ahrefs allows you to schedule regular audits of the website for various bugs, including 404 errors.
Soft Error 404 – sounds harmless, but means trouble too
While analyzing your website, you may come across a bug which is described in Google Search Console as “Submitted URL seems to be a Soft 404”. What does it mean? The subpage doesn’t exist, but the browser still receives a status code of 200. It also happens that the user is shown a page with little or almost empty content. Such a result will be indexed in the search results, which will mean that the Google robot, instead of indexing existing pages we want to, will return to a non-existent subpage and try to index it again and again. This means that we lose a valuable crawl budget on subpages that neither the user nor the robot should have access to.
How to deal with these errors?
Well, it depends (yeah, I know it’s not funny anymore, but I can’t hold myself back). ;) In general, Google's recommendations are clear: if we don’t care about a given subpage in terms of position in search results, we can leave it as is. Single pages with 404 errors are found and removed from the Google index automatically.
However, if we don’t want to lose the link juice of incoming links, we need to set a permanent redirect 301. It’s only worth remembering that it should lead to the subpage that was thematically closest to the deleted one, consistent with the user's intention.
You can also properly configure the .htaccess file, automatically directing traffic from the error page to the home page of our website. But in my opinion, this is not the optimal solution.
What if a 404 error can become your friend?
As you know, there is no such thing as one right answer to a problem in SEO. And sometimes it turns out that what is apparently a mistake may turn out to be your asset. This also happens to sites with 404 errors that creative web masters have decided to use for the benefit of the website. How? After all, you can design a 404 page with great graphics that draw users' attention. You can also direct users to different links on the page that they can go to, or place an internal search engine that allows users to type in the phrase they are looking for.
Don't forget about a sense of humor. And it may turn out that with the creative use of 404 pages you’ll improve user experience and increase your conversion rate. ;) Just look at these examples of clever 404 pages.