Making Decision When to Block in Robots.txt, 301, 404 or Ignore Errors and Warnings

Countless times I have been asked by a Boss, CEO or a Client why we see so many errors and warnings in Google Search Console, Moz, Conductor, Brightedge or Botify Reports; and what should we do about them. Often times the solution to the issue could be more damaging than the issue itself.

When Do We Address Errors and Warnings, and How Best to Deal with Them?

It's not an answer that is very straightforward, really. As any SEO will respond with "It Depends". It depends on so many factors going into the issue from cause, amount pages, impact on traffic and revenue those pages have, and so on.

I'll walk you through a decision making process that might help you make the best decision on what to do in these situations.

What are Errors/Warnings?

First, take a look at the issue that is being brought up. Are they Warnings or Errors? 

If you are seeing Errors, then there is a strong chance the issue should receive a priority, but don't go running up and down the hallways of the office screaming "Fire" just yet. We need to look as what exactly is going on before sounding the alarms.

Using one of my own sites as an example, Google Search Console sent me an email that there is an increase in Crawl Errors. I logged in, clicked on "Coverage" to find that the errors had a substantial increase. At some point every SEO has been through this same scenario. The key is to dig through and understand exactly what the error is, and start to identify what the cause is.

The few most common errors you will find in Google Search Console are:

• Server error (5xx)
• Submitted URL not found (404)
• Submitted URL blocked by robots.txt
• Submitted URL has crawl issue
• Submitted URL seems to be a Soft 404
• Submitted URL marked ‘noindex’

Iit's good to familiarize yourself with your GSC Errors before any such notifications are sent. URLs marked "noindex", blocked by robots.txt and not founf (404) are usually known issues that you can ignore. If you are going through your errors for the first time, it's good to spot check directories or sets of pages generated dynamically to understand what might be causing the errors. If they are something to be alarmed by, then it's a good time to discuss this with engineering, or if you develop yourself, to understand what the cause is and come up a best solution to fix.

Server error (5xx), seems to be a Soft 404, and has a crawl issue are all errors that should be addressed immediately. However, you should fully understand how many pages are causing the errors, what directory those pages are in, and what impact those pages are having before bringing them up with engineering, your boss, CEO or client.

Understanding the Impact

Let's say we have a set of pages in a directory that are causing 500 or soft 404 errors. In this case we see one real estate listing and a set of pages under the "events" directory. This is telling us that the system building out those events is having connectivity issues (server errors are usually a connection from web to database). 

When looking at the server logs for these pages, there is an issue with a javascript call on those pages that is causing the 500 error. An easy fix, but what would be the impact for the company?

Logging into Google Analytics and pulling up all pages that include "events" the session percentage of the total for the event pages in a 3 month span is 18% and last year for the same 3 months was 23% with a avg revenue value and conversion rate higher than any other set of pages for the site, and in general. These numbers show that the impact on the server error is potentially effecting the percentage of traffic negatively and the value of those pages is too great to let the error go. If the value of the pages were lower, and the YoY numbers were the same, then a lower priority on the fix could be set. 

Soft 404s could have the same impact as a server error, with a different cause and more involved fix. Soft 404s are usually caused from pages that throw a 200 (page is ok) code, but the page has little to no content on it. Google translates this a page that is in error and gives it a soft 404, rather than saying is doesn't exist.

Understanding server codes and reading your Google Analytics is extremely important in these cases. Furthermore, understanding web log data and being able to identify issues is just as equally important in determining an issue, level of effort to resolve, and impact to the business both negatively and positively (once resolved). 

With this knowledge, you can go into any meeting with your engineering team, your boss, CEO or client and articulate what it is you're seeing, demonstrate your knowledge of the issue, impact on the business and level of effort it will take to resolve.