Google Clarifies The Index, Follow Meta Tag

In response to questions about the meta robots tag featuring an index, follow directive, Google’s John Mueller sheds light on how Google actually deals with this aspect.

 

Understanding the Robots Meta Tag

 

Let’s break down the technical jargon. The HTML meta element serves as a communicator of information that machines, like Googlebot, can understand. Among various meta elements, the Robots Meta Element stands out because it guides search engine crawlers. The information communicated by the robots meta tag is a set of instructions telling robot crawlers what to do.

One commonly used instruction is the “noindex, nofollow” meta tag:

<meta name=”robots” content=”noindex, nofollow”>

This essentially tells search engine crawlers not to include the webpage’s content in search results and not to follow any links. On the other side, the widely used “index, follow” meta tag instructs search engines to include the content in search results and to follow all links.

 

Effect of Leaving Out the Index Tag

 

Imagine this as a missing piece of information puzzle. A Reddit user was puzzled about a missing ‘index’ tag in meta snippets. Mueller’s response was enlightening – both the “index” and “follow” instructions are ignored by Google. It’s not as critical as it may seem, and it doesn’t function the way many think it does.

 

Why Google Ignores Robots Index & Follow

 

Google ignores the “index, follow” meta tag because indexing content and following links are already default behaviors for search engine robots. Google’s guidelines emphasize that these default values, “index, follow,” don’t need explicit mention. If a particular directive isn’t explicitly listed in Google’s valid directives, Googlebot will simply ignore it.

 

Is Index, Follow Completely Ineffective?

 

Now, let’s address a common misconception. For Googlebot, using <meta name=”robots” content=”index, follow”> is like telling a chef you want salt when they already add salt by default. It’s redundant and gets overlooked. Bing, another search engine, has a similar stance with some nuanced differences.

In summary, while the “index, follow” directive might not be as impactful as perceived, it’s crucial to communicate clearly in SEO practices. Clarity helps dispel misconceptions about the functionality of certain meta tags.

 

Quotes from John Mueller:

 

• The “index” robots meta tag has no function (at least in Google) – it’s completely ignored. Also “follow”.
• Google has documentation where listed the meta tags that have functions. You can use anything else; it’ll be ignored.
• By default, we assume “index”, but if needed you can use <meta name=”robots” content=”index”> to explicitly state that we may index the page.
• By default, we assume “follow”, but you can explicitly state “follow” if so desired.