What is the X-Robots-Tag?

Search engines employ the X-Robots-Tag HTTP header to indicate to bots how to crawl and index the content of a webpage.

Unlike meta tags placed within the HTML of a page, the X-Robots-Tag is part of the HTTP response header provided by the web server. This tag is particularly useful for advanced directives and when controlling the indexing of non-HTML files like PDFs or images.

It instructs web crawlers, such as Googlebot, on whether to follow links, index a page, or archive content.

Importance of X-Robots-Tag

The X-Robots-Tag is important for several reasons:

  • Advanced Control: It provides more granular control over how search engines interact with your content, especially for non-HTML files.
  • Non-HTML File Management: Allows you to manage the indexing and crawling of files like PDFs, images, and other resources that can’t have meta tags.
  • Supplement to Meta Tags: Works alongside meta tags to offer comprehensive control over the crawling and indexing behavior of your entire site.

Which Directives are Compatible with the X-Robots-Tag?

The X-Robots-Tag supports several directives commonly used in meta tags:

  • noindex: Prevents the page or file from being indexed by search engines.
  • nofollow: Prevents search engines from following the links on the page.
  • noarchive: Prevents the cached copy of the page from being stored.
  • nosnippet: Stops search engines from showing a snippet of the page in search results.
  • noodp/noydir: Prevents the use of Open Directory Project (DMOZ) descriptions in search results.

How to Set Up the X-Robots-Tag?

Setting up the X-Robots-Tag involves modifying your server’s HTTP headers. Here’s how you can do it:

1. Apache

Insert the following lines into your .htaccess file:

<FilesMatch “\.(pdf|doc|jpg)$”>Header set X-Robots-Tag “noindex, nofollow”

</FilesMatch>

2. Nginx

Insert the following configuration into your server block:

location ~* \.(pdf|doc|jpg)$ {add_header X-Robots-Tag “noindex, nofollow”;}

Where to Find the X-Robots-Tag?

You can verify the presence of the X-Robots-Tag by examining your website’s HTTP headers. Tools like cURL or browser developer tools can help you inspect the headers. For example, using cURL:

curl -I https://example.com/file.pdf

This command will display the HTTP headers, allowing you to check for the X-Robots-Tag.