What are robot meta tags?
Robots meta directives (sometimes called “meta tags”) are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file directives give bots suggestions for how to crawl a website’s pages
There are two types of robots meta directives:
Those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag).
[1] Using the robots meta tag: The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. Place the robots meta tag in the <head> section of a given page, like this:
<!DOCTYPE html> <html><head> <meta name="robots" content="noindex" /> (…) </head> <body>(…)</body> </html>
There are following ways for Indexing & No Indexing in Search Engine like Google , Yahoo or Bing
[a] For No Indexing Web Pages: This method used in website developer mode
Syntax as –
[b] For Indexing Web Pages: This method used in website production mode,
Syntax as –
[2] Using the X-Robots-Tag HTTP header:
The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag. Here’s an example of an HTTP response with an X-Robots-Tag instructing crawlers not to index a page:
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)
Multiple X-Robots-Tag headers can be combined within the HTTP response, or you can specify a comma-separated list of directives. Here's an example of an HTTP header response which has a noarchive X-Robots-Tag combined with an unavailable_after X-Robots-Tag.
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noarchive
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PSTThe different robots meta tag values
The following values (‘parameters’) can be placed on their own, or together in the content attribute of tag (separated by a comma), to control how search engines interact with your page.
- index: Allow search engines to add the page to their index, so that it can be discovered by people searching.Note: When there are no directives relating to indexing, this is assumed to be the default.
- noindex: Disallow search engines from adding this page to their index, and therefore disallow them from showing it in their results.
- follow: Tells the search engines that it may follow links on the page, to discover other pages.
- nofollow: Tells the search engines robots not to ‘endorse’ (pass equity through) any links on the page. Note that this includes all links on the page, including, e.g., those in navigation elements, links to images or other resources, and so on.
- none: A shortcut for
noindex, nofollow.
- all: A shortcut for
index, follow.
- noimageindex: Disallow search engines from indexing images on the page.
- noarchive: Prevents the search engines from showing a cached copy of this page in their search results listings.
- nocache: Same as
noarchive, but only used by MSN/Live.
- nosnippet: Prevents the search engines from showing a text or video (as
meta description) of this page in the search results, and prevents them from showing a cached copy of this page in their search results listings. - nositelinkssearchbox: Prevents the search engine from showing an inline search box for your site.
- nopagereadaloud: Prevents the search engine from reading your page’s content aloud via voice services/results.
- notranslate: Prevents search engines from showing translations of the page in their search results.
- max-snippet:
[number]Sets a maximum number of characters for the meta description. - max-video-preview:
[number]Sets a maximum number of seconds for a video in a preview.
- max-image-preview:
[setting]Sets a maximum image size for use in a preview (none,standardorlarge). - rating: Indicates that a page contains adult material.
- unavailable_after: Tells search engines a date/time after which they should not show it in search results; a ‘timed’ version of
noindex. - noyaca: Prevents the search results snippet from using the page description from the Yandex Directory.
- noydir: Blocks Yahoo from using the description for this page in the Yahoo directory as the snippet for your page in the search results.