The robots meta tag is used to control search engine crawlers' inclusion and link tracking of web pages. It tells search engines whether to allow inclusion of pages (index/noindex) and whether to allow tracking of links in pages (follow/nofollow) by adding directives in the HTML page (head> area). Common usage scenarios include: 1. Use noindex when you do not want the page to be included; 2. Use nofollow when you prevent crawlers from tracking specific links; 3. Multiple instructions can be combined to achieve finer control. Unlike robots.txt, robots.txt is used to restrict crawlers from accessing website directories, while robots meta tags are page-level controls, and the two work better together. Notes include: non-compliant crawlers may ignore settings, Google may still display page information due to external links, and HTTP versions need to be configured separately.

Search engine crawlers are one of the key factors in whether website content can be discovered. You may have written the content of the web page, but if you do not set up the robots meta tag for HTML, search engines may ignore you and even include content that should not be included. How do you want to control the crawler to crawl your page? The key lies in this meta tag.

What is robots meta tag?

robots meta tag is a directive placed in the area of the HTML page, telling search engine crawlers how to deal with the current page. It's not mandatory, but most mainstream search engines respect this setting.

The basic writing method is as follows:

 <meta name="robots" content="noindex, nofollow">

• name="robots" means that this is the instruction for all crawlers to see.
• The value in content determines the specific behavior.

Common content values include:

• index : Allows to include pages
• noindex : Do not include pages
• follow : Track the links on the page
• nofollow : Don't track links

Common usage scenarios and suggestions

Don't want the page to be included? Use noindex

If you have some pages that you don't want to appear in search results, such as "Thank you page", "Internal test page", or "Repeat content page", you can add:

 <meta name="robots" content="noindex">

In this way, search engines will not add these pages to the index and users will not be able to search.

Tip: If you just want to prevent Google from inclusion, you can also submit a "removal" request through Google Search Console, but using meta tags is a more general approach.

Don't want the crawler to track the links on the page? Add nofollow

Sometimes you have some links on your page, such as user links, advertising links in the comments section, or links that you don't want to pass weights, you can use:

 <meta name="robots" content="nofollow">

This way the search engines will not continue to crawl along these links.

For example: You have a forum page, and many of the links posted by users are spam links. Adding nofollow can prevent search engines from taking these links as recommendations.

Multiple instructions can be used in combination

You can use multiple behaviors at the same time, such as:

 <meta name="robots" content="noindex, nofollow">

It means: Don't include this page, and don't track the links inside.

It can also be the other way around:

 <meta name="robots" content="index, follow">

This is the default behavior for most pages, but it is sometimes safer to write it explicitly.

What's the difference with robots.txt?

robots.txt is a file in the root directory of the website, used to control whether the crawler can access certain directories or files. It is more like a "gate guard" telling the crawlers where they cannot enter.

robots meta tag is a page-level control, which is more like a "tip sign in the room", telling the crawler whether the contents of the room can be included and whether they can continue to explore.

The two work better together. For example, you use robots.txt to prohibit crawlers from accessing the /admin/ directory, and at the same time use noindex in some pages to prevent them from being included.

Things to note and places to ignore

• Not all crawlers follow : Some unruly crawlers may ignore your settings, so you can't expect it to protect sensitive information.
• Google has its own judgment : Even if you add noindex , Google may still display your page title and URL for some reason (such as too many external links).
• The HTTPS page and the HTTP page should be set separately : If you have two versions, HTTPS and HTTP, their meta tags should be configured separately, otherwise there may be problems.

Basically that's it. Setting robots meta tags is not complicated, but it is easy to ignore details. If used well, it can help you control more carefully how search engines crawl your content.

The above is the detailed content of HTML `robot` Meta Tags: Controlling Search Engine Crawlers. For more information, please follow other related articles on the PHP Chinese website!