You want to avoid using your robots.txt file to exclude individual pages from search results.
Why? Because, with that exclusion method, search engines don’t count or share the value of a page when you might want them to.
Instead, we recommend using tools in the Yoast SEO meta box to exclude individual pages from search results.
The Yoast SEO meta box allows you to add a noindex—i.e., please do not index—robots meta directive to a page, thereby preventing that page from appearing in search results.
Here is what a robots meta directive looks like in a page’s source code.
Robots meta directives, a.k.a. meta tags, are small snippets of code on a page that tell search engine bots how to crawl and index the content on that page. When you use robots meta directives at the page level, search engines can still count and share the value of a page, which is why we use them instead of robots.txt file directives.
Follow the steps below to create a noindex robots meta directive using Yoast.
Exclude a Page or Post from Search Results Using the Yoast SEO Meta Box
1. To add a noindex robots meta directive, navigate to the Yoast SEO meta box on the page or post you want to exclude. Click on the cog icon to bring up the Advanced settings interface.
2. Set “Allow search engines to show this Page in search results?” to “No”. This will exclude the page from a search engine’s index by adding a noindex robots meta directive to it. Leave “Should search engines follow links on this Page?” right below the dropdown set to “Yes”. This will allow search engines to rank the page and spread its value.
3. Once you have set “Allow search engines to show this Page in search results?” to “No”, click Update in the top right corner of the page. Then, check to make sure the settings went into effect. Navigate to the front end view of the page or post. Right click and select “View Page Source”. This will pull up the source code for that page or post.
4. Don’t get intimidated if you don’t read code! Use command+F if you are on a mac and Control+F if you are on a PC. This will pull up the find function in the top right corner of your browser. Type in “noindex” and you should be able to see exactly where your noindex robots meta directive is in the source code.
If your robots meta directive reads <meta name=”robots” content =”noindex,follow”/> you are good to go.
If you can’t find this snippet of code, go back and make sure you properly configured and saved the settings discussed above in the Yoast SEO meta box on the page or post.
Once your noindex directive is in place, you don’t have to worry about your page or post appearing in search results!
Erik loves breaking down complex SEO topics into understandable instructions anyone can follow. In his role as an SEO coach, he guides Pathfinder SEO customers through the SEO process on a daily basis, giving them helpful tips, instruction, and advice along the way.