Google clarifies robots.txt best practices and explains when to use robots.txt directives and noindex tags for SEO. Don't combine robots.txt disallow with noindex tags. Use noindex when you want a ...