.Google's John Mueller answered a concern concerning why Google.com indexes web pages that are actually prohibited coming from crawling through robots.txt and also why the it is actually risk-free to neglect the associated Search Console documents concerning those crawls.Bot Visitor Traffic To Query Guideline URLs.The person asking the inquiry recorded that crawlers were actually creating web links to non-existent inquiry specification URLs (? q= xyz) to webpages with noindex meta tags that are actually likewise shut out in robots.txt. What triggered the inquiry is actually that Google.com is actually creeping the links to those web pages, acquiring blocked out by robots.txt (without seeing a noindex robotics meta tag) then receiving reported in Google Explore Console as "Indexed, though blocked through robots.txt.".The individual asked the complying with concern:." Yet listed below's the big inquiry: why would certainly Google mark pages when they can not also see the information? What's the advantage in that?".Google.com's John Mueller affirmed that if they can not creep the web page they can not observe the noindex meta tag. He additionally creates an interesting reference of the site: hunt operator, urging to dismiss the results given that the "normal" individuals won't find those outcomes.He created:." Yes, you are actually correct: if our company can not crawl the web page, we can not observe the noindex. That mentioned, if our experts can't crawl the web pages, at that point there is actually not a lot for our team to index. Therefore while you could see a few of those webpages with a targeted internet site:- inquiry, the typical individual will not observe all of them, so I would not bother it. Noindex is likewise great (without robots.txt disallow), it simply indicates the Links are going to end up being actually crawled (and wind up in the Browse Console document for crawled/not catalogued-- neither of these statuses lead to issues to the remainder of the site). The integral part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the limits in operation the Web site: hunt progressed search driver for diagnostic causes. One of those explanations is actually due to the fact that it is actually not connected to the normal search index, it is actually a separate point altogether.Google's John Mueller commented on the internet site search driver in 2021:." The short answer is actually that a website: concern is certainly not suggested to become complete, neither used for diagnostics objectives.A website question is actually a specific kind of search that limits the results to a certain website. It's primarily just words website, a bowel, and then the site's domain.This concern restricts the end results to a particular site. It is actually not meant to become a comprehensive compilation of all the pages from that website.".2. Noindex tag without using a robots.txt is actually great for these kinds of conditions where a crawler is connecting to non-existent pages that are actually receiving discovered by Googlebot.3. URLs along with the noindex tag will create a "crawled/not indexed" item in Explore Console and that those will not have a bad result on the rest of the site.Go through the question and also respond to on LinkedIn:.Why would certainly Google mark webpages when they can't even see the information?Featured Graphic through Shutterstock/Krakenimages. com.