.Google.com's John Mueller addressed a concern concerning why Google marks web pages that are disallowed coming from crawling by robots.txt and why the it is actually secure to overlook the related Explore Console records regarding those crawls.Bot Visitor Traffic To Concern Parameter URLs.The individual asking the inquiry documented that crawlers were actually creating links to non-existent concern criterion URLs (? q= xyz) to pages with noindex meta tags that are actually additionally blocked out in robots.txt. What caused the question is that Google.com is crawling the web links to those web pages, receiving blocked out through robots.txt (without envisioning a noindex robots meta tag) after that obtaining shown up in Google Look Console as "Indexed, though obstructed through robots.txt.".The person talked to the observing inquiry:." Yet below is actually the significant question: why would Google.com index web pages when they can not even observe the material? What is actually the perk in that?".Google's John Mueller affirmed that if they can't creep the page they can't observe the noindex meta tag. He also helps make a fascinating reference of the internet site: search driver, advising to dismiss the results because the "normal" individuals won't observe those results.He composed:." Yes, you're correct: if our company can't crawl the web page, we can't observe the noindex. That pointed out, if we can't crawl the webpages, after that there's certainly not a great deal for our team to mark. Therefore while you could observe a few of those pages along with a targeted website:- concern, the common user will not view all of them, so I would not bother it. Noindex is actually also fine (without robots.txt disallow), it merely means the Links will certainly wind up being actually crawled (and also end up in the Search Console record for crawled/not recorded-- neither of these conditions create concerns to the rest of the internet site). The important part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the limits being used the Site: search progressed hunt driver for analysis explanations. One of those main reasons is considering that it is actually certainly not attached to the frequent search index, it is actually a separate point altogether.Google.com's John Mueller discussed the site search operator in 2021:." The short answer is actually that a site: concern is actually not suggested to be full, nor used for diagnostics reasons.A website question is a certain type of search that restricts the end results to a specific internet site. It's generally merely the word website, a bowel, and then the web site's domain name.This query restricts the outcomes to a particular website. It's certainly not suggested to be a thorough collection of all the webpages coming from that internet site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of conditions where a crawler is actually connecting to non-existent pages that are receiving discovered by Googlebot.3. Links along with the noindex tag will definitely create a "crawled/not listed" item in Search Console and also those won't possess a bad result on the rest of the site.Read the inquiry and answer on LinkedIn:.Why will Google mark web pages when they can't also observe the information?Included Graphic by Shutterstock/Krakenimages. com.