https://www.scalahosting.com/kb/how-to-use-the-robots-txt-file-to-improve-the-way-search-bots-crawl-your-website/ All useful information for hosting, billing and sales in one place - ScalaHosting Blog Tue, 15 Mar 2022 10:28:58 +0000 hourly 1 https://wordpress.org/?v=6.1.1 https://www.scalahosting.com/kb/how-to-use-the-robots-txt-file-to-improve-the-way-search-bots-crawl-your-website/#comment-2618 Tue, 15 Mar 2022 10:28:58 +0000 https://www.scalahosting.com/kb/?p=2229#comment-2618 Hello, Divya,
The line stops search engine robots from crawling a specific subfolder inside the website’s document root.
Hope this helps!

]]> https://www.scalahosting.com/kb/how-to-use-the-robots-txt-file-to-improve-the-way-search-bots-crawl-your-website/#comment-2301 Mon, 10 Jan 2022 07:47:07 +0000 https://www.scalahosting.com/kb/?p=2229#comment-2301 What is means Disallow: /go/ in robots.txt

]]>