To avoid undesirable articles inside the look for indexes, site owners can instruct spiders never to crawl particular files or directories throughout the conventional robots.txt file in the root directory with the domain. In addition, a site can be explicitly excluded from a internet search engine&#