Optimizing Drupal search results using robots.txt

You may have a website packed with great information although if search engine spiders can't crawl your site, your results may show up lower in the results listings, or not at all. A properly configured robots.txt is one key that can help make sure your site is properly crawled and help boost your search engine ranking. Some module configurations can cause the crawling of improper results, duplication, or endless looping.