There are some good bots such as Google and Bing that you may want to allow to crawl the site, while blocking all other bots.
This can be done using a robots.txt file which you place in your home directory. Below is an example of a robots.txt file that blocks all bots except for Google, MSN, Yahoo. Please do not that not all bots adhere to a robot.txt file, so it is still possible that some bots may crawl your web site in spite of this file being in place (mostly malicious bots would not follow the rules in your robots.txt file, and some use your robots.txt file to identify areas on your site that you are trying to exclude from bots)
The example below is just a basic example. Additional configuration can be added to do pattern matching and set various rules such as noindex or follow. You can find additional information online such as at http://www.robotstxt.org/
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /
User-agent: Yahoo-MMCrawler
Allow: /
User-agent: Yahoo-slurp
Allow: /
User-agent: Yahoo-slurp
Allow: /
User-agent: slurp
Allow: /
User-agent: Msnbot
Allow: /
User-agent: Msnbot
Allow: /
Content retrieved from: https://support.appliedi.net/kb/a1233/how-to-use-a-robots_txt-file-to-block-bots.aspx.