Spider

A specialized bot or script used by search engines to crawl the web (hence the term spider) analyzing and indexing web pages and other files.

Well known spiders include Googlebot, BingBot and Facebook External Hit.