Web Crawling
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, or Web spiders, Web robots, or—especially in the FOAF community—Web scutters.
This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).
Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
Related Publications
Conferences and Workshops
Odysseas Papapetrou and George Samaras (2004) "Minimizing the Network Distance in Distributed Web Crawling", Proc. of the 9th IFCIS International Conference on Cooperative Information Systems (CoopIS 2004), pp. 581- 596, Ayia Napa, Cyprus, October 2004
Odysseas Papapetrou and George Samaras (2004) "IPMicra: Toward a distributed and adaptable location aware web crawler", Local Proc. 8th East-European Conference on Advances in Databases and Information Systems (ADBIS 2004), pp. 147-159, 22-25 September, 2004, Budapest, Hungary
Odysseas Papapetrou and George Samaras (2004) "IPMicra: An IP-address Based Location Aware Distributed Web Crawler", The 2004 International Conference on Internet Computing (IC'04), pp. 694-699, June 21-24, 2004, Las Vegas, Nevada, USA
Odysseas Papapetrou, George Samaras (2004) "Distributed location aware web crawling", (Alternate Track Papers & Posters), 13th International World Wide Web Conference (WWW 2004), pp. 468-469, 19-22 May 2004, New York, USA
Odysseas Papapetrou, Stavros Papastavrou, George Samaras (2003) "UCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers", Proc. 7th East-European Conference on Advances in Databases and Information Systems (ADBIS 2003), pp. 133-147, September 3-6, 2003, Dresden, Germany
Odysseas Papapetrou, Stavros Papastavrou, George Samaras (2003) "Distributed Indexing of the Web Using Migrating Crawlers", Proc. of the Twelfth International World Wide Web Conference (W3 2004), 20-24 May 2003, Budapest, HUNGARY
Contact phone: +375 22 89 26 98