Anatomy of a Search Engine Crawler
(Search Engine Journal)
“When you go to a search engine and perform a search many people don’t understand how those results end up there. Some people may think that sites are submitted while others know that a piece of software finds the pages.
This article explains one piece of that puzzle: The search engine crawler.”