There are 2 main types of search engines: directories which are compiled by human editors and crawler-based search engines which send out robots/spyders to crawl from link to link through the web.
Directories
Directories, like the Open Directory Project (ODP) and Yahoo! Directory depend on humans to submit and approve listings. You submit your website and a brief description, and an editor approves it.
Crawlers, Spiders, and Bots
Crawler-based search engines find your site through crawlers (aka spiders or bots) that forever crawl from one link to another. Some of the major ones are Googlebot, MSNbot, and Inktomi Slurp. These crawlers are automated programs that seek out new and newly updated pages to add to their index. It also returns to each of those pages to re-spider them every so often. The search engine will then update its directory every couple of weeks to include the new and newly updated pages in its searchable content.
How Do Search Engines Determine Who’s On Top?
Say you go to Google in hopes of finding an attorney who practices trademark law in Pennsylvania. If you were to try a generic query like “Attorney,” you’d get about 30.7 million pages that include the term. Of course, most aren’t relevant or helpful to your specific need, so you might try again with “Pennsylvania Attorney” or “Trademark Attorney” until you found a site that meets your needs.
With a generic, one word query, the search engine doesn’t understand what you’re looking for, so it returns a large quantity of results. When you use several keywords in a search, the search engine can exclude results that it determines probably aren’t what you are looking for to return a smaller number of more relevant sites.
How does this happen? Search engines use algorithms to determine how they rank a page. Those algorithms take into consideration things like the location and frequency of keywords on a web page. If the search keyword appears near the top of the page, such as in the page title or first paragraph of text, search engines assume it is relevant to the page. How frequent the keyword appears in the content is also important.
Search engines also use external factors to determine a page’s relevance. One major consideration is how many pages link to your website. The search engine can then analyze what your page is about and whether other similar sites believe it to be important. Search engines can also measure click-throughs, so if a site shows up in search results, but no one ever clicks on it, it will eventually drop the site’s rank.
Join our weekly Marketing Success mailing list and receive our free 43-page Small Business Marketing Plan guide. 
