How Search Engines Find Your Website

by Krista on March 12, 2007 in Search Engine Marketing

The major search engines find your site through crawlers, spiders, and bots (as in “robots”). Crawlers, spiders, and bots (as in “robots”) are automated programs that seek out new and newly updated pages to add to their database of content. Some of the major ones are Googlebot (Google), MSNbot (MSN), and Inktomi Slurp (Yahoo).

These crawlers visit a website and collect certain information from that website. From that site, they can organize your site based on other websites with similar content.

They also return to each of those pages to re-spider them every so often. The search engine will then update its directory every couple of weeks to include the new and newly updated pages in its searchable content.

How does a crawler know what sites to visit? There are a few primary ways to let crawlers know that you have a new site or that you’ve recently updated your site.

Get Inbound Links
Search engines primarily find other sites by following links from other pages, so the best way to ensure you are included is to have lots of links from sites that are already indexed.

Use Free Site Submit Tools
Google, Yahoo, and MSN all have free submission tools.

  1. Yahoo: https://siteexplorer.search.yahoo.com/submit
  2. Google: http://www.google.com/addurl/
  3. MSN: http://submitit.bcentral.com/msnsubmit.htm
  4. MSN Live search: http://search.msn.com.sg/docs/submit.aspx

Submit an RSS feed
RSS Feeds are a special type of file designed for content publishers. Their primary purpose is to keep track of the most recently added content to the site and let others (people, computers) know that the site has been updated. Many web developers include RSS feeds within their web designs and virtually all blogging platforms include RSS feeds. Whenever you publish something new, you can submit your feed to sites like http://www.pingoat.com/ or http://pingomatic.com/.

Submit a SiteMap
Google allows website owners to submit a list of all the pages of their site through their SiteMaps protocol. Like RSS Feeds, Google’s SiteMaps are a special type of file that targets search engines and notifies them of recently updated pages. You can download a tool from Google to create your own SiteMap in the specified format at http://www.google.com/webmasters/sitemaps/

Written by: Krista

Random Posts

    No comments yet.

    Leave a Reply