Web Crawlers and Search Engines

Search Engines are made up of three distinct functions.

They are:

1 - Search and results pages
2 - Web Crawlers
3 - Indexing programs and data centers

We, the search engine user, see the Search page and its results.

Before we can see a result the search engine has to find the page. They do that with specialized programs called web crawlers. Web crawlers go online and try to visit every single page on the web every month. As they visit these pages they pass the information, and in fact the entire page, back to the data centers. The information is placed in an index. The copy of the page is stored on several different servers in different places around the world.

Web crawlers find pages by following links from other pages they visit. They also find pages by looking at lists created when a person submits a web page to the search engine to be added to the index.

The last time I looked into submitting pages to search engines to be indexed it could take as long as 6 months for the search engines to find you.

The way to be found the fastest is to have a link to your page added to a page you know the search engine has already indexed. Using this method your new page will be found the very next time the web crawler makes its trip through the web.

To the best of my knowledge, search engines crawl the web once every month. They do not publish any schedule. The articles I have read on this issue say that they vary the time of their crawls from month to month for reasons they see no need to explain.

You may recall I cautioned you earlier not to participate in link farms. However, if you know some one with a web site that has been indexed, asking them to post a single link to you on their site will be all you need to get found in the very next web crawl.

How do you know if a friend’s site has been indexed? The easiest way is to go to the search engine you are interested in and type in the site’s web address into the search box without the www. For example, for a web address of www.website.com, type in website.com . If the search engine returns a result for that page, then that page has been indexed.

Depending on the service you use to host your web site, the hosting site will have a variety of tools to submit your pages to be indexed. Some are better than others. I have never seen most so I can not offer many opinions.

Once you have a page that has been indexed, any page that links to or from that page will automatically be found on the next web crawl.

To get found in 15 minutes after posting your page you can always use Sponsored Links programs. I strongly recommend these and will be going into them in great detail over the next few days.