From the title you're probably thinking, "What does this have to do with internet marketing?" Actually, it has everything to do with internet marketing. And it has to do with search engines or you may know it as web search engines. Still don't know? Of course you have heard of Google, Yahoo and others that bring your website to the notice of the prospective customers. They present information to your potential customers initiating a search.

So, what are these robots, spiders and crawlers? Robots or bots are software applications that run automated tasks over the Internet. Our friends at Wikipedia state, "...bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone." These robots are also being used as an access to content for media delivery.

Spiders are automated programs run by the search engine systems and are used to index websites. Basically it works like this, when you submit your website to a search engine a spider or automated program will index your entire site. It does so by reading the content or your site, your site's meta tags and will follow the links that the site connects. The spider then returns all that information back to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well. Some spiders will only index a certain number of pages on your site, so don't create a site with 500 pages!

The spider will periodically return to your site to check for any information that has changed. The frequency with which this happens is determined by the moderators of the search engine.

A spider is almost like a book where it contains the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may index up to a million pages a day.

Crawlers or web crawlers are similar to spiders but as explained in Wikipedia, "...are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a website, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from web pages, such as harvesting e-mail addresses (usually for spam)."

What Does All of This Mean?

When you ask a search engine to locate information, it will actually search through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.

One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing (this is where you repeat your keywords or phrases to manipulate the relevancy or prominence of your resources.) Then the algorithms analyze the way that pages link to other pages in the Web.

A search engine can determine what a page is about and whether the keywords or phrases of the linked pages are the same as the keywords on the original page. How do they accomplish this? By checking how pages link to each other. It is basically a check and balance system.

With this knowledge you now understand the meaning of "search engine optimization." You will be better equipped in your content writing with no need to fear...robots, spiders and crawlers.