SEO Crawla

Home > Web Crawler Information > Web Crawler Questions > What and where does a web crawler look on a web page? Example: Yahoo images, how do these photos get found?

What and where does a web crawler look on a web page? Example: Yahoo images, how do these photos get found?

When you do a search, the results are diplayed. There is a bold heading and text under. Where does a web crawler find this info.Where would I put this info on my webpage? Same with photos?

Public Comments

  1. A robot or web crawler will look at everything unless you DON'T tell it to look at something. So if you are not using a meta tag in the <head></head> section like:



    <meta name="robots" content="noindex,nofollow">



    or you don't have a robots.txt sitting in your root directory with instructions not to follow certain directories, then web crawlers should be indexing the stuff.
  2. They will look at everything on your website unless you have created an index telling it what or what not to look at.
Powered by Yahoo! Answers