SEO Crawla

why some web pages can be accessed by a search engine spider but not a browser?

some web pages, a search engine robot or spider can access, but a normal browser is forbidden, maybe you need to register. why does this happen?
but why a web crawler can access them?
unfortunately, both of you are out of focus.
some web sites will ask you to register if you want to proceed to browse. but as a web crawler identity, registration is not needed.

Public Comments

1. They are most likely access restricted sites. In other words you would need a password.
2. A web crawler accesses all links in a web page, whether the browser displays it or not.
   
   
   
   Consider this tag:
   
   <a href="http://www.yahoo.com">Click here to go to Yahoo!</a>
   
   
   
   A web crawler sees the part of the href and it can go there, the browser displays 'Click here to go to Yahoo!' and the user can click on it. But f you take this away and not put anything between the anchor tags <a> and </a>, the web crawler will still see 'http://www.yahoo.com' but the browser won't display anything and the user can't click on anything.
   
   
   
   There is, however, a solution: instead of clicking, you can just type http://www.yahoo.com on the address bar of the browser and you'll go to that link.
   
   
   
   Hope this helps. =)