SEO Crawla

please how can I get the information about spider web and robot ?. I would like to build search engine .?

I am searched about sites which contain the technology of search engine and web spider but I didn't got it ? please what is the method to build search engine by spider and robot use php language ? if you can provide me about all necessary information to do that and I shll be thanks .

Public Comments

1. Use search engine capabilities to find specific info.
   
   robotstxt.org