SEO Crawla

Can i use Curl to impersonate a search engine robot/spider to access a web site?

if not, what utility can i use to do this?

Public Comments

1. Yes. curl will do it so will wget.