Lucas Ng's blog

Currently, crawlers can’t fill out registration forms, nor do they support cookies. Given that, we need to be able to circumvent your registration page in order to successfully crawl your site. The easiest way to do this is to configure your webservers to