Polite crawling

A steady, low request rate per host is the usual sign of a crawler that is trying not to cause harm.

Read the full version