Well-behaved crawlers cap how many pages they fetch per host and how deep they follow links, so a site cannot trap them forever.
Read the full version