Lab notes

Short notes on the web, crawlers and video.

• Making video shoppable
• What robots.txt promises
• Serverless and slow clients
• Caching at the edge
• Crawl budgets
• Reading user agents
• Who owns an IP address
• Polite crawling
• Sitemaps
• Measuring compliance