626fb73a58
- Add CrawlerConfig with ProxyConfig, delay, and retry settings - Support HTTP_PROXY/HTTPS_PROXY environment variable parsing - Add proxy_required validation with clear error message - Add get_crawler_config singleton for configuration access Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>