starmen1265
123e307522
- Proxy configuration from HTTP_PROXY/HTTPS_PROXY - Random delay (1-3s) between requests - Exponential backoff retry (3 attempts) - x-zse-96 signature handled by browser All 44 crawler tests passing Requirements: NET-01, NET-02, NET-03 completed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
89 lines
3.0 KiB
Markdown
89 lines
3.0 KiB
Markdown
---
|
||
phase: 08-crawler
|
||
plan: 03
|
||
subsystem: crawler
|
||
tags: [proxy, rate-limiter, retry, anti-crawl, playwright]
|
||
|
||
requires:
|
||
- phase: 08-01
|
||
provides: ZhihuCrawler base class
|
||
provides:
|
||
- CrawlerConfig for proxy/rate limiting configuration
|
||
- Random delay (1-3s) between requests
|
||
- Exponential backoff retry (3 attempts)
|
||
- Automatic x-zse-96 signature handling via browser
|
||
|
||
tech-stack:
|
||
added:
|
||
- tenacity (retry library)
|
||
patterns:
|
||
- Environment variable proxy configuration
|
||
- Exponential backoff retry pattern
|
||
- Random delay for rate limiting
|
||
|
||
key-files:
|
||
created:
|
||
- kb_workflow/crawler/config.py
|
||
- tests/test_crawler/test_proxy_config.py
|
||
- tests/test_crawler/test_rate_limiter.py
|
||
- tests/test_crawler/test_api_intercept.py
|
||
modified:
|
||
- kb_workflow/crawler/zhihu_crawler.py
|
||
- kb_workflow/crawler/models.py
|
||
- tests/test_crawler/test_comment_crawler.py
|
||
key-decisions:
|
||
- Mandatory proxy configuration (configurable via proxy_required flag)
|
||
- Use tenacity library for retry with exponential backoff
|
||
- x-zse-96 signature handled automatically by browser via route.fetch()
|
||
- Random delay range: 1-3 seconds (configurable)
|
||
requirements-completed: [NET-01, NET-02, NET-03]
|
||
duration: 15min
|
||
completed: 2026-03-17
|
||
---
|
||
|
||
## Summary
|
||
|
||
实现了网络反爬措施：强制代理配置、请求频率控制、有限重试机制，以及利用浏览器自动处理 x-zse-96 签名。
|
||
|
||
### What Was Built
|
||
|
||
**CrawlerConfig 类** (`kb_workflow/crawler/config.py`)
|
||
- `ProxyConfig` - 代理配置模型
|
||
- 从环境变量 `HTTP_PROXY`/`HTTPS_PROXY` 读取代理设置
|
||
- `proxy_required` 标志控制是否强制代理（默认 `True`）
|
||
- 随机延迟配置（`min_delay`, `max_delay`）
|
||
- 重试配置（`max_retries`, `retry_min_wait`, `retry_max_wait`）
|
||
|
||
**ZhihuCrawler 增强**
|
||
- `_start_browser_with_proxy()` - 启动浏览器并应用代理配置
|
||
- `_apply_random_delay()` - 应用随机延迟（1-3 秒）
|
||
- `_crawl_with_retry()` - 带指数退避的重试逻辑
|
||
- 浏览器自动处理 x-zse-96 签名（通过 `route.fetch()`）
|
||
|
||
### Test Coverage
|
||
|
||
**44 个测试全部通过**
|
||
- `test_proxy_config.py` - 11 个代理配置测试
|
||
- `test_rate_limiter.py` - 8 个频率控制测试
|
||
- `test_api_intercept.py` - 4 个 API 拦截测试
|
||
- 其他测试文件保持通过
|
||
|
||
### Commits
|
||
|
||
1. `626fb73` feat(08-03): create CrawlerConfig for proxy and rate limiting
|
||
2. `9e1452d` feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler
|
||
|
||
### Key Decisions
|
||
|
||
1. **强制代理**: 默认 `proxy_required=True`，未配置代理时抛出 `ValueError`
|
||
2. **重试机制**: 使用 `tenacity` 库，最多 3 次重试，间隔指数递增（2s → 10s）
|
||
3. **签名处理**: 利用浏览器 `route.fetch()` 自动携带 x-zse-96 签名，无需手动逆向
|
||
|
||
### Requirements Completed
|
||
|
||
- **NET-01**: 用户可以通过环境变量配置代理 IP ✓
|
||
- **NET-02**: 不配置代理时，爬虫拒绝运行并提示错误 ✓
|
||
- **NET-03**: 每次请求间有随机延迟（1-3秒），失败重试 3 次 ✓
|
||
|
||
## Self-Check: PASSED
|