docs(08-03): complete anti-crawl measures plan
- Proxy configuration from HTTP_PROXY/HTTPS_PROXY - Random delay (1-3s) between requests - Exponential backoff retry (3 attempts) - x-zse-96 signature handled by browser All 44 crawler tests passing Requirements: NET-01, NET-02, NET-03 completed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -23,9 +23,9 @@
|
||||
|
||||
### 网络与反爬
|
||||
|
||||
- [ ] **NET-01**: 支持代理 IP 配置
|
||||
- [ ] **NET-02**: 请求频率控制,避免触发反爬
|
||||
- [ ] **NET-03**: 自动获取请求签名(x-zse-96),无需手动逆向
|
||||
- [x] **NET-01**: 支持代理 IP 配置
|
||||
- [x] **NET-02**: 请求频率控制,避免触发反爬
|
||||
- [x] **NET-03**: 自动获取请求签名(x-zse-96),无需手动逆向
|
||||
|
||||
### Pipeline 整合
|
||||
|
||||
@@ -119,9 +119,9 @@
|
||||
| CONT-02 | Phase 8 | Complete |
|
||||
| CONT-03 | Phase 8 | Complete |
|
||||
| CONT-04 | Phase 8 | Complete |
|
||||
| NET-01 | Phase 8 | Pending |
|
||||
| NET-02 | Phase 8 | Pending |
|
||||
| NET-03 | Phase 8 | Pending |
|
||||
| NET-01 | Phase 8 | Complete |
|
||||
| NET-02 | Phase 8 | Complete |
|
||||
| NET-03 | Phase 8 | Complete |
|
||||
| PIPE-01 | Phase 9 | Pending |
|
||||
| PIPE-02 | Phase 9 | Pending |
|
||||
| PIPE-03 | Phase 9 | Pending |
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
phase: 08-crawler
|
||||
plan: 03
|
||||
subsystem: crawler
|
||||
tags: [proxy, rate-limiter, retry, anti-crawl, playwright]
|
||||
|
||||
requires:
|
||||
- phase: 08-01
|
||||
provides: ZhihuCrawler base class
|
||||
provides:
|
||||
- CrawlerConfig for proxy/rate limiting configuration
|
||||
- Random delay (1-3s) between requests
|
||||
- Exponential backoff retry (3 attempts)
|
||||
- Automatic x-zse-96 signature handling via browser
|
||||
|
||||
tech-stack:
|
||||
added:
|
||||
- tenacity (retry library)
|
||||
patterns:
|
||||
- Environment variable proxy configuration
|
||||
- Exponential backoff retry pattern
|
||||
- Random delay for rate limiting
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- kb_workflow/crawler/config.py
|
||||
- tests/test_crawler/test_proxy_config.py
|
||||
- tests/test_crawler/test_rate_limiter.py
|
||||
- tests/test_crawler/test_api_intercept.py
|
||||
modified:
|
||||
- kb_workflow/crawler/zhihu_crawler.py
|
||||
- kb_workflow/crawler/models.py
|
||||
- tests/test_crawler/test_comment_crawler.py
|
||||
key-decisions:
|
||||
- Mandatory proxy configuration (configurable via proxy_required flag)
|
||||
- Use tenacity library for retry with exponential backoff
|
||||
- x-zse-96 signature handled automatically by browser via route.fetch()
|
||||
- Random delay range: 1-3 seconds (configurable)
|
||||
requirements-completed: [NET-01, NET-02, NET-03]
|
||||
duration: 15min
|
||||
completed: 2026-03-17
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
实现了网络反爬措施:强制代理配置、请求频率控制、有限重试机制,以及利用浏览器自动处理 x-zse-96 签名。
|
||||
|
||||
### What Was Built
|
||||
|
||||
**CrawlerConfig 类** (`kb_workflow/crawler/config.py`)
|
||||
- `ProxyConfig` - 代理配置模型
|
||||
- 从环境变量 `HTTP_PROXY`/`HTTPS_PROXY` 读取代理设置
|
||||
- `proxy_required` 标志控制是否强制代理(默认 `True`)
|
||||
- 随机延迟配置(`min_delay`, `max_delay`)
|
||||
- 重试配置(`max_retries`, `retry_min_wait`, `retry_max_wait`)
|
||||
|
||||
**ZhihuCrawler 增强**
|
||||
- `_start_browser_with_proxy()` - 启动浏览器并应用代理配置
|
||||
- `_apply_random_delay()` - 应用随机延迟(1-3 秒)
|
||||
- `_crawl_with_retry()` - 带指数退避的重试逻辑
|
||||
- 浏览器自动处理 x-zse-96 签名(通过 `route.fetch()`)
|
||||
|
||||
### Test Coverage
|
||||
|
||||
**44 个测试全部通过**
|
||||
- `test_proxy_config.py` - 11 个代理配置测试
|
||||
- `test_rate_limiter.py` - 8 个频率控制测试
|
||||
- `test_api_intercept.py` - 4 个 API 拦截测试
|
||||
- 其他测试文件保持通过
|
||||
|
||||
### Commits
|
||||
|
||||
1. `626fb73` feat(08-03): create CrawlerConfig for proxy and rate limiting
|
||||
2. `9e1452d` feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler
|
||||
|
||||
### Key Decisions
|
||||
|
||||
1. **强制代理**: 默认 `proxy_required=True`,未配置代理时抛出 `ValueError`
|
||||
2. **重试机制**: 使用 `tenacity` 库,最多 3 次重试,间隔指数递增(2s → 10s)
|
||||
3. **签名处理**: 利用浏览器 `route.fetch()` 自动携带 x-zse-96 签名,无需手动逆向
|
||||
|
||||
### Requirements Completed
|
||||
|
||||
- **NET-01**: 用户可以通过环境变量配置代理 IP ✓
|
||||
- **NET-02**: 不配置代理时,爬虫拒绝运行并提示错误 ✓
|
||||
- **NET-03**: 每次请求间有随机延迟(1-3秒),失败重试 3 次 ✓
|
||||
|
||||
## Self-Check: PASSED
|
||||
Reference in New Issue
Block a user