docs: define v1.3 anti-crawler requirements
This commit is contained in:
@@ -70,6 +70,42 @@
|
||||
- [x] **PIP-03**: 删除独立的 kb_pipeline 目录
|
||||
- [x] **PIP-04**: 更新 CLI 入口导入路径
|
||||
|
||||
## v1.3 Requirements (Current)
|
||||
|
||||
反爬虫优化,增强爬虫反检测能力,降低被封风险,提升爬取稳定性。
|
||||
|
||||
### Proxy Pool (代理池管理)
|
||||
|
||||
- [ ] **PROXY-01**: 用户可以配置代理池 — 支持从文件或环境变量加载多个代理地址
|
||||
- [ ] **PROXY-02**: 系统自动轮换代理 — 每次请求自动选择下一个可用代理
|
||||
- [ ] **PROXY-03**: 代理健康检查 — 检测代理可用性,自动移除失效代理
|
||||
- [ ] **PROXY-04**: 代理失败回退 — 当前代理失败时自动切换到下一个代理
|
||||
|
||||
### Behavior Camouflage (请求行为伪装)
|
||||
|
||||
- [ ] **BEHAV-01**: 随机请求延迟 — 请求间隔在可配置范围内随机化(默认 1-5 秒)
|
||||
- [ ] **BEHAV-02**: User-Agent 轮换 — 每次请求使用不同的真实浏览器 UA
|
||||
- [ ] **BEHAV-03**: 请求头随机化 — Accept-Language、Accept-Encoding 等头部随机化
|
||||
- [ ] **BEHAV-04**: 模拟人类浏览模式 — 支持页面滚动、鼠标移动等行为模拟(Playwright)
|
||||
|
||||
### CAPTCHA Detection (验证码检测)
|
||||
|
||||
- [ ] **CAPT-01**: 验证码页面检测 — 自动检测知乎/B站验证码页面
|
||||
- [ ] **CAPT-02**: 验证码触发告警 — 检测到验证码时记录日志并暂停爬取
|
||||
- [ ] **CAPT-03**: 验证码状态恢复 — 支持手动处理后继续爬取
|
||||
|
||||
### Error Recovery (错误恢复机制)
|
||||
|
||||
- [ ] **ERR-01**: 智能重试策略 — 根据错误类型选择不同的重试策略(403/429/超时)
|
||||
- [ ] **ERR-02**: 指数退避 + 抖动 — 重试间隔使用指数退避并添加随机抖动
|
||||
- [ ] **ERR-03**: 错误分类处理 — 区分临时错误和永久错误,避免无效重试
|
||||
- [ ] **ERR-04**: 断点续传增强 — 记录失败位置,支持从失败点恢复
|
||||
|
||||
### Platform Adaptation (平台适配)
|
||||
|
||||
- [ ] **PLAT-01**: 知乎反爬适配 — 针对知乎的反爬策略优化(登录态检测、请求频率限制)
|
||||
- [ ] **PLAT-02**: B站反爬适配 — 针对 B站的反爬策略优化(视频/专栏元数据获取)
|
||||
|
||||
## v2 Requirements (Deferred)
|
||||
|
||||
推迟到后续版本的需求。
|
||||
@@ -158,4 +194,31 @@
|
||||
|
||||
---
|
||||
*Requirements defined: 2026-03-17*
|
||||
*Last updated: 2026-03-17 after v1.2 roadmap created*
|
||||
*Last updated: 2026-03-18 after v1.3 requirements added*
|
||||
|
||||
### v1.3 Traceability
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| PROXY-01 | Phase 10 | Pending |
|
||||
| PROXY-02 | Phase 10 | Pending |
|
||||
| PROXY-03 | Phase 10 | Pending |
|
||||
| PROXY-04 | Phase 10 | Pending |
|
||||
| BEHAV-01 | Phase 11 | Pending |
|
||||
| BEHAV-02 | Phase 11 | Pending |
|
||||
| BEHAV-03 | Phase 11 | Pending |
|
||||
| BEHAV-04 | Phase 11 | Pending |
|
||||
| CAPT-01 | Phase 12 | Pending |
|
||||
| CAPT-02 | Phase 12 | Pending |
|
||||
| CAPT-03 | Phase 12 | Pending |
|
||||
| ERR-01 | Phase 13 | Pending |
|
||||
| ERR-02 | Phase 13 | Pending |
|
||||
| ERR-03 | Phase 13 | Pending |
|
||||
| ERR-04 | Phase 13 | Pending |
|
||||
| PLAT-01 | Phase 14 | Pending |
|
||||
| PLAT-02 | Phase 14 | Pending |
|
||||
|
||||
**v1.3 Coverage:**
|
||||
- Requirements: 17 total
|
||||
- Mapped to phases: 17
|
||||
- Unmapped: 0 ✓
|
||||
|
||||
Reference in New Issue
Block a user