From d7f32b6b938b784e1146926169530688f22401e0 Mon Sep 17 00:00:00 2001 From: starmen1265 Date: Wed, 18 Mar 2026 22:35:48 +0800 Subject: [PATCH] chore: remove phases 11-13 (Behavior Camouflage, CAPTCHA Detection, Error Recovery) --- .planning/ROADMAP.md | 90 +++++++++++--------------------------------- .planning/STATE.md | 6 +-- 2 files changed, 24 insertions(+), 72 deletions(-) diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 9c98c65..3f285c5 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -2,15 +2,15 @@ ## Milestones -- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15) -- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16) -- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17) +- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2011-03-15) +- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2011-03-16) +- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2011-03-17) - 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress) ## Phases
-✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2026-03-15 +✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2011-03-15 ### Phase 1: Foundation **Goal**: 项目基础设施和状态管理 @@ -49,7 +49,7 @@ Plans:
-✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2026-03-16 +✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2011-03-16 ### Phase 5: 归档与目录优化 **Goal**: 统一中间目录结构，添加自动归档步骤，优化最终目录布局 @@ -79,7 +79,7 @@ Plans:
-✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17 +✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2011-03-17 ### Phase 7: 登录认证体系 **Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式，实现 Cookie 持久化和过期检测 @@ -135,54 +135,9 @@ Plans: - [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换 - [ ] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换 -#### Phase 11: Behavior Camouflage (请求行为伪装) -**Goal**: 通过随机化请求特征模拟真实用户行为，降低被反爬系统检测的风险 -**Depends on**: Phase 10 -**Requirements**: BEHAV-01, BEHAV-02, BEHAV-03, BEHAV-04 -**Success Criteria** (what must be TRUE): - 1. 连续请求之间有随机的延迟间隔（默认 1-5 秒） - 2. 每次请求使用不同的真实浏览器 User-Agent - 3. 请求头中的 Accept-Language、Accept-Encoding 等字段随机化 - 4. 支持通过 Playwright 模拟页面滚动和鼠标移动行为 -**Plans**: TBD - -Plans: -- [ ] 11-01: 请求延迟与 UA 轮换 (BEHAV-01, BEHAV-02) -- [ ] 11-02: 请求头随机化 (BEHAV-03) -- [ ] 11-03: Playwright 行为模拟 (BEHAV-04) - -#### Phase 12: CAPTCHA Detection (验证码检测) -**Goal**: 实现验证码页面检测机制，在触发验证码时优雅地暂停并通知用户 -**Depends on**: Phase 11 -**Requirements**: CAPT-01, CAPT-02, CAPT-03 -**Success Criteria** (what must be TRUE): - 1. 系统能自动识别知乎和 B 站的验证码页面 - 2. 检测到验证码时记录日志并暂停当前爬取任务 - 3. 用户手动处理验证码后可以继续爬取 -**Plans**: TBD - -Plans: -- [ ] 12-01: 验证码页面检测与告警 (CAPT-01, CAPT-02) -- [ ] 12-02: 手动恢复机制 (CAPT-03) - -#### Phase 13: Error Recovery (错误恢复机制) -**Goal**: 建立智能的错误分类和重试策略，提升爬取任务的健壮性 -**Depends on**: Phase 12 -**Requirements**: ERR-01, ERR-02, ERR-03, ERR-04 -**Success Criteria** (what must be TRUE): - 1. 403/429/超时等不同错误类型采用不同的重试策略 - 2. 重试间隔使用指数退避并添加随机抖动，避免请求风暴 - 3. 永久性错误（如内容不存在）不再重试，记录原因后跳过 - 4. 失败位置被记录，用户可以从失败点恢复爬取 -**Plans**: TBD - -Plans: -- [ ] 13-01: 错误分类与重试策略 (ERR-01, ERR-02, ERR-03) -- [ ] 13-02: 断点续传增强 (ERR-04) - -#### Phase 14: Platform Adaptation (平台适配) +#### Phase 11: Platform Adaptation (平台适配) **Goal**: 针对知乎和 B 站的反爬策略进行专项优化，整合前面所有反爬能力 -**Depends on**: Phase 13 +**Depends on**: Phase 11 **Requirements**: PLAT-01, PLAT-02 **Success Criteria** (what must be TRUE): 1. 知乎爬取针对登录态检测和请求频率限制进行了优化 @@ -191,8 +146,8 @@ Plans: **Plans**: TBD Plans: -- [ ] 14-01: 知乎反爬适配 (PLAT-01) -- [ ] 14-02: B 站反爬适配 (PLAT-02) +- [ ] 11-01: 知乎反爬适配 (PLAT-01) +- [ ] 11-02: B 站反爬适配 (PLAT-02) ## Progress @@ -201,17 +156,14 @@ Phases execute in numeric order: 1 → ... → 9 → 10 → 11 → 12 → 13 → | Phase | Milestone | Plans Complete | Status | Completed | |-------|-----------|----------------|--------|-----------| -| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 | -| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 | -| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 | -| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 | -| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 | -| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 | -| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 | -| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 | -| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 | -| 10. Proxy Pool | 2/2 | Complete | 2026-03-18 | - | -| 11. Behavior Camouflage | v1.3 | 0/3 | Not started | - | -| 12. CAPTCHA Detection | v1.3 | 0/2 | Not started | - | -| 13. Error Recovery | v1.3 | 0/2 | Not started | - | -| 14. Platform Adaptation | v1.3 | 0/2 | Not started | - | +| 1. Foundation | v1.0 | 3/3 | Complete | 2011-03-15 | +| 2. Adapters | v1.0 | 2/2 | Complete | 2011-03-15 | +| 3. Processors | v1.0 | 3/3 | Complete | 2011-03-15 | +| 4. Pipeline | v1.0 | 2/2 | Complete | 2011-03-15 | +| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2011-03-16 | +| 6. 代码整合 | v1.1 | 3/3 | Complete | 2011-03-16 | +| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2011-03-17 | +| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2011-03-17 | +| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2011-03-17 | +| 10. Proxy Pool | 2/2 | Complete | 2011-03-18 | - | +| 11. Platform Adaptation | v1.3 | 0/2 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index c65f376..bf9c7b1 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,10 +4,10 @@ milestone: v1.3 milestone_name: Anti-Crawler Optimization status: executing stopped_at: Completed 10-02-PLAN.md -last_updated: "2026-03-18T08:55:13.067Z" +last_updated: "2026-03-18T14:35:35.015Z" last_activity: 2026-03-18 — 10-01 proxy pool core complete progress: - total_phases: 14 + total_phases: 11 completed_phases: 9 total_plans: 29 completed_plans: 28 @@ -25,7 +25,7 @@ See: .planning/PROJECT.md (updated 2026-03-18) ## Current Position -Phase: 10 of 14 (Proxy Pool) +Phase: 10 of 11 (Proxy Pool) Plan: 1 of 2 in current phase Status: In progress Last activity: 2026-03-18 — 10-01 proxy pool core complete