diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 13359d7..5cf53cb 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -4,7 +4,8 @@ - ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15) - ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16) -- 🚧 **v1.2 Zhihu Crawler** - Phases 7-9 (in progress) +- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17) +- 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress) ## Phases @@ -77,11 +78,10 @@ Plans: -### 🚧 v1.2 Zhihu Crawler (In Progress) +
+✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17 -**Milestone Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题,支持自动登录维持 - -#### Phase 7: 登录认证体系 +### Phase 7: 登录认证体系 **Goal**: 建立知乎登录认证机制,支持二维码扫码和手动 Cookie 两种方式,实现 Cookie 持久化和过期检测 **Depends on**: Phase 6 **Requirements**: AUTH-01, AUTH-02, AUTH-03, AUTH-04 @@ -92,12 +92,7 @@ Plans: 4. Cookie 过期时系统自动检测并提示用户重新登录 **Plans**: 3 plans (all complete) -Plans: -- [x] 07-01-PLAN.md — 二维码扫码登录核心逻辑 (AUTH-01) + Wave 0 测试脚手架 -- [x] 07-02-PLAN.md — Cookie 导入和持久化管理 (AUTH-02, AUTH-03) -- [x] 07-03-PLAN.md — Cookie 验证 + CLI login 子命令 (AUTH-04) - -#### Phase 8: 内容爬取实现 +### Phase 8: 内容爬取实现 **Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能,包含网络反爬措施 **Depends on**: Phase 7 **Requirements**: CONT-01, CONT-02, CONT-03, CONT-04, NET-01, NET-02, NET-03 @@ -107,14 +102,9 @@ Plans: 3. 用户可以选择是否爬取内容下的评论 4. 爬取结果包含作者信息、发布时间、点赞数等完整元数据 5. 用户可以配置代理 IP 以降低被封风险 -**Plans**: 3 plans in 2 waves +**Plans**: 3 plans (all complete) -Plans: -- [ ] 08-01-PLAN.md — 问答和专栏内容爬取 (CONT-01, CONT-02, CONT-04) — Wave 1 -- [ ] 08-02-PLAN.md — 评论爬取功能 (CONT-03) — Wave 1 (parallel) -- [ ] 08-03-PLAN.md — 网络反爬措施 (NET-01, NET-02, NET-03) — Wave 2 - -#### Phase 9: Pipeline 整合 +### Phase 9: Pipeline 整合 **Goal**: 将新爬虫实现整合进现有 Pipeline 架构,替换原有知乎处理器 **Depends on**: Phase 8 **Requirements**: PIPE-01, PIPE-02, PIPE-03 @@ -122,15 +112,92 @@ Plans: 1. 知乎链接使用新爬虫处理器,无需手动设置 Cookie 2. 新处理器与现有 Pipeline 架构完全兼容 3. 支持断点续传和幂等执行(与其他处理器行为一致) +**Plans**: 1 plan (complete) + +
+ +### 🚧 v1.3 Anti-Crawler Optimization (In Progress) + +**Milestone Goal:** 增强爬虫反检测能力,降低被封风险,提升爬取稳定性 + +#### Phase 10: Proxy Pool (代理池管理) +**Goal**: 建立代理池基础设施,支持多代理配置、自动轮换和健康检查 +**Depends on**: Phase 9 +**Requirements**: PROXY-01, PROXY-02, PROXY-03, PROXY-04 +**Success Criteria** (what must be TRUE): + 1. 用户可以从配置文件或环境变量加载多个代理地址 + 2. 每次请求自动选择下一个可用代理,实现负载均衡 + 3. 代理不可用时自动检测并标记为失效 + 4. 当前代理失败时自动切换到备用代理继续请求 **Plans**: TBD Plans: -- [ ] 09-01: ZhihuProcessor 替换和整合 (PIPE-01, PIPE-02, PIPE-03) +- [ ] 10-01: 代理池核心实现 (PROXY-01, PROXY-02) +- [ ] 10-02: 健康检查与故障切换 (PROXY-03, PROXY-04) + +#### Phase 11: Behavior Camouflage (请求行为伪装) +**Goal**: 通过随机化请求特征模拟真实用户行为,降低被反爬系统检测的风险 +**Depends on**: Phase 10 +**Requirements**: BEHAV-01, BEHAV-02, BEHAV-03, BEHAV-04 +**Success Criteria** (what must be TRUE): + 1. 连续请求之间有随机的延迟间隔(默认 1-5 秒) + 2. 每次请求使用不同的真实浏览器 User-Agent + 3. 请求头中的 Accept-Language、Accept-Encoding 等字段随机化 + 4. 支持通过 Playwright 模拟页面滚动和鼠标移动行为 +**Plans**: TBD + +Plans: +- [ ] 11-01: 请求延迟与 UA 轮换 (BEHAV-01, BEHAV-02) +- [ ] 11-02: 请求头随机化 (BEHAV-03) +- [ ] 11-03: Playwright 行为模拟 (BEHAV-04) + +#### Phase 12: CAPTCHA Detection (验证码检测) +**Goal**: 实现验证码页面检测机制,在触发验证码时优雅地暂停并通知用户 +**Depends on**: Phase 11 +**Requirements**: CAPT-01, CAPT-02, CAPT-03 +**Success Criteria** (what must be TRUE): + 1. 系统能自动识别知乎和 B 站的验证码页面 + 2. 检测到验证码时记录日志并暂停当前爬取任务 + 3. 用户手动处理验证码后可以继续爬取 +**Plans**: TBD + +Plans: +- [ ] 12-01: 验证码页面检测与告警 (CAPT-01, CAPT-02) +- [ ] 12-02: 手动恢复机制 (CAPT-03) + +#### Phase 13: Error Recovery (错误恢复机制) +**Goal**: 建立智能的错误分类和重试策略,提升爬取任务的健壮性 +**Depends on**: Phase 12 +**Requirements**: ERR-01, ERR-02, ERR-03, ERR-04 +**Success Criteria** (what must be TRUE): + 1. 403/429/超时等不同错误类型采用不同的重试策略 + 2. 重试间隔使用指数退避并添加随机抖动,避免请求风暴 + 3. 永久性错误(如内容不存在)不再重试,记录原因后跳过 + 4. 失败位置被记录,用户可以从失败点恢复爬取 +**Plans**: TBD + +Plans: +- [ ] 13-01: 错误分类与重试策略 (ERR-01, ERR-02, ERR-03) +- [ ] 13-02: 断点续传增强 (ERR-04) + +#### Phase 14: Platform Adaptation (平台适配) +**Goal**: 针对知乎和 B 站的反爬策略进行专项优化,整合前面所有反爬能力 +**Depends on**: Phase 13 +**Requirements**: PLAT-01, PLAT-02 +**Success Criteria** (what must be TRUE): + 1. 知乎爬取针对登录态检测和请求频率限制进行了优化 + 2. B 站爬取针对视频/专栏元数据获取进行了优化 + 3. 两个平台的爬取成功率相比 v1.2 有明显提升 +**Plans**: TBD + +Plans: +- [ ] 14-01: 知乎反爬适配 (PLAT-01) +- [ ] 14-02: B 站反爬适配 (PLAT-02) ## Progress **Execution Order:** -Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 +Phases execute in numeric order: 1 → ... → 9 → 10 → 11 → 12 → 13 → 14 | Phase | Milestone | Plans Complete | Status | Completed | |-------|-----------|----------------|--------|-----------| @@ -141,5 +208,10 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → | 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 | | 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 | | 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 | -| 8. 内容爬取实现 | v1.2 | 0/3 | Ready to execute | - | -| 9. Pipeline 整合 | v1.2 | 0/1 | Not started | - | +| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 | +| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 | +| 10. Proxy Pool | v1.3 | 0/2 | Ready to plan | - | +| 11. Behavior Camouflage | v1.3 | 0/3 | Not started | - | +| 12. CAPTCHA Detection | v1.3 | 0/2 | Not started | - | +| 13. Error Recovery | v1.3 | 0/2 | Not started | - | +| 14. Platform Adaptation | v1.3 | 0/2 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 6289308..ee1ed32 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,14 +2,14 @@ gsd_state_version: 1.0 milestone: v1.3 milestone_name: Anti-Crawler Optimization -status: planning -stopped_at: Defining requirements +status: ready_to_plan +stopped_at: Roadmap created, ready to plan Phase 10 last_updated: "2026-03-18T00:00:00.000Z" -last_activity: 2026-03-18 — Milestone v1.3 started +last_activity: 2026-03-18 — v1.3 roadmap created (Phases 10-14) progress: - total_phases: 0 + total_phases: 5 completed_phases: 0 - total_plans: 0 + total_plans: 11 completed_plans: 0 percent: 0 --- @@ -21,27 +21,32 @@ progress: See: .planning/PROJECT.md (updated 2026-03-18) **Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复 -**Current focus:** v1.3 Anti-Crawler Optimization — 反爬优化 +**Current focus:** Phase 10 - Proxy Pool (代理池管理) ## Current Position -Phase: Not started (defining requirements) -Plan: — -Status: Defining requirements -Last activity: 2026-03-18 — Milestone v1.3 started +Phase: 10 of 14 (Proxy Pool) +Plan: 0 of 2 in current phase +Status: Ready to plan +Last activity: 2026-03-18 — v1.3 roadmap created (Phases 10-14, 11 plans) -Progress: [░░░░░░░░░░] 0% (v1.3 milestone - requirements phase) +Progress: [██████████░░░░░░░░░░] 50% (23/46 plans complete across all milestones) ## Performance Metrics **Velocity:** -- Total plans completed: 0 (v1.3) +- Total plans completed: 23 (v1.0-v1.2) +- v1.3 plans: 0/11 - Average duration: — - Total execution time: — | Plan | Duration | Tasks | Files | Date | |------|----------|-------|-------|------| -| (no plans yet) | +| (v1.3 plans pending) | + +**Recent Trend:** +- v1.0-v1.2 completed smoothly +- Trend: Stable ## Accumulated Context @@ -76,14 +81,18 @@ v1.2 decisions: - Cookie expiration triggers automatic QR code login flow - Comments appended to content in Markdown format with '## 热门评论' section +v1.3 decisions: +- Phase 10-14 覆盖 17 个需求:代理池、行为伪装、验证码检测、错误恢复、平台适配 +- 每个阶段 2-3 个计划,总计 11 个计划 + ### Blockers/Concerns None. ## Session Continuity -Last session: — -Stopped at: Defining requirements +Last session: 2026-03-18 +Stopped at: Roadmap created, ready to plan Phase 10 Resume file: None -**Next step:** Define v1.3 requirements and create roadmap +**Next step:** Run `/gsd:plan-phase 10` to create first plan