From 9e30e042a847b599522e4a0d358d0c92f7764bba Mon Sep 17 00:00:00 2001 From: starmen1265 Date: Wed, 18 Mar 2026 23:19:25 +0800 Subject: [PATCH] docs: create v1.4 Zhihu Anti-Crawler roadmap (1 phase, 2 plans) --- .planning/ROADMAP.md | 47 ++++++++++++++++++++++++----- .planning/STATE.md | 71 ++++++++++++++++---------------------------- 2 files changed, 65 insertions(+), 53 deletions(-) diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 9c8fbe2..b2704eb 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -6,6 +6,7 @@ - ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16) - ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17) - ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18) +- 🚧 **v1.4 Zhihu Anti-Crawler** - Phase 11 (in progress) ## Phases @@ -95,19 +96,49 @@ Plans: - 惰性冷却检查而非后台定时器 - ProxyPool 参数优先于 config.proxy 配置 -**Known Gaps (deferred):** -- BEHAV-01~04: 请求行为伪装 -- CAPT-01~03: 验证码检测 -- ERR-01~04: 错误恢复机制 -- PLAT-01~02: 平台适配 - --- -## Next Milestone +## 🚧 v1.4 Zhihu Anti-Crawler (In Progress) -Run `/gsd:new-milestone` to plan the next version. +**Milestone Goal:** 绕过知乎反爬检测，让 Playwright 爬虫能正常工作 + +### Phase 11: Stealth Fingerprinting (指纹伪装) +**Goal**: 隐藏 Playwright 自动化特征，让爬虫通过知乎的浏览器指纹检测 +**Depends on**: Phase 10 +**Requirements**: STEALTH-01, STEALTH-02, STEALTH-03, STEALTH-04, VERIFY-01, VERIFY-02 +**Success Criteria** (what must be TRUE): + 1. ZhihuCrawler 访问问答页面时不再被 40362 错误拦截 + 2. 爬虫的 navigator.webdriver 属性对知乎不可见 + 3. Playwright 特有的 JS 属性（__playwright, pwInspect 等）被移除 + 4. 现有功能（登录、Cookie 持久化、内容爬取）保持正常工作 +**Plans**: TBD + +Plans: +- [ ] 11-01: 指纹伪装脚本实现 — 隐藏 webdriver、移除 Playwright 特有属性、模拟 Chrome 特性 +- [ ] 11-02: ZhihuCrawler 集成与验证 — 注入伪装脚本，验证爬取功能正常 + +--- + +## Progress + +**Execution Order:** +Phases execute in numeric order: 1 → 2 → ... → 10 → 11 + +| Phase | Milestone | Plans Complete | Status | Completed | +|-------|-----------|----------------|--------|-----------| +| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 | +| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 | +| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 | +| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 | +| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 | +| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 | +| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 | +| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 | +| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 | +| 10. Proxy Pool | v1.3 | 2/2 | Complete | 2026-03-18 | +| 11. Stealth Fingerprinting | v1.4 | 0/2 | Not started | - | --- diff --git a/.planning/STATE.md b/.planning/STATE.md index 9ee353b..3d7f682 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,12 +4,12 @@ milestone: v1.4 milestone_name: Zhihu Anti-Crawler status: planning stopped_at: — -last_updated: "2026-03-18T15:30:00.000Z" -last_activity: 2026-03-18 — Milestone v1.4 started +last_updated: "2026-03-18T16:00:00.000Z" +last_activity: 2026-03-18 — Roadmap created for v1.4 progress: - total_phases: 0 + total_phases: 1 completed_phases: 0 - total_plans: 0 + total_plans: 2 completed_plans: 0 percent: 0 --- @@ -21,31 +21,33 @@ progress: See: .planning/PROJECT.md (updated 2026-03-18) **Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复 -**Current focus:** Phase 10 - Proxy Pool (代理池管理) +**Current focus:** Phase 11 - Stealth Fingerprinting (指纹伪装) ## Current Position -Phase: Not started (defining requirements) -Plan: — -Status: Defining requirements -Last activity: 2026-03-18 — Milestone v1.4 started +Phase: 1 of 1 (Phase 11: Stealth Fingerprinting) +Plan: 0 of 2 +Status: Ready to plan +Last activity: 2026-03-18 — Roadmap created for v1.4 + +Progress: [░░░░░░░░░░] 0% ## Performance Metrics **Velocity:** -- Total plans completed: 24 (v1.0-v1.3) -- v1.3 plans: 1/11 -- Average duration: — -- Total execution time: — +- Total plans completed: 26 (v1.0-v1.3) +- v1.3 plans: 2/2 +- Average duration: ~5min +- Total execution time: ~2.5 hours | Plan | Duration | Tasks | Files | Date | |------|----------|-------|-------|------| | 10-01 | 3min | 2 | 2 | 2026-03-18 | +| 10-02 | 4min | 2 | 2 | 2026-03-18 | **Recent Trend:** -- v1.0-v1.2 completed smoothly +- v1.0-v1.3 completed smoothly - Trend: Stable -| Phase 10-proxy-pool P02 | 4min | 2 tasks | 2 files | ## Accumulated Context @@ -58,35 +60,14 @@ Key decisions from v1.0 (preserved): - 顺序执行 — 简单性和可调试性 - CLI subparsers — 可扩展命令结构 -v1.1 decisions: -- 中间目录统一到 `~/work/intermediate/analysis/` -- 归档只保留 md 文件，不归档 json 文件 -- 最终目录按平台分组，不再按时间创建子目录 - -v1.2 decisions: -- 知乎登录策略：二维码扫码 + 手动 Cookie 降级 -- 参考 MediaCrawler 实现，自实现而非直接依赖 -- 使用 playwright 作为浏览器自动化框架 -- 整合进现有 Pipeline 架构 -- Cookie 文件权限设置为 600 保证安全性 -- 保存 Cookie 后清除 get_settings 缓存确保配置刷新 -- 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时 -- 多个 canvas 选择器降级提高 QR 码检测鲁棒性 -- login CLI 命令支持三种模式：扫码登录、导入 Cookie、状态检查 -- Use Playwright request API for direct comment API calls instead of route interception -- Only fetch first-level comments (no nested replies) -- Use hot sorting (order=hot) for popular comments -- ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default -- Cookie expiration triggers automatic QR code login flow -- Comments appended to content in Markdown format with '## 热门评论' section - v1.3 decisions: -- Phase 10-14 覆盖 17 个需求：代理池、行为伪装、验证码检测、错误恢复、平台适配 -- 每个阶段 2-3 个计划，总计 11 个计划 - 代理池使用索引指针实现 Round-Robin，惰性检查冷却恢复 -- 所有代理失效时回退到第一个代理继续尝试 -- [Phase 10-proxy-pool]: ProxyPool parameter takes priority over config.proxy when both are provided -- [Phase 10-proxy-pool]: Health feedback methods added to ZhihuCrawler for future automatic proxy switching +- ProxyPool 参数优先于 config.proxy 配置 + +v1.4 scope: +- 只针对知乎反爬检测 +- 使用最小改动（指纹伪装）先验证效果 +- 行为伪装（鼠标轨迹等）暂不实现，如需要再加 ### Blockers/Concerns @@ -94,8 +75,8 @@ None. ## Session Continuity -Last session: 2026-03-18T08:51:02.420Z -Stopped at: Completed 10-02-PLAN.md +Last session: 2026-03-18 +Stopped at: Roadmap created for v1.4 Resume file: None -**Next step:** Run `/gsd:execute-phase 10` to continue with 10-02 plan +**Next step:** Run `/gsd:plan-phase 11` to create first plan