docs: create v1.4 Zhihu Anti-Crawler roadmap (1 phase, 2 plans)
This commit is contained in:
+39
-8
@@ -6,6 +6,7 @@
|
|||||||
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16)
|
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16)
|
||||||
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17)
|
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17)
|
||||||
- ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18)
|
- ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18)
|
||||||
|
- 🚧 **v1.4 Zhihu Anti-Crawler** - Phase 11 (in progress)
|
||||||
|
|
||||||
## Phases
|
## Phases
|
||||||
|
|
||||||
@@ -95,19 +96,49 @@ Plans:
|
|||||||
- 惰性冷却检查而非后台定时器
|
- 惰性冷却检查而非后台定时器
|
||||||
- ProxyPool 参数优先于 config.proxy 配置
|
- ProxyPool 参数优先于 config.proxy 配置
|
||||||
|
|
||||||
**Known Gaps (deferred):**
|
|
||||||
- BEHAV-01~04: 请求行为伪装
|
|
||||||
- CAPT-01~03: 验证码检测
|
|
||||||
- ERR-01~04: 错误恢复机制
|
|
||||||
- PLAT-01~02: 平台适配
|
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Next Milestone
|
## 🚧 v1.4 Zhihu Anti-Crawler (In Progress)
|
||||||
|
|
||||||
Run `/gsd:new-milestone` to plan the next version.
|
**Milestone Goal:** 绕过知乎反爬检测,让 Playwright 爬虫能正常工作
|
||||||
|
|
||||||
|
### Phase 11: Stealth Fingerprinting (指纹伪装)
|
||||||
|
**Goal**: 隐藏 Playwright 自动化特征,让爬虫通过知乎的浏览器指纹检测
|
||||||
|
**Depends on**: Phase 10
|
||||||
|
**Requirements**: STEALTH-01, STEALTH-02, STEALTH-03, STEALTH-04, VERIFY-01, VERIFY-02
|
||||||
|
**Success Criteria** (what must be TRUE):
|
||||||
|
1. ZhihuCrawler 访问问答页面时不再被 40362 错误拦截
|
||||||
|
2. 爬虫的 navigator.webdriver 属性对知乎不可见
|
||||||
|
3. Playwright 特有的 JS 属性(__playwright, pwInspect 等)被移除
|
||||||
|
4. 现有功能(登录、Cookie 持久化、内容爬取)保持正常工作
|
||||||
|
**Plans**: TBD
|
||||||
|
|
||||||
|
Plans:
|
||||||
|
- [ ] 11-01: 指纹伪装脚本实现 — 隐藏 webdriver、移除 Playwright 特有属性、模拟 Chrome 特性
|
||||||
|
- [ ] 11-02: ZhihuCrawler 集成与验证 — 注入伪装脚本,验证爬取功能正常
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Progress
|
||||||
|
|
||||||
|
**Execution Order:**
|
||||||
|
Phases execute in numeric order: 1 → 2 → ... → 10 → 11
|
||||||
|
|
||||||
|
| Phase | Milestone | Plans Complete | Status | Completed |
|
||||||
|
|-------|-----------|----------------|--------|-----------|
|
||||||
|
| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||||
|
| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||||
|
| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||||
|
| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||||
|
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||||
|
| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||||
|
| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||||
|
| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||||
|
| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 |
|
||||||
|
| 10. Proxy Pool | v1.3 | 2/2 | Complete | 2026-03-18 |
|
||||||
|
| 11. Stealth Fingerprinting | v1.4 | 0/2 | Not started | - |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+26
-45
@@ -4,12 +4,12 @@ milestone: v1.4
|
|||||||
milestone_name: Zhihu Anti-Crawler
|
milestone_name: Zhihu Anti-Crawler
|
||||||
status: planning
|
status: planning
|
||||||
stopped_at: —
|
stopped_at: —
|
||||||
last_updated: "2026-03-18T15:30:00.000Z"
|
last_updated: "2026-03-18T16:00:00.000Z"
|
||||||
last_activity: 2026-03-18 — Milestone v1.4 started
|
last_activity: 2026-03-18 — Roadmap created for v1.4
|
||||||
progress:
|
progress:
|
||||||
total_phases: 0
|
total_phases: 1
|
||||||
completed_phases: 0
|
completed_phases: 0
|
||||||
total_plans: 0
|
total_plans: 2
|
||||||
completed_plans: 0
|
completed_plans: 0
|
||||||
percent: 0
|
percent: 0
|
||||||
---
|
---
|
||||||
@@ -21,31 +21,33 @@ progress:
|
|||||||
See: .planning/PROJECT.md (updated 2026-03-18)
|
See: .planning/PROJECT.md (updated 2026-03-18)
|
||||||
|
|
||||||
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
||||||
**Current focus:** Phase 10 - Proxy Pool (代理池管理)
|
**Current focus:** Phase 11 - Stealth Fingerprinting (指纹伪装)
|
||||||
|
|
||||||
## Current Position
|
## Current Position
|
||||||
|
|
||||||
Phase: Not started (defining requirements)
|
Phase: 1 of 1 (Phase 11: Stealth Fingerprinting)
|
||||||
Plan: —
|
Plan: 0 of 2
|
||||||
Status: Defining requirements
|
Status: Ready to plan
|
||||||
Last activity: 2026-03-18 — Milestone v1.4 started
|
Last activity: 2026-03-18 — Roadmap created for v1.4
|
||||||
|
|
||||||
|
Progress: [░░░░░░░░░░] 0%
|
||||||
|
|
||||||
## Performance Metrics
|
## Performance Metrics
|
||||||
|
|
||||||
**Velocity:**
|
**Velocity:**
|
||||||
- Total plans completed: 24 (v1.0-v1.3)
|
- Total plans completed: 26 (v1.0-v1.3)
|
||||||
- v1.3 plans: 1/11
|
- v1.3 plans: 2/2
|
||||||
- Average duration: —
|
- Average duration: ~5min
|
||||||
- Total execution time: —
|
- Total execution time: ~2.5 hours
|
||||||
|
|
||||||
| Plan | Duration | Tasks | Files | Date |
|
| Plan | Duration | Tasks | Files | Date |
|
||||||
|------|----------|-------|-------|------|
|
|------|----------|-------|-------|------|
|
||||||
| 10-01 | 3min | 2 | 2 | 2026-03-18 |
|
| 10-01 | 3min | 2 | 2 | 2026-03-18 |
|
||||||
|
| 10-02 | 4min | 2 | 2 | 2026-03-18 |
|
||||||
|
|
||||||
**Recent Trend:**
|
**Recent Trend:**
|
||||||
- v1.0-v1.2 completed smoothly
|
- v1.0-v1.3 completed smoothly
|
||||||
- Trend: Stable
|
- Trend: Stable
|
||||||
| Phase 10-proxy-pool P02 | 4min | 2 tasks | 2 files |
|
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -58,35 +60,14 @@ Key decisions from v1.0 (preserved):
|
|||||||
- 顺序执行 — 简单性和可调试性
|
- 顺序执行 — 简单性和可调试性
|
||||||
- CLI subparsers — 可扩展命令结构
|
- CLI subparsers — 可扩展命令结构
|
||||||
|
|
||||||
v1.1 decisions:
|
|
||||||
- 中间目录统一到 `~/work/intermediate/analysis/`
|
|
||||||
- 归档只保留 md 文件,不归档 json 文件
|
|
||||||
- 最终目录按平台分组,不再按时间创建子目录
|
|
||||||
|
|
||||||
v1.2 decisions:
|
|
||||||
- 知乎登录策略:二维码扫码 + 手动 Cookie 降级
|
|
||||||
- 参考 MediaCrawler 实现,自实现而非直接依赖
|
|
||||||
- 使用 playwright 作为浏览器自动化框架
|
|
||||||
- 整合进现有 Pipeline 架构
|
|
||||||
- Cookie 文件权限设置为 600 保证安全性
|
|
||||||
- 保存 Cookie 后清除 get_settings 缓存确保配置刷新
|
|
||||||
- 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时
|
|
||||||
- 多个 canvas 选择器降级提高 QR 码检测鲁棒性
|
|
||||||
- login CLI 命令支持三种模式:扫码登录、导入 Cookie、状态检查
|
|
||||||
- Use Playwright request API for direct comment API calls instead of route interception
|
|
||||||
- Only fetch first-level comments (no nested replies)
|
|
||||||
- Use hot sorting (order=hot) for popular comments
|
|
||||||
- ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default
|
|
||||||
- Cookie expiration triggers automatic QR code login flow
|
|
||||||
- Comments appended to content in Markdown format with '## 热门评论' section
|
|
||||||
|
|
||||||
v1.3 decisions:
|
v1.3 decisions:
|
||||||
- Phase 10-14 覆盖 17 个需求:代理池、行为伪装、验证码检测、错误恢复、平台适配
|
|
||||||
- 每个阶段 2-3 个计划,总计 11 个计划
|
|
||||||
- 代理池使用索引指针实现 Round-Robin,惰性检查冷却恢复
|
- 代理池使用索引指针实现 Round-Robin,惰性检查冷却恢复
|
||||||
- 所有代理失效时回退到第一个代理继续尝试
|
- ProxyPool 参数优先于 config.proxy 配置
|
||||||
- [Phase 10-proxy-pool]: ProxyPool parameter takes priority over config.proxy when both are provided
|
|
||||||
- [Phase 10-proxy-pool]: Health feedback methods added to ZhihuCrawler for future automatic proxy switching
|
v1.4 scope:
|
||||||
|
- 只针对知乎反爬检测
|
||||||
|
- 使用最小改动(指纹伪装)先验证效果
|
||||||
|
- 行为伪装(鼠标轨迹等)暂不实现,如需要再加
|
||||||
|
|
||||||
### Blockers/Concerns
|
### Blockers/Concerns
|
||||||
|
|
||||||
@@ -94,8 +75,8 @@ None.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-03-18T08:51:02.420Z
|
Last session: 2026-03-18
|
||||||
Stopped at: Completed 10-02-PLAN.md
|
Stopped at: Roadmap created for v1.4
|
||||||
Resume file: None
|
Resume file: None
|
||||||
|
|
||||||
**Next step:** Run `/gsd:execute-phase 10` to continue with 10-02 plan
|
**Next step:** Run `/gsd:plan-phase 11` to create first plan
|
||||||
|
|||||||
Reference in New Issue
Block a user