diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index 6582f8a..61cdccd 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -1,16 +1,15 @@ # kb-skills 工作流重构 -## Current Milestone: v1.2 Zhihu Crawler +## Current Milestone: v1.3 Anti-Crawler Optimization -**Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题,支持自动登录维持 +**Goal:** 增强爬虫反检测能力,降低被封风险,提升爬取稳定性 **Target features:** -- 二维码扫码登录 + 手动 Cookie 降级 -- Cookie 持久化存储 -- 问答/专栏/文章内容爬取 -- 评论爬取支持 -- 代理 IP 支持 -- 整合进现有 Pipeline +- 代理 IP 轮换机制 +- 请求行为伪装(随机间隔、模拟人类行为) +- 验证码检测与处理 +- 错误恢复机制(自动重试、指数退避) +- 知乎 + B站 双平台支持 ## What This Is @@ -62,17 +61,25 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx - ✓ OCR 文字识别 — ocr_processor.py - ✓ AI 深度分析 — deep_analyzer.py +**v1.2 Zhihu Crawler (2026-03-17):** +- ✓ **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码,用户扫码 +- ✓ **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie +- ✓ **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie,自动复用 +- ✓ **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据 +- ✓ **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据 +- ✓ **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论 +- ✓ **ZHIHU-07**: 代理 IP 支持 — 降低被封风险 +- ✓ **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器 + ### Active -**v1.2 Zhihu Crawler:** -- [ ] **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码,用户扫码 -- [ ] **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie -- [ ] **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie,自动复用 -- [ ] **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据 -- [ ] **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据 -- [ ] **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论 -- [ ] **ZHIHU-07**: 代理 IP 支持 — 降低被封风险 -- [ ] **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器 +**v1.3 Anti-Crawler Optimization:** +- [ ] **AC-01**: 代理池管理 — 支持配置代理池,自动轮换 IP +- [ ] **AC-02**: 请求行为伪装 — 随机化请求间隔,模拟人类行为 +- [ ] **AC-03**: 验证码检测 — 检测验证码页面,触发告警/暂停 +- [ ] **AC-04**: 错误恢复机制 — 自动重试、指数退避、断点续传增强 +- [ ] **AC-05**: 知乎反爬适配 — 针对知乎的反爬策略优化 +- [ ] **AC-06**: B站反爬适配 — 针对 B站的反爬策略优化 ### Deferred @@ -133,4 +140,4 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx | 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 | --- -*Last updated: 2026-03-17 after v1.2 milestone started* +*Last updated: 2026-03-18 after v1.3 milestone started* diff --git a/.planning/STATE.md b/.planning/STATE.md index f2baa43..6289308 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,54 +1,47 @@ --- gsd_state_version: 1.0 -milestone: v1.2 -milestone_name: Zhihu Crawler -status: completed -stopped_at: Completed 09-01 ZhihuAdapter integration -last_updated: "2026-03-17T15:25:04.074Z" -last_activity: 2026-03-17 — Phase 09-00 Test scaffolding complete +milestone: v1.3 +milestone_name: Anti-Crawler Optimization +status: planning +stopped_at: Defining requirements +last_updated: "2026-03-18T00:00:00.000Z" +last_activity: 2026-03-18 — Milestone v1.3 started progress: - total_phases: 9 - completed_phases: 8 - total_plans: 27 - completed_plans: 26 - percent: 100 + total_phases: 0 + completed_phases: 0 + total_plans: 0 + completed_plans: 0 + percent: 0 --- # Project State ## Project Reference -See: .planning/PROJECT.md (updated 2026-03-17) +See: .planning/PROJECT.md (updated 2026-03-18) **Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复 -**Current focus:** v1.2 Zhihu Crawler — Phase 9 执行中 +**Current focus:** v1.3 Anti-Crawler Optimization — 反爬优化 ## Current Position -Phase: 9 of 9 (Pipeline Integration - 管道整合) -Plan: 01 of 04 -Status: Phase 09-00 Wave 0 complete -Last activity: 2026-03-17 — Phase 09-00 Test scaffolding complete +Phase: Not started (defining requirements) +Plan: — +Status: Defining requirements +Last activity: 2026-03-18 — Milestone v1.3 started -Progress: [██████████] 100% (v1.2 milestone - Phase 9 in progress) +Progress: [░░░░░░░░░░] 0% (v1.3 milestone - requirements phase) ## Performance Metrics **Velocity:** -- Total plans completed: 3 (v1.2) -- Average duration: 11 min -- Total execution time: 32 min +- Total plans completed: 0 (v1.3) +- Average duration: — +- Total execution time: — | Plan | Duration | Tasks | Files | Date | |------|----------|-------|-------|------| -| 07-01 | 15min | 2 | 5 | 2026-03-17 | -| 07-02 | 5min | 2 | 4 | 2026-03-17 | -| 07-03 | 12min | 3 | 4 | 2026-03-17 | -| Phase 08 P01 | 3min | 2 | 8 | 2026-03-17 | -| Phase 08 P02 | 5min | 2 | 3 | 2026-03-17 | -| Phase 08 P03 | 15min | 2 | 5 | 2026-03-17 | -| 09-00 | 2min | 1 | 1 | 2026-03-17 | -| Phase 09-pipeline-integration P01 | 11min | 3 tasks | 2 files | +| (no plans yet) | ## Accumulated Context @@ -76,12 +69,12 @@ v1.2 decisions: - 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时 - 多个 canvas 选择器降级提高 QR 码检测鲁棒性 - login CLI 命令支持三种模式:扫码登录、导入 Cookie、状态检查 -- [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception -- [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies) -- [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments -- [Phase 09-01]: ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default -- [Phase 09-01]: Cookie expiration triggers automatic QR code login flow -- [Phase 09-01]: Comments appended to content in Markdown format with '## 热门评论' section +- Use Playwright request API for direct comment API calls instead of route interception +- Only fetch first-level comments (no nested replies) +- Use hot sorting (order=hot) for popular comments +- ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default +- Cookie expiration triggers automatic QR code login flow +- Comments appended to content in Markdown format with '## 热门评论' section ### Blockers/Concerns @@ -89,8 +82,8 @@ None. ## Session Continuity -Last session: 2026-03-17T15:16:21.489Z -Stopped at: Completed 09-01 ZhihuAdapter integration +Last session: — +Stopped at: Defining requirements Resume file: None -**Next step:** Continue with Phase 09-01 ZhihuAdapter integration +**Next step:** Define v1.3 requirements and create roadmap