From 9077a783e8b166cb45e52c25e65a1a4067c18aae Mon Sep 17 00:00:00 2001 From: starmen1265 Date: Tue, 17 Mar 2026 16:16:38 +0800 Subject: [PATCH] docs: start milestone v1.2 Zhihu Crawler --- .planning/PROJECT.md | 45 ++++++++++++++++++++--------- .planning/STATE.md | 69 +++++++++++++++++++------------------------- 2 files changed, 61 insertions(+), 53 deletions(-) diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index a056896..6582f8a 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -1,15 +1,16 @@ # kb-skills 工作流重构 -## Current Milestone: v1.1 Architecture Refactor +## Current Milestone: v1.2 Zhihu Crawler -**Goal:** 简化代码结构和优化文件组织 — 统一中间目录、添加归档步骤、整合元数据技能和 Pipeline 模块 +**Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题，支持自动登录维持 **Target features:** -- 中间目录统一到 `~/work/intermediate/analysis/` -- 分析完成后自动归档到最终目录 -- 最终目录按平台分目录，文件前缀加时间戳 -- kb-bilibili-metadata + kb-zhihu-metadata 整合进 kb-workflow -- kb-pipeline 合并进 kb-workflow +- 二维码扫码登录 + 手动 Cookie 降级 +- Cookie 持久化存储 +- 问答/专栏/文章内容爬取 +- 评论爬取支持 +- 代理 IP 支持 +- 整合进现有 Pipeline ## What This Is @@ -28,6 +29,13 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx ### Validated +**v1.1 Architecture Refactor (2026-03-16):** +- ✓ **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/` +- ✓ **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录 +- ✓ **REF-03**: 最终目录结构 — 按平台分目录，文件前缀加时间戳 +- ✓ **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow +- ✓ **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow + **v1.0 管道架构 (2026-03-15):** - ✓ **PIPE-01**: 统一的管道模型 — `kb_pipeline/pipeline.py` - ✓ **PIPE-02**: 步骤级状态管理 — `WorkflowStateManager` @@ -56,12 +64,15 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx ### Active -**v1.1 Architecture Refactor:** -- [ ] **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/` -- [ ] **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录 -- [ ] **REF-03**: 最终目录结构 — 按平台分目录，文件前缀加时间戳 -- [ ] **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow -- [ ] **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow +**v1.2 Zhihu Crawler:** +- [ ] **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码，用户扫码 +- [ ] **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie +- [ ] **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie，自动复用 +- [ ] **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据 +- [ ] **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据 +- [ ] **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论 +- [ ] **ZHIHU-07**: 代理 IP 支持 — 降低被封风险 +- [ ] **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器 ### Deferred @@ -69,6 +80,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx - [ ] **ADV-02**: 并行步骤执行 — 支持独立步骤的并行执行 - [ ] **ADV-03**: 说话者分离默认启用 — HF_TOKEN 配置后自动使用 - [ ] **EXT-01**: 微信公众号处理 — 支持微信公众号文章 +- [ ] **ZHIHU-F01**: 问题下所有回答 — 爬取问题的所有热门回答 +- [ ] **ZHIHU-F02**: 作者全部内容 — 爬取某作者的所有文章/回答 ### Out of Scope @@ -77,6 +90,10 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx - 用户界面 — 命令行运行 - 多语言支持 — 中文即可 - 小红书 — 当前不支持 +- 知乎会员付费内容 — 不爬取付费内容 +- 知乎视频 — v1.2 不支持 +- 大规模爬取 — 仅用于个人知识库 +- 付费打码服务 — 使用二维码登录替代 ## Context @@ -116,4 +133,4 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx | 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 | --- -*Last updated: 2026-03-16 after v1.1 milestone started* +*Last updated: 2026-03-17 after v1.2 milestone started* diff --git a/.planning/STATE.md b/.planning/STATE.md index aa5376a..4f99a8a 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,58 +1,43 @@ --- gsd_state_version: 1.0 -milestone: v1.1 -milestone_name: Architecture Refactor -status: completed -stopped_at: Completed 06-03 Gap Closure -last_updated: "2026-03-16T16:39:54.753Z" -last_activity: 2026-03-17 — 06-03 Gap closure completed +milestone: v1.2 +milestone_name: Zhihu Crawler +status: planning +stopped_at: Defining requirements +last_updated: "2026-03-17T10:00:00.000Z" +last_activity: 2026-03-17 — Milestone v1.2 started progress: - total_phases: 6 - completed_phases: 6 - total_plans: 18 - completed_plans: 18 - percent: 100 + total_phases: 0 + completed_phases: 0 + total_plans: 0 + completed_plans: 0 + percent: 0 --- # Project State ## Project Reference -See: .planning/PROJECT.md (updated 2026-03-16) +See: .planning/PROJECT.md (updated 2026-03-17) **Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复 -**Current focus:** v1.1 Architecture Refactor — Phase 6 代码整合 COMPLETE +**Current focus:** v1.2 Zhihu Crawler — 需求定义中 ## Current Position -Phase: 6 of 6 (代码整合) - COMPLETE -Plan: 3 of 3 in current phase - COMPLETE -Status: 06-03 Complete -Last activity: 2026-03-17 — 06-03 Gap closure completed +Phase: Not started (defining requirements) +Plan: — +Status: Defining requirements +Last activity: 2026-03-17 — Milestone v1.2 started -Progress: [██████████] 100% (v1.1 milestone) +Progress: [░░░░░░░░░░] 0% (v1.2 milestone) ## Performance Metrics **Velocity:** -- Total plans completed: 5 (v1.1) -- Average duration: 4 min -- Total execution time: 20 min - -**By Phase:** - -| Phase | Plans | Total | Avg/Plan | -|-------|-------|-------|----------| -| 5. 归档与目录优化 | 3/3 | 9 min | 3 min | -| 6. 代码整合 | 3/3 | 18 min | 6 min | - -**Recent Trend:** -- Last 5 plans: 5 min -- Trend: Stable - -*Updated after each plan completion* -| Phase 06 P01 | 7min | 4 tasks | 10 files | -| Phase 06 P03 | 4min | 6 tasks | 6 files | +- Total plans completed: 0 (v1.2) +- Average duration: — +- Total execution time: — ## Accumulated Context @@ -79,14 +64,20 @@ v1.1 decisions: - [Phase 06-03]: External dependencies (kb-deep-analyzer) use lazy sys.path manipulation inside methods, not at module top-level - [Phase 06-03]: Pipeline class exported at package level for cleaner imports +v1.2 decisions: +- 知乎登录策略：二维码扫码 + 手动 Cookie 降级 +- 参考 MediaCrawler 实现，自实现而非直接依赖 +- 使用 playwright 作为浏览器自动化框架 +- 整合进现有 Pipeline 架构 + ### Blockers/Concerns None. ## Session Continuity -Last session: 2026-03-16T16:35:38.000Z -Stopped at: Completed 06-03 Gap Closure +Last session: 2026-03-17T10:00:00.000Z +Stopped at: Defining requirements Resume file: None -**Next step:** v1.1 Architecture Refactor complete - all phases finished +**Next step:** Define requirements → Create roadmap