docs: start milestone v1.2 Zhihu Crawler
This commit is contained in:
+31
-14
@@ -1,15 +1,16 @@
|
|||||||
# kb-skills 工作流重构
|
# kb-skills 工作流重构
|
||||||
|
|
||||||
## Current Milestone: v1.1 Architecture Refactor
|
## Current Milestone: v1.2 Zhihu Crawler
|
||||||
|
|
||||||
**Goal:** 简化代码结构和优化文件组织 — 统一中间目录、添加归档步骤、整合元数据技能和 Pipeline 模块
|
**Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题,支持自动登录维持
|
||||||
|
|
||||||
**Target features:**
|
**Target features:**
|
||||||
- 中间目录统一到 `~/work/intermediate/analysis/`
|
- 二维码扫码登录 + 手动 Cookie 降级
|
||||||
- 分析完成后自动归档到最终目录
|
- Cookie 持久化存储
|
||||||
- 最终目录按平台分目录,文件前缀加时间戳
|
- 问答/专栏/文章内容爬取
|
||||||
- kb-bilibili-metadata + kb-zhihu-metadata 整合进 kb-workflow
|
- 评论爬取支持
|
||||||
- kb-pipeline 合并进 kb-workflow
|
- 代理 IP 支持
|
||||||
|
- 整合进现有 Pipeline
|
||||||
|
|
||||||
## What This Is
|
## What This Is
|
||||||
|
|
||||||
@@ -28,6 +29,13 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
|||||||
|
|
||||||
### Validated
|
### Validated
|
||||||
|
|
||||||
|
**v1.1 Architecture Refactor (2026-03-16):**
|
||||||
|
- ✓ **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/`
|
||||||
|
- ✓ **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录
|
||||||
|
- ✓ **REF-03**: 最终目录结构 — 按平台分目录,文件前缀加时间戳
|
||||||
|
- ✓ **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow
|
||||||
|
- ✓ **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow
|
||||||
|
|
||||||
**v1.0 管道架构 (2026-03-15):**
|
**v1.0 管道架构 (2026-03-15):**
|
||||||
- ✓ **PIPE-01**: 统一的管道模型 — `kb_pipeline/pipeline.py`
|
- ✓ **PIPE-01**: 统一的管道模型 — `kb_pipeline/pipeline.py`
|
||||||
- ✓ **PIPE-02**: 步骤级状态管理 — `WorkflowStateManager`
|
- ✓ **PIPE-02**: 步骤级状态管理 — `WorkflowStateManager`
|
||||||
@@ -56,12 +64,15 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
|||||||
|
|
||||||
### Active
|
### Active
|
||||||
|
|
||||||
**v1.1 Architecture Refactor:**
|
**v1.2 Zhihu Crawler:**
|
||||||
- [ ] **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/`
|
- [ ] **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码,用户扫码
|
||||||
- [ ] **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录
|
- [ ] **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie
|
||||||
- [ ] **REF-03**: 最终目录结构 — 按平台分目录,文件前缀加时间戳
|
- [ ] **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie,自动复用
|
||||||
- [ ] **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow
|
- [ ] **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据
|
||||||
- [ ] **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow
|
- [ ] **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据
|
||||||
|
- [ ] **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论
|
||||||
|
- [ ] **ZHIHU-07**: 代理 IP 支持 — 降低被封风险
|
||||||
|
- [ ] **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器
|
||||||
|
|
||||||
### Deferred
|
### Deferred
|
||||||
|
|
||||||
@@ -69,6 +80,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
|||||||
- [ ] **ADV-02**: 并行步骤执行 — 支持独立步骤的并行执行
|
- [ ] **ADV-02**: 并行步骤执行 — 支持独立步骤的并行执行
|
||||||
- [ ] **ADV-03**: 说话者分离默认启用 — HF_TOKEN 配置后自动使用
|
- [ ] **ADV-03**: 说话者分离默认启用 — HF_TOKEN 配置后自动使用
|
||||||
- [ ] **EXT-01**: 微信公众号处理 — 支持微信公众号文章
|
- [ ] **EXT-01**: 微信公众号处理 — 支持微信公众号文章
|
||||||
|
- [ ] **ZHIHU-F01**: 问题下所有回答 — 爬取问题的所有热门回答
|
||||||
|
- [ ] **ZHIHU-F02**: 作者全部内容 — 爬取某作者的所有文章/回答
|
||||||
|
|
||||||
### Out of Scope
|
### Out of Scope
|
||||||
|
|
||||||
@@ -77,6 +90,10 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
|||||||
- 用户界面 — 命令行运行
|
- 用户界面 — 命令行运行
|
||||||
- 多语言支持 — 中文即可
|
- 多语言支持 — 中文即可
|
||||||
- 小红书 — 当前不支持
|
- 小红书 — 当前不支持
|
||||||
|
- 知乎会员付费内容 — 不爬取付费内容
|
||||||
|
- 知乎视频 — v1.2 不支持
|
||||||
|
- 大规模爬取 — 仅用于个人知识库
|
||||||
|
- 付费打码服务 — 使用二维码登录替代
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -116,4 +133,4 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
|||||||
| 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 |
|
| 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 |
|
||||||
|
|
||||||
---
|
---
|
||||||
*Last updated: 2026-03-16 after v1.1 milestone started*
|
*Last updated: 2026-03-17 after v1.2 milestone started*
|
||||||
|
|||||||
+30
-39
@@ -1,58 +1,43 @@
|
|||||||
---
|
---
|
||||||
gsd_state_version: 1.0
|
gsd_state_version: 1.0
|
||||||
milestone: v1.1
|
milestone: v1.2
|
||||||
milestone_name: Architecture Refactor
|
milestone_name: Zhihu Crawler
|
||||||
status: completed
|
status: planning
|
||||||
stopped_at: Completed 06-03 Gap Closure
|
stopped_at: Defining requirements
|
||||||
last_updated: "2026-03-16T16:39:54.753Z"
|
last_updated: "2026-03-17T10:00:00.000Z"
|
||||||
last_activity: 2026-03-17 — 06-03 Gap closure completed
|
last_activity: 2026-03-17 — Milestone v1.2 started
|
||||||
progress:
|
progress:
|
||||||
total_phases: 6
|
total_phases: 0
|
||||||
completed_phases: 6
|
completed_phases: 0
|
||||||
total_plans: 18
|
total_plans: 0
|
||||||
completed_plans: 18
|
completed_plans: 0
|
||||||
percent: 100
|
percent: 0
|
||||||
---
|
---
|
||||||
|
|
||||||
# Project State
|
# Project State
|
||||||
|
|
||||||
## Project Reference
|
## Project Reference
|
||||||
|
|
||||||
See: .planning/PROJECT.md (updated 2026-03-16)
|
See: .planning/PROJECT.md (updated 2026-03-17)
|
||||||
|
|
||||||
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
||||||
**Current focus:** v1.1 Architecture Refactor — Phase 6 代码整合 COMPLETE
|
**Current focus:** v1.2 Zhihu Crawler — 需求定义中
|
||||||
|
|
||||||
## Current Position
|
## Current Position
|
||||||
|
|
||||||
Phase: 6 of 6 (代码整合) - COMPLETE
|
Phase: Not started (defining requirements)
|
||||||
Plan: 3 of 3 in current phase - COMPLETE
|
Plan: —
|
||||||
Status: 06-03 Complete
|
Status: Defining requirements
|
||||||
Last activity: 2026-03-17 — 06-03 Gap closure completed
|
Last activity: 2026-03-17 — Milestone v1.2 started
|
||||||
|
|
||||||
Progress: [██████████] 100% (v1.1 milestone)
|
Progress: [░░░░░░░░░░] 0% (v1.2 milestone)
|
||||||
|
|
||||||
## Performance Metrics
|
## Performance Metrics
|
||||||
|
|
||||||
**Velocity:**
|
**Velocity:**
|
||||||
- Total plans completed: 5 (v1.1)
|
- Total plans completed: 0 (v1.2)
|
||||||
- Average duration: 4 min
|
- Average duration: —
|
||||||
- Total execution time: 20 min
|
- Total execution time: —
|
||||||
|
|
||||||
**By Phase:**
|
|
||||||
|
|
||||||
| Phase | Plans | Total | Avg/Plan |
|
|
||||||
|-------|-------|-------|----------|
|
|
||||||
| 5. 归档与目录优化 | 3/3 | 9 min | 3 min |
|
|
||||||
| 6. 代码整合 | 3/3 | 18 min | 6 min |
|
|
||||||
|
|
||||||
**Recent Trend:**
|
|
||||||
- Last 5 plans: 5 min
|
|
||||||
- Trend: Stable
|
|
||||||
|
|
||||||
*Updated after each plan completion*
|
|
||||||
| Phase 06 P01 | 7min | 4 tasks | 10 files |
|
|
||||||
| Phase 06 P03 | 4min | 6 tasks | 6 files |
|
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -79,14 +64,20 @@ v1.1 decisions:
|
|||||||
- [Phase 06-03]: External dependencies (kb-deep-analyzer) use lazy sys.path manipulation inside methods, not at module top-level
|
- [Phase 06-03]: External dependencies (kb-deep-analyzer) use lazy sys.path manipulation inside methods, not at module top-level
|
||||||
- [Phase 06-03]: Pipeline class exported at package level for cleaner imports
|
- [Phase 06-03]: Pipeline class exported at package level for cleaner imports
|
||||||
|
|
||||||
|
v1.2 decisions:
|
||||||
|
- 知乎登录策略:二维码扫码 + 手动 Cookie 降级
|
||||||
|
- 参考 MediaCrawler 实现,自实现而非直接依赖
|
||||||
|
- 使用 playwright 作为浏览器自动化框架
|
||||||
|
- 整合进现有 Pipeline 架构
|
||||||
|
|
||||||
### Blockers/Concerns
|
### Blockers/Concerns
|
||||||
|
|
||||||
None.
|
None.
|
||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-03-16T16:35:38.000Z
|
Last session: 2026-03-17T10:00:00.000Z
|
||||||
Stopped at: Completed 06-03 Gap Closure
|
Stopped at: Defining requirements
|
||||||
Resume file: None
|
Resume file: None
|
||||||
|
|
||||||
**Next step:** v1.1 Architecture Refactor complete - all phases finished
|
**Next step:** Define requirements → Create roadmap
|
||||||
|
|||||||
Reference in New Issue
Block a user