docs: start milestone v1.2 Zhihu Crawler
This commit is contained in:
+31
-14
@@ -1,15 +1,16 @@
|
||||
# kb-skills 工作流重构
|
||||
|
||||
## Current Milestone: v1.1 Architecture Refactor
|
||||
## Current Milestone: v1.2 Zhihu Crawler
|
||||
|
||||
**Goal:** 简化代码结构和优化文件组织 — 统一中间目录、添加归档步骤、整合元数据技能和 Pipeline 模块
|
||||
**Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题,支持自动登录维持
|
||||
|
||||
**Target features:**
|
||||
- 中间目录统一到 `~/work/intermediate/analysis/`
|
||||
- 分析完成后自动归档到最终目录
|
||||
- 最终目录按平台分目录,文件前缀加时间戳
|
||||
- kb-bilibili-metadata + kb-zhihu-metadata 整合进 kb-workflow
|
||||
- kb-pipeline 合并进 kb-workflow
|
||||
- 二维码扫码登录 + 手动 Cookie 降级
|
||||
- Cookie 持久化存储
|
||||
- 问答/专栏/文章内容爬取
|
||||
- 评论爬取支持
|
||||
- 代理 IP 支持
|
||||
- 整合进现有 Pipeline
|
||||
|
||||
## What This Is
|
||||
|
||||
@@ -28,6 +29,13 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
|
||||
### Validated
|
||||
|
||||
**v1.1 Architecture Refactor (2026-03-16):**
|
||||
- ✓ **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/`
|
||||
- ✓ **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录
|
||||
- ✓ **REF-03**: 最终目录结构 — 按平台分目录,文件前缀加时间戳
|
||||
- ✓ **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow
|
||||
- ✓ **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow
|
||||
|
||||
**v1.0 管道架构 (2026-03-15):**
|
||||
- ✓ **PIPE-01**: 统一的管道模型 — `kb_pipeline/pipeline.py`
|
||||
- ✓ **PIPE-02**: 步骤级状态管理 — `WorkflowStateManager`
|
||||
@@ -56,12 +64,15 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
|
||||
### Active
|
||||
|
||||
**v1.1 Architecture Refactor:**
|
||||
- [ ] **REF-01**: 中间目录调整 — 统一到 `~/work/intermediate/analysis/`
|
||||
- [ ] **REF-02**: 归档步骤 — 分析完成后自动归档到最终目录
|
||||
- [ ] **REF-03**: 最终目录结构 — 按平台分目录,文件前缀加时间戳
|
||||
- [ ] **REF-04**: 元数据技能整合 — kb-bilibili-metadata + kb-zhihu-metadata → kb-workflow
|
||||
- [ ] **REF-05**: Pipeline 整合 — kb-pipeline 合并进 kb-workflow
|
||||
**v1.2 Zhihu Crawler:**
|
||||
- [ ] **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码,用户扫码
|
||||
- [ ] **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie
|
||||
- [ ] **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie,自动复用
|
||||
- [ ] **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据
|
||||
- [ ] **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据
|
||||
- [ ] **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论
|
||||
- [ ] **ZHIHU-07**: 代理 IP 支持 — 降低被封风险
|
||||
- [ ] **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器
|
||||
|
||||
### Deferred
|
||||
|
||||
@@ -69,6 +80,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
- [ ] **ADV-02**: 并行步骤执行 — 支持独立步骤的并行执行
|
||||
- [ ] **ADV-03**: 说话者分离默认启用 — HF_TOKEN 配置后自动使用
|
||||
- [ ] **EXT-01**: 微信公众号处理 — 支持微信公众号文章
|
||||
- [ ] **ZHIHU-F01**: 问题下所有回答 — 爬取问题的所有热门回答
|
||||
- [ ] **ZHIHU-F02**: 作者全部内容 — 爬取某作者的所有文章/回答
|
||||
|
||||
### Out of Scope
|
||||
|
||||
@@ -77,6 +90,10 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
- 用户界面 — 命令行运行
|
||||
- 多语言支持 — 中文即可
|
||||
- 小红书 — 当前不支持
|
||||
- 知乎会员付费内容 — 不爬取付费内容
|
||||
- 知乎视频 — v1.2 不支持
|
||||
- 大规模爬取 — 仅用于个人知识库
|
||||
- 付费打码服务 — 使用二维码登录替代
|
||||
|
||||
## Context
|
||||
|
||||
@@ -116,4 +133,4 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
| 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 |
|
||||
|
||||
---
|
||||
*Last updated: 2026-03-16 after v1.1 milestone started*
|
||||
*Last updated: 2026-03-17 after v1.2 milestone started*
|
||||
|
||||
+30
-39
@@ -1,58 +1,43 @@
|
||||
---
|
||||
gsd_state_version: 1.0
|
||||
milestone: v1.1
|
||||
milestone_name: Architecture Refactor
|
||||
status: completed
|
||||
stopped_at: Completed 06-03 Gap Closure
|
||||
last_updated: "2026-03-16T16:39:54.753Z"
|
||||
last_activity: 2026-03-17 — 06-03 Gap closure completed
|
||||
milestone: v1.2
|
||||
milestone_name: Zhihu Crawler
|
||||
status: planning
|
||||
stopped_at: Defining requirements
|
||||
last_updated: "2026-03-17T10:00:00.000Z"
|
||||
last_activity: 2026-03-17 — Milestone v1.2 started
|
||||
progress:
|
||||
total_phases: 6
|
||||
completed_phases: 6
|
||||
total_plans: 18
|
||||
completed_plans: 18
|
||||
percent: 100
|
||||
total_phases: 0
|
||||
completed_phases: 0
|
||||
total_plans: 0
|
||||
completed_plans: 0
|
||||
percent: 0
|
||||
---
|
||||
|
||||
# Project State
|
||||
|
||||
## Project Reference
|
||||
|
||||
See: .planning/PROJECT.md (updated 2026-03-16)
|
||||
See: .planning/PROJECT.md (updated 2026-03-17)
|
||||
|
||||
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
||||
**Current focus:** v1.1 Architecture Refactor — Phase 6 代码整合 COMPLETE
|
||||
**Current focus:** v1.2 Zhihu Crawler — 需求定义中
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 6 of 6 (代码整合) - COMPLETE
|
||||
Plan: 3 of 3 in current phase - COMPLETE
|
||||
Status: 06-03 Complete
|
||||
Last activity: 2026-03-17 — 06-03 Gap closure completed
|
||||
Phase: Not started (defining requirements)
|
||||
Plan: —
|
||||
Status: Defining requirements
|
||||
Last activity: 2026-03-17 — Milestone v1.2 started
|
||||
|
||||
Progress: [██████████] 100% (v1.1 milestone)
|
||||
Progress: [░░░░░░░░░░] 0% (v1.2 milestone)
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
**Velocity:**
|
||||
- Total plans completed: 5 (v1.1)
|
||||
- Average duration: 4 min
|
||||
- Total execution time: 20 min
|
||||
|
||||
**By Phase:**
|
||||
|
||||
| Phase | Plans | Total | Avg/Plan |
|
||||
|-------|-------|-------|----------|
|
||||
| 5. 归档与目录优化 | 3/3 | 9 min | 3 min |
|
||||
| 6. 代码整合 | 3/3 | 18 min | 6 min |
|
||||
|
||||
**Recent Trend:**
|
||||
- Last 5 plans: 5 min
|
||||
- Trend: Stable
|
||||
|
||||
*Updated after each plan completion*
|
||||
| Phase 06 P01 | 7min | 4 tasks | 10 files |
|
||||
| Phase 06 P03 | 4min | 6 tasks | 6 files |
|
||||
- Total plans completed: 0 (v1.2)
|
||||
- Average duration: —
|
||||
- Total execution time: —
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -79,14 +64,20 @@ v1.1 decisions:
|
||||
- [Phase 06-03]: External dependencies (kb-deep-analyzer) use lazy sys.path manipulation inside methods, not at module top-level
|
||||
- [Phase 06-03]: Pipeline class exported at package level for cleaner imports
|
||||
|
||||
v1.2 decisions:
|
||||
- 知乎登录策略:二维码扫码 + 手动 Cookie 降级
|
||||
- 参考 MediaCrawler 实现,自实现而非直接依赖
|
||||
- 使用 playwright 作为浏览器自动化框架
|
||||
- 整合进现有 Pipeline 架构
|
||||
|
||||
### Blockers/Concerns
|
||||
|
||||
None.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-03-16T16:35:38.000Z
|
||||
Stopped at: Completed 06-03 Gap Closure
|
||||
Last session: 2026-03-17T10:00:00.000Z
|
||||
Stopped at: Defining requirements
|
||||
Resume file: None
|
||||
|
||||
**Next step:** v1.1 Architecture Refactor complete - all phases finished
|
||||
**Next step:** Define requirements → Create roadmap
|
||||
|
||||
Reference in New Issue
Block a user