- Archive roadmap to milestones/v1.5-ROADMAP.md - Archive requirements to milestones/v1.5-REQUIREMENTS.md - Update ROADMAP.md with collapsed v1.5 entry - Update PROJECT.md with validated requirements - Add semantic segmentation to backlog Delivered: B站视频简化处理流程 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# Requirements Archive: v1.5 B站字幕简化处理
|
||||
|
||||
**Archived:** 2026-03-21
|
||||
**Status:** SHIPPED
|
||||
|
||||
For current requirements, see `.planning/REQUIREMENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
# Requirements: kb-skills v1.5
|
||||
|
||||
**Defined:** 2026-03-19
|
||||
**Core Value:** 简化 B 站视频处理流程，生成易读的 Markdown 文档
|
||||
|
||||
## v1.5 Requirements
|
||||
|
||||
### SRT Processing
|
||||
|
||||
- [x] **SRT-01**: 系统能解析 SRT 字幕文件格式
|
||||
- [x] **SRT-02**: 系统能移除时间戳，提取纯文本内容
|
||||
- [x] **SRT-03**: 系统能根据内容主题变化自动划分语义段落
|
||||
- [x] **SRT-04**: 系统能使用 LLM 根据上下文修正语音识别错误
|
||||
|
||||
### Output
|
||||
|
||||
- [x] **OUT-01**: 系统能生成结构化的 Markdown 文件
|
||||
- [x] **OUT-02**: 系统能在 Markdown 中保留视频元数据（标题、作者等）
|
||||
- [ ] **OUT-03**: 系统能保留原有的元数据处理功能（BilibiliAdapter）
|
||||
- [x] **OUT-04**: 系统能继续生成视频标签（基于元数据）
|
||||
|
||||
### Scope Change
|
||||
|
||||
- [ ] **SCOPE-01**: 移除深度观点分析功能（DeepAnalyzer）
|
||||
- [ ] **SCOPE-02**: 仅针对 B 站视频，不影响其他平台处理器
|
||||
|
||||
## Out of Scope
|
||||
|
||||
| Feature | Reason |
|
||||
|---------|--------|
|
||||
| 知乎内容处理 | 本次只改 B 站 |
|
||||
| 观点提炼 | 明确移除 |
|
||||
| 并行处理 | 不需要 |
|
||||
| 进度显示 | Deferred |
|
||||
|
||||
## Traceability
|
||||
|
||||
Which phases cover which requirements. Updated during roadmap creation.
|
||||
|
||||
| Requirement | Phase | Status |
|
||||
|-------------|-------|--------|
|
||||
| SRT-01 | Phase 12 | Complete |
|
||||
| SRT-02 | Phase 12 | Complete |
|
||||
| SRT-03 | Phase 12 | Complete |
|
||||
| SRT-04 | Phase 12 | Complete |
|
||||
| OUT-01 | Phase 12 | Complete |
|
||||
| OUT-02 | Phase 12 | Complete |
|
||||
| OUT-03 | Phase 12 | Pending |
|
||||
| OUT-04 | Phase 12 | Complete |
|
||||
| SCOPE-01 | Phase 12 | Pending |
|
||||
| SCOPE-02 | Phase 12 | Pending |
|
||||
|
||||
**Coverage:**
|
||||
- v1.5 requirements: 10 total
|
||||
- Mapped to phases: 10
|
||||
- Unmapped: 0 ✓
|
||||
|
||||
---
|
||||
*Requirements defined: 2026-03-19*
|
||||
*Last updated: 2026-03-19 after initial definition*
|
||||
@@ -0,0 +1,184 @@
|
||||
# Roadmap: kb-skills
|
||||
|
||||
## Milestones
|
||||
|
||||
- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15)
|
||||
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16)
|
||||
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17)
|
||||
- ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18)
|
||||
- ✅ **v1.4 Zhihu Anti-Crawler** - Phase 11 (shipped 2026-03-18)
|
||||
- ◆ **v1.5 B站字幕简化处理** - Phase 12 (shipped 2026-03-19)
|
||||
|
||||
## Phases
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2026-03-15</summary>
|
||||
|
||||
### Phase 1: Foundation
|
||||
**Goal**: 项目基础设施和状态管理
|
||||
**Plans**: 3 plans
|
||||
|
||||
Plans:
|
||||
- [x] 01-01: WorkflowStateManager 状态管理
|
||||
- [x] 01-02: LinkParser URL 解析
|
||||
- [x] 01-03: IdempotencyChecker 幂等检查
|
||||
|
||||
### Phase 2: Adapters
|
||||
**Goal**: 平台元数据适配器
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [x] 02-01: BilibiliAdapter B站适配器
|
||||
- [x] 02-02: ZhihuAdapter 知乎适配器
|
||||
|
||||
### Phase 3: Processors
|
||||
**Goal**: 管道处理器架构
|
||||
**Plans**: 3 plans
|
||||
|
||||
Plans:
|
||||
- [x] 03-01: BaseProcessor 基类
|
||||
- [x] 03-02: 具体处理器实现
|
||||
- [x] 03-03: Schema 验证
|
||||
|
||||
### Phase 4: Pipeline
|
||||
**Goal**: 处理器编排和 CLI
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [x] 04-01: Pipeline 编排引擎
|
||||
- [x] 04-02: CLI process 命令
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2026-03-16</summary>
|
||||
|
||||
### Phase 5: 归档与目录优化
|
||||
**Goal**: 统一中间目录结构，添加自动归档步骤，优化最终目录布局
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 6: 代码整合
|
||||
**Goal**: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17</summary>
|
||||
|
||||
### Phase 7: 登录认证体系
|
||||
**Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 8: 内容爬取实现
|
||||
**Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 9: Pipeline 整合
|
||||
**Goal**: 将新爬虫实现整合进现有 Pipeline 架构
|
||||
**Plans**: 1 plan (complete)
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.3 Proxy Pool (Phase 10) - SHIPPED 2026-03-18</summary>
|
||||
|
||||
### Phase 10: Proxy Pool (代理池管理)
|
||||
**Goal**: 建立代理池基础设施，支持多代理配置、自动轮换和健康检查
|
||||
**Requirements**: PROXY-01, PROXY-02, PROXY-03, PROXY-04
|
||||
**Plans**: 2 plans (all complete)
|
||||
|
||||
Plans:
|
||||
- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换
|
||||
- [x] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
||||
|
||||
**Key Decisions:**
|
||||
- 使用索引指针而非 itertools.cycle 实现轮换
|
||||
- 惰性冷却检查而非后台定时器
|
||||
- ProxyPool 参数优先于 config.proxy 配置
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## ✅ v1.4 Zhihu Anti-Crawler (SHIPPED 2026-03-18)
|
||||
|
||||
**Milestone Goal:** 绕过知乎反爬检测，让 Playwright 爬虫能正常工作
|
||||
|
||||
### Phase 11: Stealth Fingerprinting (指纹伪装)
|
||||
**Goal**: 隐藏 Playwright 自动化特征，让爬虫通过知乎的浏览器指纹检测
|
||||
**Depends on**: Phase 10
|
||||
**Requirements**: STEALTH-01, STEALTH-02, STEALTH-03, STEALTH-04, VERIFY-01, VERIFY-02
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. ZhihuCrawler 访问问答页面时不再被 40362 错误拦截
|
||||
2. 爬虫的 navigator.webdriver 属性对知乎不可见
|
||||
3. Playwright 特有的 JS 属性（__playwright, pwInspect 等）被移除
|
||||
4. 现有功能（登录、Cookie 持久化、内容爬取）保持正常工作
|
||||
**Plans**: 2 plans (all complete)
|
||||
|
||||
Plans:
|
||||
- [x] 11-01-PLAN.md — 创建 StealthConfig 模块和单元测试
|
||||
- [x] 11-02-PLAN.md — 集成 stealth 到 ZhihuCrawler 并验证
|
||||
|
||||
**Key Decisions:**
|
||||
- 使用 playwright-stealth 库进行指纹伪装
|
||||
- stealth_config=None 表示默认启用 stealth (向后兼容)
|
||||
- launch 时添加 --disable-blink-features=AutomationControlled 参数
|
||||
|
||||
---
|
||||
|
||||
## ✅ v1.5 B站字幕简化处理 (SHIPPED 2026-03-19)
|
||||
|
||||
**Milestone Goal:** 简化 B 站视频处理流程，基于 SRT 字幕生成易读的 Markdown 文档
|
||||
|
||||
### Phase 12: SRT 简化处理器
|
||||
**Goal**: 实现 SRT 字幕处理，生成简化的 Markdown 文档，保留元数据和标签，移除深度分析
|
||||
**Depends on**: Phase 4 (Pipeline)
|
||||
**Requirements**: SRT-01, SRT-02, SRT-03, SRT-04, OUT-01, OUT-02, OUT-03, OUT-04, SCOPE-01, SCOPE-02
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户运行 `kb-workflow process <bilibili-url>` 后生成 Markdown 文件而非深度分析报告
|
||||
2. Markdown 文件包含视频元数据（标题、作者、时长）
|
||||
3. 元数据处理功能保持不变（BilibiliAdapter 正常工作）
|
||||
4. 视频标签继续正常生成
|
||||
5. 字幕文本已去除时间戳，按语义分段
|
||||
6. 明显的语音识别错误已被 LLM 纠正
|
||||
7. DeepAnalyzer 相关代码已移除或禁用
|
||||
8. 知乎等其他平台处理器不受影响
|
||||
**Plans**: 2 plans (all complete)
|
||||
|
||||
Plans:
|
||||
- [x] 12-01-PLAN.md — SRT 处理模块（parser、segmenter、corrector）和 SRTProcessor 实现
|
||||
- [x] 12-02-PLAN.md — Pipeline 集成、平台路由、测试验证
|
||||
|
||||
**Key Decisions:**
|
||||
- SRT 处理模块独立于 DeepAnalyzer，位于 kb_workflow/srt/
|
||||
- 语义分段基于简单规则（时间间隔 + 长度限制），不依赖 LLM
|
||||
- ASR 纠错可选，AI 不可用时返回原文
|
||||
- B 站视频使用 SRTProcessor，知乎/文章继续使用 AnalyzeProcessor
|
||||
- DeepAnalyzer 保留但添加弃用警告（仅针对 B 站视频）
|
||||
|
||||
---
|
||||
|
||||
## Progress
|
||||
|
||||
**Execution Order:**
|
||||
Phases execute in numeric order: 1 → 2 → ... → 10 → 11 → 12
|
||||
|
||||
| Phase | Milestone | Plans Complete | Status | Completed |
|
||||
|-------|-----------|----------------|--------|-----------|
|
||||
| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||
| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||
| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||
| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||
| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||
| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||
| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 |
|
||||
| 10. Proxy Pool | v1.3 | 2/2 | Complete | 2026-03-18 |
|
||||
| 11. Stealth Fingerprinting | v1.4 | 2/2 | Complete | 2026-03-18 |
|
||||
| 12. SRT 简化处理器 | v1.5 | Complete | 2026-03-19 | 2026-03-19 |
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-03-19*
|
||||
Reference in New Issue
Block a user