169 lines
7.3 KiB
Markdown
169 lines
7.3 KiB
Markdown
---
|
||
gsd_state_version: 1.0
|
||
milestone: v1.4
|
||
milestone_name: Zhihu Anti-Crawler
|
||
status: executing
|
||
stopped_at: Completed 14-02 parse error retry and fallback
|
||
last_updated: "2026-03-23T14:39:06.904Z"
|
||
last_activity: 2026-03-22 — Phase 13 Plan 07 completed (SRTProcessor integration)
|
||
progress:
|
||
total_phases: 15
|
||
completed_phases: 11
|
||
total_plans: 44
|
||
completed_plans: 41
|
||
percent: 31
|
||
---
|
||
|
||
# Project State
|
||
|
||
## Project Reference
|
||
|
||
See: .planning/PROJECT.md (updated 2026-03-21)
|
||
|
||
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
|
||
**Current focus:** 语义分段改进 — 使用 LLM 实现智能分段
|
||
|
||
## Current Position
|
||
|
||
Phase: Phase 13 - 核心 LLM 分段功能
|
||
Plan: 07 (completed)
|
||
Status: In progress
|
||
Last activity: 2026-03-22 — Phase 13 Plan 07 completed (SRTProcessor integration)
|
||
|
||
Progress: [██░░░░░░░░░] 31% (Phase 13/15, 7/7 plans complete)
|
||
|
||
### Phase 13 Goal
|
||
实现基于 LLM 的智能语义分段，识别视频类型并生成主题标题
|
||
|
||
### Phase 13 Requirements (18 total)
|
||
- VIDEO-01, VIDEO-02, VIDEO-03 (视频类型识别) ✅
|
||
- BOUND-01, BOUND-02, BOUND-03, BOUND-04 (语义边界检测) ✅
|
||
- COMP-01, COMP-02, COMP-03 (内容完整性保护) ✅
|
||
- TITLE-01, TITLE-02, TITLE-03 (主题标题生成) ✅
|
||
- SENT-01, SENT-02, SENT-03 (中文句边界检测) ✅
|
||
- STRAT-01, STRAT-02, STRAT-03 (分段策略) ✅
|
||
|
||
## Accumulated Context
|
||
|
||
### Decisions
|
||
|
||
Key decisions from v1.0-v1.5 (preserved):
|
||
- 管道模型 — BaseProcessor + Pipeline 架构
|
||
- 步骤级状态 — WorkflowStateManager
|
||
- Schema 验证 — Pydantic v2
|
||
- 顺序执行 — 简单性和可调试性
|
||
- CLI subparsers — 可扩展命令结构
|
||
- SRT 独立模块 — kb_workflow/srt/ 职责分离
|
||
- ASR 纠错可选 — AI 不可用时返回原文
|
||
|
||
v1.6 Phase 13 decisions (from 13-CONTEXT.md):
|
||
- **数量控制**: 采用建议式数量引导（5-8段），不强制执行
|
||
- **视频类型策略**: 无类型区分 — 使用统一 Prompt，不针对不同类型设计不同策略
|
||
- **句边界保护**: 纯 Prompt 约束 — 依赖 LLM 理解，不进行代码后处理
|
||
- **论点完整性**: 一般性指令 — 不提供具体示例或标记词识别
|
||
- **并列结构**: 不特别指定 — 让 LLM 根据整体语义自然判断
|
||
- **中文句边界**: Prompt 中强调 — 列举所有中文句末标点
|
||
- **标题范围**: LLM 自主判断 — 哪些分段需要标题由 LLM 决定
|
||
- **标题风格**: 简洁描述型 — 5-15字，如「机器学习概述」
|
||
- **标题格式**: H2 (##) — 与现有格式保持一致
|
||
- **标题缺失处理**: 跳过标题 — 无法生成标题时设为 null
|
||
- **模块位置**: `kb_workflow/srt/semantic_engine.py`
|
||
- **现有 SemanticSegmenter**: 完全替代 — 新代码完全替代旧代码
|
||
- **SRTProcessor 集成**: 配置开关控制 — `enable_semantic_segmentation` 参数
|
||
- **Pydantic Schema 位置**: `semantic_engine.py` 内
|
||
- [Phase 13]: 确认 semantic_engine.py 已包含内容完整性保护指令（在之前的计划中实现）
|
||
- [Phase 13]: 遵守用户决策：不添加并列结构特殊处理指令
|
||
- [Phase 13]: 视频类型关键词映射 - 使用 VIDEO_TYPE_KEYWORDS 字典定义 5 种类型
|
||
- [Phase 13]: 默认视频类型设置为知识类
|
||
- [Phase 13]: 标题长度验证使用 Pydantic field_validator (5-15字)
|
||
- [Phase 13]: start_index/end_index 表示字符位置(0-based)不是字幕索引(1-based)
|
||
- [Phase 13]: Lazy AI Client 初始化采用 ASRCorrector 模式
|
||
- [Phase 13 Plan 06]: Strategy Pattern 架构 - SegmentationStrategy Protocol 接口定义策略契约
|
||
- [Phase 13 Plan 06]: 统一分段策略 - UnifiedSegmentationStrategy 适用于所有视频类型（遵守用户决策）
|
||
- [Phase 13 Plan 06]: 策略注入模式 - SemanticSegmentationEngine 通过构造函数注入策略，支持测试和扩展
|
||
- [Phase 13]: Strategy Pattern with Protocol interface for extensibility (STRAT-01)
|
||
- [Phase 13]: Unified segmentation strategy for all video types (per user decision, STRAT-02)
|
||
- [Phase 13]: Strategy injection via constructor for easy testing and extension (STRAT-03)
|
||
- [Phase 14-robustness-cost]: 缓存键使用 SHA256 内容哈希，确保相同内容精确匹配
|
||
- [Phase 14-robustness-cost]: 缓存文件包含 version 字段（1.0），支持未来格式变更
|
||
- [Phase 14-robustness-cost]: 缓存失败时返回 None，不中断主流程（错误容错）
|
||
- [Phase 14]: ValidationError checked before ValueError (inherits from ValueError)
|
||
- [Phase 14]: JSON parse errors (JSONDecodeError, ValueError) are recoverable - retry 1 time
|
||
- [Phase 14]: Pydantic validation errors (ValidationError) are non-recoverable - immediate fallback
|
||
- [Phase 14]: Parse failures log LLM output fragment (first 500 chars) for diagnosis
|
||
|
||
### Technical Context
|
||
|
||
**架构集成:**
|
||
- 新增 `SemanticSegmentationEngine` 模块（`kb_workflow/srt/semantic_engine.py`）
|
||
- 替代现有 `SemanticSegmenter`（规则分段）
|
||
- 在 `SRTProcessor` 中添加 LLM 分支（`enable_semantic_segmentation` 参数）
|
||
|
||
**技术栈:**
|
||
- SimpleAIClient（已集成 zhipu/openai/minimax）
|
||
- Pydantic v2（LLM 响应验证）
|
||
- Protocol（SegmentationStrategy 接口）
|
||
|
||
**数据流:**
|
||
```
|
||
SRT Input → Parse Subtitles → SemanticSegmentationEngine (LLM) →
|
||
Extract Paragraphs → ASR Correction → Generate Markdown
|
||
```
|
||
|
||
### Known Risks
|
||
|
||
**高优先级风险:**
|
||
1. **上下文窗口溢出** — 长视频（>1小时）字幕超10万字符，需分块处理
|
||
2. **LLM 输出格式不稳定** — 需多层 fallback 策略
|
||
3. **分段边界破坏完整性** — 需句边界保护机制
|
||
4. **分段不一致性** — LLM 随机性导致多次运行结果不同
|
||
5. **成本失控** — 高频 LLM 调用可能导致费用激增
|
||
|
||
**缓解策略:**
|
||
- Phase 13: 实现 Pydantic 验证 + 句边界保护（Prompt 约束）
|
||
- Phase 14: 实现分块处理 + 缓存 + 温度控制 + 降级策略
|
||
- Phase 15: 输出格式优化 + 配置开关
|
||
|
||
### Blockers/Concerns
|
||
|
||
None.
|
||
|
||
## Performance Metrics
|
||
|
||
| Phase | Plan | Duration | Tasks | Files | Date |
|
||
|-------|------|----------|-------|-------|------|
|
||
| 13 | 01 | 35min | 1 | 4 | 2026-03-22 |
|
||
| 13 | 02 | 4min | 1 | 2 | 2026-03-22 |
|
||
| 13 | 03 | 5min | 1 | 1 | 2026-03-22 |
|
||
| 13 | 04 | 2min | 1 | 2 | 2026-03-22 |
|
||
| 13 | 05 | 2min | 1 | 2 | 2026-03-22 |
|
||
| 13 | 06 | 12min | 1 | 2 | 2026-03-22 |
|
||
| 13 | 07 | 4min | 1 | 4 | 2026-03-22 |
|
||
| Phase 14-robustness-cost P03 | 72 | 1 tasks | 3 files |
|
||
| Phase 14-robustness-cost P00 | 3min | 3 tasks | 3 files |
|
||
| Phase 14-robustness-cost P01 | 219 | 1 tasks | 2 files |
|
||
| Phase 14 P02 | 220 | 1 tasks | 2 files |
|
||
|
||
## Session Continuity
|
||
|
||
Last session: 2026-03-23T14:39:06.903Z
|
||
Stopped at: Completed 14-02 parse error retry and fallback
|
||
Resume file: None
|
||
|
||
**Next step:** Phase 13 complete, continue to Phase 14 (鲁棒性) or Phase 15 (优化)
|
||
|
||
## Phase 13 Plan Structure
|
||
|
||
**Wave 1** (并行执行，无依赖):
|
||
- Plan 01: 视频类型识别（TDD）- VIDEO-01, VIDEO-02, VIDEO-03
|
||
- Plan 02: 语义边界检测（TDD）- BOUND-01, BOUND-02, BOUND-03, BOUND-04
|
||
- Plan 03: 内容完整性保护（TDD）- COMP-01, COMP-02, COMP-03
|
||
- Plan 04: 主题标题生成（TDD）- TITLE-01, TITLE-02, TITLE-03
|
||
- Plan 05: 中文句边界检测（TDD）- SENT-01, SENT-02, SENT-03
|
||
|
||
**Wave 2** (依赖 Wave 1):
|
||
- Plan 06: 可扩展策略架构 - STRAT-01, STRAT-02, STRAT-03
|
||
- Plan 07: SRTProcessor 集成 - CONFIG-03
|
||
|
||
**Total**: 7 plans (5 TDD + 2 execute)
|