Commit Graph

428 Commits

Author SHA1 Message Date
starmen1265 adc1aa3b1d test(13-03): add content completeness protection tests
- 添加 TestSentenceBoundaryProtection 测试类(COMP-01)
  - test_sentence_boundary_protection: 验证分段边界在标点符号后
  - test_sentence_boundary_in_prompt: 验证 Prompt 包含句边界指令

- 添加 TestArgumentCompleteness 测试类(COMP-02)
  - test_argument_completeness: 验证论点结构完整性
  - test_argument_completeness_in_prompt: 验证 Prompt 包含论点完整性指令

- 添加 TestParallelStructurePreservation 测试类(COMP-03)
  - test_parallel_structure_preserved: 验证并列结构不被错误拆分
  - test_no_special_handling_for_parallel_in_prompt: 验证遵守用户决策(不特别指定并列结构处理)

所有测试通过(29/29),semantic_engine.py 已包含所需的内容完整性保护指令:
- 第 337 行:保持句子完整性,列举所有中文句末标点(。?!……)
- 第 340 行:保持论点完整性,不在前提-结论结构中间截断
- 不包含并列结构特殊处理(符合用户决策)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:26:14 +08:00
starmen1265 759503f48a feat(13-05): 实现中文句边界检测指令
更新 SEGMENTATION_PROMPT,增强句边界检测:
- 列举所有中文句末标点(。?!……)
- 强调「自然停顿处」和「语义停顿」
- 说明「不仅依赖标点符号」

添加测试验证句边界检测效果:
- test_chinese_sentence_boundary_punctuations
- test_prompt_emphasizes_semantic_pause
- test_prompt_not_only_punctuation
- test_sentence_boundary_in_segmentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:25:40 +08:00
starmen1265 321b097492 docs(13-04): complete plan summary and state updates
- Create 13-04-SUMMARY.md with execution details
- Update STATE.md with position and performance metrics
- Update ROADMAP.md with plan progress (1/7 complete)
- Mark TITLE-01, TITLE-02, TITLE-03 requirements complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:25:37 +08:00
starmen1265 00319f523f feat(13-01): implement video type detection and Pydantic models
- 创建 SemanticSegmentationEngine 类
- 实现 VideoSegment 和 SegmentationResult Pydantic 模型
  - VideoSegment: title(可选), content(必填), start_index, end_index
  - SegmentationResult: video_type, segments[], actual_segment_count
  - 添加标题长度验证器(5-15字)
- 实现 _infer_video_type() 方法
  - 从元数据推断视频类型(知识类/故事类/访谈类/新闻类/vlog)
  - 支持从 title, category, tags 推断
  - 默认类型为"知识类"
- 实现 _build_prompt() 方法
  - 包含视频类型信息
  - 包含完整的分段要求(数量控制、句边界保护、标题生成等)
  - 包含 JSON 输出格式规范
- 添加 extract_json_from_llm_response() 工具函数
- 添加 AIClientProtocol 协议定义
- 添加 deferred-items.md 记录超出范围的测试

所有计划 13-01 测试通过(10个测试)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:25:21 +08:00
starmen1265 8c2b3e079a feat(13-04): 实现主题标题生成功能
实现 VideoSegment.title 字段和验证,Prompt 中的标题生成指令。

功能:
- VideoSegment.title: Optional[str] 字段
- @field_validator 验证标题长度(不超过 15 字)
- SEGMENTATION_PROMPT 包含标题生成指令
- LLM 自主判断是否需要标题(主题/章节有标题,论点/观点无标题)
- 测试验证标题生成和省略

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:24:58 +08:00
starmen1265 9862da7085 test(13-01): add failing tests for video type detection
- 创建 conftest.py 共享 fixtures(MockAIClient, sample_segments)
- 创建 test_semantic_engine.py 包含 8 个测试用例
  - test_video_type_detection_from_title: 测试从标题推断类型
  - test_video_type_detection_from_category: 测试从分类推断类型
  - test_video_type_fallback_to_default: 测试默认类型回退
  - test_video_type_in_prompt: 测试类型出现在 Prompt 中
  - test_prompt_format: 测试 Prompt 格式
  - test_video_segment_model: 测试 VideoSegment 模型
  - test_segmentation_result_model: 测试 SegmentationResult 模型
  - test_model_validation: 测试字段验证

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-22 00:23:40 +08:00
starmen1265 47cd84b30a docs(state): record phase 13 planning complete 2026-03-22 00:21:40 +08:00
starmen1265 2bcaf05d52 docs(13): create phase 13 plan (7 plans) 2026-03-22 00:21:30 +08:00
starmen1265 da4b2d563a docs(13): research phase 13 LLM semantic segmentation 2026-03-22 00:19:23 +08:00
starmen1265 eecaccf288 docs(state): record phase 13 context session 2026-03-22 00:14:08 +08:00
starmen1265 6ae74f423d docs(13): capture phase context 2026-03-22 00:14:06 +08:00
starmen1265 ea2d6e4740 docs: create milestone v1.6 roadmap (3 phases, 30 requirements) 2026-03-22 00:02:53 +08:00
starmen1265 6279ab4268 docs: define milestone v1.6 requirements (30 requirements) 2026-03-21 23:58:15 +08:00
starmen1265 440c885448 docs: complete v1.6 semantic segmentation research 2026-03-21 23:53:55 +08:00
starmen1265 cf858027d4 docs: start milestone v1.6 语义分段改进 2026-03-21 23:43:58 +08:00
starmen1265 c8dcc3e09b chore: complete v1.5 milestone - B站字幕简化处理
PR Test / lint (pull_request) Successful in 2s
- Archive roadmap to milestones/v1.5-ROADMAP.md
- Archive requirements to milestones/v1.5-REQUIREMENTS.md
- Update ROADMAP.md with collapsed v1.5 entry
- Update PROJECT.md with validated requirements
- Add semantic segmentation to backlog

Delivered: B站视频简化处理流程

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-21 23:24:59 +08:00
starmen1265 d039537f1d docs(backlog): add semantic segmentation requirement
Add requirement for improving SRT semantic segmentation:
- Current rule-based approach cannot identify topic transitions
- Document 4 potential solutions with trade-offs
- Recommended approach: two-stage with optional LLM

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v1.5
2026-03-21 23:20:19 +08:00
starmen1265 a2ca45523b fix(srt): add punctuation requirement to ASR correction prompt
- Update CORRECTION_PROMPT to explicitly require adding punctuation
- Fix missing punctuation in thunderstorm and lightning sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-21 23:20:07 +08:00
starmen1265 0752f8b38f debug: add API response logging to investigate content vs reasoning_content
PR Test / lint (pull_request) Successful in 2s
2026-03-21 22:47:15 +08:00
starmen1265 4d2645876b fix(srt): add disable_reasoning parameter for ASR correction and include title in archive filename
PR Test / lint (pull_request) Successful in 1s
- SimpleAIClient: Add disable_reasoning parameter to control AI thinking mode
- ASRCorrector: Use disable_reasoning=True to get direct correction results
- ArchiveProcessor: Extract title from frontmatter and include in filename

File naming format: {YYYYMMDD}_{content_id}_{title}_analysis.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-21 22:19:14 +08:00
starmen1265 9e007432eb docs: resolve debug deep-analyzer-wrong-output
PR Test / lint (pull_request) Successful in 3s
2026-03-21 21:41:30 +08:00
starmen1265 7ac4937c48 fix(deep-analyzer, srt): correct Zhipu API thinking parameter and fix metadata handling
Root cause:
1. thinking.type="disabled" caused ASR correction accuracy degradation
2. SRTProcessor didn't flatten nested metadata structure
3. Low-quality auto-generated summary section
4. duration field type handling issue

Changes:
- DeepAnalyzer: Remove thinking.type="disabled" to enable default thinking mode
- SRTProcessor: Add metadata flattening for nested structures
- SRTProcessor: Remove low-quality summary section generation
- SRTProcessor: Add duration type checking (string or int)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-21 21:41:29 +08:00
starmen1265 1f599f0c60 fix: correct Zhipu API thinking parameter and add non-interactive archive mode
PR Test / lint (pull_request) Successful in 2s
2026-03-20 00:44:41 +08:00
starmen1265 7903690882 fix(deep-analyzer): add disable_reasoning parameter to GLM API calls
PR Test / lint (pull_request) Successful in 1s
2026-03-20 00:28:33 +08:00
starmen1265 fbdfc2d68a fix(12-02): add SRTProcessor to processors/__init__.py exports
PR Test / lint (pull_request) Successful in 1s
2026-03-20 00:16:53 +08:00
starmen1265 04f071b3bd docs(phase-12): complete phase execution - srt-simplifier
PR Test / lint (pull_request) Successful in 2s
2026-03-20 00:07:06 +08:00
starmen1265 a04d5c1ddb docs(12-02): complete Pipeline Integration plan
Phase 12 complete:
- B站视频使用 SRTProcessor 替代 AnalyzeProcessor
- Pipeline 平台路由逻辑实现
- 24 个测试全部通过
- DeepAnalyzer 添加弃用警告

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:48:30 +08:00
starmen1265 a175a3242c fix(12-02): add deprecation warning to DeepAnalyzer
- Add logging warning in DeepAnalyzer.__init__
- Warn that Bilibili video should use SRTProcessor instead
- Keep functionality intact for Zhihu and other platforms

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:46:08 +08:00
starmen1265 98709476ab test(12-02): add Bilibili video pipeline end-to-end tests
- Test Bilibili video uses SRTProcessor (not AnalyzeProcessor)
- Test Bilibili article uses AnalyzeProcessor (not SRTProcessor)
- Test Zhihu answer uses AnalyzeProcessor
- Test Markdown output format verification

All 4 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:45:49 +08:00
starmen1265 b3b6498a71 feat(12-02): update CLI processor registration
- Import SRTProcessor from processors module
- Add SRTProcessor to processors list
- Update PipelineConfig.steps to include 'srt' step
- Pipeline will select SRTProcessor or AnalyzeProcessor based on platform

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:44:05 +08:00
starmen1265 8d41a1b324 feat(12-02): modify Pipeline for platform-specific processing flow
- Add Bilibili video detection: platform=bilibili and content_type=video
- Skip analyze step for Bilibili videos, use srt step instead
- Skip srt step for non-Bilibili-video content
- Add SRTInput import and step input creation logic
- Update _populate_skipped_step_results for srt step
- Add output_file, paragraphs, tags, corrected to result fields

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:43:39 +08:00
starmen1265 b312612e6c docs(12-01): complete SRT Processing Module plan
- Create 12-01-SUMMARY.md with implementation details
- Update STATE.md with current position (Phase 12, 50% complete)
- Update ROADMAP.md with plan progress
- Mark requirements SRT-01~04, OUT-01~02, OUT-04 as complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:41:13 +08:00
starmen1265 812f016b1b chore(12-01): update srt module exports
- Add SemanticSegmenter and ASRCorrector to __all__
- Maintain lazy loading for optional dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:39:20 +08:00
starmen1265 87b1919483 feat(12-01): implement SRTProcessor and Markdown generation
- Add SRTInput/SRTOutput schemas to pipeline
- Create SRTProcessor following BaseProcessor pattern
- Generate Markdown with YAML frontmatter
- Support metadata extraction and formatting
- Handle missing metadata gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:38:56 +08:00
starmen1265 5e00632f68 feat(12-01): implement ASRCorrector for speech recognition error correction
- Use LLM to correct obvious ASR errors (homophones, missing words)
- Batch processing for long text (5000 chars per batch)
- Fallback to original text when AI unavailable
- Simple sentence-based text splitting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:35:53 +08:00
starmen1265 757793bfa9 test(12-01): add failing tests for ASRCorrector
- Test obvious homophone error correction
- Test contextual error correction
- Test no overcorrection on correct text
- Test AI unavailable fallback
- Test batch processing for long text
- Test empty text handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:35:02 +08:00
starmen1265 a350693863 feat(12-01): implement SemanticSegmenter for subtitle segmentation
- Merge short segments into paragraphs
- Split on time gaps (>3 seconds by default)
- Enforce max paragraph length (500 chars by default)
- Simple rule-based approach (no LLM needed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:34:18 +08:00
starmen1265 81a7dc558b test(12-01): add failing tests for SemanticSegmenter
- Test merging short segments into paragraphs
- Test segmentation on topic change (time gap)
- Test max paragraph length constraint
- Test empty segments handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:33:43 +08:00
starmen1265 c259e0ac2f feat(12-01): implement SRT parser module
- Add Segment dataclass for subtitle segments
- Add parse_srt() function to parse SRT content string
- Add parse_srt_file() function to parse from file path
- Support standard SRT format with timestamps
- Handle special characters and multi-line subtitles

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:32:57 +08:00
starmen1265 7023a57644 test(12-01): add failing tests for SRT parser module
- Test standard SRT format parsing
- Test special character handling
- Test error handling for invalid/empty content
- Test file parsing functionality

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:31:36 +08:00
starmen1265 40337940d8 docs(12-02): fix autonomous flag for checkpoint task 2026-03-19 23:28:06 +08:00
starmen1265 56039c5956 docs(12-srt-simplifier): create phase 12 plans 2026-03-19 23:25:43 +08:00
starmen1265 978613461b docs: create milestone v1.5 requirements and roadmap 2026-03-19 23:18:50 +08:00
starmen1265 d69b85cb1e docs: start milestone v1.5 B站字幕简化处理 2026-03-19 23:13:43 +08:00
starmen1265 7915808413 fix(cli): use json mode for model_dump serialization
PR Test / lint (pull_request) Successful in 2s
Ensure Pydantic model serializes to JSON-compatible format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 22:59:36 +08:00
starmen1265 fa46062698 chore: archive v1.4 milestone - clean up moved files
PR Test / lint (pull_request) Successful in 2s
- Delete .planning/REQUIREMENTS.md (archived to milestones/)
- Delete .planning/phases/11-stealth-fingerprinting/* (moved to milestones/v1.4-phases/)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 00:05:27 +08:00
starmen1265 0e558d2e3d chore: archive v1.4 phase directories 2026-03-19 00:05:03 +08:00
starmen1265 a58fbfb4a7 chore: complete v1.4 Zhihu Anti-Crawler milestone v1.4 2026-03-19 00:04:04 +08:00
starmen1265 679b7d1440 docs(phase-11): complete phase execution - stealth fingerprinting 2026-03-18 23:59:31 +08:00
starmen1265 b81eec4284 docs(11-02): complete stealth integration plan
- Created 11-02-SUMMARY.md with execution results
- Updated STATE.md: v1.4 milestone complete, 100% progress
- Updated ROADMAP.md: Phase 11 complete, all plans done
- Updated REQUIREMENTS.md: VERIFY-01, VERIFY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 23:56:09 +08:00