fix/crawler-proxy-config #34

Merged
starmen1265 merged 19 commits from fix/crawler-proxy-config into main 2026-03-20 00:12:04 +08:00

19 Commits

Author SHA1 Message Date
starmen1265 04f071b3bd docs(phase-12): complete phase execution - srt-simplifier
PR Test / lint (pull_request) Successful in 2s
2026-03-20 00:07:06 +08:00
starmen1265 a04d5c1ddb docs(12-02): complete Pipeline Integration plan
Phase 12 complete:
- B站视频使用 SRTProcessor 替代 AnalyzeProcessor
- Pipeline 平台路由逻辑实现
- 24 个测试全部通过
- DeepAnalyzer 添加弃用警告

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:48:30 +08:00
starmen1265 a175a3242c fix(12-02): add deprecation warning to DeepAnalyzer
- Add logging warning in DeepAnalyzer.__init__
- Warn that Bilibili video should use SRTProcessor instead
- Keep functionality intact for Zhihu and other platforms

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:46:08 +08:00
starmen1265 98709476ab test(12-02): add Bilibili video pipeline end-to-end tests
- Test Bilibili video uses SRTProcessor (not AnalyzeProcessor)
- Test Bilibili article uses AnalyzeProcessor (not SRTProcessor)
- Test Zhihu answer uses AnalyzeProcessor
- Test Markdown output format verification

All 4 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:45:49 +08:00
starmen1265 b3b6498a71 feat(12-02): update CLI processor registration
- Import SRTProcessor from processors module
- Add SRTProcessor to processors list
- Update PipelineConfig.steps to include 'srt' step
- Pipeline will select SRTProcessor or AnalyzeProcessor based on platform

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:44:05 +08:00
starmen1265 8d41a1b324 feat(12-02): modify Pipeline for platform-specific processing flow
- Add Bilibili video detection: platform=bilibili and content_type=video
- Skip analyze step for Bilibili videos, use srt step instead
- Skip srt step for non-Bilibili-video content
- Add SRTInput import and step input creation logic
- Update _populate_skipped_step_results for srt step
- Add output_file, paragraphs, tags, corrected to result fields

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:43:39 +08:00
starmen1265 b312612e6c docs(12-01): complete SRT Processing Module plan
- Create 12-01-SUMMARY.md with implementation details
- Update STATE.md with current position (Phase 12, 50% complete)
- Update ROADMAP.md with plan progress
- Mark requirements SRT-01~04, OUT-01~02, OUT-04 as complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:41:13 +08:00
starmen1265 812f016b1b chore(12-01): update srt module exports
- Add SemanticSegmenter and ASRCorrector to __all__
- Maintain lazy loading for optional dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:39:20 +08:00
starmen1265 87b1919483 feat(12-01): implement SRTProcessor and Markdown generation
- Add SRTInput/SRTOutput schemas to pipeline
- Create SRTProcessor following BaseProcessor pattern
- Generate Markdown with YAML frontmatter
- Support metadata extraction and formatting
- Handle missing metadata gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:38:56 +08:00
starmen1265 5e00632f68 feat(12-01): implement ASRCorrector for speech recognition error correction
- Use LLM to correct obvious ASR errors (homophones, missing words)
- Batch processing for long text (5000 chars per batch)
- Fallback to original text when AI unavailable
- Simple sentence-based text splitting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:35:53 +08:00
starmen1265 757793bfa9 test(12-01): add failing tests for ASRCorrector
- Test obvious homophone error correction
- Test contextual error correction
- Test no overcorrection on correct text
- Test AI unavailable fallback
- Test batch processing for long text
- Test empty text handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:35:02 +08:00
starmen1265 a350693863 feat(12-01): implement SemanticSegmenter for subtitle segmentation
- Merge short segments into paragraphs
- Split on time gaps (>3 seconds by default)
- Enforce max paragraph length (500 chars by default)
- Simple rule-based approach (no LLM needed)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:34:18 +08:00
starmen1265 81a7dc558b test(12-01): add failing tests for SemanticSegmenter
- Test merging short segments into paragraphs
- Test segmentation on topic change (time gap)
- Test max paragraph length constraint
- Test empty segments handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:33:43 +08:00
starmen1265 c259e0ac2f feat(12-01): implement SRT parser module
- Add Segment dataclass for subtitle segments
- Add parse_srt() function to parse SRT content string
- Add parse_srt_file() function to parse from file path
- Support standard SRT format with timestamps
- Handle special characters and multi-line subtitles

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:32:57 +08:00
starmen1265 7023a57644 test(12-01): add failing tests for SRT parser module
- Test standard SRT format parsing
- Test special character handling
- Test error handling for invalid/empty content
- Test file parsing functionality

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-19 23:31:36 +08:00
starmen1265 40337940d8 docs(12-02): fix autonomous flag for checkpoint task 2026-03-19 23:28:06 +08:00
starmen1265 56039c5956 docs(12-srt-simplifier): create phase 12 plans 2026-03-19 23:25:43 +08:00
starmen1265 978613461b docs: create milestone v1.5 requirements and roadmap 2026-03-19 23:18:50 +08:00
starmen1265 d69b85cb1e docs: start milestone v1.5 B站字幕简化处理 2026-03-19 23:13:43 +08:00