diff --git a/.planning/STATE.md b/.planning/STATE.md index 8600754..d9e9ecc 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,13 +5,13 @@ milestone_name: milestone current_plan: 3 status: executing stopped_at: Phase 3 context gathered -last_updated: "2026-03-13T10:50:04.038Z" +last_updated: "2026-03-13T11:29:09.319Z" last_activity: 2026-03-13 progress: total_phases: 4 - completed_phases: 2 - total_plans: 6 - completed_plans: 6 + completed_phases: 3 + total_plans: 7 + completed_plans: 7 percent: 8 --- diff --git a/.planning/phases/03-processing-skills/03-01-SUMMARY.md b/.planning/phases/03-processing-skills/03-01-SUMMARY.md new file mode 100644 index 0000000..f8aed32 --- /dev/null +++ b/.planning/phases/03-processing-skills/03-01-SUMMARY.md @@ -0,0 +1,158 @@ +--- +phase: 03-processing-skills +plan: 01 +subsystem: processing +tags: + - asr + - whisper + - pydantic + - pyannote + - aiohttp + - pytest + + - tenacity + + - ffmpeg + + - whisper.cpp + +requires: + - phase: 02 (platform content download) +provides: + - asr + - official subtitle + - whisper transcription + - speaker diarization +affects: + - phase: 04 (analysis pipeline) +tech-stack: + added: + - pydantic v2: Schema validation + - aiohttp for B站 API calls + - pyannote.audio (optional, speaker diarization) + - whisper.cpp for ASR + - ffmpeg for audio conversion + - pytest/pytest-asyncio for testing + patterns: + - Processor pattern: ASRProcessor class with process() method + - Schema inheritance: ASRInput/ASROutput extend StepInput/StepOutput + - State management: Integration with WorkflowStateManager + - Retry mechanism: tenacity-based retry for network errors + + - Temporary file cleanup: Automatic deletion of *_audio.wav files + +key-files: + created: + - kb_pipeline/processors/__init__.py + - kb_pipeline/processors/asr.py + - tests/test_processors/__init__.py + - tests/test_processors/conftest.py + - tests/test_processors/test_asr.py + modified: + - kb_pipeline/schemas.py + +key-decisions: + - Used single ASRProcessor class for all ASR functionality instead of separate modules + - Implemented _try_official_subtitle for B站字幕下载 with automatic fallback to Whisper + - Used Literal type for tool field to enforce valid values only + - Implemented _run_whisper_basic with ffmpeg audio conversion and whisper-cli subprocess + - Implemented _run_whisper_with_diarization with pyannote.audio integration + - Implemented state recording with WorkflowStateManager + - Added helper functions: get_whisper_cpp_path, convert_to_wav, run_diarization, assign_speakers_to_segments + +patterns-established: + - ASRProcessor: Processor class pattern following Phase 1/2 established patterns + - Schema: StepInput/StepOutput inheritance pattern from Phase 1 + - State: WorkflowStateManager integration for status persistence + - Error handling: Structured error handling with proper fallback + +requirements-completed: [] +metrics: + duration: 10 + completed: "2026-03-13T11:16:26Z" +--- + +# Phase 03 Plan 01: ASRProcessor Summary + +ASRProcessor unified processor for video-to-text conversion, combining B站 official subtitle download with Whisper.cpp automatic speech recognition. + +Official subtitles are prioritized for quality, with fallback to local Whisper processing when unavailable. +Speaker diarization is automatically enabled when HF_TOKEN is configured. + +Output includes txt/srt/json formats with automatic state persistence. +## Performance +- **Duration:** 10 min +- **Started:** 2026-03-13T11:16:26Z +- **Completed:** 2026-03-13T11:24:12Z +- **Tasks:** 6 (all completed) +- **Files modified:** 6 +## Accomplishments +- Created ASRInput/ASROutput schemas extending Phase 1 StepInput/StepOutput +- Implemented ASRProcessor with process() method for unified video-to-text conversion +- Integrated B站 official subtitle download with B站 API calls +- Implemented Whisper.cpp integration with automatic audio conversion +- Added speaker diarization support with HF_TOKEN auto-detection +- Implemented state recording with WorkflowStateManager +- Created comprehensive test suite with 22 test cases +## Task Commits +Each task committed atomically following TDD principles: +1. **Task 1: Define ASR Schemas** - `2039a45` (test) + - Created ASRInput and ASROutput schemas in kb_pipeline/schemas.py + - Added tests for schema validation +2. **Task 2: Create ASRProcessor with subtitle download** - `406c6cb` (feat) + - Created kb_pipeline/processors/ directory structure + - Implemented ASRProcessor class with _try_official_subtitle method + - Added B站 API integration for subtitle download +3. **Task 3: Implement Whisper.cpp integration** - (included in Task 2 commit) + - Implemented _run_whisper_basic with ffmpeg audio conversion + - Added whisper-cli subprocess execution + - Implemented temporary file cleanup +4. **Task 4: Implement speaker diarization** - (included in Task 2 commit) + - Implemented _run_whisper_with_diarization with pyannote.audio integration + - Added HF_TOKEN auto-detection logic + - Implemented run_diarization and assign_speakers_to_segments helpers +5. **Task 5: Add state recording** - (included in Task 2 commit) + - Integrated WorkflowStateManager for status persistence + - Added success/failure state recording in process() method + - Implemented output_summary format with tool and segments, speakers info +6. **Task 6: Create test infrastructure** - `6b3fae3` (test/chore) + - Created tests/test_processors/ directory with __init__.py, conftest.py + - Added comprehensive fixtures for mocking settings, whisper path, API responses + - Fixed test_state_recording_failure mock behavior +## Files Created/Modified +- `kb_pipeline/processors/__init__.py` - Processor module initialization +- `kb_pipeline/processors/asr.py` - ASRProcessor implementation (575 lines) +- `kb_pipeline/schemas.py` - Added ASRInput/ASROutput schemas +- `tests/test_processors/__init__.py` - Test module initialization +- `tests/test_processors/conftest.py` - Shared test fixtures +- `tests/test_processors/test_asr.py` - Comprehensive test suite (618 lines) +## Decisions Made +- Used single ASRProcessor class for all ASR functionality instead of separate modules +- Implemented _try_official_subtitle returning None for no subtitle (not throwing exception) +- Used Literal type for tool field to enforce valid values only +- Implemented fallback chain: official subtitle -> whisper basic -> whisper with diarization +- Added comprehensive test fixtures for mocking external dependencies +## Deviations from Plan +### Auto-fixed Issues +**1. [Rule 2 - Test] Fixed state recording test mock behavior** +- **Found during:** Task 5 (state recording tests) +- **Issue:** test_state_recording_failure test used `side_effect` on `_try_official_subtitle`, but the implementation returns `None` (not raises exception), causing `_run_whisper` to be called instead +- **Fix:** Changed test to use `return_value=None` for `_try_official_subtitle` and `side_effect=Exception` for `_run_whisper` +- **Files modified:** tests/test_processors/test_asr.py +- **Verification:** All 22 tests pass +- **Committed in:** 6b3fae3 +**Total deviations:** 1 auto-fixed +**Impact on plan:** Minor test adjustment. No scope creep - test now correctly reflects actual implementation behavior. +## Issues Encountered +None - plan executed smoothly with minor test adjustment. +## User Setup Required +As specified in PLAN.md: +- **whisper.cpp** (本地语音识别引擎) + - Installation: `git clone https://github.com/ggml-org/whisper.cpp.git ~/whisper.cpp && cd ~/whisper.cpp && make && ./models/download-ggml-model.sh base` +- Environment: `WHISPER_CPP_PATH` (optional, defaults to ~/whisper.cpp) +- **pyannote.audio** (可选，说话者分离功能) + - Requires HF_TOKEN environment variable + - User must accept model user agreements at https://huggingface.co/pyannote/speaker-diarization-3.1 +## Next Phase Readiness +Phase 3 (ASR processing) is complete. Ready for Phase 4 (Analysis & Orchestration). +**Blockers/Concerns:** None