docs(03-01): complete ASr processing plan
- Created ASRInput/ASroOutput schemas - Implemented ASRProcessor with subtitle-first strategy - Added whisper.cpp integration with auto-cleanup - Integrated speaker diarization (HF_TOKEN detection) - All 22 tests passing - Updated ROADmap progress: Phase 3: 1/1 complete - Updated state with completion timestamp - recorded execution metrics - Updated session info - Added final documentation commit
This commit is contained in:
+4
-4
@@ -5,13 +5,13 @@ milestone_name: milestone
|
|||||||
current_plan: 3
|
current_plan: 3
|
||||||
status: executing
|
status: executing
|
||||||
stopped_at: Phase 3 context gathered
|
stopped_at: Phase 3 context gathered
|
||||||
last_updated: "2026-03-13T10:50:04.038Z"
|
last_updated: "2026-03-13T11:29:09.319Z"
|
||||||
last_activity: 2026-03-13
|
last_activity: 2026-03-13
|
||||||
progress:
|
progress:
|
||||||
total_phases: 4
|
total_phases: 4
|
||||||
completed_phases: 2
|
completed_phases: 3
|
||||||
total_plans: 6
|
total_plans: 7
|
||||||
completed_plans: 6
|
completed_plans: 7
|
||||||
percent: 8
|
percent: 8
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,158 @@
|
|||||||
|
---
|
||||||
|
phase: 03-processing-skills
|
||||||
|
plan: 01
|
||||||
|
subsystem: processing
|
||||||
|
tags:
|
||||||
|
- asr
|
||||||
|
- whisper
|
||||||
|
- pydantic
|
||||||
|
- pyannote
|
||||||
|
- aiohttp
|
||||||
|
- pytest
|
||||||
|
|
||||||
|
- tenacity
|
||||||
|
|
||||||
|
- ffmpeg
|
||||||
|
|
||||||
|
- whisper.cpp
|
||||||
|
|
||||||
|
requires:
|
||||||
|
- phase: 02 (platform content download)
|
||||||
|
provides:
|
||||||
|
- asr
|
||||||
|
- official subtitle
|
||||||
|
- whisper transcription
|
||||||
|
- speaker diarization
|
||||||
|
affects:
|
||||||
|
- phase: 04 (analysis pipeline)
|
||||||
|
tech-stack:
|
||||||
|
added:
|
||||||
|
- pydantic v2: Schema validation
|
||||||
|
- aiohttp for B站 API calls
|
||||||
|
- pyannote.audio (optional, speaker diarization)
|
||||||
|
- whisper.cpp for ASR
|
||||||
|
- ffmpeg for audio conversion
|
||||||
|
- pytest/pytest-asyncio for testing
|
||||||
|
patterns:
|
||||||
|
- Processor pattern: ASRProcessor class with process() method
|
||||||
|
- Schema inheritance: ASRInput/ASROutput extend StepInput/StepOutput
|
||||||
|
- State management: Integration with WorkflowStateManager
|
||||||
|
- Retry mechanism: tenacity-based retry for network errors
|
||||||
|
|
||||||
|
- Temporary file cleanup: Automatic deletion of *_audio.wav files
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- kb_pipeline/processors/__init__.py
|
||||||
|
- kb_pipeline/processors/asr.py
|
||||||
|
- tests/test_processors/__init__.py
|
||||||
|
- tests/test_processors/conftest.py
|
||||||
|
- tests/test_processors/test_asr.py
|
||||||
|
modified:
|
||||||
|
- kb_pipeline/schemas.py
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- Used single ASRProcessor class for all ASR functionality instead of separate modules
|
||||||
|
- Implemented _try_official_subtitle for B站字幕下载 with automatic fallback to Whisper
|
||||||
|
- Used Literal type for tool field to enforce valid values only
|
||||||
|
- Implemented _run_whisper_basic with ffmpeg audio conversion and whisper-cli subprocess
|
||||||
|
- Implemented _run_whisper_with_diarization with pyannote.audio integration
|
||||||
|
- Implemented state recording with WorkflowStateManager
|
||||||
|
- Added helper functions: get_whisper_cpp_path, convert_to_wav, run_diarization, assign_speakers_to_segments
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- ASRProcessor: Processor class pattern following Phase 1/2 established patterns
|
||||||
|
- Schema: StepInput/StepOutput inheritance pattern from Phase 1
|
||||||
|
- State: WorkflowStateManager integration for status persistence
|
||||||
|
- Error handling: Structured error handling with proper fallback
|
||||||
|
|
||||||
|
requirements-completed: []
|
||||||
|
metrics:
|
||||||
|
duration: 10
|
||||||
|
completed: "2026-03-13T11:16:26Z"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 03 Plan 01: ASRProcessor Summary
|
||||||
|
|
||||||
|
ASRProcessor unified processor for video-to-text conversion, combining B站 official subtitle download with Whisper.cpp automatic speech recognition.
|
||||||
|
|
||||||
|
Official subtitles are prioritized for quality, with fallback to local Whisper processing when unavailable.
|
||||||
|
Speaker diarization is automatically enabled when HF_TOKEN is configured.
|
||||||
|
|
||||||
|
Output includes txt/srt/json formats with automatic state persistence.
|
||||||
|
## Performance
|
||||||
|
- **Duration:** 10 min
|
||||||
|
- **Started:** 2026-03-13T11:16:26Z
|
||||||
|
- **Completed:** 2026-03-13T11:24:12Z
|
||||||
|
- **Tasks:** 6 (all completed)
|
||||||
|
- **Files modified:** 6
|
||||||
|
## Accomplishments
|
||||||
|
- Created ASRInput/ASROutput schemas extending Phase 1 StepInput/StepOutput
|
||||||
|
- Implemented ASRProcessor with process() method for unified video-to-text conversion
|
||||||
|
- Integrated B站 official subtitle download with B站 API calls
|
||||||
|
- Implemented Whisper.cpp integration with automatic audio conversion
|
||||||
|
- Added speaker diarization support with HF_TOKEN auto-detection
|
||||||
|
- Implemented state recording with WorkflowStateManager
|
||||||
|
- Created comprehensive test suite with 22 test cases
|
||||||
|
## Task Commits
|
||||||
|
Each task committed atomically following TDD principles:
|
||||||
|
1. **Task 1: Define ASR Schemas** - `2039a45` (test)
|
||||||
|
- Created ASRInput and ASROutput schemas in kb_pipeline/schemas.py
|
||||||
|
- Added tests for schema validation
|
||||||
|
2. **Task 2: Create ASRProcessor with subtitle download** - `406c6cb` (feat)
|
||||||
|
- Created kb_pipeline/processors/ directory structure
|
||||||
|
- Implemented ASRProcessor class with _try_official_subtitle method
|
||||||
|
- Added B站 API integration for subtitle download
|
||||||
|
3. **Task 3: Implement Whisper.cpp integration** - (included in Task 2 commit)
|
||||||
|
- Implemented _run_whisper_basic with ffmpeg audio conversion
|
||||||
|
- Added whisper-cli subprocess execution
|
||||||
|
- Implemented temporary file cleanup
|
||||||
|
4. **Task 4: Implement speaker diarization** - (included in Task 2 commit)
|
||||||
|
- Implemented _run_whisper_with_diarization with pyannote.audio integration
|
||||||
|
- Added HF_TOKEN auto-detection logic
|
||||||
|
- Implemented run_diarization and assign_speakers_to_segments helpers
|
||||||
|
5. **Task 5: Add state recording** - (included in Task 2 commit)
|
||||||
|
- Integrated WorkflowStateManager for status persistence
|
||||||
|
- Added success/failure state recording in process() method
|
||||||
|
- Implemented output_summary format with tool and segments, speakers info
|
||||||
|
6. **Task 6: Create test infrastructure** - `6b3fae3` (test/chore)
|
||||||
|
- Created tests/test_processors/ directory with __init__.py, conftest.py
|
||||||
|
- Added comprehensive fixtures for mocking settings, whisper path, API responses
|
||||||
|
- Fixed test_state_recording_failure mock behavior
|
||||||
|
## Files Created/Modified
|
||||||
|
- `kb_pipeline/processors/__init__.py` - Processor module initialization
|
||||||
|
- `kb_pipeline/processors/asr.py` - ASRProcessor implementation (575 lines)
|
||||||
|
- `kb_pipeline/schemas.py` - Added ASRInput/ASROutput schemas
|
||||||
|
- `tests/test_processors/__init__.py` - Test module initialization
|
||||||
|
- `tests/test_processors/conftest.py` - Shared test fixtures
|
||||||
|
- `tests/test_processors/test_asr.py` - Comprehensive test suite (618 lines)
|
||||||
|
## Decisions Made
|
||||||
|
- Used single ASRProcessor class for all ASR functionality instead of separate modules
|
||||||
|
- Implemented _try_official_subtitle returning None for no subtitle (not throwing exception)
|
||||||
|
- Used Literal type for tool field to enforce valid values only
|
||||||
|
- Implemented fallback chain: official subtitle -> whisper basic -> whisper with diarization
|
||||||
|
- Added comprehensive test fixtures for mocking external dependencies
|
||||||
|
## Deviations from Plan
|
||||||
|
### Auto-fixed Issues
|
||||||
|
**1. [Rule 2 - Test] Fixed state recording test mock behavior**
|
||||||
|
- **Found during:** Task 5 (state recording tests)
|
||||||
|
- **Issue:** test_state_recording_failure test used `side_effect` on `_try_official_subtitle`, but the implementation returns `None` (not raises exception), causing `_run_whisper` to be called instead
|
||||||
|
- **Fix:** Changed test to use `return_value=None` for `_try_official_subtitle` and `side_effect=Exception` for `_run_whisper`
|
||||||
|
- **Files modified:** tests/test_processors/test_asr.py
|
||||||
|
- **Verification:** All 22 tests pass
|
||||||
|
- **Committed in:** 6b3fae3
|
||||||
|
**Total deviations:** 1 auto-fixed
|
||||||
|
**Impact on plan:** Minor test adjustment. No scope creep - test now correctly reflects actual implementation behavior.
|
||||||
|
## Issues Encountered
|
||||||
|
None - plan executed smoothly with minor test adjustment.
|
||||||
|
## User Setup Required
|
||||||
|
As specified in PLAN.md:
|
||||||
|
- **whisper.cpp** (本地语音识别引擎)
|
||||||
|
- Installation: `git clone https://github.com/ggml-org/whisper.cpp.git ~/whisper.cpp && cd ~/whisper.cpp && make && ./models/download-ggml-model.sh base`
|
||||||
|
- Environment: `WHISPER_CPP_PATH` (optional, defaults to ~/whisper.cpp)
|
||||||
|
- **pyannote.audio** (可选，说话者分离功能)
|
||||||
|
- Requires HF_TOKEN environment variable
|
||||||
|
- User must accept model user agreements at https://huggingface.co/pyannote/speaker-diarization-3.1
|
||||||
|
## Next Phase Readiness
|
||||||
|
Phase 3 (ASR processing) is complete. Ready for Phase 4 (Analysis & Orchestration).
|
||||||
|
**Blockers/Concerns:** None
|
||||||
Reference in New Issue
Block a user