- Add ArchiveInput creation in Pipeline.run()
- Import ArchiveInput in pipeline.py
- Add archive_file and skipped to output field recording
- Add ArchiveProcessor to CLI processors list
- Update PipelineConfig steps to include archive
- Change default output_dir from cwd to settings.intermediate_path
- Reorder settings initialization before output_dir assignment
- Maintain output_dir parameter for testing/custom override
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add intermediate_path field with default ~/work/intermediate/analysis/
- Follow pydantic-settings naming convention for env var override
- Users can customize via KB_INTERMEDIATE_PATH environment variable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _normalize_metadata() to flatten nested metadata structure
- Fix generate_frontmatter() to use correct metadata fields
- Change _sanitize_filename() to return standardized "bilibili_{bvid}" format
Fixes: analysis.md frontmatter showing "未命名"/0/"未知" instead of actual values
Fixes: audio filename using video title instead of platform_bvid format
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle Whisper transcription format (array of segments with text)
- Extract full text from transcription array before passing to DeepAnalyzer
- Fix offsets parsing to support both dict and list formats
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add download_video() to BilibiliAdapter using yt-dlp
- Add bilibili_cookie config to Settings
- Update DownloadProcessor to call video download for bilibili videos
- Add idempotency check to ASRProcessor for existing subtitle files
- Fix indentation error in asr.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- When content_type is video and no official subtitle available:
- Download video using yt-dlp
- Extract audio for Whisper fallback
- Update content_file with actual video path
This enables the pipeline to handle videos that don't have
official Bilibili subtitles.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- When a step is skipped (already completed), ask user if they want to re-run
- If skipping, scan content_dir for existing files and populate results
- Support both new naming (platform_id_subtitle.json) and old naming (platform_id.json)
- This fixes the issue where analyze step couldn't find subtitle files
when ASR step was skipped
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add content_dir field to ASRInput schema
- Pass content_dir from pipeline when creating ASRInput
- Modify _try_official_subtitle to use content_dir for output
- Modify _run_whisper_basic to use content_dir for output
- Add validation for empty video_path in Whisper fallback
Root cause: DownloadProcessor sets content_file="" for video types,
causing Path("").parent to resolve to current directory instead of
content directory. Subtitle files were written to wrong location.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parse duration strings like "12:34" or "01:23:45" into seconds.
Previously only integer duration values were handled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BilibiliAdapter._fetch_metadata_with_retry() now calls the underlying
async methods directly instead of sync wrappers that use asyncio.run().
This prevents "asyncio.run() cannot be called from a running event loop"
error when the adapter is called from within an already-running event loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add async run() method to Pipeline class
- Execute processors sequentially with state persistence
- Handle step-specific input types (Parse, Download, ASR, Analyze)
- Support skip_completed for resumable workflows
- Return PipelineOutput with results and content_dir
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pipeline class initializes with processors list
- Validates processor STEP_NAME matches config.steps
- _parse_url method parses URLs using LinkParser
- _get_content_dir creates content directory from parsed URL
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeProcessor wraps existing DeepAnalyzer for pipeline integration
- Loads metadata and content from JSON files
- Generates analysis reports as Markdown with frontmatter
- Records state on success and failure
- Supports both video and article content types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeInput extends StepInput with metadata_file, content_file, content_type, output_file
- AnalyzeOutput extends StepOutput with analysis_file, summary, main_topics, key_insights, tags, stats
- Supports both video and article content types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor downloads article/answer/zhuanlan content
- Video content is skipped (handled by ASR in Phase 3)
- Content files saved with correct naming convention
- DownloadProcessor exported from kb_pipeline.processors
- All 17 tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor inherits from BaseProcessor
- Fetches metadata from Bilibili and Zhihu adapters
- Saves metadata to content_dir as JSON
- Records state on success/failure
- Skips step if already completed (idempotency)
- All metadata tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseProcessor wraps LinkParser for URL parsing
- Returns ParseOutput with platform, content_type, content_id
- Records state when state_manager is provided
- Handles unknown URLs with success=False
- Export BaseProcessor, ParseProcessor, ASRProcessor from __init__.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseInput extends StepInput with url field
- ParseOutput extends StepOutput with platform, content_type, content_id, resolved_url, original_url
- Tests for schema validation and inheritance
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test BaseProcessor cannot be instantiated directly
- Test subclass must implement process() method
- Test state_manager attribute is provided
- Test STEP_NAME class attribute requirement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create processors module with ASRProcessor class
- Implement _try_official_subtitle for B站字幕下载
- Support process() method with subtitle-first strategy
- Add helper functions for whisper.cpp integration
- All 5 processor tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ASRInput schema with video_path and content_type fields
- Add ASROutput schema with tool/has_subtitle/output_files/segments/text/duration_seconds/speakers
- Create test_processors directory with conftest fixtures
- Add comprehensive schema tests (8 test cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implement ZhihuAdapter class inheriting PlatformAdapter
- Support answer URL content extraction
- Support zhuanlan (article) URL content extraction
- Add cookie-based authentication from KBSettings
- Detect login page redirects (expired cookies)
- Return AuthError for missing/expired cookies
- Format content with YAML frontmatter
- Update package exports to include ZhihuAdapter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test answer content extraction
- Test zhuanlan (article) content extraction
- Test cookie-based authentication
- Test login page detection
- Test AuthError handling for missing/expired cookies
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test for abstract class instantiation prevention
- Add test for incomplete implementation detection
- Add test for complete implementation instantiation
- Add test for async method signatures
- Add shared fixtures for platform tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LinkParser and ParsedLink to __init__.py exports
- Add IdempotencyChecker to __init__.py exports
- Update __all__ list with new public APIs
- All 23 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create IdempotencyChecker class with check_exists method
- Check for metadata JSON and content Markdown files
- Return tuple of (all_exist, existing_files)
- Follow naming convention: {platform}_{content_id}_{type}.{ext}
- Add get_expected_files helper method
- Add 8 unit tests covering all scenarios
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ParsedLink dataclass with platform, content_type, content_id fields
- Implement LinkParser class with regex-based URL parsing
- Support Bilibili video/article URLs (with/without www)
- Support Zhihu answer/zhuanlan URLs
- Support b23.tv short link resolution with requests.head
- Return None for unrecognized URLs (no exceptions)
- Add 15 unit tests covering all supported URL types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- retry_on_network_error decorator for automatic retry
- Max 3 attempts with exponential backoff (2s, 4s, max 10s)
- Only retries on NetworkError
- Logs retry attempts via before_sleep_log
- Reraises original exception after max attempts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add WorkflowStateManager to package exports for easier imports.
Users can now use: from kb_pipeline import WorkflowStateManager
All 20 tests pass with 90% coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ErrorType enum with NETWORK, VALIDATION, SYSTEM, AUTH
- KBError base class with to_dict method for serialization
- NetworkError with status_code and auto-generated suggestion
- ValidationError with validation_errors list
- SystemError with system_message
- AuthError with auth_type
All errors include step_name, input_summary, error_type, message,
and suggestion for rich debugging context.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create kb_pipeline package structure
- Add test fixtures in conftest.py
- Add 28 test cases for StepInput, StepOutput, StepStatus, StepRecord, WorkflowState
- Add JSON serialization/deserialization tests
- TDD RED phase: tests fail because schemas.py not yet implemented
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
starmen1265