Commit Graph

163 Commits

Author SHA1 Message Date
starmen1265 4b4ed9cab6 docs(04-analysis-orchestration): create phase plans 2026-03-13 22:50:28 +08:00
starmen1265 debd53961a docs(04): research phase 04 domain 2026-03-13 22:43:32 +08:00
starmen1265 777fe38a9d docs(state): record phase 4 context session 2026-03-13 22:37:52 +08:00
starmen1265 62c33cdfd5 docs(04): capture phase context 2026-03-13 22:37:52 +08:00
starmen1265 6e4582f7ad docs(phase-03): complete phase execution 2026-03-13 19:39:56 +08:00
starmen1265 cb20090162 docs(03-01): complete ASr processing plan
- Created ASRInput/ASroputput schemas extending StepInput/StepOutput
- Implemented ASRProcessor with subtitle-first strategy
- Added Whisper.cpp integration with auto-cleanup
- Integrated speaker diarization (HF_TOKEN detection)
- All 22 tests passing
- Updated ROADMAP progress: Phase 3: 1/1 complete
- Updated state with completion timestamp and- Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 19:34:36 +08:00
starmen1265 d8f4df9149 docs(03-01): complete ASr processing plan
- Created ASRInput/ASroOutput schemas
- Implemented ASRProcessor with subtitle-first strategy
- Added whisper.cpp integration with auto-cleanup
- Integrated speaker diarization (HF_TOKEN detection)
- All 22 tests passing

- Updated ROADmap progress: Phase 3: 1/1 complete
- Updated state with completion timestamp

- recorded execution metrics

- Updated session info
- Added final documentation commit
2026-03-13 19:29:27 +08:00
starmen1265 6b3fae3543 test(03-01): fix state recording test mock behavior
- Fix test_state_recording_failure to use correct mock pattern
- _try_official_subtitle returning None triggers _Whisper` fallback
- Ensure proper exception propagation for accurate failure recording

- All 22 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 19:27:21 +08:00
starmen1265 406c6cbbc6 feat(03-01): implement ASRProcessor with official subtitle download
- Create processors module with ASRProcessor class
- Implement _try_official_subtitle for B站字幕下载
- Support process() method with subtitle-first strategy
- Add helper functions for whisper.cpp integration

- All 5 processor tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 19:22:23 +08:00
starmen1265 b2330c32c8 test(03-01): add ASRInput and ASROutput schemas
- Add ASRInput schema with video_path and content_type fields
- Add ASROutput schema with tool/has_subtitle/output_files/segments/text/duration_seconds/speakers
- Create test_processors directory with conftest fixtures
- Add comprehensive schema tests (8 test cases)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 19:20:17 +08:00
starmen1265 2039a45bf0 docs(03): create ASR processing plan 2026-03-13 19:11:29 +08:00
starmen1265 eaa68acb9d docs(03): add research and validation strategy 2026-03-13 19:06:19 +08:00
starmen1265 46bd809bf6 docs(phase-03): research ASR processing domain 2026-03-13 19:05:28 +08:00
starmen1265 acb6e15621 docs(state): record phase 3 context session 2026-03-13 18:50:40 +08:00
starmen1265 2148f1632a docs(03): capture phase context 2026-03-13 18:49:57 +08:00
starmen1265 cfd61d6cf6 docs(phase-02): complete phase execution 2026-03-13 18:32:20 +08:00
starmen1265 45df259047 docs(02-02): complete bilibili adapter plan
- Create 02-02-SUMMARY.md with execution results
- Update STATE.md with decisions and session info
- Update ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:27:51 +08:00
starmen1265 d7f1ddde47 docs(02-03): complete zhihu adapter plan
- ZhihuAdapter supports answer and zhuanlan content extraction
- Cookie-based authentication with login page detection
- Markdown output with YAML frontmatter
- 13 new tests, all 35 platform tests passing
- Test coverage: 86%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:27:10 +08:00
starmen1265 c4beb3ef3c feat(02-03): implement ZhihuAdapter for answer and zhuanlan content
- Implement ZhihuAdapter class inheriting PlatformAdapter
- Support answer URL content extraction
- Support zhuanlan (article) URL content extraction
- Add cookie-based authentication from KBSettings
- Detect login page redirects (expired cookies)
- Return AuthError for missing/expired cookies
- Format content with YAML frontmatter
- Update package exports to include ZhihuAdapter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:24:55 +08:00
starmen1265 a6f4d34625 test(02-03): add failing tests for ZhihuAdapter
- Test answer content extraction
- Test zhuanlan (article) content extraction
- Test cookie-based authentication
- Test login page detection
- Test AuthError handling for missing/expired cookies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:24:37 +08:00
starmen1265 d17fbd8118 test(02-02): add article content tests with YAML frontmatter
- Add test for article content returning Markdown format
- Add test for YAML frontmatter with required fields
- Add test for video content returning empty (ASR handling)
- Add test for unsupported content type error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:24:17 +08:00
starmen1265 b74f6a0c73 feat(02-02): implement BilibiliAdapter for video and article metadata
- Implement PlatformAdapter interface for Bilibili platform
- Add video metadata extraction (BV号)
- Add article metadata extraction (CV号)
- Add automatic retry on network errors (3 attempts)
- Add state recording for failed steps
- Add content type identification from URL
- Add output filename generation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:23:11 +08:00
starmen1265 3f42bfaf0d test(02-02): add failing tests for PlatformAdapter base class
- Add test for abstract class instantiation prevention
- Add test for incomplete implementation detection
- Add test for complete implementation instantiation
- Add test for async method signatures
- Add shared fixtures for platform tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:19:24 +08:00
starmen1265 ae7dea9581 docs(02-01): complete link parser & idempotency plan
- Created LinkParser for unified URL parsing (B站/知乎)
- Created IdempotencyChecker for file existence checking
- All 23 new tests passing, 95% coverage
- Requirements PLAT-01, PLAT-02, PLAT-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:16:04 +08:00
starmen1265 b8339c636a feat(02-01): export LinkParser and IdempotencyChecker from package root
- Add LinkParser and ParsedLink to __init__.py exports
- Add IdempotencyChecker to __init__.py exports
- Update __all__ list with new public APIs
- All 23 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:14:06 +08:00
starmen1265 bd29425d28 feat(02-01): add IdempotencyChecker for file existence checking
- Create IdempotencyChecker class with check_exists method
- Check for metadata JSON and content Markdown files
- Return tuple of (all_exist, existing_files)
- Follow naming convention: {platform}_{content_id}_{type}.{ext}
- Add get_expected_files helper method
- Add 8 unit tests covering all scenarios

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:13:35 +08:00
starmen1265 dd94f068e7 feat(02-01): add LinkParser and ParsedLink for unified URL parsing
- Create ParsedLink dataclass with platform, content_type, content_id fields
- Implement LinkParser class with regex-based URL parsing
- Support Bilibili video/article URLs (with/without www)
- Support Zhihu answer/zhuanlan URLs
- Support b23.tv short link resolution with requests.head
- Return None for unrecognized URLs (no exceptions)
- Add 15 unit tests covering all supported URL types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 18:12:16 +08:00
starmen1265 8a8959cebe docs(02): add phase 2 plans 2026-03-13 18:04:48 +08:00
starmen1265 cc6c915f10 fix(02): revise plan 02-02 based on checker feedback 2026-03-13 18:02:24 +08:00
starmen1265 65e7c276d0 docs(02): create phase 2 plans for platform skills 2026-03-13 17:57:10 +08:00
starmen1265 3d262ff230 docs(02): research phase 2 platform skills 2026-03-13 17:49:06 +08:00
starmen1265 b8b2f72311 docs(state): record phase 2 context session 2026-03-13 17:43:15 +08:00
starmen1265 ea82f5d965 docs(02): capture phase 2 context 2026-03-13 17:43:03 +08:00
starmen1265 8736f240ec docs(phase-01): complete phase execution with verification
Phase 1 Foundation verified with 5/5 must-haves passed.
All 9 requirements covered: PIPE-01~03, REL-01~03, VAL-01, MNT-02~03.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:28:35 +08:00
starmen1265 b7db451d9d docs(01-03): complete error handler & config manager plan
- Structured error types with rich context (KBError, NetworkError, etc.)
- Configuration management via pydantic-settings from ~/.kb_env
- Automatic retry mechanism with tenacity (3 attempts, exponential backoff)
- All 100 tests passing
- Requirements REL-02, MNT-02, MNT-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:22:58 +08:00
starmen1265 770225439b feat(01-03): export all public APIs from package root
- Export error types (KBError, NetworkError, ValidationError, SystemError, AuthError, ErrorType)
- Export config (KBSettings, get_settings)
- Export retry (retry_on_network_error)
- Clean import interface for users

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:20:38 +08:00
starmen1265 74adaf3a79 feat(01-03): implement retry mechanism with tenacity
- retry_on_network_error decorator for automatic retry
- Max 3 attempts with exponential backoff (2s, 4s, max 10s)
- Only retries on NetworkError
- Logs retry attempts via before_sleep_log
- Reraises original exception after max attempts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:19:42 +08:00
starmen1265 46f1724e06 docs(01-01): complete WorkflowStateManager plan
Summary:
- WorkflowStateManager with atomic write and backup rotation
- 90% test coverage (20 tests)
- 5 min execution time

State updates:
- Plan counter advanced to 3
- Progress: 67% (2/3 plans complete)
- Requirements marked complete: PIPE-02, REL-01

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:19:37 +08:00
starmen1265 c87dd8b803 test(01-03): add failing tests for retry mechanism
- Test retry on NetworkError (up to 3 times)
- Test no retry on non-NetworkError (ValidationError)
- Test reraise after max attempts
- Test preserved return value on success
- Test exponential backoff configuration
- Test cause exception preservation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:17:51 +08:00
starmen1265 0dbb0c4cb6 refactor(01-01): export WorkflowStateManager from kb_pipeline
Add WorkflowStateManager to package exports for easier imports.
Users can now use: from kb_pipeline import WorkflowStateManager

All 20 tests pass with 90% coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:17:24 +08:00
starmen1265 44b5d18113 feat(01-03): implement configuration management with pydantic-settings
- KBSettings class loading from ~/.kb_env file
- Required fields: base_path, archive_path
- Optional fields: git_root, git_user, git_email
- Secret fields: hf_token, zhihu_cookie (SecretStr)
- get_settings() singleton with lru_cache
- env_prefix "KB_" for environment variables
- Tests adjusted for proper env isolation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:17:10 +08:00
starmen1265 624cd76538 feat(01-01): implement WorkflowStateManager with atomic write
GREEN phase: Implement state management with:
- Atomic write pattern (temp file + os.replace)
- Backup rotation (workflow_state.json -> .bak)
- Backup fallback on corruption
- get_or_create idempotent creation

Fix test to properly create backup before corruption test.

All 20 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:16:30 +08:00
starmen1265 e789d81230 test(01-03): add failing tests for configuration management
- Test KBSettings required fields (base_path, archive_path)
- Test KBSettings optional fields with defaults
- Test secret fields (hf_token, zhihu_cookie) as SecretStr
- Test missing required fields raises ValidationError
- Test get_settings returns singleton via lru_cache
- Test env vars with KB_ prefix are read

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:15:35 +08:00
starmen1265 246e4038b4 test(01-01): add failing tests for WorkflowStateManager
RED phase: 20 test cases for:
- WorkflowState.is_step_completed (5 tests)
- WorkflowState.mark_step_* methods (4 tests)
- WorkflowStateManager atomic write (4 tests)
- WorkflowStateManager backup fallback (4 tests)
- WorkflowStateManager.get_or_create (3 tests)

11 tests fail due to missing kb_pipeline.state module.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:15:02 +08:00
starmen1265 d8cfee3bf1 feat(01-03): implement structured error handling
- ErrorType enum with NETWORK, VALIDATION, SYSTEM, AUTH
- KBError base class with to_dict method for serialization
- NetworkError with status_code and auto-generated suggestion
- ValidationError with validation_errors list
- SystemError with system_message
- AuthError with auth_type

All errors include step_name, input_summary, error_type, message,
and suggestion for rich debugging context.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:14:51 +08:00
starmen1265 840559075a test(01-03): add failing tests for error handling
- Test ErrorType enum (NETWORK, VALIDATION, SYSTEM, AUTH)
- Test KBError base class with to_dict method
- Test NetworkError with status_code and auto-suggestion
- Test ValidationError with validation_errors list
- Test SystemError with system_message
- Test AuthError with auth_type
- Test error class hierarchy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:13:50 +08:00
starmen1265 3cec1cafaf docs(01-02): complete core pipeline schemas plan
- Create 01-02-SUMMARY.md with execution results
- Update STATE.md: Plan 2 of 3 completed, 8% progress
- Mark requirements complete: PIPE-01, PIPE-03, VAL-01, REL-03
- Update ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:11:30 +08:00
starmen1265 7ffaafd615 refactor(01-02): add comprehensive tests for full coverage
- Add tests for WorkflowState methods (mark_step_started, mark_step_completed, mark_step_failed)
- Add tests for StepResult schema
- Add tests for StepDefinition schema with dependencies
- Achieve 100% test coverage on kb_pipeline/schemas.py
- Total 37 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:08:51 +08:00
starmen1265 2caddd792e feat(01-02): implement core pipeline schemas
- Add StepStatus enum (PENDING, RUNNING, COMPLETED, FAILED, SKIPPED)
- Add StepInput with content_id and platform validation
- Add StepOutput with success and error_message fields
- Add StepResult for step execution results
- Add StepRecord for step execution records
- Add WorkflowState with step management methods
- Add StepDefinition for step configuration
- Add pytest.ini for PYTHONPATH configuration
- TDD GREEN phase: all 28 tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:07:09 +08:00
starmen1265 85413bc1c6 test(01-02): add failing tests for core pipeline schemas
- Create kb_pipeline package structure
- Add test fixtures in conftest.py
- Add 28 test cases for StepInput, StepOutput, StepStatus, StepRecord, WorkflowState
- Add JSON serialization/deserialization tests
- TDD RED phase: tests fail because schemas.py not yet implemented

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:05:37 +08:00