fix/ci-deploy-reset #9

Merged
starmen1265 merged 96 commits from fix/ci-deploy-reset into main 2026-03-15 01:05:14 +08:00
Owner
No description provided.
starmen1265 added 95 commits 2026-03-15 01:04:46 +08:00
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dev-workflow multi-agent orchestration skill is no longer needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Nyquist validation contract for Phase 1 Foundation.
Covers test infrastructure, sampling rate, and per-task verification map.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update VALIDATION.md test file paths to use tests/test_pipeline/
instead of tests/unit/ to match PLAN.md definitions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change depends_on from [01-02] to ["02"] for consistency
with plan frontmatter format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create kb_pipeline package structure
- Add test fixtures in conftest.py
- Add 28 test cases for StepInput, StepOutput, StepStatus, StepRecord, WorkflowState
- Add JSON serialization/deserialization tests
- TDD RED phase: tests fail because schemas.py not yet implemented

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add StepStatus enum (PENDING, RUNNING, COMPLETED, FAILED, SKIPPED)
- Add StepInput with content_id and platform validation
- Add StepOutput with success and error_message fields
- Add StepResult for step execution results
- Add StepRecord for step execution records
- Add WorkflowState with step management methods
- Add StepDefinition for step configuration
- Add pytest.ini for PYTHONPATH configuration
- TDD GREEN phase: all 28 tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tests for WorkflowState methods (mark_step_started, mark_step_completed, mark_step_failed)
- Add tests for StepResult schema
- Add tests for StepDefinition schema with dependencies
- Achieve 100% test coverage on kb_pipeline/schemas.py
- Total 37 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 01-02-SUMMARY.md with execution results
- Update STATE.md: Plan 2 of 3 completed, 8% progress
- Mark requirements complete: PIPE-01, PIPE-03, VAL-01, REL-03
- Update ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test ErrorType enum (NETWORK, VALIDATION, SYSTEM, AUTH)
- Test KBError base class with to_dict method
- Test NetworkError with status_code and auto-suggestion
- Test ValidationError with validation_errors list
- Test SystemError with system_message
- Test AuthError with auth_type
- Test error class hierarchy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ErrorType enum with NETWORK, VALIDATION, SYSTEM, AUTH
- KBError base class with to_dict method for serialization
- NetworkError with status_code and auto-generated suggestion
- ValidationError with validation_errors list
- SystemError with system_message
- AuthError with auth_type

All errors include step_name, input_summary, error_type, message,
and suggestion for rich debugging context.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RED phase: 20 test cases for:
- WorkflowState.is_step_completed (5 tests)
- WorkflowState.mark_step_* methods (4 tests)
- WorkflowStateManager atomic write (4 tests)
- WorkflowStateManager backup fallback (4 tests)
- WorkflowStateManager.get_or_create (3 tests)

11 tests fail due to missing kb_pipeline.state module.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test KBSettings required fields (base_path, archive_path)
- Test KBSettings optional fields with defaults
- Test secret fields (hf_token, zhihu_cookie) as SecretStr
- Test missing required fields raises ValidationError
- Test get_settings returns singleton via lru_cache
- Test env vars with KB_ prefix are read

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GREEN phase: Implement state management with:
- Atomic write pattern (temp file + os.replace)
- Backup rotation (workflow_state.json -> .bak)
- Backup fallback on corruption
- get_or_create idempotent creation

Fix test to properly create backup before corruption test.

All 20 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- KBSettings class loading from ~/.kb_env file
- Required fields: base_path, archive_path
- Optional fields: git_root, git_user, git_email
- Secret fields: hf_token, zhihu_cookie (SecretStr)
- get_settings() singleton with lru_cache
- env_prefix "KB_" for environment variables
- Tests adjusted for proper env isolation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add WorkflowStateManager to package exports for easier imports.
Users can now use: from kb_pipeline import WorkflowStateManager

All 20 tests pass with 90% coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test retry on NetworkError (up to 3 times)
- Test no retry on non-NetworkError (ValidationError)
- Test reraise after max attempts
- Test preserved return value on success
- Test exponential backoff configuration
- Test cause exception preservation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary:
- WorkflowStateManager with atomic write and backup rotation
- 90% test coverage (20 tests)
- 5 min execution time

State updates:
- Plan counter advanced to 3
- Progress: 67% (2/3 plans complete)
- Requirements marked complete: PIPE-02, REL-01

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- retry_on_network_error decorator for automatic retry
- Max 3 attempts with exponential backoff (2s, 4s, max 10s)
- Only retries on NetworkError
- Logs retry attempts via before_sleep_log
- Reraises original exception after max attempts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Export error types (KBError, NetworkError, ValidationError, SystemError, AuthError, ErrorType)
- Export config (KBSettings, get_settings)
- Export retry (retry_on_network_error)
- Clean import interface for users

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Structured error types with rich context (KBError, NetworkError, etc.)
- Configuration management via pydantic-settings from ~/.kb_env
- Automatic retry mechanism with tenacity (3 attempts, exponential backoff)
- All 100 tests passing
- Requirements REL-02, MNT-02, MNT-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 Foundation verified with 5/5 must-haves passed.
All 9 requirements covered: PIPE-01~03, REL-01~03, VAL-01, MNT-02~03.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ParsedLink dataclass with platform, content_type, content_id fields
- Implement LinkParser class with regex-based URL parsing
- Support Bilibili video/article URLs (with/without www)
- Support Zhihu answer/zhuanlan URLs
- Support b23.tv short link resolution with requests.head
- Return None for unrecognized URLs (no exceptions)
- Add 15 unit tests covering all supported URL types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create IdempotencyChecker class with check_exists method
- Check for metadata JSON and content Markdown files
- Return tuple of (all_exist, existing_files)
- Follow naming convention: {platform}_{content_id}_{type}.{ext}
- Add get_expected_files helper method
- Add 8 unit tests covering all scenarios

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LinkParser and ParsedLink to __init__.py exports
- Add IdempotencyChecker to __init__.py exports
- Update __all__ list with new public APIs
- All 23 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Created LinkParser for unified URL parsing (B站/知乎)
- Created IdempotencyChecker for file existence checking
- All 23 new tests passing, 95% coverage
- Requirements PLAT-01, PLAT-02, PLAT-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test for abstract class instantiation prevention
- Add test for incomplete implementation detection
- Add test for complete implementation instantiation
- Add test for async method signatures
- Add shared fixtures for platform tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implement PlatformAdapter interface for Bilibili platform
- Add video metadata extraction (BV号)
- Add article metadata extraction (CV号)
- Add automatic retry on network errors (3 attempts)
- Add state recording for failed steps
- Add content type identification from URL
- Add output filename generation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test for article content returning Markdown format
- Add test for YAML frontmatter with required fields
- Add test for video content returning empty (ASR handling)
- Add test for unsupported content type error

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test answer content extraction
- Test zhuanlan (article) content extraction
- Test cookie-based authentication
- Test login page detection
- Test AuthError handling for missing/expired cookies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implement ZhihuAdapter class inheriting PlatformAdapter
- Support answer URL content extraction
- Support zhuanlan (article) URL content extraction
- Add cookie-based authentication from KBSettings
- Detect login page redirects (expired cookies)
- Return AuthError for missing/expired cookies
- Format content with YAML frontmatter
- Update package exports to include ZhihuAdapter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ZhihuAdapter supports answer and zhuanlan content extraction
- Cookie-based authentication with login page detection
- Markdown output with YAML frontmatter
- 13 new tests, all 35 platform tests passing
- Test coverage: 86%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 02-02-SUMMARY.md with execution results
- Update STATE.md with decisions and session info
- Update ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ASRInput schema with video_path and content_type fields
- Add ASROutput schema with tool/has_subtitle/output_files/segments/text/duration_seconds/speakers
- Create test_processors directory with conftest fixtures
- Add comprehensive schema tests (8 test cases)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create processors module with ASRProcessor class
- Implement _try_official_subtitle for B站字幕下载
- Support process() method with subtitle-first strategy
- Add helper functions for whisper.cpp integration

- All 5 processor tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix test_state_recording_failure to use correct mock pattern
- _try_official_subtitle returning None triggers _Whisper` fallback
- Ensure proper exception propagation for accurate failure recording

- All 22 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Created ASRInput/ASroOutput schemas
- Implemented ASRProcessor with subtitle-first strategy
- Added whisper.cpp integration with auto-cleanup
- Integrated speaker diarization (HF_TOKEN detection)
- All 22 tests passing

- Updated ROADmap progress: Phase 3: 1/1 complete
- Updated state with completion timestamp

- recorded execution metrics

- Updated session info
- Added final documentation commit
- Created ASRInput/ASroputput schemas extending StepInput/StepOutput
- Implemented ASRProcessor with subtitle-first strategy
- Added Whisper.cpp integration with auto-cleanup
- Integrated speaker diarization (HF_TOKEN detection)
- All 22 tests passing
- Updated ROADMAP progress: Phase 3: 1/1 complete
- Updated state with completion timestamp and- Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Issues addressed:
1. Remove non-existent 04-06 from depends_on → [04-04] only
2. Add CLI 'process' subcommand implementation task (was missing)
3. Fix verify command syntax (pytest instead of malformed python -c)
4. Rewrite Task 2 action with proper test code structure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test BaseProcessor cannot be instantiated directly
- Test subclass must implement process() method
- Test state_manager attribute is provided
- Test STEP_NAME class attribute requirement

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseInput extends StepInput with url field
- ParseOutput extends StepOutput with platform, content_type, content_id, resolved_url, original_url
- Tests for schema validation and inheritance

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseProcessor wraps LinkParser for URL parsing
- Returns ParseOutput with platform, content_type, content_id
- Records state when state_manager is provided
- Handles unknown URLs with success=False
- Export BaseProcessor, ParseProcessor, ASRProcessor from __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- BaseProcessor abstract class with unified interface
- ParseProcessor wrapping LinkParser for URL parsing
- ParseInput/ParseOutput schemas for parse step
- All tests passing (18 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD RED phase:
- Test DownloadInput validates required fields (content_dir, content_type)
- Test DownloadOutput validates required fields (metadata_file, content_file, title, author)
- Test DownloadOutput handles both video and article content types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadInput extends StepInput with content_dir and content_type
- DownloadOutput extends StepOutput with metadata_file, content_file, title, author, duration
- Supports video, article, answer, zhuanlan content types
- All schema validation tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD RED phase:
- Test DownloadProcessor fetches Bilibili video metadata correctly
- Test DownloadProcessor fetches Zhihu answer metadata correctly
- Test DownloadProcessor records state on success
- Test DownloadProcessor records state on failure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor inherits from BaseProcessor
- Fetches metadata from Bilibili and Zhihu adapters
- Saves metadata to content_dir as JSON
- Records state on success/failure
- Skips step if already completed (idempotency)
- All metadata tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor downloads article/answer/zhuanlan content
- Video content is skipped (handled by ASR in Phase 3)
- Content files saved with correct naming convention
- DownloadProcessor exported from kb_pipeline.processors
- All 17 tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test schema validation for AnalyzeInput (metadata_file, content_file, content_type, output_file)
- Test schema validation for AnalyzeOutput (analysis_file, summary, main_topics, key_insights, tags, stats)
- Test AnalyzeProcessor integration with DeepAnalyzer
- Test state recording on success and failure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeInput extends StepInput with metadata_file, content_file, content_type, output_file
- AnalyzeOutput extends StepOutput with analysis_file, summary, main_topics, key_insights, tags, stats
- Supports both video and article content types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeProcessor wraps existing DeepAnalyzer for pipeline integration
- Loads metadata and content from JSON files
- Generates analysis reports as Markdown with frontmatter
- Records state on success and failure
- Supports both video and article content types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add AnalyzeProcessor to __init__.py exports
- Update module docstring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeProcessor wraps DeepAnalyzer for AI-powered content analysis
- Analysis reports saved as Markdown with YAML frontmatter
- All 15 tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test PipelineInput validates url field (required)
- Test PipelineOutput contains results dict
- Test PipelineConfig has default steps list
- Test PipelineConfig allows custom output_dir

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- PipelineInput: url (required), output_dir, skip_completed
- PipelineOutput: success, error_message, results, content_dir
- PipelineConfig: steps, output_dir, skip_completed, max_retries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pipeline class initializes with processors list
- Validates processor STEP_NAME matches config.steps
- _parse_url method parses URLs using LinkParser
- _get_content_dir creates content directory from parsed URL

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test process --help shows usage
- Test process <url> calls Pipeline.run()
- Test invalid URL returns error
- Test --output-dir flag
- Test --no-skip flag

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add process_command function that executes full pipeline
- Add subparser for 'process' command with URL argument
- Add --output-dir and --skip-completed flags
- Add --no-skip flag to re-run all steps
- Maintain backward compatibility with legacy arguments

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test full workflow execution from URL to analysis
- Test workflow error handling
- Test state persistence
- Test workflow resumability (skip completed steps)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add async run() method to Pipeline class
- Execute processors sequentially with state persistence
- Handle step-specific input types (Parse, Download, ASR, Analyze)
- Support skip_completed for resumable workflows
- Return PipelineOutput with results and content_dir

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CLI 'process' subcommand for single-command workflow execution
- End-to-end integration tests (16 tests passing)
- State persistence and resumability verified

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Archive ROADMAP and REQUIREMENTS to milestones/
- Delete original ROADMAP.md and REQUIREMENTS.md
- Update PROJECT.md with v1.0 validated requirements
- Update STATE.md for milestone complete status
- Update MILESTONES.md with key accomplishments
- Add VERIFICATION.md for phase 04

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Simplify to single-command workflow: kb-workflow process <url>
- Document new CLI parameters and options
- Add architecture overview
- Remove deprecated multi-step manual process

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
starmen1265 added 1 commit 2026-03-15 01:04:58 +08:00
Merge branch 'main' into fix/ci-deploy-reset
PR Test / lint (pull_request) Successful in 1s
f7569af007
starmen1265 merged commit 264dbe1cde into main 2026-03-15 01:05:14 +08:00
This repo is archived. You cannot comment on pull requests.
No Reviewers
No Label
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starmen1265/kb-skills#9