docs(04-analysis-orchestration): create phase plans
This commit is contained in:
@@ -0,0 +1,288 @@
|
||||
---
|
||||
phase: 04-analysis-orchestration
|
||||
plan: 05
|
||||
type: execute
|
||||
wave: 5
|
||||
depends_on: [04-04]
|
||||
files_modified:
|
||||
- README.md
|
||||
- tests/test_integration/test_e2e_workflow.py
|
||||
autonomous: false
|
||||
requirements: [MNT-01]
|
||||
user_setup: []
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "End-to-end workflow completes successfully"
|
||||
- "Analysis report is saved to content directory"
|
||||
- "workflow_state.json correctly records all steps"
|
||||
artifacts:
|
||||
- path: "README.md"
|
||||
provides: "Project documentation"
|
||||
contains: "## Installation"
|
||||
- path: "tests/test_integration/test_e2e_workflow.py"
|
||||
provides: "End-to-end integration test"
|
||||
contains: "test_e2e"
|
||||
key_links:
|
||||
- from: "kb_pipeline/pipeline.py"
|
||||
to: "kb_pipeline/processors"
|
||||
via: "import processors"
|
||||
pattern: "from.*processor import"
|
||||
- from: "kb-workflow/kb_workflow.py"
|
||||
to: "kb_pipeline"
|
||||
via: "import pipeline"
|
||||
pattern: "from.*pipeline import"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Create end-to-end integration test and documentation ( project overview, installation, and and usage.
|
||||
|
||||
Purpose: Verify the complete workflow works and provide clear project documentation for maintainability.
|
||||
Output: README.md with project overview, tests/test_integration/test_e2e_workflow.py
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/04-analysis-orchestration/04-CONTEXT.md
|
||||
@.planning/phases/04-analysis-orchestration/04-RESEARCH.md
|
||||
@.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md
|
||||
|
||||
<interfaces>
|
||||
<!-- Key types and contracts from existing code. Executor should use these directly. -->
|
||||
|
||||
From kb_pipeline/pipeline.py:
|
||||
```python
|
||||
class Pipeline:
|
||||
"""Pipeline orchestration engine"""
|
||||
def __init__(self, ...): # From 04-01
|
||||
...
|
||||
```
|
||||
|
||||
From kb_pipeline/processors/base.py:
|
||||
```python
|
||||
class BaseProcessor(ABC):
|
||||
STEP_NAME: str
|
||||
async def process(self, input_data: StepInput) -> StepOutput
|
||||
```
|
||||
|
||||
from kb_pipeline/processors/parse.py:
|
||||
```python
|
||||
class ParseProcessor(BaseProcessor):
|
||||
STEP_NAME = "parse"
|
||||
async def process(self, input_data: ParseInput) -> ParseOutput
|
||||
```
|
||||
|
||||
from kb_pipeline/processors/download.py:
|
||||
```python
|
||||
class DownloadProcessor(BaseProcessor):
|
||||
STEP_NAME = "download"
|
||||
async def process(self, input_data: DownloadInput) -> DownloadOutput
|
||||
```
|
||||
|
||||
from kb_pipeline/processors/asr.py:
|
||||
```python
|
||||
class ASRProcessor:
|
||||
STEP_NAME = "asr"
|
||||
async def process(self, input_data: ASRInput) -> ASROutput
|
||||
```
|
||||
|
||||
from kb_pipeline/processors/analyze.py
|
||||
```python
|
||||
class AnalyzeProcessor(BaseProcessor):
|
||||
STEP_NAME = "analyze"
|
||||
async def process(self, input_data: AnalyzeInput) -> AnalyOutput
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Create README.md</name>
|
||||
<files>README.md</files>
|
||||
<behavior>
|
||||
- Test 1: README.md exists and contains project overview
|
||||
- Test 2: README.md contains installation instructions
|
||||
</behavior>
|
||||
<action>
|
||||
Create `README.md` in project root directory with the following sections:
|
||||
|
||||
**## kb-skills**
|
||||
|
||||
知识库工作流工具,从链接到分析报告的端到端自动化处理系统。
|
||||
|
||||
### 格 Features
|
||||
|
||||
- 统一 Processor 模式: Parse -> Download -> ASR -> Analyze
|
||||
- 状态管理: workflow_state.json
|
||||
- 配置管理: 通过 ~/.kb_env
|
||||
- 输出: Markdown 格式分析报告
|
||||
|
||||
- 幂等性保证: 重复执行不会产生重复文件
|
||||
|
||||
- 断点续传: 中断后可继续
|
||||
|
||||
- 模块化设计: 新人能在30分钟内理解代码结构
|
||||
- 清晰的项目结构
|
||||
|
||||
- 统一的 CLI 入
|
||||
|
||||
entry
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# 安装依赖
|
||||
pip install kb-pipeline
|
||||
```
|
||||
|
||||
验证安装:
|
||||
```bash
|
||||
# 测试
|
||||
pytest tests/test_pipeline/test_schemas.py tests/test_pipeline/test_state.py tests/test_pipeline/test_config.py tests/test_pipeline/test_errors.py tests/test_pipeline/test_retry.py
|
||||
```
|
||||
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "import os; content = open('README.md', 'r', encoding='utf-8'); print(content[:100]) > 0"
|
||||
</verify>
|
||||
<done>README.md created with project overview, installation, and, usage sections, docstrings</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Create end-to-end integration test</name>
|
||||
<files>tests/test_integration/test_e2e_workflow.py</files>
|
||||
<behavior>
|
||||
- Test 1: Full workflow executes from URL to analysis
|
||||
- Test 2: Workflow handles errors gracefully
|
||||
- Test 3: State is persisted correctly
|
||||
- Test 4: Workflow is resumable
|
||||
</behavior>
|
||||
<action>
|
||||
Create `tests/test_integration/test_e2e_workflow.py` with comprehensive end-to-end tests.
|
||||
|
||||
**Test structure:**
|
||||
```python
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from unittest.mock import AsyncMock, patch, MagicMock
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
|
||||
from kb_pipeline.pipeline import Pipeline
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_content_dir(tmp_path):
|
||||
"""Create temp content directory"""
|
||||
content_dir.mkdir(parents=True, exist_ok=True)
|
||||
yield content_dir
|
||||
|
||||
# Create test workflow state
|
||||
state_path = content_dir / "workflow_state.json"
|
||||
state = WorkflowState(
|
||||
content_id="test123",
|
||||
platform="bilibili",
|
||||
source_url="https://test.url"
|
||||
)
|
||||
state_manager = WorkflowStateManager(content_dir)
|
||||
yield state_manager
|
||||
|
||||
# Create real pipeline
|
||||
settings = get_settings()
|
||||
pipeline = Pipeline(
|
||||
processors=[
|
||||
ParseProcessor(state_manager),
|
||||
DownloadProcessor(state_manager),
|
||||
ASRProcessor(state_manager),
|
||||
AnalyzeProcessor(state_manager),
|
||||
],
|
||||
output_dir=content_dir,
|
||||
config=PipelineConfig(
|
||||
steps=["parse", "download", "asr", "analyze"],
|
||||
)
|
||||
)
|
||||
yield pipeline
|
||||
# Mock workflow run
|
||||
result = await pipeline.run("https://test.url")
|
||||
|
||||
# Verify success
|
||||
assert result["parse"]["success"]
|
||||
assert result["download"]["success"]
|
||||
assert result["asr"]["success"] or result["asr"]["skipped"] # ASR not applicable for text content
|
||||
assert result["analyze"]["success"]
|
||||
|
||||
# Verify state was saved
|
||||
state = state_manager.load()
|
||||
assert state is not None
|
||||
assert state.is_step_completed("parse")
|
||||
assert state.is_step_completed("download")
|
||||
if content_type in ["video", "zvideo"]:
|
||||
assert state.is_step_completed("asr")
|
||||
assert not state.is_step_completed("asr")
|
||||
assert state.is_step_completed("analyze")
|
||||
|
||||
# Verify output files exist
|
||||
assert (content_dir / "bilibili_test123_metadata.json").exists()
|
||||
assert (content_dir / "bilibili_test123_content.json").exists() or result["content_type"] == "video":
|
||||
assert (content_dir / "bilibili_test123_analysis.md").exists()
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_integration/test_e2e_workflow.py -v</automated>
|
||||
</verify>
|
||||
<done>End-to-end workflow tested with state persistence</done>
|
||||
</task>
|
||||
|
||||
<task type="checkpoint:human-verify" gate="blocking">
|
||||
<what-built>
|
||||
End-to-end workflow (Phase 1-2/3):
|
||||
- Pipeline Engine with 4 processors (Parse, Download, ASR, Analyze)
|
||||
- CLI entry point (kb-workflow process command)
|
||||
- DeepAnalyzer integration for analysis reports saved as Markdown
|
||||
- State persistence via workflow_state.json
|
||||
</what-built>
|
||||
DeepAnalyzer integration:
|
||||
- B站视频/专栏链接 → parse, download, ASR, analyze
|
||||
- Error recovery with retry
|
||||
- Progress logging with simple format
|
||||
- Documentation in README.md and docstrings
|
||||
</how-to-verify>
|
||||
1. Run `pytest tests/test_integration/test_e2e_workflow.py -v -x`
|
||||
2. Visit https://github.com/your-org/kb-skills
|
||||
3. Run `kb-workflow process <url>` to test
|
||||
4. Run `kb-workflow process --help` to see full usage
|
||||
</how-to-verify>
|
||||
<resume-signal>Type "approved" or describe any issues found
|
||||
release-signal>Type: "approved" to indicate you should proceed to Phase 4 for finalize the documentation
|
||||
</how-to-verify>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- All tests pass
|
||||
- End-to-end workflow tested with state persistence
|
||||
- README.md exists with project overview
|
||||
- tests pass
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- README.md complete with project overview, installation, and
|
||||
usage
|
||||
- End-to-end integration test passes
|
||||
- workflow_state.json correctly records all steps
|
||||
- Documentation is clear and comprehensive
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/04-analysis-orchestration/04-05-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user