docs(04-analysis-orchestration): create phase plans

This commit is contained in:
2026-03-13 22:50:28 +08:00
parent debd53961a
commit 4b4ed9cab6
6 changed files with 1730 additions and 0 deletions
@@ -0,0 +1,658 @@
---
phase: 04-analysis-orchestration
plan: 04
type: execute
wave: 4
depends_on: [04-01, 04-02, 04-03]
files_modified:
- kb_pipeline/pipeline.py
- kb_pipeline/processors/__init__.py
- kb-workflow/kb_workflow.py
- tests/test_pipeline/test_pipeline.py
- tests/test_cli/test_process_command.py
autonomous: true
requirements: [MNT-01]
user_setup: []
must_haves:
truths:
- "Pipeline orchestrestr executes full workflow from URL to analysis"
- "Single command triggers entire workflow"
- "Progress is visible through simple logging"
- "New developers can understand code in 30 minutes"
artifacts:
- path: "kb_pipeline/pipeline.py"
provides: "Pipeline engine class"
min_lines: 100
- path: "kb-workflow/kb_workflow.py"
provides: "CLI entry point with process subcommand"
min_lines: 50
- path: "tests/test_pipeline/test_pipeline.py"
provides: "Pipeline tests"
min_lines: 100
- path: "tests/test_cli/test_process_command.py"
provides: "CLI tests"
key_links:
- from: "kb_pipeline/pipeline.py"
to: "kb_pipeline/processors"
via: "Pipeline.run()"
pattern: "Pipeline\\("
- from: "kb-workflow/kb_workflow.py"
to: "kb_pipeline.processors"
via: "import"
pattern: "from.* import"
- from: "kb_pipeline/processors" import ParseProcessor, DownloadProcessor, ASrProcessor, AnalyzeProcessor
pattern: "from.* import.*"
---
<objective>
Create Pipeline engine to orchest all processors and execute the complete workflow from URL to analysis report.
Purpose: Enable single-command workflow execution with automatic state management and error recovery, and and new developer to easily understand and code结构.
Output: Pipeline engine class, process subcommand, and CLI entry point.
</objective>
<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-analysis-orchestration/04-CONTEXT.md
@.planning/phases/04-analysis-orchestration/04-RESEARCH.md
@.planning/phases/04-analysis-orchestration/04-01-SUMMARY.md
@.planning/phases/04-analysis-orchestration/04-02-SUMMARY.md
@.planning/phases/04-analysis-orchestration/04-03-SUMMARY.md
<interfaces>
<!-- Key types and contracts from existing code. Executor should use these directly. -->
From kb_pipeline/processors/base.py:
```python
class BaseProcessor(ABC):
STEP_NAME: str
async def process(self, input_data: StepInput) -> StepOutput:
```
From kb_pipeline/processors/parse.py:
```python
class ParseProcessor(BaseProcessor):
STEP_NAME = "parse"
async def process(self, input_data: ParseInput) -> ParseOutput
```
from kb_pipeline/processors/download.py:
```python
class DownloadProcessor(BaseProcessor):
STEP_NAME = "download"
async def process(self, input_data: DownloadInput) -> DownloadOutput
```
from kb_pipeline/processors/asr.py:
```python
class ASRProcessor:
STEP_NAME = "asr"
async def process(self, input_data: ASRInput) -> ASROutput
```
from kb_pipeline/processors/analyze.py:
```python
class AnalyzeProcessor(BaseProcessor):
STEP_NAME = "analyze"
async def process(self, input_data: AnalyInput) -> AnalyOutput
```
from kb_pipeline/schemas.py:
```python
class StepInput(BaseModel):
content_id: str = platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str]
completed_at: datetime
class ParseInput(StepInput):
url: str
class ParseOutput(StepOutput):
parsed: ParsedLink
content_dir: Path
metadata_file: Optional[str] = None
content_file: Optional[str] = None
content_type: str = "video" # video needs AS
# article不需要
content_file: Optional[str] = None # 从 download步骤获取
content_type: str # "video" or "article"
download_input.content_dir
download_output(
success=True,
metadata_file=f"{content_dir}/{platform}_{content_id}_metadata.json",
content_file=content_file,
title=metadata.get('title', '未知'),
author=metadata.get('author', '未知'),
duration=metadata.get('duration') # seconds, )
return state.mark_step_completed(
self.STEP_NAME, f"title={title}, author={author}, type={content_type}"
)
self.state_manager.save(state)
return output
```
from kb_pipeline.state import WorkflowStateManager
from kb_pipeline.config import get_settings
from kb_pipeline.errors import PipelineError
from kb_pipeline.processors.parse import ParseProcessor
from kb_pipeline.processors.download import DownloadProcessor
from kb_pipeline.processors.asr import ASRProcessor
from kb_pipeline.processors.analyze import AnalyzeProcessor
from typing import List, Optional, Dict, Any, TYPE_CHECKING
from pathlib import Path
from datetime import datetime
import logging
logger = logging.getLogger(__name__)
class PipelineConfig:
"""Pipeline configuration"""
steps: List[str] = Field(default_factory=lambda: ["parse", "download", "asr", "analyze"])
output_dir: Path
skip_completed: bool = Field(default=True, description="Skip completed steps")
max_retries: int = Field(default=3, description="Max retries for each step")
class Pipeline:
"""Pipeline engine - orchestrates multiple processors
Orchestrates the workflow:
1. Parse URL to extract platform and content ID
2. Run processors in order (parse, download, asr, analyze)
3. Handle errors and retry on failures
4. Return results
Key decisions:
- Sequential execution (not parallel - simpler to easier to debug
- State persistence after each step (via WorkflowStateManager)
- Progress logging to simple, clear
"""
def __init__(
self,
processors: List[BaseProcessor],
config: Optional[PipelineConfig] = None,
output_dir: Optional[Path] = None,
skip_completed: bool = False
):
self.processors = processors
self.config = config or PipelineConfig.default()
self.output_dir = output_dir
self.skip_completed = skip_completed
# Validate processors
for processor in self.processors:
if processor.STEP_NAME not in self.config.steps:
raise ValueError(f"Unknown processor: {processor.STEP_NAME}")
async def run(self, url: str) -> Dict[str, Any]:
"""Execute full workflow
Args:
url: URL to process
Returns:
Dict containing all step results
"""
results: Dict[str, Any] = {}
current_step = 0
total_steps = len(self.config.steps)
state: Optional[WorkflowState] = None
try:
for step_name in self.config.steps:
current_step += 1
logger.info(f"[Step {current_step}/{total_steps}] {step_name}...")
# Get processor
processor = self.processors.get(step_name)
if not processor:
raise ValueError(f"Processor not found: {step_name}")
# Get content directory from parsed URL
parsed = await self._parse_url(url)
if not parsed:
results["error"] = "Failed to parse URL"
return results
# Get or create content directory
content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
content_dir.mkdir(parents=True, exist_ok=True)
# Initialize state manager for this content
state_manager = WorkflowStateManager(content_dir)
# Run each step
previous_result = None
for step_name in self.config.steps:
# Check if already completed
if state and state.is_step_completed(step_name):
current_step += 1
logger.info(f"[Step {current_step}/{total_steps}] {step_name} - skipping (already completed)")
results[step_name] = {"status": "skipped", "message": "Already completed"}
continue
# Create step input
input_data = self._create_step_input(
step_name=step_name,
parsed=parsed,
state=state,
previous_result=previous_result
)
# Run processor
try:
output = await processor.process(input_data)
results[step_name] = {
"status": "success" if output.success else "failed",
"output": output.model_dump() if output.success else None,
"error": output.error_message
}
previous_result = output
except Exception as e:
results[step_name] = {
"status": "failed",
"error": str(e)
}
logger.error(f"[Step {current_step}/{total_steps}] {step_name} - failed: {e}")
# Don't stop pipeline on failure
raise
return results
def _parse_url(self, url: str) -> Optional[ParsedLink]:
"""Parse URL using LinkParser"""
if not url:
return None
parser = LinkParser()
parsed = parser.parse(url)
if not parsed:
logger.warning(f"Failed to parse URL: {url}")
return None
return ParsedLink
def _create_step_input(
self,
step_name: str,
parsed: ParsedLink,
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]],
) -> Dict[str, Any]:
"""Create step input based on step name and previous result"""
step_map = {
"parse": ParseProcessor,
"download": DownloadProcessor,
"asr": ASRProcessor,
"analyze": AnalyzeProcessor,
}
# Get processor instance
processor = self.processors[step_name]
if not processor:
raise ValueError(f"Unknown processor: {step_name}")
return step_map
async def _run_step(
self,
step_name: str,
input_data: Dict[str, Any],
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]]
) -> StepOutput:
"""Run a single step"""
processor = self.processors[step_name]
if not processor:
raise ValueError(f"No processor for {step_name}")
# Check if already completed
if state and state.is_step_completed(step_name):
logger.info(f"Step {step_name} already completed, skipping")
return self._create_skipped_output()
try:
# Run the
if previous_result and previous_result.get(step_name):
input_data = previous_result[step_name]
else:
raise ValueError(f"No previous result for {step_name}")
return await processor.process(input_data)
except Exception as e:
logger.error(f"Step {step_name} failed: {e}")
# Record failure
if state:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
raise
# Update previous result
previous_result[step_name] = output.model_dump()
return results
def get_output_dir(self) -> Path:
return Path(self.output_dir) / Path(self.config.output_dir)
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Define Pipeline Schemas</name>
<files>kb_pipeline/schemas.py, tests/test_pipeline/test_pipeline.py</files>
<behavior>
- Test 1: PipelineInput validates required fields
- Test 2: PipelineOutput validates required fields
- Test 3: PipelineConfig default steps are correct
</behavior>
<action>
Add PipelineInput, PipelineOutput, and PipelineConfig to kb_pipeline/schemas.py.
**PipelineInput fields (extends nothing - all fields inherited from StepInput):**
- content_dir: str - Directory to save downloaded content
- content_type: str - Type of content (video, article, answer, zhuanlan)
- skip_completed: bool - Whether to skip already completed steps (default True, content_type not to ["video", "article", "answer", "zhuanlan", "download", "asr", "analyze"]
- steps: List[str] = Field(...)
output_dir: Path
skip_completed: bool = True
</action>
<verify>
<automated>pytest tests/test_pipeline/test_pipeline.py -v -x -k"test_schema" in</automated>
</verify>
<done>PipelineInput/Output schemas defined and validated</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Create Pipeline class with processor initialization</name>
<files>kb_pipeline/pipeline.py, tests/test_pipeline/test_pipeline.py</files>
<behavior>
- Test 1: Pipeline initializes with all processors
- Test 2: Pipeline.parse_url creates correct content directory
- Test 3: Pipeline.run() executes all steps in correct order
- Test 4: Pipeline.run() handles step skipping correctly
- Test 5: Pipeline.run() handles errors and continues to next step
- Test 6: Pipeline.run() records state for each step
</behavior>
<action>
Create `kb_pipeline/pipeline.py` with Pipeline class.
**Implementation:**
1. Initialize with processors list (ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor)
2. Set config with PipelineConfig.default()
3. Set output_dir
4. Initialize state_manager (WorkflowStateManager)
5. Implement run() method that:
- Creates content directory
- Runs steps in order (parse, download, asr, analyze)
- Returns results dict
6. Add helper methods:
- _parse_url(url) -> ParsedLink
- _get_content_dir(parsed)
- _run_step(step_name, parsed, state, previous_result)
- _record_state
7. Add error handling with try/except
- logger.error for error message
if self.state_manager:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
raise
def _parse_url(self, url: str) -> Optional[ParsedLink]:
"""Parse URL using LinkParser"""
if not url:
return None
parser = LinkParser()
parsed = parser.parse(url)
if not parsed:
return None
return Parsed
def _get_content_dir(self, parsed: ParsedLink) -> Path:
"""Get content directory from parsed URL"""
content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
content_dir.mkdir(parents=True, exist_ok=True)
return content_dir
def _create_step_input(
self,
step_name: str,
parsed: ParsedLink,
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]]
) -> Dict[str, Any]:
"""Create step input based on step name and previous result"""
step_map = {
"parse": ParseProcessor,
"download": DownloadProcessor,
"asr": ASRProcessor,
"analyze": AnalyzeProcessor
}
# Get processor instance
processor = self.processors[step_name]
if not processor:
raise ValueError(f"Unknown processor: {step_name}")
return step_map
async def _run_step(
self,
step_name: str,
input_data: Dict[str, Any],
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]],
) -> StepOutput:
"""Run a single step"""
processor = self.processors[step_name]
if not processor:
raise ValueError(f"No processor for {step_name}")
# Check if already completed
if state and state.is_step_completed(step_name):
logger.info(f"Step {step_name} already completed, skipping")
return self._create_skipped_output()
try:
# Run step
if previous_result and previous_result.get(step_name):
input_data = previous_result[step_name]
else:
raise ValueError(f"No previous result for {step_name}")
return await processor.process(input_data)
except Exception as e:
logger.error(f"Step {step_name} failed: {e}")
if state:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
raise
# Update previous result
previous_result[step_name] = output.model_dump()
return results
def _create_skipped_output(self) -> StepOutput:
return StepOutput(success=True, error_message="Skipped: already completed")
</task>
<task type="auto">
<name>Task 3: Add step execution logic</name>
<files>kb_pipeline/pipeline.py</files>
<action>
Extend Pipeline.run() to implement step execution logic:
1. Create content directory
2. Run steps in order
3. Handle step skipping (check state)
4. Handle errors (try/except, log and continue)
5. Record final state
6. Return results
**Step execution logic:**
```python
async def _execute_steps(
self,
parsed: ParsedLink,
state: WorkflowState,
previous_result: Dict[str, Any]
) -> None:
for step_name in self.config.steps:
step_start = datetime.now()
state.mark_step_started(step_name)
logger.info(f"[Step {step_num}/{total_steps}] {step_name}...")
# Get or run processor
processor = self.processors[step_name]
if not processor:
raise ValueError(f"Unknown processor: {step_name}")
# Get step input
input_data = self._create_step_input(
step_name=step_name,
parsed=parsed,
state=state,
previous_result=previous_result.get(step_name)
)
# Run processor
try:
output = await processor.process(input_data)
previous_result[step_name] = output.model_dump()
results[step_name] = {
"status": "success" if output.success else "failed",
"output": output.model_dump() if output.success else None,
"error": output.error_message
}
previous_result = output
except Exception as e:
results[step_name] = {
"status": "failed",
"error": str(e)
}
logger.error(f"[Step {step_num}/{total_steps}] {step_name} - failed: {e}")
# Record failure
if state:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
# Stop on failure
raise
return results
</action>
<verify>
<automated>pytest tests/test_pipeline/test_pipeline.py -v -x</automated>
</verify>
<done>Pipeline engine executes workflow with state persistence</done>
</task>
<task type="auto">
<name>Task 4: Add CLI process command</name>
<files>kb-workflow/kb_workflow.py</files>
<action>
Extend kb-workflow/kb_workflow.py to add process subcommand.
**CLI additions:**
- New argument: 'url' with help='URL to process'
- New argument: '--output-dir', '-o' with help='Output directory (default: KB_BASE_PATH/content)
- New argument: '--no-analyze', action='store_true', help='Skip analysis step')
- Set default to process function that handles the command
- Call Pipeline().run(url, output_dir, skip_completed=no)
- Print success message
- If error: print error message and exit(1)
**Update existing help text and kb_workflow.py to document the new command.
</action>
<verify>
<automated>python kb-workflow/kb_workflow.py --help | grep process</automated>
</verify>
<done>CLI process command added and functional</done>
</task>
<task type="auto">
<name>Task 5: Create Pipeline tests</name>
<files>tests/test_pipeline/test_pipeline.py,files>
<action>
Create comprehensive tests for Pipeline class.
**Test cases:**
1. Test pipeline initialization
2. Test URL parsing and content dir creation
3. Test step execution order
4. Test step skipping
5. Test error handling and6. Test state persistence
7. Test full workflow (mocked)
Use fixtures for mocking processors, state_manager, settings.
</action>
<verify>
<automated>pytest tests/test_pipeline/test_pipeline.py -v</automated>
</verify>
<done>Pipeline class fully tested</done>
</task>
</tasks>
<verification>
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py tests/test_cli/test_process_command.py -v`
- Pipeline.run() executes workflow correctly
- State persistence works
- Error handling works
- Step skipping works
</verification>
<success_criteria>
- Pipeline class orchestrates all processors
- Pipeline.run() executes workflow from URL to analysis
- Content directory created correctly
- Steps executed in order
- State persisted after each step
- Errors handled with retry
- Completed steps skipped
- CLI process command works
</success_criteria>
<output>
After completion, create `.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md`
</output>