21 KiB
phase: 04-analysis-orchestration plan: 04 type: execute wave: 4 depends_on: [04-01, 04-02, 04-03] files_modified:
- kb_pipeline/pipeline.py
- kb_pipeline/processors/__init__.py
- kb-workflow/kb_workflow.py
- tests/test_pipeline/test_pipeline.py
- tests/test_cli/test_process_command.py autonomous: true requirements: [MNT-01] user_setup: []
must_haves: truths: - "Pipeline orchestrestr executes full workflow from URL to analysis" - "Single command triggers entire workflow" - "Progress is visible through simple logging" - "New developers can understand code in 30 minutes" artifacts: - path: "kb_pipeline/pipeline.py" provides: "Pipeline engine class" min_lines: 100 - path: "kb-workflow/kb_workflow.py" provides: "CLI entry point with process subcommand" min_lines: 50 - path: "tests/test_pipeline/test_pipeline.py" provides: "Pipeline tests" min_lines: 100 - path: "tests/test_cli/test_process_command.py" provides: "CLI tests" key_links: - from: "kb_pipeline/pipeline.py" to: "kb_pipeline/processors" via: "Pipeline.run()" pattern: "Pipeline\(" - from: "kb-workflow/kb_workflow.py" to: "kb_pipeline.processors" via: "import" pattern: "from.* import" - from: "kb_pipeline/processors" import ParseProcessor, DownloadProcessor, ASrProcessor, AnalyzeProcessor pattern: "from.* import.*"
Create Pipeline engine to orchest all processors and execute the complete workflow from URL to analysis report.Purpose: Enable single-command workflow execution with automatic state management and error recovery, and and new developer to easily understand and code结构. Output: Pipeline engine class, process subcommand, and CLI entry point.
<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-CONTEXT.md @.planning/phases/04-analysis-orchestration/04-RESEARCH.md@.planning/phases/04-analysis-orchestration/04-01-SUMMARY.md @.planning/phases/04-analysis-orchestration/04-02-SUMMARY.md @.planning/phases/04-analysis-orchestration/04-03-SUMMARY.md
From kb_pipeline/processors/base.py:
class BaseProcessor(ABC):
STEP_NAME: str
async def process(self, input_data: StepInput) -> StepOutput:
From kb_pipeline/processors/parse.py:
class ParseProcessor(BaseProcessor):
STEP_NAME = "parse"
async def process(self, input_data: ParseInput) -> ParseOutput
from kb_pipeline/processors/download.py:
class DownloadProcessor(BaseProcessor):
STEP_NAME = "download"
async def process(self, input_data: DownloadInput) -> DownloadOutput
from kb_pipeline/processors/asr.py:
class ASRProcessor:
STEP_NAME = "asr"
async def process(self, input_data: ASRInput) -> ASROutput
from kb_pipeline/processors/analyze.py:
class AnalyzeProcessor(BaseProcessor):
STEP_NAME = "analyze"
async def process(self, input_data: AnalyInput) -> AnalyOutput
from kb_pipeline/schemas.py:
class StepInput(BaseModel):
content_id: str = platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str]
completed_at: datetime
class ParseInput(StepInput):
url: str
class ParseOutput(StepOutput):
parsed: ParsedLink
content_dir: Path
metadata_file: Optional[str] = None
content_file: Optional[str] = None
content_type: str = "video" # video needs AS
# article不需要
content_file: Optional[str] = None # 从 download步骤获取
content_type: str # "video" or "article"
download_input.content_dir
download_output(
success=True,
metadata_file=f"{content_dir}/{platform}_{content_id}_metadata.json",
content_file=content_file,
title=metadata.get('title', '未知'),
author=metadata.get('author', '未知'),
duration=metadata.get('duration') # seconds, )
return state.mark_step_completed(
self.STEP_NAME, f"title={title}, author={author}, type={content_type}"
)
self.state_manager.save(state)
return output
from kb_pipeline.state import WorkflowStateManager from kb_pipeline.config import get_settings from kb_pipeline.errors import PipelineError
from kb_pipeline.processors.parse import ParseProcessor from kb_pipeline.processors.download import DownloadProcessor from kb_pipeline.processors.asr import ASRProcessor from kb_pipeline.processors.analyze import AnalyzeProcessor
from typing import List, Optional, Dict, Any, TYPE_CHECKING from pathlib import Path from datetime import datetime
import logging
logger = logging.getLogger(name)
class PipelineConfig: """Pipeline configuration""" steps: List[str] = Field(default_factory=lambda: ["parse", "download", "asr", "analyze"]) output_dir: Path skip_completed: bool = Field(default=True, description="Skip completed steps") max_retries: int = Field(default=3, description="Max retries for each step")
class Pipeline: """Pipeline engine - orchestrates multiple processors
Orchestrates the workflow:
1. Parse URL to extract platform and content ID
2. Run processors in order (parse, download, asr, analyze)
3. Handle errors and retry on failures
4. Return results
Key decisions:
- Sequential execution (not parallel - simpler to easier to debug
- State persistence after each step (via WorkflowStateManager)
- Progress logging to simple, clear
"""
def __init__(
self,
processors: List[BaseProcessor],
config: Optional[PipelineConfig] = None,
output_dir: Optional[Path] = None,
skip_completed: bool = False
):
self.processors = processors
self.config = config or PipelineConfig.default()
self.output_dir = output_dir
self.skip_completed = skip_completed
# Validate processors
for processor in self.processors:
if processor.STEP_NAME not in self.config.steps:
raise ValueError(f"Unknown processor: {processor.STEP_NAME}")
async def run(self, url: str) -> Dict[str, Any]:
"""Execute full workflow
Args:
url: URL to process
Returns:
Dict containing all step results
"""
results: Dict[str, Any] = {}
current_step = 0
total_steps = len(self.config.steps)
state: Optional[WorkflowState] = None
try:
for step_name in self.config.steps:
current_step += 1
logger.info(f"[Step {current_step}/{total_steps}] {step_name}...")
# Get processor
processor = self.processors.get(step_name)
if not processor:
raise ValueError(f"Processor not found: {step_name}")
# Get content directory from parsed URL
parsed = await self._parse_url(url)
if not parsed:
results["error"] = "Failed to parse URL"
return results
# Get or create content directory
content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
content_dir.mkdir(parents=True, exist_ok=True)
# Initialize state manager for this content
state_manager = WorkflowStateManager(content_dir)
# Run each step
previous_result = None
for step_name in self.config.steps:
# Check if already completed
if state and state.is_step_completed(step_name):
current_step += 1
logger.info(f"[Step {current_step}/{total_steps}] {step_name} - skipping (already completed)")
results[step_name] = {"status": "skipped", "message": "Already completed"}
continue
# Create step input
input_data = self._create_step_input(
step_name=step_name,
parsed=parsed,
state=state,
previous_result=previous_result
)
# Run processor
try:
output = await processor.process(input_data)
results[step_name] = {
"status": "success" if output.success else "failed",
"output": output.model_dump() if output.success else None,
"error": output.error_message
}
previous_result = output
except Exception as e:
results[step_name] = {
"status": "failed",
"error": str(e)
}
logger.error(f"[Step {current_step}/{total_steps}] {step_name} - failed: {e}")
# Don't stop pipeline on failure
raise
return results
def _parse_url(self, url: str) -> Optional[ParsedLink]:
"""Parse URL using LinkParser"""
if not url:
return None
parser = LinkParser()
parsed = parser.parse(url)
if not parsed:
logger.warning(f"Failed to parse URL: {url}")
return None
return ParsedLink
def _create_step_input(
self,
step_name: str,
parsed: ParsedLink,
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]],
) -> Dict[str, Any]:
"""Create step input based on step name and previous result"""
step_map = {
"parse": ParseProcessor,
"download": DownloadProcessor,
"asr": ASRProcessor,
"analyze": AnalyzeProcessor,
}
# Get processor instance
processor = self.processors[step_name]
if not processor:
raise ValueError(f"Unknown processor: {step_name}")
return step_map
async def _run_step(
self,
step_name: str,
input_data: Dict[str, Any],
state: Optional[WorkflowState],
previous_result: Optional[Dict[str, Any]]
) -> StepOutput:
"""Run a single step"""
processor = self.processors[step_name]
if not processor:
raise ValueError(f"No processor for {step_name}")
# Check if already completed
if state and state.is_step_completed(step_name):
logger.info(f"Step {step_name} already completed, skipping")
return self._create_skipped_output()
try:
# Run the
if previous_result and previous_result.get(step_name):
input_data = previous_result[step_name]
else:
raise ValueError(f"No previous result for {step_name}")
return await processor.process(input_data)
except Exception as e:
logger.error(f"Step {step_name} failed: {e}")
# Record failure
if state:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
raise
# Update previous result
previous_result[step_name] = output.model_dump()
return results
def get_output_dir(self) -> Path:
return Path(self.output_dir) / Path(self.config.output_dir)
Task 1: Define Pipeline Schemas
kb_pipeline/schemas.py, tests/test_pipeline/test_pipeline.py
- Test 1: PipelineInput validates required fields
- Test 2: PipelineOutput validates required fields
- Test 3: PipelineConfig default steps are correct
Add PipelineInput, PipelineOutput, and PipelineConfig to kb_pipeline/schemas.py.
PipelineInput fields (extends nothing - all fields inherited from StepInput):
-
content_dir: str - Directory to save downloaded content
-
content_type: str - Type of content (video, article, answer, zhuanlan)
-
skip_completed: bool - Whether to skip already completed steps (default True, content_type not to ["video", "article", "answer", "zhuanlan", "download", "asr", "analyze"]
-
steps: List[str] = Field(...)
output_dir: Path skip_completed: bool = True pytest tests/test_pipeline/test_pipeline.py -v -x -k"test_schema" in PipelineInput/Output schemas defined and validated
Implementation:
-
Initialize with processors list (ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor)
-
Set config with PipelineConfig.default()
-
Set output_dir
-
Initialize state_manager (WorkflowStateManager)
-
Implement run() method that:
- Creates content directory
- Runs steps in order (parse, download, asr, analyze)
- Returns results dict
-
Add helper methods:
- _parse_url(url) -> ParsedLink
- _get_content_dir(parsed)
- _run_step(step_name, parsed, state, previous_result)
- _record_state
-
Add error handling with try/except
- logger.error for error message if self.state_manager: state.mark_step_failed(step_name, str(e)) self.state_manager.save(state) raise
def _parse_url(self, url: str) -> Optional[ParsedLink]: """Parse URL using LinkParser""" if not url: return None
parser = LinkParser() parsed = parser.parse(url) if not parsed: return None return Parseddef get_content_dir(self, parsed: ParsedLink) -> Path: """Get content directory from parsed URL""" content_dir = self.output_dir / f"{parsed.platform}{parsed.content_id}" content_dir.mkdir(parents=True, exist_ok=True) return content_dir
def _create_step_input( self, step_name: str, parsed: ParsedLink, state: Optional[WorkflowState], previous_result: Optional[Dict[str, Any]] ) -> Dict[str, Any]: """Create step input based on step name and previous result""" step_map = { "parse": ParseProcessor, "download": DownloadProcessor, "asr": ASRProcessor, "analyze": AnalyzeProcessor }
# Get processor instance processor = self.processors[step_name] if not processor: raise ValueError(f"Unknown processor: {step_name}") return step_mapasync def _run_step( self, step_name: str, input_data: Dict[str, Any], state: Optional[WorkflowState], previous_result: Optional[Dict[str, Any]], ) -> StepOutput: """Run a single step""" processor = self.processors[step_name] if not processor: raise ValueError(f"No processor for {step_name}")
# Check if already completed if state and state.is_step_completed(step_name): logger.info(f"Step {step_name} already completed, skipping") return self._create_skipped_output() try: # Run step if previous_result and previous_result.get(step_name): input_data = previous_result[step_name] else: raise ValueError(f"No previous result for {step_name}") return await processor.process(input_data) except Exception as e: logger.error(f"Step {step_name} failed: {e}") if state: state.mark_step_failed(step_name, str(e)) self.state_manager.save(state) raise # Update previous result previous_result[step_name] = output.model_dump() return resultsdef _create_skipped_output(self) -> StepOutput: return StepOutput(success=True, error_message="Skipped: already completed")
- Create content directory
- Run steps in order
- Handle step skipping (check state)
- Handle errors (try/except, log and continue)
- Record final state
- Return results
Step execution logic:
async def _execute_steps(
self,
parsed: ParsedLink,
state: WorkflowState,
previous_result: Dict[str, Any]
) -> None:
for step_name in self.config.steps:
step_start = datetime.now()
state.mark_step_started(step_name)
logger.info(f"[Step {step_num}/{total_steps}] {step_name}...")
# Get or run processor
processor = self.processors[step_name]
if not processor:
raise ValueError(f"Unknown processor: {step_name}")
# Get step input
input_data = self._create_step_input(
step_name=step_name,
parsed=parsed,
state=state,
previous_result=previous_result.get(step_name)
)
# Run processor
try:
output = await processor.process(input_data)
previous_result[step_name] = output.model_dump()
results[step_name] = {
"status": "success" if output.success else "failed",
"output": output.model_dump() if output.success else None,
"error": output.error_message
}
previous_result = output
except Exception as e:
results[step_name] = {
"status": "failed",
"error": str(e)
}
logger.error(f"[Step {step_num}/{total_steps}] {step_name} - failed: {e}")
# Record failure
if state:
state.mark_step_failed(step_name, str(e))
self.state_manager.save(state)
# Stop on failure
raise
return results
</action>
<verify>
<automated>pytest tests/test_pipeline/test_pipeline.py -v -x</automated>
</verify>
<done>Pipeline engine executes workflow with state persistence</done>
</task>
<task type="auto">
<name>Task 4: Add CLI process command</name>
<files>kb-workflow/kb_workflow.py</files>
<action>
Extend kb-workflow/kb_workflow.py to add process subcommand.
**CLI additions:**
- New argument: 'url' with help='URL to process'
- New argument: '--output-dir', '-o' with help='Output directory (default: KB_BASE_PATH/content)
- New argument: '--no-analyze', action='store_true', help='Skip analysis step')
- Set default to process function that handles the command
- Call Pipeline().run(url, output_dir, skip_completed=no)
- Print success message
- If error: print error message and exit(1)
**Update existing help text and kb_workflow.py to document the new command.
</action>
<verify>
<automated>python kb-workflow/kb_workflow.py --help | grep process</automated>
</verify>
<done>CLI process command added and functional</done>
</task>
<task type="auto">
<name>Task 5: Create Pipeline tests</name>
<files>tests/test_pipeline/test_pipeline.py,files>
<action>
Create comprehensive tests for Pipeline class.
**Test cases:**
1. Test pipeline initialization
2. Test URL parsing and content dir creation
3. Test step execution order
4. Test step skipping
5. Test error handling and6. Test state persistence
7. Test full workflow (mocked)
Use fixtures for mocking processors, state_manager, settings.
</action>
<verify>
<automated>pytest tests/test_pipeline/test_pipeline.py -v</automated>
</verify>
<done>Pipeline class fully tested</done>
</task>
</tasks>
<verification>
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py tests/test_cli/test_process_command.py -v`
- Pipeline.run() executes workflow correctly
- State persistence works
- Error handling works
- Step skipping works
</verification>
<success_criteria>
- Pipeline class orchestrates all processors
- Pipeline.run() executes workflow from URL to analysis
- Content directory created correctly
- Steps executed in order
- State persisted after each step
- Errors handled with retry
- Completed steps skipped
- CLI process command works
</success_criteria>
<output>
After completion, create `.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md`
</output>