9.2 KiB
9.2 KiB
phase: 04-analysis-orchestration plan: 04 type: execute wave: 4 depends_on: [04-01, 04-02, 04-03] files_modified:
- kb_pipeline/schemas.py
- kb_pipeline/pipeline.py
- tests/test_pipeline/test_pipeline.py autonomous: true requirements: [MNT-01] user_setup: []
must_haves: truths: - "Pipeline schemas (PipelineInput/Output/Config) are defined" - "Pipeline class initializes with processors list" - "Pipeline orchestrates processor execution" artifacts: - path: "kb_pipeline/schemas.py" provides: "PipelineInput/Output/Config schemas" contains: "class PipelineInput" - path: "kb_pipeline/pipeline.py" provides: "Pipeline engine class" contains: "class Pipeline" min_lines: 50 - path: "tests/test_pipeline/test_pipeline.py" provides: "Pipeline unit tests" contains: "test_pipeline" key_links: - from: "kb_pipeline/pipeline.py" to: "kb_pipeline/processors" via: "process() calls" pattern: "processor.process("
Create Pipeline schemas and Pipeline class foundation to orchestrate processor execution.Purpose: Establish the core orchestration engine that coordinates all processors in a unified workflow. Output: PipelineInput/Output/Config schemas, Pipeline class with processor initialization and step execution logic.
<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-context.md @.planning/phases/04-analysis-orchestration/04-research.md@.planning/phases/04-analysis-orchestration/04-01-summary.md @.planning/phases/04-analysis-orchestration/04-02-summary.md @.planning/phases/04-analysis-orchestration/04-03-summary.md
From kb_pipeline/processors/base.py:
class BaseProcessor(ABC):
STEP_NAME: str
async def process(self, input_data: StepInput) -> StepOutput
from kb_pipeline/processors/parse.py:
class ParseProcessor(BaseProcessor):
STEP_NAME = "parse"
async def process(self, input_data: ParseInput) -> ParseOutput
from kb_pipeline/processors/download.py:
class DownloadProcessor(BaseProcessor):
STEP_NAME = "download"
async def process(self, input_data: DownloadInput) -> DownloadOutput
from kb_pipeline/processors/asr.py:
class ASRProcessor:
STEP_NAME = "asr"
async def process(self, input_data: ASRInput) -> ASROutput
from kb_pipeline/processors/analyze.py:
class AnalyzeProcessor(BaseProcessor):
STEP_NAME = "analyze"
async def process(self, input_data: AnalyzeInput) -> AnalyzeOutput
from kb_pipeline/schemas.py:
class StepInput(BaseModel):
content_id: str = Field(..., min_length=1)
platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str] = None
completed_at: datetime = Field(default_factory=datetime.now)
class ParseInput(StepInput):
url: str = Field(..., min_length=1, description="URL to parse")
class ParseOutput(StepOutput):
platform: str
content_type: str
content_id: str
resolved_url: str
original_url: str
from kb_pipeline/state.py:
class WorkflowStateManager:
STATE_FILENAME = "workflow_state.json"
def __init__(self, content_dir: Path): ...
def load(self) -> Optional[WorkflowState]: ...
def save(self, state: WorkflowState) -> None: ...
def get_or_create(self, content_id: str, platform: str, source_url: str) -> WorkflowState: ...
from kb_pipeline/config.py:
def get_settings() -> KBSettings: ...
PipelineInput fields:
class PipelineInput(BaseModel):
"""Pipeline input - URL to process"""
url: str = Field(..., min_length=1, description="URL to process")
output_dir: Optional[Path] = Field(None, description="Output directory for content")
skip_completed: bool = Field(default=True, description="Skip already completed steps")
PipelineOutput fields:
class PipelineOutput(BaseModel):
"""Pipeline output - results from all steps"""
success: bool
error_message: Optional[str] = None
results: Dict[str, Any] = Field(default_factory=dict)
content_dir: Optional[Path] = None
PipelineConfig fields:
class PipelineConfig(BaseModel):
"""Pipeline configuration"""
steps: List[str] = Field(
default_factory=lambda: ["parse", "download", "asr", "analyze"],
description="Steps to execute in order"
)
output_dir: Optional[Path] = Field(None, description="Base output directory")
skip_completed: bool = Field(default=True, description="Skip completed steps")
max_retries: int = Field(default=3, description="Max retries per step")
Create test file tests/test_pipeline/test_pipeline.py with schema validation tests. pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_schema" PipelineInput/Output/Config schemas defined and validated
Task 2: Create Pipeline class with processor initialization kb_pipeline/pipeline.py, tests/test_pipeline/test_pipeline.py - Test 1: Pipeline initializes with all processors - Test 2: Pipeline.parse_url creates correct content directory - Test 3: Pipeline validates processor STEP_NAME matches config.steps - Test 4: Pipeline handles missing processor gracefully Create `kb_pipeline/pipeline.py` with Pipeline class.Implementation:
from typing import List, Optional, Dict, Any
from pathlib import Path
import logging
from kb_pipeline.config import get_settings
from kb_pipeline.schemas import PipelineConfig, PipelineInput, PipelineOutput
from kb_pipeline.state import WorkflowStateManager
from kb_pipeline.link_parser import LinkParser, ParsedLink
from kb_pipeline.processors.base import BaseProcessor
logger = logging.getLogger(__name__)
class Pipeline:
"""Pipeline engine - orchestrates multiple processors
Orchestrates the workflow:
1. Parse URL to extract platform and content ID
2. Run processors in order (parse, download, asr, analyze)
3. Handle errors and state persistence
4. Return results
Key design decisions:
- Sequential execution (simpler, easier to debug)
- State persistence after each step (via WorkflowStateManager)
- Simple progress logging
"""
def __init__(
self,
processors: List[BaseProcessor],
config: Optional[PipelineConfig] = None,
output_dir: Optional[Path] = None,
):
self.processors = {p.STEP_NAME: p for p in processors}
self.config = config or PipelineConfig()
self.output_dir = output_dir or Path.cwd()
self.settings = get_settings()
# Validate processors
for step_name in self.processors:
if step_name not in self.config.steps:
raise ValueError(f"Unknown processor: {step_name}")
def _parse_url(self, url: str) -> Optional[ParsedLink]:
"""Parse URL using LinkParser"""
if not url:
return None
parser = LinkParser()
parsed = parser.parse(url)
if not parsed:
logger.warning(f"Failed to parse URL: {url}")
return None
return parsed
def _get_content_dir(self, parsed: ParsedLink) -> Path:
"""Get content directory from parsed URL"""
content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
content_dir.mkdir(parents=True, exist_ok=True)
return content_dir
Create comprehensive tests with mocked processors. pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_init or test_parse" Pipeline class initializes correctly and processor validation
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py -v -x` - PipelineInput/Output/Config schemas exist and validate - Pipeline class initializes with processors - URL parsing creates correct content directory - Processor validation works<success_criteria>
- PipelineInput/Output/Config schemas defined in schemas.py
- Pipeline class exists in pipeline.py
- Pipeline initializes with processor list
- URL parsing creates content directory
- All tests pass </success_criteria>