---
phase: 04-analysis-orchestration
plan: 04
type: execute
wave: 4
depends_on: [04-01, 04-02, 04-03]
files_modified:
- kb_pipeline/schemas.py
- kb_pipeline/pipeline.py
- tests/test_pipeline/test_pipeline.py
autonomous: true
requirements: [MNT-01]
user_setup: []
must_haves:
truths:
- "Pipeline schemas (PipelineInput/Output/Config) are defined"
- "Pipeline class initializes with processors list"
- "Pipeline orchestrates processor execution"
artifacts:
- path: "kb_pipeline/schemas.py"
provides: "PipelineInput/Output/Config schemas"
contains: "class PipelineInput"
- path: "kb_pipeline/pipeline.py"
provides: "Pipeline engine class"
contains: "class Pipeline"
min_lines: 50
- path: "tests/test_pipeline/test_pipeline.py"
provides: "Pipeline unit tests"
contains: "test_pipeline"
key_links:
- from: "kb_pipeline/pipeline.py"
to: "kb_pipeline/processors"
via: "process() calls"
pattern: "processor\.process\("
---
Create Pipeline schemas and Pipeline class foundation to orchestrate processor execution.
Purpose: Establish the core orchestration engine that coordinates all processors in a unified workflow.
Output: PipelineInput/Output/Config schemas, Pipeline class with processor initialization and step execution logic.
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-analysis-orchestration/04-context.md
@.planning/phases/04-analysis-orchestration/04-research.md
@.planning/phases/04-analysis-orchestration/04-01-summary.md
@.planning/phases/04-analysis-orchestration/04-02-summary.md
@.planning/phases/04-analysis-orchestration/04-03-summary.md
From kb_pipeline/processors/base.py:
```python
class BaseProcessor(ABC):
STEP_NAME: str
async def process(self, input_data: StepInput) -> StepOutput
```
from kb_pipeline/processors/parse.py:
```python
class ParseProcessor(BaseProcessor):
STEP_NAME = "parse"
async def process(self, input_data: ParseInput) -> ParseOutput
```
from kb_pipeline/processors/download.py:
```python
class DownloadProcessor(BaseProcessor):
STEP_NAME = "download"
async def process(self, input_data: DownloadInput) -> DownloadOutput
```
from kb_pipeline/processors/asr.py:
```python
class ASRProcessor:
STEP_NAME = "asr"
async def process(self, input_data: ASRInput) -> ASROutput
```
from kb_pipeline/processors/analyze.py:
```python
class AnalyzeProcessor(BaseProcessor):
STEP_NAME = "analyze"
async def process(self, input_data: AnalyzeInput) -> AnalyzeOutput
```
from kb_pipeline/schemas.py:
```python
class StepInput(BaseModel):
content_id: str = Field(..., min_length=1)
platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str] = None
completed_at: datetime = Field(default_factory=datetime.now)
class ParseInput(StepInput):
url: str = Field(..., min_length=1, description="URL to parse")
class ParseOutput(StepOutput):
platform: str
content_type: str
content_id: str
resolved_url: str
original_url: str
```
from kb_pipeline/state.py:
```python
class WorkflowStateManager:
STATE_FILENAME = "workflow_state.json"
def __init__(self, content_dir: Path): ...
def load(self) -> Optional[WorkflowState]: ...
def save(self, state: WorkflowState) -> None: ...
def get_or_create(self, content_id: str, platform: str, source_url: str) -> WorkflowState: ...
```
from kb_pipeline/config.py:
```python
def get_settings() -> KBSettings: ...
```
Task 1: Define Pipeline Schemas
kb_pipeline/schemas.py, tests/test_pipeline/test_pipeline.py
- Test 1: PipelineInput validates url field (required)
- Test 2: PipelineOutput contains results dict
- Test 3: PipelineConfig has default steps list
- Test 4: PipelineConfig allows custom output_dir
Add PipelineInput, PipelineOutput, and PipelineConfig to kb_pipeline/schemas.py.
**PipelineInput fields:**
```python
class PipelineInput(BaseModel):
"""Pipeline input - URL to process"""
url: str = Field(..., min_length=1, description="URL to process")
output_dir: Optional[Path] = Field(None, description="Output directory for content")
skip_completed: bool = Field(default=True, description="Skip already completed steps")
```
**PipelineOutput fields:**
```python
class PipelineOutput(BaseModel):
"""Pipeline output - results from all steps"""
success: bool
error_message: Optional[str] = None
results: Dict[str, Any] = Field(default_factory=dict)
content_dir: Optional[Path] = None
```
**PipelineConfig fields:**
```python
class PipelineConfig(BaseModel):
"""Pipeline configuration"""
steps: List[str] = Field(
default_factory=lambda: ["parse", "download", "asr", "analyze"],
description="Steps to execute in order"
)
output_dir: Optional[Path] = Field(None, description="Base output directory")
skip_completed: bool = Field(default=True, description="Skip completed steps")
max_retries: int = Field(default=3, description="Max retries per step")
```
Create test file tests/test_pipeline/test_pipeline.py with schema validation tests.
pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_schema"
PipelineInput/Output/Config schemas defined and validated
Task 2: Create Pipeline class with processor initialization
kb_pipeline/pipeline.py, tests/test_pipeline/test_pipeline.py
- Test 1: Pipeline initializes with all processors
- Test 2: Pipeline.parse_url creates correct content directory
- Test 3: Pipeline validates processor STEP_NAME matches config.steps
- Test 4: Pipeline handles missing processor gracefully
Create `kb_pipeline/pipeline.py` with Pipeline class.
**Implementation:**
```python
from typing import List, Optional, Dict, Any
from pathlib import Path
import logging
from kb_pipeline.config import get_settings
from kb_pipeline.schemas import PipelineConfig, PipelineInput, PipelineOutput
from kb_pipeline.state import WorkflowStateManager
from kb_pipeline.link_parser import LinkParser, ParsedLink
from kb_pipeline.processors.base import BaseProcessor
logger = logging.getLogger(__name__)
class Pipeline:
"""Pipeline engine - orchestrates multiple processors
Orchestrates the workflow:
1. Parse URL to extract platform and content ID
2. Run processors in order (parse, download, asr, analyze)
3. Handle errors and state persistence
4. Return results
Key design decisions:
- Sequential execution (simpler, easier to debug)
- State persistence after each step (via WorkflowStateManager)
- Simple progress logging
"""
def __init__(
self,
processors: List[BaseProcessor],
config: Optional[PipelineConfig] = None,
output_dir: Optional[Path] = None,
):
self.processors = {p.STEP_NAME: p for p in processors}
self.config = config or PipelineConfig()
self.output_dir = output_dir or Path.cwd()
self.settings = get_settings()
# Validate processors
for step_name in self.processors:
if step_name not in self.config.steps:
raise ValueError(f"Unknown processor: {step_name}")
def _parse_url(self, url: str) -> Optional[ParsedLink]:
"""Parse URL using LinkParser"""
if not url:
return None
parser = LinkParser()
parsed = parser.parse(url)
if not parsed:
logger.warning(f"Failed to parse URL: {url}")
return None
return parsed
def _get_content_dir(self, parsed: ParsedLink) -> Path:
"""Get content directory from parsed URL"""
content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
content_dir.mkdir(parents=True, exist_ok=True)
return content_dir
```
Create comprehensive tests with mocked processors.
pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_init or test_parse"
Pipeline class initializes correctly and processor validation
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py -v -x`
- PipelineInput/Output/Config schemas exist and validate
- Pipeline class initializes with processors
- URL parsing creates correct content directory
- Processor validation works
- PipelineInput/Output/Config schemas defined in schemas.py
- Pipeline class exists in pipeline.py
- Pipeline initializes with processor list
- URL parsing creates content directory
- All tests pass