kb-skills/.planning/phases/04-analysis-orchestration/04-04-PLAN.md

---
phase: 04-analysis-orchestration
plan: 04
type: execute
wave: 4
depends_on: [04-01, 04-02, 04-03]
files_modified:
  - kb_pipeline/schemas.py
  - kb_pipeline/pipeline.py
  - tests/test_pipeline/test_pipeline.py
autonomous: true
requirements: [MNT-01]
user_setup: []

must_haves:
  truths:
    - "Pipeline schemas (PipelineInput/Output/Config) are defined"
    - "Pipeline class initializes with processors list"
    - "Pipeline orchestrates processor execution"
  artifacts:
    - path: "kb_pipeline/schemas.py"
      provides: "PipelineInput/Output/Config schemas"
      contains: "class PipelineInput"
    - path: "kb_pipeline/pipeline.py"
      provides: "Pipeline engine class"
      contains: "class Pipeline"
      min_lines: 50
    - path: "tests/test_pipeline/test_pipeline.py"
      provides: "Pipeline unit tests"
      contains: "test_pipeline"
  key_links:
    - from: "kb_pipeline/pipeline.py"
      to: "kb_pipeline/processors"
      via: "process() calls"
      pattern: "processor\.process\("
---

<objective>
Create Pipeline schemas and Pipeline class foundation to orchestrate processor execution.

Purpose: Establish the core orchestration engine that coordinates all processors in a unified workflow.
Output: PipelineInput/Output/Config schemas, Pipeline class with processor initialization and step execution logic.
</objective>

<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-analysis-orchestration/04-context.md
@.planning/phases/04-analysis-orchestration/04-research.md

@.planning/phases/04-analysis-orchestration/04-01-summary.md
@.planning/phases/04-analysis-orchestration/04-02-summary.md
@.planning/phases/04-analysis-orchestration/04-03-summary.md

<interfaces>
<!-- Key types and contracts from existing codebase. Executor should use these directly. -->

From kb_pipeline/processors/base.py:
```python
class BaseProcessor(ABC):
    STEP_NAME: str
    async def process(self, input_data: StepInput) -> StepOutput
```

from kb_pipeline/processors/parse.py:
```python
class ParseProcessor(BaseProcessor):
    STEP_NAME = "parse"
    async def process(self, input_data: ParseInput) -> ParseOutput
```

from kb_pipeline/processors/download.py:
```python
class DownloadProcessor(BaseProcessor):
    STEP_NAME = "download"
    async def process(self, input_data: DownloadInput) -> DownloadOutput
```

from kb_pipeline/processors/asr.py:
```python
class ASRProcessor:
    STEP_NAME = "asr"
    async def process(self, input_data: ASRInput) -> ASROutput
```

from kb_pipeline/processors/analyze.py:
```python
class AnalyzeProcessor(BaseProcessor):
    STEP_NAME = "analyze"
    async def process(self, input_data: AnalyzeInput) -> AnalyzeOutput
```

from kb_pipeline/schemas.py:
```python
class StepInput(BaseModel):
    content_id: str = Field(..., min_length=1)
    platform: Literal["bilibili", "zhihu"]

class StepOutput(BaseModel):
    success: bool
    error_message: Optional[str] = None
    completed_at: datetime = Field(default_factory=datetime.now)

class ParseInput(StepInput):
    url: str = Field(..., min_length=1, description="URL to parse")

class ParseOutput(StepOutput):
    platform: str
    content_type: str
    content_id: str
    resolved_url: str
    original_url: str
```

from kb_pipeline/state.py:
```python
class WorkflowStateManager:
    STATE_FILENAME = "workflow_state.json"
    def __init__(self, content_dir: Path): ...
    def load(self) -> Optional[WorkflowState]: ...
    def save(self, state: WorkflowState) -> None: ...
    def get_or_create(self, content_id: str, platform: str, source_url: str) -> WorkflowState: ...
```

from kb_pipeline/config.py:
```python
def get_settings() -> KBSettings: ...
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Define Pipeline Schemas</name>
  <files>kb_pipeline/schemas.py, tests/test_pipeline/test_pipeline.py</files>
  <behavior>
    - Test 1: PipelineInput validates url field (required)
    - Test 2: PipelineOutput contains results dict
    - Test 3: PipelineConfig has default steps list
    - Test 4: PipelineConfig allows custom output_dir
  </behavior>
  <action>
Add PipelineInput, PipelineOutput, and PipelineConfig to kb_pipeline/schemas.py.

**PipelineInput fields:**
```python
class PipelineInput(BaseModel):
    """Pipeline input - URL to process"""
    url: str = Field(..., min_length=1, description="URL to process")
    output_dir: Optional[Path] = Field(None, description="Output directory for content")
    skip_completed: bool = Field(default=True, description="Skip already completed steps")
```

**PipelineOutput fields:**
```python
class PipelineOutput(BaseModel):
    """Pipeline output - results from all steps"""
    success: bool
    error_message: Optional[str] = None
    results: Dict[str, Any] = Field(default_factory=dict)
    content_dir: Optional[Path] = None
```

**PipelineConfig fields:**
```python
class PipelineConfig(BaseModel):
    """Pipeline configuration"""
    steps: List[str] = Field(
        default_factory=lambda: ["parse", "download", "asr", "analyze"],
        description="Steps to execute in order"
    )
    output_dir: Optional[Path] = Field(None, description="Base output directory")
    skip_completed: bool = Field(default=True, description="Skip completed steps")
    max_retries: int = Field(default=3, description="Max retries per step")
```

Create test file tests/test_pipeline/test_pipeline.py with schema validation tests.
  </action>
  <verify>
    <automated>pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_schema"</automated>
  </verify>
  <done>PipelineInput/Output/Config schemas defined and validated</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Create Pipeline class with processor initialization</name>
  <files>kb_pipeline/pipeline.py, tests/test_pipeline/test_pipeline.py</files>
  <behavior>
    - Test 1: Pipeline initializes with all processors
    - Test 2: Pipeline.parse_url creates correct content directory
    - Test 3: Pipeline validates processor STEP_NAME matches config.steps
    - Test 4: Pipeline handles missing processor gracefully
  </behavior>
  <action>
Create `kb_pipeline/pipeline.py` with Pipeline class.

**Implementation:**
```python
from typing import List, Optional, Dict, Any
from pathlib import Path
import logging

from kb_pipeline.config import get_settings
from kb_pipeline.schemas import PipelineConfig, PipelineInput, PipelineOutput
from kb_pipeline.state import WorkflowStateManager
from kb_pipeline.link_parser import LinkParser, ParsedLink
from kb_pipeline.processors.base import BaseProcessor

logger = logging.getLogger(__name__)


class Pipeline:
    """Pipeline engine - orchestrates multiple processors

    Orchestrates the workflow:
    1. Parse URL to extract platform and content ID
    2. Run processors in order (parse, download, asr, analyze)
    3. Handle errors and state persistence
    4. Return results

    Key design decisions:
    - Sequential execution (simpler, easier to debug)
    - State persistence after each step (via WorkflowStateManager)
    - Simple progress logging
    """

    def __init__(
        self,
        processors: List[BaseProcessor],
        config: Optional[PipelineConfig] = None,
        output_dir: Optional[Path] = None,
    ):
        self.processors = {p.STEP_NAME: p for p in processors}
        self.config = config or PipelineConfig()
        self.output_dir = output_dir or Path.cwd()
        self.settings = get_settings()

        # Validate processors
        for step_name in self.processors:
            if step_name not in self.config.steps:
                raise ValueError(f"Unknown processor: {step_name}")

    def _parse_url(self, url: str) -> Optional[ParsedLink]:
        """Parse URL using LinkParser"""
        if not url:
            return None
        parser = LinkParser()
        parsed = parser.parse(url)
        if not parsed:
            logger.warning(f"Failed to parse URL: {url}")
            return None
        return parsed

    def _get_content_dir(self, parsed: ParsedLink) -> Path:
        """Get content directory from parsed URL"""
        content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
        content_dir.mkdir(parents=True, exist_ok=True)
        return content_dir
```

Create comprehensive tests with mocked processors.
  </action>
  <verify>
    <automated>pytest tests/test_pipeline/test_pipeline.py -v -x -k "test_init or test_parse"</automated>
  </verify>
  <done>Pipeline class initializes correctly and processor validation</done>
</task>

</tasks>

<verification>
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py -v -x`
- PipelineInput/Output/Config schemas exist and validate
- Pipeline class initializes with processors
- URL parsing creates correct content directory
- Processor validation works
</verification>

<success_criteria>
- PipelineInput/Output/Config schemas defined in schemas.py
- Pipeline class exists in pipeline.py
- Pipeline initializes with processor list
- URL parsing creates content directory
- All tests pass
</success_criteria>

<output>
After completion, create `.planning/phases/04-analysis-orchestration/04-04-SUMmary.md`
</output>