This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
kb-skills/.planning/phases/04-analysis-orchestration/04-04-PLAN.md
T

21 KiB


phase: 04-analysis-orchestration plan: 04 type: execute wave: 4 depends_on: [04-01, 04-02, 04-03] files_modified:

  • kb_pipeline/pipeline.py
  • kb_pipeline/processors/__init__.py
  • kb-workflow/kb_workflow.py
  • tests/test_pipeline/test_pipeline.py
  • tests/test_cli/test_process_command.py autonomous: true requirements: [MNT-01] user_setup: []

must_haves: truths: - "Pipeline orchestrestr executes full workflow from URL to analysis" - "Single command triggers entire workflow" - "Progress is visible through simple logging" - "New developers can understand code in 30 minutes" artifacts: - path: "kb_pipeline/pipeline.py" provides: "Pipeline engine class" min_lines: 100 - path: "kb-workflow/kb_workflow.py" provides: "CLI entry point with process subcommand" min_lines: 50 - path: "tests/test_pipeline/test_pipeline.py" provides: "Pipeline tests" min_lines: 100 - path: "tests/test_cli/test_process_command.py" provides: "CLI tests" key_links: - from: "kb_pipeline/pipeline.py" to: "kb_pipeline/processors" via: "Pipeline.run()" pattern: "Pipeline\(" - from: "kb-workflow/kb_workflow.py" to: "kb_pipeline.processors" via: "import" pattern: "from.* import" - from: "kb_pipeline/processors" import ParseProcessor, DownloadProcessor, ASrProcessor, AnalyzeProcessor pattern: "from.* import.*"

Create Pipeline engine to orchest all processors and execute the complete workflow from URL to analysis report.

Purpose: Enable single-command workflow execution with automatic state management and error recovery, and and new developer to easily understand and code结构. Output: Pipeline engine class, process subcommand, and CLI entry point.

<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-CONTEXT.md @.planning/phases/04-analysis-orchestration/04-RESEARCH.md

@.planning/phases/04-analysis-orchestration/04-01-SUMMARY.md @.planning/phases/04-analysis-orchestration/04-02-SUMMARY.md @.planning/phases/04-analysis-orchestration/04-03-SUMMARY.md

From kb_pipeline/processors/base.py:

class BaseProcessor(ABC):
    STEP_NAME: str
    async def process(self, input_data: StepInput) -> StepOutput:

From kb_pipeline/processors/parse.py:

class ParseProcessor(BaseProcessor):
    STEP_NAME = "parse"
    async def process(self, input_data: ParseInput) -> ParseOutput

from kb_pipeline/processors/download.py:

class DownloadProcessor(BaseProcessor):
    STEP_NAME = "download"
    async def process(self, input_data: DownloadInput) -> DownloadOutput

from kb_pipeline/processors/asr.py:

class ASRProcessor:
    STEP_NAME = "asr"
    async def process(self, input_data: ASRInput) -> ASROutput

from kb_pipeline/processors/analyze.py:

class AnalyzeProcessor(BaseProcessor):
    STEP_NAME = "analyze"
    async def process(self, input_data: AnalyInput) -> AnalyOutput

from kb_pipeline/schemas.py:

class StepInput(BaseModel):
    content_id: str =    platform: Literal["bilibili", "zhihu"]

class StepOutput(BaseModel):
    success: bool
    error_message: Optional[str]
    completed_at: datetime

class ParseInput(StepInput):
    url: str

class ParseOutput(StepOutput):
    parsed: ParsedLink
    content_dir: Path
    metadata_file: Optional[str] = None
    content_file: Optional[str] = None
    content_type: str = "video"  # video needs AS

    # article不需要

    content_file: Optional[str] = None  # 从 download步骤获取

    content_type: str  # "video" or "article"
    download_input.content_dir
    download_output(
        success=True,
        metadata_file=f"{content_dir}/{platform}_{content_id}_metadata.json",
        content_file=content_file,
        title=metadata.get('title', '未知'),
        author=metadata.get('author', '未知'),
        duration=metadata.get('duration')  # seconds,    )

    return state.mark_step_completed(
        self.STEP_NAME, f"title={title}, author={author}, type={content_type}"
    )
        self.state_manager.save(state)
    return output

from kb_pipeline.state import WorkflowStateManager from kb_pipeline.config import get_settings from kb_pipeline.errors import PipelineError

from kb_pipeline.processors.parse import ParseProcessor from kb_pipeline.processors.download import DownloadProcessor from kb_pipeline.processors.asr import ASRProcessor from kb_pipeline.processors.analyze import AnalyzeProcessor

from typing import List, Optional, Dict, Any, TYPE_CHECKING from pathlib import Path from datetime import datetime

import logging

logger = logging.getLogger(name)

class PipelineConfig: """Pipeline configuration""" steps: List[str] = Field(default_factory=lambda: ["parse", "download", "asr", "analyze"]) output_dir: Path skip_completed: bool = Field(default=True, description="Skip completed steps") max_retries: int = Field(default=3, description="Max retries for each step")

class Pipeline: """Pipeline engine - orchestrates multiple processors

Orchestrates the workflow:
1. Parse URL to extract platform and content ID
2. Run processors in order (parse, download, asr, analyze)
3. Handle errors and retry on failures
4. Return results

Key decisions:
- Sequential execution (not parallel - simpler to easier to debug
- State persistence after each step (via WorkflowStateManager)
- Progress logging to simple, clear
"""

def __init__(
    self,
    processors: List[BaseProcessor],
    config: Optional[PipelineConfig] = None,
    output_dir: Optional[Path] = None,
    skip_completed: bool = False
):
    self.processors = processors
    self.config = config or PipelineConfig.default()
    self.output_dir = output_dir

    self.skip_completed = skip_completed

    # Validate processors
    for processor in self.processors:
        if processor.STEP_NAME not in self.config.steps:
            raise ValueError(f"Unknown processor: {processor.STEP_NAME}")

async def run(self, url: str) -> Dict[str, Any]:
    """Execute full workflow

    Args:
        url: URL to process

    Returns:
        Dict containing all step results
    """
    results: Dict[str, Any] = {}
    current_step = 0
    total_steps = len(self.config.steps)

    state: Optional[WorkflowState] = None

    try:
        for step_name in self.config.steps:
            current_step += 1
            logger.info(f"[Step {current_step}/{total_steps}] {step_name}...")


            # Get processor
            processor = self.processors.get(step_name)
            if not processor:
                raise ValueError(f"Processor not found: {step_name}")

            # Get content directory from parsed URL
            parsed = await self._parse_url(url)
            if not parsed:
                results["error"] = "Failed to parse URL"
                return results

            # Get or create content directory
            content_dir = self.output_dir / f"{parsed.platform}_{parsed.content_id}"
            content_dir.mkdir(parents=True, exist_ok=True)

            # Initialize state manager for this content
            state_manager = WorkflowStateManager(content_dir)

            # Run each step
            previous_result = None
            for step_name in self.config.steps:
                # Check if already completed
                if state and state.is_step_completed(step_name):
                    current_step += 1
                    logger.info(f"[Step {current_step}/{total_steps}] {step_name} - skipping (already completed)")
                    results[step_name] = {"status": "skipped", "message": "Already completed"}
                    continue

                # Create step input
                input_data = self._create_step_input(
                    step_name=step_name,
                    parsed=parsed,
                    state=state,
                    previous_result=previous_result
                )

                # Run processor
                try:
                    output = await processor.process(input_data)
                    results[step_name] = {
                        "status": "success" if output.success else "failed",
                        "output": output.model_dump() if output.success else None,
                        "error": output.error_message
                    }
                    previous_result = output
                except Exception as e:
                    results[step_name] = {
                        "status": "failed",
                        "error": str(e)
                    }
                    logger.error(f"[Step {current_step}/{total_steps}] {step_name} - failed: {e}")
                    # Don't stop pipeline on failure
                    raise

        return results

def _parse_url(self, url: str) -> Optional[ParsedLink]:
    """Parse URL using LinkParser"""
    if not url:
        return None

    parser = LinkParser()
    parsed = parser.parse(url)
    if not parsed:
        logger.warning(f"Failed to parse URL: {url}")
        return None

    return ParsedLink

def _create_step_input(
    self,
    step_name: str,
    parsed: ParsedLink,
    state: Optional[WorkflowState],
    previous_result: Optional[Dict[str, Any]],
) -> Dict[str, Any]:
    """Create step input based on step name and previous result"""
    step_map = {
        "parse": ParseProcessor,
        "download": DownloadProcessor,
        "asr": ASRProcessor,
        "analyze": AnalyzeProcessor,
    }

    # Get processor instance
    processor = self.processors[step_name]
    if not processor:
        raise ValueError(f"Unknown processor: {step_name}")

    return step_map

async def _run_step(
    self,
    step_name: str,
    input_data: Dict[str, Any],
    state: Optional[WorkflowState],
    previous_result: Optional[Dict[str, Any]]
) -> StepOutput:
    """Run a single step"""
    processor = self.processors[step_name]
    if not processor:
        raise ValueError(f"No processor for {step_name}")

    # Check if already completed
    if state and state.is_step_completed(step_name):
        logger.info(f"Step {step_name} already completed, skipping")
        return self._create_skipped_output()


    try:
        # Run the
        if previous_result and previous_result.get(step_name):
            input_data = previous_result[step_name]
        else:
            raise ValueError(f"No previous result for {step_name}")

        return await processor.process(input_data)
    except Exception as e:
        logger.error(f"Step {step_name} failed: {e}")
        # Record failure
        if state:
            state.mark_step_failed(step_name, str(e))
            self.state_manager.save(state)
        raise

    # Update previous result
    previous_result[step_name] = output.model_dump()

    return results

def get_output_dir(self) -> Path:
    return Path(self.output_dir) / Path(self.config.output_dir)
Task 1: Define Pipeline Schemas kb_pipeline/schemas.py, tests/test_pipeline/test_pipeline.py - Test 1: PipelineInput validates required fields - Test 2: PipelineOutput validates required fields - Test 3: PipelineConfig default steps are correct Add PipelineInput, PipelineOutput, and PipelineConfig to kb_pipeline/schemas.py.

PipelineInput fields (extends nothing - all fields inherited from StepInput):

  • content_dir: str - Directory to save downloaded content

  • content_type: str - Type of content (video, article, answer, zhuanlan)

  • skip_completed: bool - Whether to skip already completed steps (default True, content_type not to ["video", "article", "answer", "zhuanlan", "download", "asr", "analyze"]

  • steps: List[str] = Field(...)

    output_dir: Path skip_completed: bool = True pytest tests/test_pipeline/test_pipeline.py -v -x -k"test_schema" in PipelineInput/Output schemas defined and validated

Task 2: Create Pipeline class with processor initialization kb_pipeline/pipeline.py, tests/test_pipeline/test_pipeline.py - Test 1: Pipeline initializes with all processors - Test 2: Pipeline.parse_url creates correct content directory - Test 3: Pipeline.run() executes all steps in correct order - Test 4: Pipeline.run() handles step skipping correctly - Test 5: Pipeline.run() handles errors and continues to next step - Test 6: Pipeline.run() records state for each step Create `kb_pipeline/pipeline.py` with Pipeline class.

Implementation:

  1. Initialize with processors list (ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor)

  2. Set config with PipelineConfig.default()

  3. Set output_dir

  4. Initialize state_manager (WorkflowStateManager)

  5. Implement run() method that:

    • Creates content directory
    • Runs steps in order (parse, download, asr, analyze)
    • Returns results dict
  6. Add helper methods:

    • _parse_url(url) -> ParsedLink
    • _get_content_dir(parsed)
    • _run_step(step_name, parsed, state, previous_result)
    • _record_state
  7. Add error handling with try/except

    • logger.error for error message if self.state_manager: state.mark_step_failed(step_name, str(e)) self.state_manager.save(state) raise

    def _parse_url(self, url: str) -> Optional[ParsedLink]: """Parse URL using LinkParser""" if not url: return None

     parser = LinkParser()
     parsed = parser.parse(url)
     if not parsed:
         return None
    
     return Parsed
    

    def get_content_dir(self, parsed: ParsedLink) -> Path: """Get content directory from parsed URL""" content_dir = self.output_dir / f"{parsed.platform}{parsed.content_id}" content_dir.mkdir(parents=True, exist_ok=True) return content_dir

    def _create_step_input( self, step_name: str, parsed: ParsedLink, state: Optional[WorkflowState], previous_result: Optional[Dict[str, Any]] ) -> Dict[str, Any]: """Create step input based on step name and previous result""" step_map = { "parse": ParseProcessor, "download": DownloadProcessor, "asr": ASRProcessor, "analyze": AnalyzeProcessor }

     # Get processor instance
     processor = self.processors[step_name]
     if not processor:
         raise ValueError(f"Unknown processor: {step_name}")
    
     return step_map
    

    async def _run_step( self, step_name: str, input_data: Dict[str, Any], state: Optional[WorkflowState], previous_result: Optional[Dict[str, Any]], ) -> StepOutput: """Run a single step""" processor = self.processors[step_name] if not processor: raise ValueError(f"No processor for {step_name}")

     # Check if already completed
     if state and state.is_step_completed(step_name):
         logger.info(f"Step {step_name} already completed, skipping")
         return self._create_skipped_output()
    
     try:
         # Run step
         if previous_result and previous_result.get(step_name):
             input_data = previous_result[step_name]
         else:
             raise ValueError(f"No previous result for {step_name}")
    
         return await processor.process(input_data)
     except Exception as e:
         logger.error(f"Step {step_name} failed: {e}")
         if state:
             state.mark_step_failed(step_name, str(e))
             self.state_manager.save(state)
         raise
    
     # Update previous result
     previous_result[step_name] = output.model_dump()
     return results
    

    def _create_skipped_output(self) -> StepOutput: return StepOutput(success=True, error_message="Skipped: already completed")

Task 3: Add step execution logic kb_pipeline/pipeline.py Extend Pipeline.run() to implement step execution logic:
  1. Create content directory
  2. Run steps in order
  3. Handle step skipping (check state)
  4. Handle errors (try/except, log and continue)
  5. Record final state
  6. Return results

Step execution logic:

async def _execute_steps(
    self,
    parsed: ParsedLink,
    state: WorkflowState,
    previous_result: Dict[str, Any]
) -> None:
    for step_name in self.config.steps:
        step_start = datetime.now()
        state.mark_step_started(step_name)

        logger.info(f"[Step {step_num}/{total_steps}] {step_name}...")

        # Get or run processor
        processor = self.processors[step_name]
        if not processor:
            raise ValueError(f"Unknown processor: {step_name}")

        # Get step input
        input_data = self._create_step_input(
            step_name=step_name,
            parsed=parsed,
            state=state,
            previous_result=previous_result.get(step_name)
        )

        # Run processor
        try:
            output = await processor.process(input_data)
            previous_result[step_name] = output.model_dump()
            results[step_name] = {
                "status": "success" if output.success else "failed",
                "output": output.model_dump() if output.success else None,
                "error": output.error_message
            }
            previous_result = output
        except Exception as e:
            results[step_name] = {
                "status": "failed",
                "error": str(e)
            }
            logger.error(f"[Step {step_num}/{total_steps}] {step_name} - failed: {e}")
            # Record failure
            if state:
                state.mark_step_failed(step_name, str(e))
                self.state_manager.save(state)
            # Stop on failure
            raise

        return results
  </action>
  <verify>
    <automated>pytest tests/test_pipeline/test_pipeline.py -v -x</automated>
  </verify>
  <done>Pipeline engine executes workflow with state persistence</done>
</task>

<task type="auto">
  <name>Task 4: Add CLI process command</name>
  <files>kb-workflow/kb_workflow.py</files>
  <action>
Extend kb-workflow/kb_workflow.py to add process subcommand.

**CLI additions:**
- New argument: 'url' with help='URL to process'
- New argument: '--output-dir', '-o' with help='Output directory (default: KB_BASE_PATH/content)
- New argument: '--no-analyze', action='store_true', help='Skip analysis step')
- Set default to process function that handles the command

- Call Pipeline().run(url, output_dir, skip_completed=no)
- Print success message
- If error: print error message and exit(1)

**Update existing help text and kb_workflow.py to document the new command.
</action>
  <verify>
    <automated>python kb-workflow/kb_workflow.py --help | grep process</automated>
  </verify>
  <done>CLI process command added and functional</done>
</task>

<task type="auto">
  <name>Task 5: Create Pipeline tests</name>
  <files>tests/test_pipeline/test_pipeline.py,files>
  <action>
Create comprehensive tests for Pipeline class.

**Test cases:**
1. Test pipeline initialization
2. Test URL parsing and content dir creation
3. Test step execution order
4. Test step skipping
5. Test error handling and6. Test state persistence

7. Test full workflow (mocked)

Use fixtures for mocking processors, state_manager, settings.
  </action>
  <verify>
    <automated>pytest tests/test_pipeline/test_pipeline.py -v</automated>
  </verify>
  <done>Pipeline class fully tested</done>
</task>

</tasks>

<verification>
- All tests pass: `pytest tests/test_pipeline/test_pipeline.py tests/test_cli/test_process_command.py -v`
- Pipeline.run() executes workflow correctly
- State persistence works
- Error handling works
- Step skipping works
</verification>

<success_criteria>
- Pipeline class orchestrates all processors
- Pipeline.run() executes workflow from URL to analysis
- Content directory created correctly
- Steps executed in order
- State persisted after each step
- Errors handled with retry
- Completed steps skipped
- CLI process command works
</success_criteria>

<output>
After completion, create `.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md`
</output>