kb-skills/.planning/phases/04-analysis-orchestration/04-05-PLAN.md

---

phase: 04-analysis-orchestration plan: 05 type: execute wave: 5 depends_on: [04-04] files_modified:

• kb-workflow/kb_workflow.py
• tests/test_cli/test_process_command.py
• tests/test_integration/test_e2e_workflow.py autonomous: false requirements: [MNT-01] user_setup: []

must_haves: truths: - "CLI 'process' subcommand is available" - "kb-workflow process executes full pipeline" - "End-to-end workflow completes successfully" - "Analysis report is saved to content directory" - "workflow_state.json correctly records all steps" artifacts: - path: "kb-workflow/kb_workflow.py" provides: "CLI entry point with process subcommand" contains: "def process_command" - path: "tests/test_cli/test_process_command.py" provides: "CLI command tests" contains: "test_process_command" - path: "tests/test_integration/test_e2e_workflow.py" provides: "End-to-end integration test" contains: "test_e2e" key_links: - from: "kb-workflow/kb_workflow.py" to: "kb_pipeline" via: "import Pipeline" pattern: "from kb_pipeline" - from: "kb_pipeline/pipeline.py" to: "kb_pipeline/processors" via: "Pipeline.run()" pattern: "Pipeline("

Add CLI 'process' subcommand and create end-to-end integration tests.

Purpose: Provide the single-command entry point kb-workflow process <url> as specified in CONTEXT.md, and verify the complete workflow works. Output: CLI process subcommand in kb_workflow.py, CLI tests, E2E integration tests.

<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-CONTEXT.md @.planning/phases/04-analysis-orchestration/04-RESEARCH.md

@.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md

from kb_pipeline/pipeline.py:

class Pipeline:
    """Pipeline orchestration engine"""
    def __init__(
        self,
        processors: List[BaseProcessor],
        config: Optional[PipelineConfig] = None,
        output_dir: Optional[Path] = None,
    ): ...
    async def run(self, url: str) -> PipelineOutput: ...

from kb_pipeline/schemas.py:

class PipelineInput(BaseModel):
    url: str = Field(..., min_length=1)
    output_dir: Optional[Path] = None
    skip_completed: bool = Field(default=True)

class PipelineOutput(BaseModel):
    success: bool
    error_message: Optional[str] = None
    results: Dict[str, Any] = Field(default_factory=dict)
    content_dir: Optional[Path] = None

class PipelineConfig(BaseModel):
    steps: List[str] = Field(default_factory=lambda: ["parse", "download", "asr", "analyze"])
    output_dir: Optional[Path] = None
    skip_completed: bool = Field(default=True)
    max_retries: int = Field(default=3)

from kb_pipeline/processors/base.py:

class BaseProcessor(ABC):
    STEP_NAME: str
    async def process(self, input_data: StepInput) -> StepOutput

from kb-workflow/kb_workflow.py (existing CLI pattern):

def main():
    parser = argparse.ArgumentParser(...)
    parser.add_argument('--check', action='store_true', ...)
    # ... existing arguments ...

Task 1: Add CLI 'process' subcommand kb-workflow/kb_workflow.py, tests/test_cli/test_process_command.py - Test 1: `kb-workflow process --help` shows usage - Test 2: `kb-workflow process ` calls Pipeline.run() - Test 3: Invalid URL returns appropriate error - Test 4: --output-dir flag sets custom output directory Add `process` subcommand to kb-workflow/kb_workflow.py using argparse subparsers.

Implementation requirements:

1. Add subparser for process command:

subparsers = parser.add_subparsers(dest='command', help='Available commands')

# Process subcommand
process_parser = subparsers.add_parser('process', help='Process URL through full pipeline')
process_parser.add_argument('url', type=str, help='URL to process')
process_parser.add_argument('--output-dir', '-o', type=str, help='Output directory for content')
process_parser.add_argument('--skip-completed', action='store_true', default=True, help='Skip completed steps')
process_parser.add_argument('--no-skip', action='store_false', dest='skip_completed', help='Re-run all steps')

2. Add process_command function:

def process_command(url: str, output_dir: Optional[str] = None, skip_completed: bool = True) -> Dict[str, Any]:
    """Execute full pipeline for a URL"""
    import asyncio
    from kb_pipeline.pipeline import Pipeline
    from kb_pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor
    from kb_pipeline.schemas import PipelineConfig
    from kb_pipeline.state import WorkflowStateManager

    # Parse URL first to get platform and content_id
    # ... implementation ...

    # Create pipeline and run
    async def run_pipeline():
        # Initialize processors and pipeline
        # ... implementation ...

    result = asyncio.run(run_pipeline())
    return result

3. Update main() to dispatch to process_command when args.command == 'process'

4. Create tests/test_cli/test_process_command.py with mocked Pipeline tests. pytest tests/test_cli/test_process_command.py -v -x CLI process subcommand implemented and tested

Task 2: Create end-to-end integration test tests/test_integration/test_e2e_workflow.py - Test 1: Full workflow executes from URL to analysis - Test 2: Workflow handles errors gracefully - Test 3: State is persisted correctly - Test 4: Workflow is resumable (skip completed steps) Create `tests/test_integration/test_e2e_workflow.py` with comprehensive end-to-end tests.

Test file structure:

"""End-to-end integration tests for the complete workflow."""
import pytest
import pytest_asyncio
from unittest.mock import AsyncMock, patch, MagicMock
from pathlib import Path
from typing import Dict, Any

from kb_pipeline.pipeline import Pipeline
from kb_pipeline.schemas import PipelineConfig, WorkflowState
from kb_pipeline.state import WorkflowStateManager


@pytest.fixture
def temp_content_dir(tmp_path):
    """Create temp content directory for testing."""
    content_dir = tmp_path / "content"
    content_dir.mkdir(parents=True, exist_ok=True)
    yield content_dir


@pytest.fixture
def mock_state_manager(temp_content_dir):
    """Create mock workflow state manager."""
    return WorkflowStateManager(temp_content_dir)


@pytest.fixture
def mock_processors():
    """Create mock processors for testing."""
    processors = []

    for step_name in ["parse", "download", "asr", "analyze"]:
        processor = MagicMock()
        processor.STEP_NAME = step_name
        processor.process = AsyncMock(return_value=MagicMock(
            success=True,
            error_message=None,
        ))
        processors.append(processor)

    return processors


@pytest.mark.asyncio
async def test_full_workflow_executes_successfully(temp_content_dir, mock_processors):
    """Test that full workflow executes from URL to analysis."""
    config = PipelineConfig(
        steps=["parse", "download", "asr", "analyze"],
        output_dir=temp_content_dir,
    )

    pipeline = Pipeline(
        processors=mock_processors,
        config=config,
        output_dir=temp_content_dir,
    )

    result = await pipeline.run("https://www.bilibili.com/video/BV1test123")

    assert result.success
    for processor in mock_processors:
        processor.process.assert_called_once()


@pytest.mark.asyncio
async def test_workflow_handles_errors_gracefully(temp_content_dir):
    """Test that workflow handles errors gracefully."""
    # Create processor that fails
    failing_processor = MagicMock()
    failing_processor.STEP_NAME = "parse"
    failing_processor.process = AsyncMock(return_value=MagicMock(
        success=False,
        error_message="Network error",
    ))

    config = PipelineConfig(
        steps=["parse"],
        output_dir=temp_content_dir,
    )

    pipeline = Pipeline(
        processors=[failing_processor],
        config=config,
        output_dir=temp_content_dir,
    )

    result = await pipeline.run("https://invalid.url")

    assert not result.success
    assert result.error_message is not None


@pytest.mark.asyncio
async def test_state_persisted_correctly(temp_content_dir, mock_state_manager):
    """Test that state is persisted correctly after each step."""
    # ... test implementation
    pass


@pytest.mark.asyncio
async def test_workflow_resumable(temp_content_dir):
    """Test that workflow can resume (skip completed steps)."""
    # Create state with parse already completed
    state_manager = WorkflowStateManager(temp_content_dir)
    state = state_manager.get_or_create(
        content_id="test123",
        platform="bilibili",
        source_url="https://test.url"
    )
    state.mark_step_completed("parse")
    state_manager.save(state)

    # Create processors
    parse_processor = MagicMock()
    parse_processor.STEP_NAME = "parse"
    parse_processor.process = AsyncMock()

    download_processor = MagicMock()
    download_processor.STEP_NAME = "download"
    download_processor.process = AsyncMock(return_value=MagicMock(success=True))

    config = PipelineConfig(
        steps=["parse", "download"],
        output_dir=temp_content_dir,
        skip_completed=True,
    )

    pipeline = Pipeline(
        processors=[parse_processor, download_processor],
        config=config,
        output_dir=temp_content_dir,
    )

    # Run pipeline - parse should be skipped
    result = await pipeline.run("https://www.bilibili.com/video/BV1test123")

    # Parse should NOT be called (skipped)
    parse_processor.process.assert_not_called()
    # Download should be called
    download_processor.process.assert_called_once()

pytest tests/test_integration/test_e2e_workflow.py -v End-to-end workflow tested with state persistence End-to-end workflow (Phase 4): - Pipeline Engine with 4 processors (Parse, Download, ASR, Analyze) - CLI entry point with `kb-workflow process ` command - DeepAnalyzer integration for analysis reports saved as Markdown - State persistence via workflow_state.json 1. Run `pytest tests/test_cli/test_process_command.py tests/test_integration/test_e2e_workflow.py -v -x` 2. Run `python -m kb_workflow.kb_workflow process --help` to see usage 3. (Optional) Run `kb-workflow process ` with a real URL to test end-to-end Type "approved" to indicate successful completion, or describe issues found - All tests pass: `pytest tests/test_cli/ tests/test_integration/ -v` - CLI process subcommand available: `python -m kb_workflow.kb_workflow process --help` - End-to-end workflow tested with state persistence

<success_criteria>

• CLI process subcommand implemented in kb_workflow.py
• CLI tests pass
• End-to-end integration test passes
• workflow_state.json correctly records all steps
• Documentation is clear and comprehensive </success_criteria>

After completion, create `.planning/phases/04-analysis-orchestration/04-05-SUMMARY.md`