kb-skills/.planning/phases/04-analysis-orchestration/04-05-PLAN.md

---
phase: 04-analysis-orchestration
plan: 05
type: execute
wave: 5
depends_on: [04-04]
files_modified:
  - kb-workflow/kb_workflow.py
  - tests/test_cli/test_process_command.py
  - tests/test_integration/test_e2e_workflow.py
autonomous: false
requirements: [MNT-01]
user_setup: []

must_haves:
  truths:
    - "CLI 'process' subcommand is available"
    - "kb-workflow process <url> executes full pipeline"
    - "End-to-end workflow completes successfully"
    - "Analysis report is saved to content directory"
    - "workflow_state.json correctly records all steps"
  artifacts:
    - path: "kb-workflow/kb_workflow.py"
      provides: "CLI entry point with process subcommand"
      contains: "def process_command"
    - path: "tests/test_cli/test_process_command.py"
      provides: "CLI command tests"
      contains: "test_process_command"
    - path: "tests/test_integration/test_e2e_workflow.py"
      provides: "End-to-end integration test"
      contains: "test_e2e"
  key_links:
    - from: "kb-workflow/kb_workflow.py"
      to: "kb_pipeline"
      via: "import Pipeline"
      pattern: "from kb_pipeline"
    - from: "kb_pipeline/pipeline.py"
      to: "kb_pipeline/processors"
      via: "Pipeline.run()"
      pattern: "Pipeline\("
---

<objective>
Add CLI 'process' subcommand and create end-to-end integration tests.

Purpose: Provide the single-command entry point `kb-workflow process <url>` as specified in CONTEXT.md, and verify the complete workflow works.
Output: CLI process subcommand in kb_workflow.py, CLI tests, E2E integration tests.
</objective>

<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-analysis-orchestration/04-CONTEXT.md
@.planning/phases/04-analysis-orchestration/04-RESEARCH.md

@.planning/phases/04-analysis-orchestration/04-04-SUMMARY.md

<interfaces>
<!-- Key types and contracts from existing codebase. Executor should use these directly. -->

from kb_pipeline/pipeline.py:
```python
class Pipeline:
    """Pipeline orchestration engine"""
    def __init__(
        self,
        processors: List[BaseProcessor],
        config: Optional[PipelineConfig] = None,
        output_dir: Optional[Path] = None,
    ): ...
    async def run(self, url: str) -> PipelineOutput: ...
```

from kb_pipeline/schemas.py:
```python
class PipelineInput(BaseModel):
    url: str = Field(..., min_length=1)
    output_dir: Optional[Path] = None
    skip_completed: bool = Field(default=True)

class PipelineOutput(BaseModel):
    success: bool
    error_message: Optional[str] = None
    results: Dict[str, Any] = Field(default_factory=dict)
    content_dir: Optional[Path] = None

class PipelineConfig(BaseModel):
    steps: List[str] = Field(default_factory=lambda: ["parse", "download", "asr", "analyze"])
    output_dir: Optional[Path] = None
    skip_completed: bool = Field(default=True)
    max_retries: int = Field(default=3)
```

from kb_pipeline/processors/base.py:
```python
class BaseProcessor(ABC):
    STEP_NAME: str
    async def process(self, input_data: StepInput) -> StepOutput
```

from kb-workflow/kb_workflow.py (existing CLI pattern):
```python
def main():
    parser = argparse.ArgumentParser(...)
    parser.add_argument('--check', action='store_true', ...)
    # ... existing arguments ...
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add CLI 'process' subcommand</name>
  <files>kb-workflow/kb_workflow.py, tests/test_cli/test_process_command.py</files>
  <behavior>
    - Test 1: `kb-workflow process --help` shows usage
    - Test 2: `kb-workflow process <url>` calls Pipeline.run()
    - Test 3: Invalid URL returns appropriate error
    - Test 4: --output-dir flag sets custom output directory
  </behavior>
  <action>
Add `process` subcommand to kb-workflow/kb_workflow.py using argparse subparsers.

**Implementation requirements:**

1. Add subparser for `process` command:
```python
subparsers = parser.add_subparsers(dest='command', help='Available commands')

# Process subcommand
process_parser = subparsers.add_parser('process', help='Process URL through full pipeline')
process_parser.add_argument('url', type=str, help='URL to process')
process_parser.add_argument('--output-dir', '-o', type=str, help='Output directory for content')
process_parser.add_argument('--skip-completed', action='store_true', default=True, help='Skip completed steps')
process_parser.add_argument('--no-skip', action='store_false', dest='skip_completed', help='Re-run all steps')
```

2. Add `process_command` function:
```python
def process_command(url: str, output_dir: Optional[str] = None, skip_completed: bool = True) -> Dict[str, Any]:
    """Execute full pipeline for a URL"""
    import asyncio
    from kb_pipeline.pipeline import Pipeline
    from kb_pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor
    from kb_pipeline.schemas import PipelineConfig
    from kb_pipeline.state import WorkflowStateManager

    # Parse URL first to get platform and content_id
    # ... implementation ...

    # Create pipeline and run
    async def run_pipeline():
        # Initialize processors and pipeline
        # ... implementation ...

    result = asyncio.run(run_pipeline())
    return result
```

3. Update `main()` to dispatch to `process_command` when `args.command == 'process'`

4. Create tests/test_cli/test_process_command.py with mocked Pipeline tests.
  </action>
  <verify>
    <automated>pytest tests/test_cli/test_process_command.py -v -x</automated>
  </verify>
  <done>CLI process subcommand implemented and tested</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Create end-to-end integration test</name>
  <files>tests/test_integration/test_e2e_workflow.py</files>
  <behavior>
    - Test 1: Full workflow executes from URL to analysis
    - Test 2: Workflow handles errors gracefully
    - Test 3: State is persisted correctly
    - Test 4: Workflow is resumable (skip completed steps)
  </behavior>
  <action>
Create `tests/test_integration/test_e2e_workflow.py` with comprehensive end-to-end tests.

**Test file structure:**
```python
"""End-to-end integration tests for the complete workflow."""
import pytest
import pytest_asyncio
from unittest.mock import AsyncMock, patch, MagicMock
from pathlib import Path
from typing import Dict, Any

from kb_pipeline.pipeline import Pipeline
from kb_pipeline.schemas import PipelineConfig, WorkflowState
from kb_pipeline.state import WorkflowStateManager


@pytest.fixture
def temp_content_dir(tmp_path):
    """Create temp content directory for testing."""
    content_dir = tmp_path / "content"
    content_dir.mkdir(parents=True, exist_ok=True)
    yield content_dir


@pytest.fixture
def mock_state_manager(temp_content_dir):
    """Create mock workflow state manager."""
    return WorkflowStateManager(temp_content_dir)


@pytest.fixture
def mock_processors():
    """Create mock processors for testing."""
    processors = []

    for step_name in ["parse", "download", "asr", "analyze"]:
        processor = MagicMock()
        processor.STEP_NAME = step_name
        processor.process = AsyncMock(return_value=MagicMock(
            success=True,
            error_message=None,
        ))
        processors.append(processor)

    return processors


@pytest.mark.asyncio
async def test_full_workflow_executes_successfully(temp_content_dir, mock_processors):
    """Test that full workflow executes from URL to analysis."""
    config = PipelineConfig(
        steps=["parse", "download", "asr", "analyze"],
        output_dir=temp_content_dir,
    )

    pipeline = Pipeline(
        processors=mock_processors,
        config=config,
        output_dir=temp_content_dir,
    )

    result = await pipeline.run("https://www.bilibili.com/video/BV1test123")

    assert result.success
    for processor in mock_processors:
        processor.process.assert_called_once()


@pytest.mark.asyncio
async def test_workflow_handles_errors_gracefully(temp_content_dir):
    """Test that workflow handles errors gracefully."""
    # Create processor that fails
    failing_processor = MagicMock()
    failing_processor.STEP_NAME = "parse"
    failing_processor.process = AsyncMock(return_value=MagicMock(
        success=False,
        error_message="Network error",
    ))

    config = PipelineConfig(
        steps=["parse"],
        output_dir=temp_content_dir,
    )

    pipeline = Pipeline(
        processors=[failing_processor],
        config=config,
        output_dir=temp_content_dir,
    )

    result = await pipeline.run("https://invalid.url")

    assert not result.success
    assert result.error_message is not None


@pytest.mark.asyncio
async def test_state_persisted_correctly(temp_content_dir, mock_state_manager):
    """Test that state is persisted correctly after each step."""
    # ... test implementation
    pass


@pytest.mark.asyncio
async def test_workflow_resumable(temp_content_dir):
    """Test that workflow can resume (skip completed steps)."""
    # Create state with parse already completed
    state_manager = WorkflowStateManager(temp_content_dir)
    state = state_manager.get_or_create(
        content_id="test123",
        platform="bilibili",
        source_url="https://test.url"
    )
    state.mark_step_completed("parse")
    state_manager.save(state)

    # Create processors
    parse_processor = MagicMock()
    parse_processor.STEP_NAME = "parse"
    parse_processor.process = AsyncMock()

    download_processor = MagicMock()
    download_processor.STEP_NAME = "download"
    download_processor.process = AsyncMock(return_value=MagicMock(success=True))

    config = PipelineConfig(
        steps=["parse", "download"],
        output_dir=temp_content_dir,
        skip_completed=True,
    )

    pipeline = Pipeline(
        processors=[parse_processor, download_processor],
        config=config,
        output_dir=temp_content_dir,
    )

    # Run pipeline - parse should be skipped
    result = await pipeline.run("https://www.bilibili.com/video/BV1test123")

    # Parse should NOT be called (skipped)
    parse_processor.process.assert_not_called()
    # Download should be called
    download_processor.process.assert_called_once()
```
  </action>
  <verify>
    <automated>pytest tests/test_integration/test_e2e_workflow.py -v</automated>
  </verify>
  <done>End-to-end workflow tested with state persistence</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <what-built>
End-to-end workflow (Phase 4):
- Pipeline Engine with 4 processors (Parse, Download, ASR, Analyze)
- CLI entry point with `kb-workflow process <url>` command
- DeepAnalyzer integration for analysis reports saved as Markdown
- State persistence via workflow_state.json
  </what-built>
  <how-to-verify>
1. Run `pytest tests/test_cli/test_process_command.py tests/test_integration/test_e2e_workflow.py -v -x`
2. Run `python -m kb_workflow.kb_workflow process --help` to see usage
3. (Optional) Run `kb-workflow process <url>` with a real URL to test end-to-end
  </how-to-verify>
  <resume-signal>Type "approved" to indicate successful completion, or describe issues found</resume-signal>
</task>

</tasks>

<verification>
- All tests pass: `pytest tests/test_cli/ tests/test_integration/ -v`
- CLI process subcommand available: `python -m kb_workflow.kb_workflow process --help`
- End-to-end workflow tested with state persistence
</verification>

<success_criteria>
- CLI `process` subcommand implemented in kb_workflow.py
- CLI tests pass
- End-to-end integration test passes
- workflow_state.json correctly records all steps
- Documentation is clear and comprehensive
</success_criteria>

<output>
After completion, create `.planning/phases/04-analysis-orchestration/04-05-SUMMARY.md`
</output>