---
phase: 04-analysis-orchestration
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- kb_pipeline/processors/base.py
- kb_pipeline/processors/parse.py
- kb_pipeline/processors/__init__.py
- kb_pipeline/schemas.py
- tests/test_processors/test_base.py
- tests/test_processors/test_parse.py
autonomous: true
requirements: [MNT-01]
user_setup: []
must_haves:
truths:
- "BaseProcessor defines unified interface for all processors"
- "ParseProcessor correctly parses Bilibili and Zhihu URLs"
- "All processors inherit from BaseProcessor with consistent process() method"
artifacts:
- path: "kb_pipeline/processors/base.py"
provides: "BaseProcessor abstract class"
contains: "class BaseProcessor"
min_lines: 80
- path: "kb_pipeline/processors/parse.py"
provides: "ParseProcessor implementation"
contains: "class ParseProcessor"
min_lines: 60
- path: "kb_pipeline/schemas.py"
provides: "ParseInput/ParseOutput schemas"
contains: "class ParseInput"
key_links:
- from: "kb_pipeline/processors/parse.py"
to: "kb_pipeline/link_parser.py"
via: "LinkParser.parse()"
pattern: "LinkParser\(\)\.parse"
- from: "kb_pipeline/processors/base.py"
to: "kb_pipeline/state.py"
via: "WorkflowStateManager integration"
pattern: "state_manager\."
---
Create unified BaseProcessor interface and ParseProcessor to establish the foundation for the pipeline architecture.
Purpose: Provide consistent processor interface that all steps (parse, download, asr, analyze) will follow, enabling unified state management and error handling.
Output: BaseProcessor abstract class, ParseProcessor implementation, and related schemas.
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-analysis-orchestration/04-CONTEXT.md
@.planning/phases/04-analysis-orchestration/04-RESEARCH.md
From kb_pipeline/schemas.py:
```python
class StepStatus(str, Enum):
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
SKIPPED = "skipped"
class StepInput(BaseModel):
content_id: str = Field(..., min_length=1)
platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str] = None
completed_at: datetime = Field(default_factory=datetime.now)
class WorkflowState(BaseModel):
def is_step_completed(self, step_name: str) -> bool: ...
def mark_step_started(self, step_name: str) -> None: ...
def mark_step_completed(self, step_name: str, output_summary: Optional[str] = None) -> None: ...
def mark_step_failed(self, step_name: str, error_message: str) -> None: ...
```
From kb_pipeline/state.py:
```python
class WorkflowStateManager:
STATE_FILENAME = "workflow_state.json"
def __init__(self, content_dir: Path): ...
def load(self) -> Optional[WorkflowState]: ...
def save(self, state: WorkflowState) -> None: ...
def get_or_create(self, content_id: str, platform: str, source_url: str) -> WorkflowState: ...
```
From kb_pipeline/link_parser.py:
```python
@dataclass
class ParsedLink:
platform: str
content_type: str
content_id: str
original_url: str
resolved_url: str = ""
class LinkParser:
def parse(self, url: str) -> Optional[ParsedLink]: ...
```
From kb_pipeline/processors/asr.py (existing pattern reference):
```python
class ASRProcessor:
STEP_NAME = "asr"
def __init__(self, state_manager: Optional[WorkflowStateManager] = None):
self.settings = get_settings()
self.state_manager = state_manager
async def process(self, input_data: ASRInput) -> ASROutput: ...
```
Task 1: Define BaseProcessor abstract class
kb_pipeline/processors/base.py, tests/test_processors/test_base.py
- Test 1: BaseProcessor cannot be instantiated directly (abstract)
- Test 2: Subclass must implement process() method
- Test 3: BaseProcessor provides state_manager attribute
- Test 4: BaseProcessor provides STEP_NAME class attribute
Create `kb_pipeline/processors/base.py` with BaseProcessor abstract class:
1. Define abstract base class with:
- STEP_NAME: str class attribute (must be overridden by subclass)
- __init__(self, state_manager: Optional[WorkflowStateManager] = None)
- settings: KBSettings from get_settings()
- @abstractmethod process(self, input_data: StepInput) -> StepOutput
2. Use ABC (Abstract Base Class) from abc module
3. Import types from typing: Optional, TypeVar, Generic
4. Keep it minimal - no complex template methods, just the interface contract
Reference existing ASRProcessor pattern but extract the common interface.
Do NOT add retry logic - that belongs in concrete implementations.
Do NOT add complex inheritance hierarchies - keep it flat and simple.
pytest tests/test_processors/test_base.py -v -x
BaseProcessor abstract class exists with process() method, cannot be instantiated directly
Task 2: Add ParseInput and ParseOutput schemas
kb_pipeline/schemas.py, tests/test_pipeline/test_schemas.py
- Test 1: ParseInput validates url field (required, min_length=1)
- Test 2: ParseOutput contains platform, content_type, content_id, resolved_url
- Test 3: ParseOutput inherits success and error_message from StepOutput
Add ParseInput and ParseOutput schemas to `kb_pipeline/schemas.py`:
```python
class ParseInput(StepInput):
"""Parse step input - URL to parse"""
url: str = Field(..., min_length=1, description="URL to parse")
# Override content_id and platform as optional for parse step
content_id: str = Field(default="", description="Will be populated after parsing")
platform: str = Field(default="", description="Will be populated after parsing")
class ParseOutput(StepOutput):
"""Parse step output"""
platform: str = Field(..., description="Detected platform")
content_type: str = Field(..., description="Content type (video, article, etc.)")
content_id: str = Field(..., description="Extracted content ID")
resolved_url: str = Field(..., description="Resolved URL after short link expansion")
original_url: str = Field(..., description="Original URL provided by user")
```
Place after ASROutput schema (around line 236).
Add tests to existing test_schemas.py file.
pytest tests/test_pipeline/test_schemas.py -v -x -k "Parse"
ParseInput and ParseOutput schemas exist and validate correctly
Task 3: Implement ParseProcessor with __init__.py export
kb_pipeline/processors/parse.py, kb_pipeline/processors/__init__.py, tests/test_processors/test_parse.py
- Test 1: ParseProcessor parses Bilibili video URL correctly
- Test 2: ParseProcessor parses Zhihu answer URL correctly
- Test 3: ParseProcessor returns ParseOutput with success=False for unknown URLs
- Test 4: ParseProcessor records state when state_manager is provided
- Test 5: ParseProcessor can be imported from kb_pipeline.processors
Create `kb_pipeline/processors/parse.py`:
1. Import BaseProcessor from .base, LinkParser from kb_pipeline.link_parser
2. Import ParseInput, ParseOutput from kb_pipeline.schemas
3. Implement ParseProcessor(BaseProcessor):
- STEP_NAME = "parse"
- async def process(self, input_data: ParseInput) -> ParseOutput:
a. Create LinkParser instance
b. Call parser.parse(input_data.url)
c. If parsed is None, return ParseOutput(success=False, error_message="Unsupported URL")
d. If parsed, return ParseOutput with parsed fields
e. Record state on success (if state_manager exists)
4. Keep implementation simple - just wrap LinkParser
5. Handle both success and failure cases with proper state recording
Then update `kb_pipeline/processors/__init__.py` to export new classes:
```python
from .base import BaseProcessor
from .parse import ParseProcessor
from .asr import ASRProcessor
__all__ = ["BaseProcessor", "ParseProcessor", "ASRProcessor"]
```
Keep existing ASRProcessor export and add new ones.
pytest tests/test_processors/test_parse.py -v -x && python -c "from kb_pipeline.processors import BaseProcessor, ParseProcessor, ASRProcessor; print('OK')"
ParseProcessor correctly parses URLs, returns structured ParseOutput, and is exported from processors module
- All tests pass: `pytest tests/test_processors/test_base.py tests/test_processors/test_parse.py -v`
- BaseProcessor is abstract and cannot be instantiated
- ParseProcessor correctly wraps LinkParser
- State recording works when state_manager is provided
- All processors can be imported from kb_pipeline.processors
- BaseProcessor abstract class defines unified interface
- ParseProcessor inherits from BaseProcessor
- ParseInput/ParseOutput schemas exist and validate
- All tests pass
- Code follows existing patterns from ASRProcessor