kb-skills/.planning/phases/04-analysis-orchestration/04-02-PLAN.md

---

phase: 04-analysis-orchestration plan: 02 type: execute wave: 2 depends_on: [04-01] files_modified:

• kb_pipeline/processors/download.py
• kb_pipeline/processors/__init__.py
• kb_pipeline/schemas.py
• tests/test_processors/test_download.py autonomous: true requirements: [MNT-01] user_setup: []

must_haves: truths: - "DownloadProcessor fetches metadata from platforms" - "DownloadProcessor downloads content (video/audio or text)" - "State is recorded after successful download" artifacts: - path: "kb_pipeline/processors/download.py" provides: "DownloadProcessor implementation" contains: "class DownloadProcessor" min_lines: 100 - path: "kb_pipeline/schemas.py" provides: "DownloadInput/DownloadOutput schemas" contains: "class DownloadInput" key_links: - from: "kb_pipeline/processors/download.py" to: "kb_pipeline/platforms/bilibili.py" via: "BilibiliAdapter" pattern: "BilibiliAdapter" - from: "kb_pipeline/processors/download.py" to: "kb_pipeline/platforms/zhihu.py" via: "ZhihuAdapter" pattern: "ZhihuAdapter" - from: "kb_pipeline/processors/download.py" to: "kb_pipeline/state.py" via: "WorkflowStateManager" pattern: "state_manager."

Create DownloadProcessor to handle metadata extraction and content downloading for all platforms.

Purpose: Unify metadata and content fetching into a single processor that works with PlatformAdapter implementations. Output: DownloadProcessor implementation with Bilibili and Zhihu adapter integration.

<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-CONTEXT.md @.planning/phases/04-analysis-orchestration/04-RESEARCH.md @.planning/phases/04-analysis-orchestration/04-01-SUMMARY.md

From kb_pipeline/schemas.py:

class StepInput(BaseModel):
    content_id: str
    platform: Literal["bilibili", "zhihu"]

class StepOutput(BaseModel):
    success: bool
    error_message: Optional[str]
    completed_at: datetime

From kb_pipeline/processors/base.py (from 04-01):

class BaseProcessor(ABC):
    STEP_NAME: str
    async def process(self, input_data: StepInput) -> StepOutput
    async def _do_process(self, input_data: StepInput) -> StepOutput

From kb_pipeline/platforms/base.py:

class PlatformAdapter(ABC):
    @abstractmethod
    async def fetch_metadata(self, content_id: str, content_type: str) -> Dict[str, Any]

    @abstractmethod
    async def fetch_content(self, content_id: str, content_type: str) -> str

    @abstractmethod
    def get_output_filename(self, content_id: str, content_type: str, suffix: str) -> str

From kb_pipeline/platforms/bilibili.py:

class BilibiliAdapter(PlatformAdapter):
    # Already implemented with fetch_metadata and fetch_content

From kb_pipeline/platforms/zhihu.py:

class ZhihuAdapter(PlatformAdapter):
    # Already implemented with fetch_metadata and fetch_content

Task 1: Define Download Schemas kb_pipeline/schemas.py, tests/test_processors/test_download.py - Test 1: DownloadInput validates required fields - Test 2: DownloadOutput validates required fields - Test 3: DownloadOutput handles both video and article content types Add DownloadInput and DownloadOutput schemas to kb_pipeline/schemas.py.

DownloadInput fields (extends StepInput):

• content_dir: str - Directory to save downloaded content
• content_type: str - Type of content (video, article, answer, zhuanlan)

DownloadOutput fields (extends StepOutput):

• metadata_file: str - Path to metadata JSON
• content_file: str - Path to content file (video/audio or text)
• title: str - Content title
• author: str - Content author
• duration: Optional[int] - Duration for video content

Follow StepInput/StepOutput pattern from Phase 1. Create test file with schema validation tests. pytest tests/test_processors/test_download.py -v -x -k "test_schema" DownloadInput/DownloadOutput schemas defined and validated with tests

Task 2: Create DownloadProcessor with metadata fetching kb_pipeline/processors/download.py, tests/test_processors/test_download.py - Test 1: DownloadProcessor fetches Bilibili video metadata correctly - Test 2: DownloadProcessor fetches Zhihu answer metadata correctly - Test 3: DownloadProcessor records state on success - Test 4: DownloadProcessor records state on failure Create `kb_pipeline/processors/download.py` with DownloadProcessor class.

Implementation:

1. Import BaseProcessor, PlatformAdapter classes
2. Import BilibiliAdapter, ZhihuAdapter from platforms
3. Create DownloadProcessor(BaseProcessor):
   • STEP_NAME = "download"
   • init with state_manager, settings
   • process() method that: a. Checks if step already completed (skip if done) b. Gets appropriate PlatformAdapter based on platform c. Calls adapter.fetch_metadata() d. Saves metadata to content_dir e. Records state on success/failure

State Recording:

• On success: mark_step_completed("download", f"title={title}, type={content_type}")
• On failure: mark_step_failed("download", error_message)

Create comprehensive tests mocking adapters and state_manager. pytest tests/test_processors/test_download.py -v -x -k "test_metadata" DownloadProcessor fetches metadata with state recording

Task 3: Add content downloading and __init__.py export kb_pipeline/processors/download.py, kb_pipeline/processors/__init__.py, tests/test_processors/test_download.py - Test 1: DownloadProcessor downloads Bilibili article content - Test 2: DownloadProcessor skips video download (handled by ASR) - Test 3: DownloadProcessor creates correct output file paths - Test 4: DownloadProcessor can be imported from kb_pipeline.processors Extend DownloadProcessor to download content and update __init__.py.

For articles/text content:

• Call adapter.fetch_content() to get text
• Save content to content_dir as Markdown file
• Update DownloadOutput with content_file path

For video content:

• Skip actual video download (handled by ASR in Phase 3)
• Set content_file to empty or indicate video needs ASR

File naming:

• Metadata: {platform}_{content_id}_metadata.json
• Content: {platform}_{content_id}_content.md (for articles)

Update tests to verify content downloading behavior.

Then update kb_pipeline/processors/__init__.py to export DownloadProcessor:

from .base import BaseProcessor
from .parse import ParseProcessor
from .download import DownloadProcessor
from .asr import ASRProcessor

__all__ = ["BaseProcessor", "ParseProcessor", "DownloadProcessor", "ASRProcessor"]

pytest tests/test_processors/test_download.py -v -x && python -c "from kb_pipeline.processors import DownloadProcessor; print('OK')" DownloadProcessor downloads content for text-based content types and is exported from processors module - All tests pass: `pytest tests/test_processors/test_download.py -v` - DownloadProcessor correctly uses PlatformAdapter implementations - State recording works for both success and failure cases - Content files are saved to correct locations - DownloadProcessor can be imported from kb_pipeline.processors

<success_criteria>

• DownloadProcessor inherits from BaseProcessor
• DownloadInput/DownloadOutput schemas exist and validate
• Metadata is fetched and saved correctly
• Text content is downloaded and saved
• Video content is marked for ASR processing
• All tests pass </success_criteria>

After completion, create `.planning/phases/04-analysis-orchestration/04-02-SUMMARY.md`