8.3 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, user_setup, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | user_setup | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 04-analysis-orchestration | 02 | execute | 2 |
|
|
true |
|
|
Purpose: Unify metadata and content fetching into a single processor that works with PlatformAdapter implementations. Output: DownloadProcessor implementation with Bilibili and Zhihu adapter integration.
<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/04-analysis-orchestration/04-CONTEXT.md @.planning/phases/04-analysis-orchestration/04-RESEARCH.md @.planning/phases/04-analysis-orchestration/04-01-SUMMARY.mdFrom kb_pipeline/schemas.py:
class StepInput(BaseModel):
content_id: str
platform: Literal["bilibili", "zhihu"]
class StepOutput(BaseModel):
success: bool
error_message: Optional[str]
completed_at: datetime
From kb_pipeline/processors/base.py (from 04-01):
class BaseProcessor(ABC):
STEP_NAME: str
async def process(self, input_data: StepInput) -> StepOutput
async def _do_process(self, input_data: StepInput) -> StepOutput
From kb_pipeline/platforms/base.py:
class PlatformAdapter(ABC):
@abstractmethod
async def fetch_metadata(self, content_id: str, content_type: str) -> Dict[str, Any]
@abstractmethod
async def fetch_content(self, content_id: str, content_type: str) -> str
@abstractmethod
def get_output_filename(self, content_id: str, content_type: str, suffix: str) -> str
From kb_pipeline/platforms/bilibili.py:
class BilibiliAdapter(PlatformAdapter):
# Already implemented with fetch_metadata and fetch_content
From kb_pipeline/platforms/zhihu.py:
class ZhihuAdapter(PlatformAdapter):
# Already implemented with fetch_metadata and fetch_content
DownloadInput fields (extends StepInput):
- content_dir: str - Directory to save downloaded content
- content_type: str - Type of content (video, article, answer, zhuanlan)
DownloadOutput fields (extends StepOutput):
- metadata_file: str - Path to metadata JSON
- content_file: str - Path to content file (video/audio or text)
- title: str - Content title
- author: str - Content author
- duration: Optional[int] - Duration for video content
Follow StepInput/StepOutput pattern from Phase 1. Create test file with schema validation tests. pytest tests/test_processors/test_download.py -v -x DownloadInput/DownloadOutput schemas defined and validated with tests
Task 2: Create DownloadProcessor with metadata fetching kb_pipeline/processors/download.py, tests/test_processors/test_download.py - Test 1: DownloadProcessor fetches Bilibili video metadata correctly - Test 2: DownloadProcessor fetches Zhihu answer metadata correctly - Test 3: DownloadProcessor records state on success - Test 4: DownloadProcessor records state on failure Create `kb_pipeline/processors/download.py` with DownloadProcessor class.Implementation:
- Import BaseProcessor, PlatformAdapter classes
- Import BilibiliAdapter, ZhihuAdapter from platforms
- Create DownloadProcessor(BaseProcessor):
- STEP_NAME = "download"
- init with state_manager, settings
- process() method that: a. Checks if step already completed (skip if done) b. Gets appropriate PlatformAdapter based on platform c. Calls adapter.fetch_metadata() d. Saves metadata to content_dir e. Records state on success/failure
State Recording:
- On success: mark_step_completed("download", f"title={title}, type={content_type}")
- On failure: mark_step_failed("download", error_message)
Create comprehensive tests mocking adapters and state_manager. pytest tests/test_processors/test_download.py -v -x DownloadProcessor fetches metadata with state recording
Task 3: Add content downloading to DownloadProcessor kb_pipeline/processors/download.py, tests/test_processors/test_download.py - Test 1: DownloadProcessor downloads Bilibili article content - Test 2: DownloadProcessor skips video download (handled by ASR) - Test 3: DownloadProcessor creates correct output file paths Extend DownloadProcessor to download content.For articles/text content:
- Call adapter.fetch_content() to get text
- Save content to content_dir as Markdown file
- Update DownloadOutput with content_file path
For video content:
- Skip actual video download (handled by ASR in Phase 3)
- Set content_file to empty or indicate video needs ASR
File naming:
- Metadata:
{platform}_{content_id}_metadata.json - Content:
{platform}_{content_id}_content.md(for articles)
Update tests to verify content downloading behavior. pytest tests/test_processors/test_download.py -v -x DownloadProcessor downloads content for text-based content types
Task 4: Update processors __init__.py kb_pipeline/processors/__init__.py Update `kb_pipeline/processors/__init__.py` to export DownloadProcessor.from .base import BaseProcessor
from .parse import ParseProcessor
from .download import DownloadProcessor
from .asr import ASRProcessor
__all__ = ["BaseProcessor", "ParseProcessor", "DownloadProcessor", "ASRProcessor"]
<success_criteria>
- DownloadProcessor inherits from BaseProcessor
- DownloadInput/DownloadOutput schemas exist and validate
- Metadata is fetched and saved correctly
- Text content is downloaded and saved
- Video content is marked for ASR processing
- All tests pass </success_criteria>