docs(04-analysis-orchestration): create phase plans
This commit is contained in:
@@ -0,0 +1,256 @@
|
||||
---
|
||||
phase: 04-analysis-orchestration
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: [04-01]
|
||||
files_modified:
|
||||
- kb_pipeline/processors/download.py
|
||||
- kb_pipeline/schemas.py
|
||||
- tests/test_processors/test_download.py
|
||||
autonomous: true
|
||||
requirements: [MNT-01]
|
||||
user_setup: []
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "DownloadProcessor fetches metadata from platforms"
|
||||
- "DownloadProcessor downloads content (video/audio or text)"
|
||||
- "State is recorded after successful download"
|
||||
artifacts:
|
||||
- path: "kb_pipeline/processors/download.py"
|
||||
provides: "DownloadProcessor implementation"
|
||||
contains: "class DownloadProcessor"
|
||||
min_lines: 100
|
||||
- path: "kb_pipeline/schemas.py"
|
||||
provides: "DownloadInput/DownloadOutput schemas"
|
||||
contains: "class DownloadInput"
|
||||
key_links:
|
||||
- from: "kb_pipeline/processors/download.py"
|
||||
to: "kb_pipeline/platforms/bilibili.py"
|
||||
via: "BilibiliAdapter"
|
||||
pattern: "BilibiliAdapter"
|
||||
- from: "kb_pipeline/processors/download.py"
|
||||
to: "kb_pipeline/platforms/zhihu.py"
|
||||
via: "ZhihuAdapter"
|
||||
pattern: "ZhihuAdapter"
|
||||
- from: "kb_pipeline/processors/download.py"
|
||||
to: "kb_pipeline/state.py"
|
||||
via: "WorkflowStateManager"
|
||||
pattern: "state_manager\\."
|
||||
---
|
||||
|
||||
<objective>
|
||||
Create DownloadProcessor to handle metadata extraction and content downloading for all platforms.
|
||||
|
||||
Purpose: Unify metadata and content fetching into a single processor that works with PlatformAdapter implementations.
|
||||
Output: DownloadProcessor implementation with Bilibili and Zhihu adapter integration.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/04-analysis-orchestration/04-CONTEXT.md
|
||||
@.planning/phases/04-analysis-orchestration/04-RESEARCH.md
|
||||
@.planning/phases/04-analysis-orchestration/04-01-SUMMARY.md
|
||||
|
||||
<interfaces>
|
||||
<!-- Key types and contracts from existing code. Executor should use these directly. -->
|
||||
|
||||
From kb_pipeline/schemas.py:
|
||||
```python
|
||||
class StepInput(BaseModel):
|
||||
content_id: str
|
||||
platform: Literal["bilibili", "zhihu"]
|
||||
|
||||
class StepOutput(BaseModel):
|
||||
success: bool
|
||||
error_message: Optional[str]
|
||||
completed_at: datetime
|
||||
```
|
||||
|
||||
From kb_pipeline/processors/base.py (from 04-01):
|
||||
```python
|
||||
class BaseProcessor(ABC):
|
||||
STEP_NAME: str
|
||||
async def process(self, input_data: StepInput) -> StepOutput
|
||||
async def _do_process(self, input_data: StepInput) -> StepOutput
|
||||
```
|
||||
|
||||
From kb_pipeline/platforms/base.py:
|
||||
```python
|
||||
class PlatformAdapter(ABC):
|
||||
@abstractmethod
|
||||
async def fetch_metadata(self, content_id: str, content_type: str) -> Dict[str, Any]
|
||||
|
||||
@abstractmethod
|
||||
async def fetch_content(self, content_id: str, content_type: str) -> str
|
||||
|
||||
@abstractmethod
|
||||
def get_output_filename(self, content_id: str, content_type: str, suffix: str) -> str
|
||||
```
|
||||
|
||||
From kb_pipeline/platforms/bilibili.py:
|
||||
```python
|
||||
class BilibiliAdapter(PlatformAdapter):
|
||||
# Already implemented with fetch_metadata and fetch_content
|
||||
```
|
||||
|
||||
From kb_pipeline/platforms/zhihu.py:
|
||||
```python
|
||||
class ZhihuAdapter(PlatformAdapter):
|
||||
# Already implemented with fetch_metadata and fetch_content
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: Define Download Schemas</name>
|
||||
<files>kb_pipeline/schemas.py, tests/test_processors/test_download.py</files>
|
||||
<behavior>
|
||||
- Test 1: DownloadInput validates required fields
|
||||
- Test 2: DownloadOutput validates required fields
|
||||
- Test 3: DownloadOutput handles both video and article content types
|
||||
</behavior>
|
||||
<action>
|
||||
Add DownloadInput and DownloadOutput schemas to kb_pipeline/schemas.py.
|
||||
|
||||
**DownloadInput fields (extends StepInput):**
|
||||
- content_dir: str - Directory to save downloaded content
|
||||
- content_type: str - Type of content (video, article, answer, zhuanlan)
|
||||
|
||||
**DownloadOutput fields (extends StepOutput):**
|
||||
- metadata_file: str - Path to metadata JSON
|
||||
- content_file: str - Path to content file (video/audio or text)
|
||||
- title: str - Content title
|
||||
- author: str - Content author
|
||||
- duration: Optional[int] - Duration for video content
|
||||
|
||||
Follow StepInput/StepOutput pattern from Phase 1.
|
||||
Create test file with schema validation tests.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_processors/test_download.py -v -x</automated>
|
||||
</verify>
|
||||
<done>DownloadInput/DownloadOutput schemas defined and validated with tests</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Create DownloadProcessor with metadata fetching</name>
|
||||
<files>kb_pipeline/processors/download.py, tests/test_processors/test_download.py</files>
|
||||
<behavior>
|
||||
- Test 1: DownloadProcessor fetches Bilibili video metadata correctly
|
||||
- Test 2: DownloadProcessor fetches Zhihu answer metadata correctly
|
||||
- Test 3: DownloadProcessor records state on success
|
||||
- Test 4: DownloadProcessor records state on failure
|
||||
</behavior>
|
||||
<action>
|
||||
Create `kb_pipeline/processors/download.py` with DownloadProcessor class.
|
||||
|
||||
**Implementation:**
|
||||
1. Import BaseProcessor, PlatformAdapter classes
|
||||
2. Import BilibiliAdapter, ZhihuAdapter from platforms
|
||||
3. Create DownloadProcessor(BaseProcessor):
|
||||
- STEP_NAME = "download"
|
||||
- __init__ with state_manager, settings
|
||||
- process() method that:
|
||||
a. Checks if step already completed (skip if done)
|
||||
b. Gets appropriate PlatformAdapter based on platform
|
||||
c. Calls adapter.fetch_metadata()
|
||||
d. Saves metadata to content_dir
|
||||
e. Records state on success/failure
|
||||
|
||||
**State Recording:**
|
||||
- On success: mark_step_completed("download", f"title={title}, type={content_type}")
|
||||
- On failure: mark_step_failed("download", error_message)
|
||||
|
||||
Create comprehensive tests mocking adapters and state_manager.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_processors/test_download.py -v -x</automated>
|
||||
</verify>
|
||||
<done>DownloadProcessor fetches metadata with state recording</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: Add content downloading to DownloadProcessor</name>
|
||||
<files>kb_pipeline/processors/download.py, tests/test_processors/test_download.py</files>
|
||||
<behavior>
|
||||
- Test 1: DownloadProcessor downloads Bilibili article content
|
||||
- Test 2: DownloadProcessor skips video download (handled by ASR)
|
||||
- Test 3: DownloadProcessor creates correct output file paths
|
||||
</behavior>
|
||||
<action>
|
||||
Extend DownloadProcessor to download content.
|
||||
|
||||
**For articles/text content:**
|
||||
- Call adapter.fetch_content() to get text
|
||||
- Save content to content_dir as Markdown file
|
||||
- Update DownloadOutput with content_file path
|
||||
|
||||
**For video content:**
|
||||
- Skip actual video download (handled by ASR in Phase 3)
|
||||
- Set content_file to empty or indicate video needs ASR
|
||||
|
||||
**File naming:**
|
||||
- Metadata: `{platform}_{content_id}_metadata.json`
|
||||
- Content: `{platform}_{content_id}_content.md` (for articles)
|
||||
|
||||
Update tests to verify content downloading behavior.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_processors/test_download.py -v -x</automated>
|
||||
</verify>
|
||||
<done>DownloadProcessor downloads content for text-based content types</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: Update processors __init__.py</name>
|
||||
<files>kb_pipeline/processors/__init__.py</files>
|
||||
<action>
|
||||
Update `kb_pipeline/processors/__init__.py` to export DownloadProcessor.
|
||||
|
||||
```python
|
||||
from .base import BaseProcessor
|
||||
from .parse import ParseProcessor
|
||||
from .download import DownloadProcessor
|
||||
from .asr import ASRProcessor
|
||||
|
||||
__all__ = ["BaseProcessor", "ParseProcessor", "DownloadProcessor", "ASRProcessor"]
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_pipeline.processors import DownloadProcessor; print('OK')"</automated>
|
||||
</verify>
|
||||
<done>DownloadProcessor exported from processors module</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- All tests pass: `pytest tests/test_processors/test_download.py -v`
|
||||
- DownloadProcessor correctly uses PlatformAdapter implementations
|
||||
- State recording works for both success and failure cases
|
||||
- Content files are saved to correct locations
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- DownloadProcessor inherits from BaseProcessor
|
||||
- DownloadInput/DownloadOutput schemas exist and validate
|
||||
- Metadata is fetched and saved correctly
|
||||
- Text content is downloaded and saved
|
||||
- Video content is marked for ASR processing
|
||||
- All tests pass
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/04-analysis-orchestration/04-02-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user