471 lines
15 KiB
Markdown
471 lines
15 KiB
Markdown
---
|
|
phase: 06-code-consolidation
|
|
plan: 02
|
|
type: execute
|
|
wave: 2
|
|
depends_on:
|
|
- 06-01
|
|
files_modified:
|
|
- kb-workflow/pipeline/__init__.py
|
|
- kb-workflow/pipeline/pipeline.py
|
|
- kb-workflow/pipeline/schemas.py
|
|
- kb-workflow/pipeline/config.py
|
|
- kb-workflow/pipeline/state.py
|
|
- kb-workflow/pipeline/errors.py
|
|
- kb-workflow/pipeline/retry.py
|
|
- kb-workflow/pipeline/idempotency.py
|
|
- kb-workflow/pipeline/link_parser.py
|
|
- kb-workflow/pipeline/processors/__init__.py
|
|
- kb-workflow/pipeline/processors/base.py
|
|
- kb-workflow/pipeline/processors/parse.py
|
|
- kb-workflow/pipeline/processors/download.py
|
|
- kb-workflow/pipeline/processors/asr.py
|
|
- kb-workflow/pipeline/processors/analyze.py
|
|
- kb-workflow/pipeline/processors/archive.py
|
|
- kb-workflow/pipeline/platforms/__init__.py
|
|
- kb-workflow/pipeline/platforms/base.py
|
|
- kb-workflow/pipeline/platforms/bilibili.py
|
|
- kb-workflow/pipeline/platforms/zhihu.py
|
|
- kb-workflow/kb_workflow.py
|
|
- tests/test_pipeline/*.py
|
|
- tests/test_processors/*.py
|
|
- tests/test_platforms/*.py
|
|
- tests/test_cli/*.py
|
|
- tests/test_integration/*.py
|
|
autonomous: true
|
|
requirements:
|
|
- PIP-01
|
|
- PIP-02
|
|
- PIP-03
|
|
- PIP-04
|
|
|
|
must_haves:
|
|
truths:
|
|
- "kb_workflow.pipeline 可正常导入"
|
|
- "kb_workflow.pipeline.processors 包含所有处理器"
|
|
- "kb_workflow.pipeline.platforms 包含所有平台适配器"
|
|
- "CLI 入口 kb_workflow.py 可正常执行"
|
|
- "所有测试通过"
|
|
- "原 kb_pipeline 目录已删除"
|
|
artifacts:
|
|
- path: "kb-workflow/pipeline/__init__.py"
|
|
provides: "pipeline 包入口，导出核心类和函数"
|
|
- path: "kb-workflow/pipeline/pipeline.py"
|
|
provides: "Pipeline 编排引擎"
|
|
- path: "kb-workflow/pipeline/schemas.py"
|
|
provides: "Pydantic Schema 定义"
|
|
- path: "kb-workflow/pipeline/processors/"
|
|
provides: "所有处理器实现"
|
|
- path: "kb-workflow/pipeline/platforms/"
|
|
provides: "所有平台适配器"
|
|
- path: "kb-workflow/kb_workflow.py"
|
|
provides: "CLI 入口"
|
|
key_links:
|
|
- from: "kb-workflow/kb_workflow.py"
|
|
to: "kb_workflow.pipeline"
|
|
via: "from kb_workflow.pipeline.xxx import ..."
|
|
- from: "tests/"
|
|
to: "kb_workflow.pipeline"
|
|
via: "from kb_workflow.pipeline.xxx import ..."
|
|
- from: "kb-workflow/pipeline/processors/base.py"
|
|
to: "kb_workflow.pipeline.config"
|
|
via: "from kb_workflow.pipeline.config import get_settings"
|
|
- from: "kb-workflow/pipeline/platforms/bilibili.py"
|
|
to: "kb_workflow.pipeline.errors"
|
|
via: "from kb_workflow.pipeline.errors import NetworkError"
|
|
</interfaces>
|
|
---
|
|
|
|
<objective>
|
|
将 kb_pipeline/ 模块合并进 kb-workflow/pipeline/，更新所有导入路径，删除原目录。
|
|
|
|
Purpose: 完成代码整合，所有代码统一在 kb_workflow 包下。
|
|
Output: kb-workflow/pipeline/ 目录包含所有 pipeline 模块，CLI 和测试使用新导入路径。
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
|
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/STATE.md
|
|
@.planning/phases/06-code-consolidation/06-CONTEXT.md
|
|
@.planning/phases/06-code-consolidation/06-RESEARCH.md
|
|
@.planning/phases/06-code-consolidation/06-01-SUMMARY.md
|
|
|
|
<interfaces>
|
|
<!-- Key interfaces executor needs from codebase -->
|
|
|
|
From kb-workflow/kb_workflow.py (需要修改的导入):
|
|
```python
|
|
# 旧代码
|
|
from kb_pipeline.config import get_settings
|
|
from kb_pipeline.pipeline import Pipeline
|
|
from kb_pipeline.processors import (
|
|
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
|
DownloadProcessor, ParseProcessor
|
|
)
|
|
from kb_pipeline.schemas import PipelineConfig
|
|
|
|
# 新代码
|
|
from kb_workflow.pipeline.config import get_settings
|
|
from kb_workflow.pipeline.pipeline import Pipeline
|
|
from kb_workflow.pipeline.processors import (
|
|
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
|
DownloadProcessor, ParseProcessor
|
|
)
|
|
from kb_workflow.pipeline.schemas import PipelineConfig
|
|
```
|
|
|
|
From kb_pipeline/__init__.py (需要移动后更新导入):
|
|
```python
|
|
# 所有 from .xxx 改为 from kb_workflow.pipeline.xxx
|
|
from kb_workflow.pipeline.schemas import (...)
|
|
from kb_workflow.pipeline.state import WorkflowStateManager
|
|
from kb_workflow.pipeline.errors import (...)
|
|
from kb_workflow.pipeline.config import KBSettings, get_settings
|
|
from kb_workflow.pipeline.retry import retry_on_network_error
|
|
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
|
|
from kb_workflow.pipeline.idempotency import IdempotencyChecker
|
|
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter
|
|
```
|
|
|
|
From kb_pipeline/processors/base.py (内部导入需要更新):
|
|
```python
|
|
# 旧
|
|
from kb_pipeline.config import get_settings
|
|
from kb_pipeline.schemas import StepInput, StepOutput
|
|
|
|
# 新
|
|
from kb_workflow.pipeline.config import get_settings
|
|
from kb_workflow.pipeline.schemas import StepInput, StepOutput
|
|
```
|
|
|
|
From kb_pipeline/platforms/bilibili.py (已更新的 metadata 导入):
|
|
```python
|
|
# 在 06-01 中已更新为:
|
|
from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor
|
|
|
|
# 还需要更新内部导入:
|
|
# 旧
|
|
from .base import PlatformAdapter
|
|
from ..errors import NetworkError, AuthError
|
|
from ..retry import retry_on_network_error
|
|
from ..state import WorkflowStateManager
|
|
|
|
# 新
|
|
from kb_workflow.pipeline.platforms.base import PlatformAdapter
|
|
from kb_workflow.pipeline.errors import NetworkError, AuthError
|
|
from kb_workflow.pipeline.retry import retry_on_network_error
|
|
from kb_workflow.pipeline.state import WorkflowStateManager
|
|
```
|
|
</interfaces>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: 创建 pipeline 目录并移动核心文件</name>
|
|
<files>
|
|
kb-workflow/pipeline/__init__.py,
|
|
kb-workflow/pipeline/pipeline.py,
|
|
kb-workflow/pipeline/schemas.py,
|
|
kb-workflow/pipeline/config.py,
|
|
kb-workflow/pipeline/state.py,
|
|
kb-workflow/pipeline/errors.py,
|
|
kb-workflow/pipeline/retry.py,
|
|
kb-workflow/pipeline/idempotency.py,
|
|
kb-workflow/pipeline/link_parser.py
|
|
</files>
|
|
<action>
|
|
1. 创建 kb-workflow/pipeline/ 目录
|
|
2. 使用 git mv 移动核心文件:
|
|
```bash
|
|
git mv kb_pipeline/pipeline.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/schemas.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/config.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/state.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/errors.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/retry.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/idempotency.py kb-workflow/pipeline/
|
|
git mv kb_pipeline/link_parser.py kb-workflow/pipeline/
|
|
```
|
|
3. 创建 kb-workflow/pipeline/__init__.py（基于原 kb_pipeline/__init__.py，更新导入路径）:
|
|
```python
|
|
"""KB Pipeline - 工作流管道基础设施
|
|
|
|
提供内容处理工作流的核心 Schema 和工具。
|
|
"""
|
|
|
|
from kb_workflow.pipeline.schemas import (
|
|
StepInput,
|
|
StepOutput,
|
|
StepResult,
|
|
StepStatus,
|
|
StepRecord,
|
|
WorkflowState,
|
|
StepDefinition,
|
|
)
|
|
from kb_workflow.pipeline.state import WorkflowStateManager
|
|
from kb_workflow.pipeline.errors import (
|
|
KBError,
|
|
NetworkError,
|
|
ValidationError,
|
|
SystemError,
|
|
AuthError,
|
|
ErrorType,
|
|
)
|
|
from kb_workflow.pipeline.config import KBSettings, get_settings
|
|
from kb_workflow.pipeline.retry import retry_on_network_error
|
|
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
|
|
from kb_workflow.pipeline.idempotency import IdempotencyChecker
|
|
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter
|
|
|
|
__all__ = [
|
|
# Schemas
|
|
"StepInput",
|
|
"StepOutput",
|
|
"StepResult",
|
|
"StepStatus",
|
|
"StepRecord",
|
|
"WorkflowState",
|
|
"StepDefinition",
|
|
# State
|
|
"WorkflowStateManager",
|
|
# Errors
|
|
"KBError",
|
|
"NetworkError",
|
|
"ValidationError",
|
|
"SystemError",
|
|
"AuthError",
|
|
"ErrorType",
|
|
# Config
|
|
"KBSettings",
|
|
"get_settings",
|
|
# Retry
|
|
"retry_on_network_error",
|
|
# Link Parser
|
|
"LinkParser",
|
|
"ParsedLink",
|
|
# Idempotency
|
|
"IdempotencyChecker",
|
|
# Platform Adapters
|
|
"PlatformAdapter",
|
|
"BilibiliAdapter",
|
|
"ZhihuAdapter",
|
|
]
|
|
|
|
__version__ = "0.1.0"
|
|
```
|
|
4. 更新核心文件内部导入（使用绝对导入）:
|
|
- pipeline.py: 所有 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx`
|
|
- schemas.py: 检查是否有内部导入需要更新
|
|
- config.py: 检查是否有内部导入需要更新
|
|
- state.py: 检查是否有内部导入需要更新
|
|
- errors.py: 通常无外部导入
|
|
- retry.py: 通常无外部导入
|
|
- idempotency.py: 通常无外部导入
|
|
- link_parser.py: 通常无外部导入
|
|
</action>
|
|
<verify>
|
|
<automated>python -c "from kb_workflow.pipeline import Pipeline, get_settings, WorkflowStateManager; print('core OK')"</automated>
|
|
</verify>
|
|
<done>pipeline 核心文件已移动到 kb-workflow/pipeline/，核心类可导入</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 2: 移动 processors 和 platforms 子目录</name>
|
|
<files>
|
|
kb-workflow/pipeline/processors/__init__.py,
|
|
kb-workflow/pipeline/processors/base.py,
|
|
kb-workflow/pipeline/processors/parse.py,
|
|
kb-workflow/pipeline/processors/download.py,
|
|
kb-workflow/pipeline/processors/asr.py,
|
|
kb-workflow/pipeline/processors/analyze.py,
|
|
kb-workflow/pipeline/processors/archive.py,
|
|
kb-workflow/pipeline/platforms/__init__.py,
|
|
kb-workflow/pipeline/platforms/base.py,
|
|
kb-workflow/pipeline/platforms/bilibili.py,
|
|
kb-workflow/pipeline/platforms/zhihu.py
|
|
</files>
|
|
<action>
|
|
1. 创建子目录并移动文件:
|
|
```bash
|
|
mkdir -p kb-workflow/pipeline/processors
|
|
mkdir -p kb-workflow/pipeline/platforms
|
|
git mv kb_pipeline/processors/*.py kb-workflow/pipeline/processors/
|
|
git mv kb_pipeline/platforms/*.py kb-workflow/pipeline/platforms/
|
|
```
|
|
|
|
2. 更新 processors 内部导入（改为绝对导入）:
|
|
在以下文件中将 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx`:
|
|
- base.py: `from kb_pipeline.config` → `from kb_workflow.pipeline.config`
|
|
- base.py: `from kb_pipeline.schemas` → `from kb_workflow.pipeline.schemas`
|
|
- parse.py, download.py, asr.py, analyze.py, archive.py: 检查并更新所有 kb_pipeline 导入
|
|
- __init__.py: 更新所有 from .xxx 或 from ..xxx 为绝对导入
|
|
|
|
3. 更新 platforms 内部导入:
|
|
- bilibili.py: 已在 06-01 更新 metadata 导入，现在更新:
|
|
- `from .base` → `from kb_workflow.pipeline.platforms.base`
|
|
- `from ..errors` → `from kb_workflow.pipeline.errors`
|
|
- `from ..retry` → `from kb_workflow.pipeline.retry`
|
|
- `from ..state` → `from kb_workflow.pipeline.state`
|
|
- `from ..config` → `from kb_workflow.pipeline.config`
|
|
- zhihu.py: 同样更新所有相对导入为绝对导入
|
|
- base.py: 检查并更新导入
|
|
- __init__.py: 更新导出
|
|
|
|
4. 更新后的 processors/__init__.py:
|
|
```python
|
|
"""Pipeline Processors
|
|
|
|
提供所有管道处理器实现。
|
|
"""
|
|
|
|
from kb_workflow.pipeline.processors.base import BaseProcessor
|
|
from kb_workflow.pipeline.processors.parse import ParseProcessor
|
|
from kb_workflow.pipeline.processors.download import DownloadProcessor
|
|
from kb_workflow.pipeline.processors.asr import ASRProcessor
|
|
from kb_workflow.pipeline.processors.analyze import AnalyzeProcessor
|
|
from kb_workflow.pipeline.processors.archive import ArchiveProcessor
|
|
|
|
__all__ = [
|
|
"BaseProcessor",
|
|
"ParseProcessor",
|
|
"DownloadProcessor",
|
|
"ASRProcessor",
|
|
"AnalyzeProcessor",
|
|
"ArchiveProcessor",
|
|
]
|
|
```
|
|
|
|
5. 更新后的 platforms/__init__.py:
|
|
```python
|
|
"""Platform Adapters
|
|
|
|
提供平台适配器接口和实现。
|
|
"""
|
|
|
|
from kb_workflow.pipeline.platforms.base import PlatformAdapter
|
|
from kb_workflow.pipeline.platforms.bilibili import BilibiliAdapter
|
|
from kb_workflow.pipeline.platforms.zhihu import ZhihuAdapter
|
|
|
|
__all__ = [
|
|
"PlatformAdapter",
|
|
"BilibiliAdapter",
|
|
"ZhihuAdapter",
|
|
]
|
|
```
|
|
</action>
|
|
<verify>
|
|
<automated>python -c "from kb_workflow.pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor, ArchiveProcessor; from kb_workflow.pipeline.platforms import BilibiliAdapter, ZhihuAdapter; print('processors and platforms OK')"</automated>
|
|
</verify>
|
|
<done>processors 和 platforms 已移动，所有处理器和适配器可导入</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 3: 更新 CLI 入口和测试文件导入</name>
|
|
<files>
|
|
kb-workflow/kb_workflow.py,
|
|
tests/test_pipeline/*.py,
|
|
tests/test_processors/*.py,
|
|
tests/test_platforms/*.py,
|
|
tests/test_cli/*.py,
|
|
tests/test_integration/*.py
|
|
</files>
|
|
<action>
|
|
1. 更新 kb-workflow/kb_workflow.py:
|
|
- 删除 sys.path.insert 操作（第 41-43 行）
|
|
- 更新所有导入:
|
|
```python
|
|
from kb_workflow.pipeline.config import get_settings
|
|
from kb_workflow.pipeline.pipeline import Pipeline
|
|
from kb_workflow.pipeline.processors import (
|
|
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
|
DownloadProcessor, ParseProcessor
|
|
)
|
|
from kb_workflow.pipeline.schemas import PipelineConfig
|
|
```
|
|
|
|
2. 使用 grep 找出所有需要更新的测试文件:
|
|
```bash
|
|
grep -rn "from kb_pipeline" tests/ --include="*.py"
|
|
```
|
|
|
|
3. 批量更新测试文件导入（使用 sed 或手动）:
|
|
- `from kb_pipeline.xxx` → `from kb_workflow.pipeline.xxx`
|
|
- 例如: `from kb_pipeline.schemas` → `from kb_workflow.pipeline.schemas`
|
|
- 例如: `from kb_pipeline.processors` → `from kb_workflow.pipeline.processors`
|
|
- 例如: `from kb_pipeline.platforms` → `from kb_workflow.pipeline.platforms`
|
|
|
|
4. 更新 tests/test_zhihu_metadata/ 中的导入（如果有）
|
|
</action>
|
|
<verify>
|
|
<automated>cd /home/starmen1265/work/kb-skills/kb-workflow && python kb_workflow.py --help</automated>
|
|
</verify>
|
|
<done>CLI 入口可执行，测试文件导入路径已更新</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 4: 删除原 kb_pipeline 目录并运行全量测试</name>
|
|
<files>
|
|
(删除) kb_pipeline/
|
|
</files>
|
|
<action>
|
|
1. 运行全量测试验证整合成功:
|
|
```bash
|
|
pytest tests/ -v --tb=short -x
|
|
```
|
|
|
|
2. 测试通过后，删除原目录:
|
|
```bash
|
|
git rm -r kb_pipeline/
|
|
```
|
|
|
|
3. 再次运行测试确认删除不影响功能:
|
|
```bash
|
|
pytest tests/ -v
|
|
```
|
|
|
|
4. 验证 CLI 功能:
|
|
```bash
|
|
cd kb-workflow && python kb_workflow.py --help
|
|
```
|
|
</action>
|
|
<verify>
|
|
<automated>pytest tests/ -v --tb=short</automated>
|
|
</verify>
|
|
<done>原 kb_pipeline 目录已删除，所有测试通过，CLI 正常工作</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
1. 核心导入验证:
|
|
- `from kb_workflow.pipeline import Pipeline, get_settings` 成功
|
|
- `from kb_workflow.pipeline.processors import ParseProcessor` 成功
|
|
- `from kb_workflow.pipeline.platforms import BilibiliAdapter` 成功
|
|
2. CLI 验证:
|
|
- `python kb-workflow/kb_workflow.py --help` 输出帮助信息
|
|
3. 测试验证:
|
|
- `pytest tests/ -v` 全部通过
|
|
4. 清理验证:
|
|
- `ls kb_pipeline/` 返回 "No such file or directory"
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- [ ] kb-workflow/pipeline/ 目录包含所有 pipeline 模块
|
|
- [ ] kb-workflow/kb_workflow.py 使用新导入路径
|
|
- [ ] 所有测试文件使用新导入路径
|
|
- [ ] 所有测试通过 (pytest tests/ -v)
|
|
- [ ] CLI 可执行 (python kb_workflow.py --help)
|
|
- [ ] 原 kb_pipeline 目录已删除
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/06-code-consolidation/06-02-SUMMARY.md`
|
|
</output>
|