docs(06): create phase 6 consolidation plans
This commit is contained in:
@@ -64,8 +64,8 @@ Plans:
|
||||
|
||||
Plans:
|
||||
- [x] 05-01-PLAN.md — 中间目录统一和配置 (INT-01, INT-02, INT-03)
|
||||
- [ ] 05-02-PLAN.md — ArchiveProcessor 归档处理器 (ARC-01, ARC-02, ARC-03, ARC-04)
|
||||
- [ ] 05-03-PLAN.md — 最终目录结构验证和测试 (OUT-01, OUT-02, OUT-03, OUT-04)
|
||||
- [x] 05-02-PLAN.md — ArchiveProcessor 归档处理器 (ARC-01, ARC-02, ARC-03, ARC-04)
|
||||
- [x] 05-03-PLAN.md — 最终目录结构验证和测试 (OUT-01, OUT-02, OUT-03, OUT-04)
|
||||
|
||||
#### Phase 6: 代码整合
|
||||
**Goal**: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow
|
||||
@@ -76,11 +76,11 @@ Plans:
|
||||
2. kb_pipeline/ 代码合并入 kb-workflow/ 保持 processors/, platforms/, schemas.py 结构
|
||||
3. 所有导入路径更新指向新位置，CLI 正常工作
|
||||
4. 原独立的 kb-bilibili-metadata、kb-zhihu-metadata、kb_pipeline 目录已删除
|
||||
**Plans**: TBD
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [ ] 06-01: 元数据技能整合
|
||||
- [ ] 06-02: Pipeline 模块整合
|
||||
- [ ] 06-01-PLAN.md — 元数据技能整合 (MET-01, MET-02, MET-03, MET-04, MET-05)
|
||||
- [ ] 06-02-PLAN.md — Pipeline 模块整合 (PIP-01, PIP-02, PIP-03, PIP-04)
|
||||
|
||||
## Progress
|
||||
|
||||
@@ -93,5 +93,5 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6
|
||||
| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||
| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 5. 归档与目录优化 | v1.1 | 1/3 | In progress | 2026-03-16 |
|
||||
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||
| 6. 代码整合 | v1.1 | 0/2 | Not started | - |
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
---
|
||||
phase: 06-code-consolidation
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- kb-workflow/metadata/__init__.py
|
||||
- kb-workflow/metadata/bilibili_extractor.py
|
||||
- kb-workflow/metadata/bilibili_metadata.py
|
||||
- kb-workflow/metadata/zhihu_extractor.py
|
||||
- kb-workflow/metadata/zhihu_metadata.py
|
||||
- kb-workflow/metadata/zhihu_article_crawler.py
|
||||
- kb-workflow/metadata/markdown_converter.py
|
||||
- kb_pipeline/platforms/bilibili.py
|
||||
- kb_pipeline/platforms/zhihu.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
- MET-01
|
||||
- MET-02
|
||||
- MET-03
|
||||
- MET-04
|
||||
- MET-05
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "kb_workflow.metadata.bilibili_extractor 可正常导入"
|
||||
- "kb_workflow.metadata.zhihu_extractor 可正常导入"
|
||||
- "kb_pipeline/platforms/ 模块可通过新路径导入 metadata 模块"
|
||||
- "原 kb-bilibili-metadata 目录已删除"
|
||||
- "原 kb-zhihu-metadata 目录已删除"
|
||||
artifacts:
|
||||
- path: "kb-workflow/metadata/__init__.py"
|
||||
provides: "metadata 包入口，导出 BilibiliExtractor 和 ZhihuPlaywrightExtractor"
|
||||
- path: "kb-workflow/metadata/bilibili_extractor.py"
|
||||
provides: "B站元数据提取器"
|
||||
- path: "kb-workflow/metadata/zhihu_extractor.py"
|
||||
provides: "知乎内容提取器"
|
||||
key_links:
|
||||
- from: "kb_pipeline/platforms/bilibili.py"
|
||||
to: "kb_workflow.metadata.bilibili_extractor"
|
||||
via: "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor"
|
||||
- from: "kb_pipeline/platforms/zhihu.py"
|
||||
to: "kb_workflow.metadata.zhihu_extractor"
|
||||
via: "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor"
|
||||
---
|
||||
|
||||
<objective>
|
||||
将独立的元数据技能模块 (kb-bilibili-metadata 和 kb-zhihu-metadata) 整合进 kb-workflow/metadata/。
|
||||
|
||||
Purpose: 统一代码结构，消除 sys.path 动态操作，使用标准包导入。
|
||||
Output: kb-workflow/metadata/ 目录包含所有元数据提取模块，platform adapters 使用标准导入。
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/06-code-consolidation/06-CONTEXT.md
|
||||
@.planning/phases/06-code-consolidation/06-RESEARCH.md
|
||||
|
||||
<interfaces>
|
||||
<!-- Key interfaces executor needs from codebase -->
|
||||
|
||||
From kb_pipeline/platforms/bilibili.py (需要修改的动态导入):
|
||||
```python
|
||||
# 旧代码 - 需要删除
|
||||
_BILIBILI_METADATA_PATH = Path(__file__).parent.parent.parent / "kb-bilibili-metadata"
|
||||
if _BILIBILI_METADATA_PATH.exists():
|
||||
sys.path.insert(0, str(_BILIBILI_METADATA_PATH))
|
||||
from bilibili_extractor import BilibiliExtractor
|
||||
|
||||
# 新代码 - 标准导入
|
||||
from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor
|
||||
```
|
||||
|
||||
From kb_pipeline/platforms/zhihu.py (需要修改的动态导入):
|
||||
```python
|
||||
# 旧代码 - 需要删除
|
||||
_kb_zhihu_path = Path(__file__).parent.parent.parent / "kb-zhihu-metadata"
|
||||
if str(_kb_zhihu_path) not in sys.path:
|
||||
sys.path.insert(0, str(_kb_zhihu_path))
|
||||
from zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
|
||||
# 新代码 - 标准导入
|
||||
from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: 创建 metadata 目录并移动 bilibili 模块</name>
|
||||
<files>
|
||||
kb-workflow/metadata/__init__.py,
|
||||
kb-workflow/metadata/bilibili_extractor.py,
|
||||
kb-workflow/metadata/bilibili_metadata.py
|
||||
</files>
|
||||
<action>
|
||||
1. 创建 kb-workflow/metadata/ 目录
|
||||
2. 使用 git mv 移动文件（保留历史）:
|
||||
- kb-bilibili-metadata/bilibili_extractor.py → kb-workflow/metadata/bilibili_extractor.py
|
||||
- kb-bilibili-metadata/bilibili_metadata.py → kb-workflow/metadata/bilibili_metadata.py
|
||||
3. 创建 kb-workflow/metadata/__init__.py，内容如下:
|
||||
```python
|
||||
"""KB Workflow Metadata Module
|
||||
|
||||
提供 B站和知乎的元数据提取功能
|
||||
"""
|
||||
|
||||
from .bilibili_extractor import BilibiliExtractor
|
||||
|
||||
__all__ = [
|
||||
"BilibiliExtractor",
|
||||
]
|
||||
```
|
||||
4. 移动后不要修改 bilibili_extractor.py 内部代码（保持原样）
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor; print('bilibili OK')"</automated>
|
||||
</verify>
|
||||
<done>kb-workflow/metadata/ 目录存在，bilibili_extractor.py 可从新路径导入</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: 移动 zhihu 模块到 metadata 目录</name>
|
||||
<files>
|
||||
kb-workflow/metadata/zhihu_extractor.py,
|
||||
kb-workflow/metadata/zhihu_metadata.py,
|
||||
kb-workflow/metadata/zhihu_article_crawler.py,
|
||||
kb-workflow/metadata/markdown_converter.py,
|
||||
kb-workflow/metadata/__init__.py
|
||||
</files>
|
||||
<action>
|
||||
1. 使用 git mv 移动文件:
|
||||
- kb-zhihu-metadata/zhihu_extractor.py → kb-workflow/metadata/zhihu_extractor.py
|
||||
- kb-zhihu-metadata/zhihu_metadata.py → kb-workflow/metadata/zhihu_metadata.py
|
||||
- kb-zhihu-metadata/zhihu_article_crawler.py → kb-workflow/metadata/zhihu_article_crawler.py
|
||||
- kb-zhihu-metadata/markdown_converter.py → kb-workflow/metadata/markdown_converter.py
|
||||
2. 更新 kb-workflow/metadata/__init__.py:
|
||||
```python
|
||||
"""KB Workflow Metadata Module
|
||||
|
||||
提供 B站和知乎的元数据提取功能
|
||||
"""
|
||||
|
||||
from .bilibili_extractor import BilibiliExtractor
|
||||
from .zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
|
||||
__all__ = [
|
||||
"BilibiliExtractor",
|
||||
"ZhihuPlaywrightExtractor",
|
||||
]
|
||||
```
|
||||
3. 移动后不要修改内部代码（保持原样）
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor; print('zhihu OK')"</automated>
|
||||
</verify>
|
||||
<done>zhihu 模块文件已移动到 kb-workflow/metadata/，可从新路径导入</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: 更新 platform adapters 导入路径</name>
|
||||
<files>
|
||||
kb_pipeline/platforms/bilibili.py,
|
||||
kb_pipeline/platforms/zhihu.py
|
||||
</files>
|
||||
<action>
|
||||
更新 kb_pipeline/platforms/bilibili.py:
|
||||
1. 删除 sys.path 动态操作代码（第 33-36 行）:
|
||||
```python
|
||||
# 删除以下代码
|
||||
_BILIBILI_METADATA_PATH = Path(__file__).parent.parent.parent / "kb-bilibili-metadata"
|
||||
if _BILIBILI_METADATA_PATH.exists():
|
||||
sys.path.insert(0, str(_BILIBILI_METADATA_PATH))
|
||||
```
|
||||
2. 将 `from bilibili_extractor import BilibiliExtractor` 改为 `from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor`
|
||||
3. 删除 try/except 包装，直接导入（因为包已存在）
|
||||
|
||||
更新 kb_pipeline/platforms/zhihu.py:
|
||||
1. 删除 sys.path 动态操作代码（第 18-20 行）:
|
||||
```python
|
||||
# 删除以下代码
|
||||
_kb_zhihu_path = Path(__file__).parent.parent.parent / "kb-zhihu-metadata"
|
||||
if str(_kb_zhihu_path) not in sys.path:
|
||||
sys.path.insert(0, str(_kb_zhihu_path))
|
||||
```
|
||||
2. 将 `from zhihu_extractor import ZhihuPlaywrightExtractor` 改为 `from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor`
|
||||
3. 删除 try/except 包装中的 ImportError 处理逻辑，直接导入
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_pipeline.platforms.bilibili import BilibiliAdapter; from kb_pipeline.platforms.zhihu import ZhihuAdapter; print('adapters OK')"</automated>
|
||||
</verify>
|
||||
<done>platform adapters 使用标准包导入，无 sys.path 操作</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: 删除原独立目录并验证</name>
|
||||
<files>
|
||||
(删除) kb-bilibili-metadata/,
|
||||
(删除) kb-zhihu-metadata/
|
||||
</files>
|
||||
<action>
|
||||
1. 运行测试验证整合成功:
|
||||
```bash
|
||||
pytest tests/test_platforms/test_bilibili.py tests/test_platforms/test_zhihu.py -v
|
||||
```
|
||||
2. 测试通过后，使用 git rm 删除原目录:
|
||||
```bash
|
||||
git rm -r kb-bilibili-metadata/ kb-zhihu-metadata/
|
||||
```
|
||||
3. 再次运行测试确认删除不影响功能:
|
||||
```bash
|
||||
pytest tests/ -v --tb=short -x
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/ -v --tb=short -x</automated>
|
||||
</verify>
|
||||
<done>原 kb-bilibili-metadata 和 kb-zhihu-metadata 目录已删除，所有测试通过</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
1. 导入验证:
|
||||
- `from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor` 成功
|
||||
- `from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor` 成功
|
||||
2. 平台适配器验证:
|
||||
- `from kb_pipeline.platforms.bilibili import BilibiliAdapter` 成功
|
||||
- `from kb_pipeline.platforms.zhihu import ZhihuAdapter` 成功
|
||||
3. 测试验证:
|
||||
- `pytest tests/ -v` 全部通过
|
||||
4. 清理验证:
|
||||
- `ls kb-bilibili-metadata/` 返回 "No such file or directory"
|
||||
- `ls kb-zhihu-metadata/` 返回 "No such file or directory"
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- [ ] kb-workflow/metadata/ 目录包含 bilibili 和 zhihu 所有模块
|
||||
- [ ] kb_pipeline/platforms/ 使用标准导入，无 sys.path 操作
|
||||
- [ ] 所有测试通过 (pytest tests/ -v)
|
||||
- [ ] 原 kb-bilibili-metadata 和 kb-zhihu-metadata 目录已删除
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/06-code-consolidation/06-01-SUMMARY.md`
|
||||
</output>
|
||||
@@ -0,0 +1,470 @@
|
||||
---
|
||||
phase: 06-code-consolidation
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on:
|
||||
- 06-01
|
||||
files_modified:
|
||||
- kb-workflow/pipeline/__init__.py
|
||||
- kb-workflow/pipeline/pipeline.py
|
||||
- kb-workflow/pipeline/schemas.py
|
||||
- kb-workflow/pipeline/config.py
|
||||
- kb-workflow/pipeline/state.py
|
||||
- kb-workflow/pipeline/errors.py
|
||||
- kb-workflow/pipeline/retry.py
|
||||
- kb-workflow/pipeline/idempotency.py
|
||||
- kb-workflow/pipeline/link_parser.py
|
||||
- kb-workflow/pipeline/processors/__init__.py
|
||||
- kb-workflow/pipeline/processors/base.py
|
||||
- kb-workflow/pipeline/processors/parse.py
|
||||
- kb-workflow/pipeline/processors/download.py
|
||||
- kb-workflow/pipeline/processors/asr.py
|
||||
- kb-workflow/pipeline/processors/analyze.py
|
||||
- kb-workflow/pipeline/processors/archive.py
|
||||
- kb-workflow/pipeline/platforms/__init__.py
|
||||
- kb-workflow/pipeline/platforms/base.py
|
||||
- kb-workflow/pipeline/platforms/bilibili.py
|
||||
- kb-workflow/pipeline/platforms/zhihu.py
|
||||
- kb-workflow/kb_workflow.py
|
||||
- tests/test_pipeline/*.py
|
||||
- tests/test_processors/*.py
|
||||
- tests/test_platforms/*.py
|
||||
- tests/test_cli/*.py
|
||||
- tests/test_integration/*.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
- PIP-01
|
||||
- PIP-02
|
||||
- PIP-03
|
||||
- PIP-04
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "kb_workflow.pipeline 可正常导入"
|
||||
- "kb_workflow.pipeline.processors 包含所有处理器"
|
||||
- "kb_workflow.pipeline.platforms 包含所有平台适配器"
|
||||
- "CLI 入口 kb_workflow.py 可正常执行"
|
||||
- "所有测试通过"
|
||||
- "原 kb_pipeline 目录已删除"
|
||||
artifacts:
|
||||
- path: "kb-workflow/pipeline/__init__.py"
|
||||
provides: "pipeline 包入口，导出核心类和函数"
|
||||
- path: "kb-workflow/pipeline/pipeline.py"
|
||||
provides: "Pipeline 编排引擎"
|
||||
- path: "kb-workflow/pipeline/schemas.py"
|
||||
provides: "Pydantic Schema 定义"
|
||||
- path: "kb-workflow/pipeline/processors/"
|
||||
provides: "所有处理器实现"
|
||||
- path: "kb-workflow/pipeline/platforms/"
|
||||
provides: "所有平台适配器"
|
||||
- path: "kb-workflow/kb_workflow.py"
|
||||
provides: "CLI 入口"
|
||||
key_links:
|
||||
- from: "kb-workflow/kb_workflow.py"
|
||||
to: "kb_workflow.pipeline"
|
||||
via: "from kb_workflow.pipeline.xxx import ..."
|
||||
- from: "tests/"
|
||||
to: "kb_workflow.pipeline"
|
||||
via: "from kb_workflow.pipeline.xxx import ..."
|
||||
- from: "kb-workflow/pipeline/processors/base.py"
|
||||
to: "kb_workflow.pipeline.config"
|
||||
via: "from kb_workflow.pipeline.config import get_settings"
|
||||
- from: "kb-workflow/pipeline/platforms/bilibili.py"
|
||||
to: "kb_workflow.pipeline.errors"
|
||||
via: "from kb_workflow.pipeline.errors import NetworkError"
|
||||
</interfaces>
|
||||
---
|
||||
|
||||
<objective>
|
||||
将 kb_pipeline/ 模块合并进 kb-workflow/pipeline/，更新所有导入路径，删除原目录。
|
||||
|
||||
Purpose: 完成代码整合，所有代码统一在 kb_workflow 包下。
|
||||
Output: kb-workflow/pipeline/ 目录包含所有 pipeline 模块，CLI 和测试使用新导入路径。
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/06-code-consolidation/06-CONTEXT.md
|
||||
@.planning/phases/06-code-consolidation/06-RESEARCH.md
|
||||
@.planning/phases/06-code-consolidation/06-01-SUMMARY.md
|
||||
|
||||
<interfaces>
|
||||
<!-- Key interfaces executor needs from codebase -->
|
||||
|
||||
From kb-workflow/kb_workflow.py (需要修改的导入):
|
||||
```python
|
||||
# 旧代码
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.pipeline import Pipeline
|
||||
from kb_pipeline.processors import (
|
||||
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
||||
DownloadProcessor, ParseProcessor
|
||||
)
|
||||
from kb_pipeline.schemas import PipelineConfig
|
||||
|
||||
# 新代码
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from kb_workflow.pipeline.pipeline import Pipeline
|
||||
from kb_workflow.pipeline.processors import (
|
||||
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
||||
DownloadProcessor, ParseProcessor
|
||||
)
|
||||
from kb_workflow.pipeline.schemas import PipelineConfig
|
||||
```
|
||||
|
||||
From kb_pipeline/__init__.py (需要移动后更新导入):
|
||||
```python
|
||||
# 所有 from .xxx 改为 from kb_workflow.pipeline.xxx
|
||||
from kb_workflow.pipeline.schemas import (...)
|
||||
from kb_workflow.pipeline.state import WorkflowStateManager
|
||||
from kb_workflow.pipeline.errors import (...)
|
||||
from kb_workflow.pipeline.config import KBSettings, get_settings
|
||||
from kb_workflow.pipeline.retry import retry_on_network_error
|
||||
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
|
||||
from kb_workflow.pipeline.idempotency import IdempotencyChecker
|
||||
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter
|
||||
```
|
||||
|
||||
From kb_pipeline/processors/base.py (内部导入需要更新):
|
||||
```python
|
||||
# 旧
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.schemas import StepInput, StepOutput
|
||||
|
||||
# 新
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from kb_workflow.pipeline.schemas import StepInput, StepOutput
|
||||
```
|
||||
|
||||
From kb_pipeline/platforms/bilibili.py (已更新的 metadata 导入):
|
||||
```python
|
||||
# 在 06-01 中已更新为:
|
||||
from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor
|
||||
|
||||
# 还需要更新内部导入:
|
||||
# 旧
|
||||
from .base import PlatformAdapter
|
||||
from ..errors import NetworkError, AuthError
|
||||
from ..retry import retry_on_network_error
|
||||
from ..state import WorkflowStateManager
|
||||
|
||||
# 新
|
||||
from kb_workflow.pipeline.platforms.base import PlatformAdapter
|
||||
from kb_workflow.pipeline.errors import NetworkError, AuthError
|
||||
from kb_workflow.pipeline.retry import retry_on_network_error
|
||||
from kb_workflow.pipeline.state import WorkflowStateManager
|
||||
```
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: 创建 pipeline 目录并移动核心文件</name>
|
||||
<files>
|
||||
kb-workflow/pipeline/__init__.py,
|
||||
kb-workflow/pipeline/pipeline.py,
|
||||
kb-workflow/pipeline/schemas.py,
|
||||
kb-workflow/pipeline/config.py,
|
||||
kb-workflow/pipeline/state.py,
|
||||
kb-workflow/pipeline/errors.py,
|
||||
kb-workflow/pipeline/retry.py,
|
||||
kb-workflow/pipeline/idempotency.py,
|
||||
kb-workflow/pipeline/link_parser.py
|
||||
</files>
|
||||
<action>
|
||||
1. 创建 kb-workflow/pipeline/ 目录
|
||||
2. 使用 git mv 移动核心文件:
|
||||
```bash
|
||||
git mv kb_pipeline/pipeline.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/schemas.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/config.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/state.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/errors.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/retry.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/idempotency.py kb-workflow/pipeline/
|
||||
git mv kb_pipeline/link_parser.py kb-workflow/pipeline/
|
||||
```
|
||||
3. 创建 kb-workflow/pipeline/__init__.py（基于原 kb_pipeline/__init__.py，更新导入路径）:
|
||||
```python
|
||||
"""KB Pipeline - 工作流管道基础设施
|
||||
|
||||
提供内容处理工作流的核心 Schema 和工具。
|
||||
"""
|
||||
|
||||
from kb_workflow.pipeline.schemas import (
|
||||
StepInput,
|
||||
StepOutput,
|
||||
StepResult,
|
||||
StepStatus,
|
||||
StepRecord,
|
||||
WorkflowState,
|
||||
StepDefinition,
|
||||
)
|
||||
from kb_workflow.pipeline.state import WorkflowStateManager
|
||||
from kb_workflow.pipeline.errors import (
|
||||
KBError,
|
||||
NetworkError,
|
||||
ValidationError,
|
||||
SystemError,
|
||||
AuthError,
|
||||
ErrorType,
|
||||
)
|
||||
from kb_workflow.pipeline.config import KBSettings, get_settings
|
||||
from kb_workflow.pipeline.retry import retry_on_network_error
|
||||
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
|
||||
from kb_workflow.pipeline.idempotency import IdempotencyChecker
|
||||
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter
|
||||
|
||||
__all__ = [
|
||||
# Schemas
|
||||
"StepInput",
|
||||
"StepOutput",
|
||||
"StepResult",
|
||||
"StepStatus",
|
||||
"StepRecord",
|
||||
"WorkflowState",
|
||||
"StepDefinition",
|
||||
# State
|
||||
"WorkflowStateManager",
|
||||
# Errors
|
||||
"KBError",
|
||||
"NetworkError",
|
||||
"ValidationError",
|
||||
"SystemError",
|
||||
"AuthError",
|
||||
"ErrorType",
|
||||
# Config
|
||||
"KBSettings",
|
||||
"get_settings",
|
||||
# Retry
|
||||
"retry_on_network_error",
|
||||
# Link Parser
|
||||
"LinkParser",
|
||||
"ParsedLink",
|
||||
# Idempotency
|
||||
"IdempotencyChecker",
|
||||
# Platform Adapters
|
||||
"PlatformAdapter",
|
||||
"BilibiliAdapter",
|
||||
"ZhihuAdapter",
|
||||
]
|
||||
|
||||
__version__ = "0.1.0"
|
||||
```
|
||||
4. 更新核心文件内部导入（使用绝对导入）:
|
||||
- pipeline.py: 所有 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx`
|
||||
- schemas.py: 检查是否有内部导入需要更新
|
||||
- config.py: 检查是否有内部导入需要更新
|
||||
- state.py: 检查是否有内部导入需要更新
|
||||
- errors.py: 通常无外部导入
|
||||
- retry.py: 通常无外部导入
|
||||
- idempotency.py: 通常无外部导入
|
||||
- link_parser.py: 通常无外部导入
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_workflow.pipeline import Pipeline, get_settings, WorkflowStateManager; print('core OK')"</automated>
|
||||
</verify>
|
||||
<done>pipeline 核心文件已移动到 kb-workflow/pipeline/，核心类可导入</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: 移动 processors 和 platforms 子目录</name>
|
||||
<files>
|
||||
kb-workflow/pipeline/processors/__init__.py,
|
||||
kb-workflow/pipeline/processors/base.py,
|
||||
kb-workflow/pipeline/processors/parse.py,
|
||||
kb-workflow/pipeline/processors/download.py,
|
||||
kb-workflow/pipeline/processors/asr.py,
|
||||
kb-workflow/pipeline/processors/analyze.py,
|
||||
kb-workflow/pipeline/processors/archive.py,
|
||||
kb-workflow/pipeline/platforms/__init__.py,
|
||||
kb-workflow/pipeline/platforms/base.py,
|
||||
kb-workflow/pipeline/platforms/bilibili.py,
|
||||
kb-workflow/pipeline/platforms/zhihu.py
|
||||
</files>
|
||||
<action>
|
||||
1. 创建子目录并移动文件:
|
||||
```bash
|
||||
mkdir -p kb-workflow/pipeline/processors
|
||||
mkdir -p kb-workflow/pipeline/platforms
|
||||
git mv kb_pipeline/processors/*.py kb-workflow/pipeline/processors/
|
||||
git mv kb_pipeline/platforms/*.py kb-workflow/pipeline/platforms/
|
||||
```
|
||||
|
||||
2. 更新 processors 内部导入（改为绝对导入）:
|
||||
在以下文件中将 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx`:
|
||||
- base.py: `from kb_pipeline.config` → `from kb_workflow.pipeline.config`
|
||||
- base.py: `from kb_pipeline.schemas` → `from kb_workflow.pipeline.schemas`
|
||||
- parse.py, download.py, asr.py, analyze.py, archive.py: 检查并更新所有 kb_pipeline 导入
|
||||
- __init__.py: 更新所有 from .xxx 或 from ..xxx 为绝对导入
|
||||
|
||||
3. 更新 platforms 内部导入:
|
||||
- bilibili.py: 已在 06-01 更新 metadata 导入，现在更新:
|
||||
- `from .base` → `from kb_workflow.pipeline.platforms.base`
|
||||
- `from ..errors` → `from kb_workflow.pipeline.errors`
|
||||
- `from ..retry` → `from kb_workflow.pipeline.retry`
|
||||
- `from ..state` → `from kb_workflow.pipeline.state`
|
||||
- `from ..config` → `from kb_workflow.pipeline.config`
|
||||
- zhihu.py: 同样更新所有相对导入为绝对导入
|
||||
- base.py: 检查并更新导入
|
||||
- __init__.py: 更新导出
|
||||
|
||||
4. 更新后的 processors/__init__.py:
|
||||
```python
|
||||
"""Pipeline Processors
|
||||
|
||||
提供所有管道处理器实现。
|
||||
"""
|
||||
|
||||
from kb_workflow.pipeline.processors.base import BaseProcessor
|
||||
from kb_workflow.pipeline.processors.parse import ParseProcessor
|
||||
from kb_workflow.pipeline.processors.download import DownloadProcessor
|
||||
from kb_workflow.pipeline.processors.asr import ASRProcessor
|
||||
from kb_workflow.pipeline.processors.analyze import AnalyzeProcessor
|
||||
from kb_workflow.pipeline.processors.archive import ArchiveProcessor
|
||||
|
||||
__all__ = [
|
||||
"BaseProcessor",
|
||||
"ParseProcessor",
|
||||
"DownloadProcessor",
|
||||
"ASRProcessor",
|
||||
"AnalyzeProcessor",
|
||||
"ArchiveProcessor",
|
||||
]
|
||||
```
|
||||
|
||||
5. 更新后的 platforms/__init__.py:
|
||||
```python
|
||||
"""Platform Adapters
|
||||
|
||||
提供平台适配器接口和实现。
|
||||
"""
|
||||
|
||||
from kb_workflow.pipeline.platforms.base import PlatformAdapter
|
||||
from kb_workflow.pipeline.platforms.bilibili import BilibiliAdapter
|
||||
from kb_workflow.pipeline.platforms.zhihu import ZhihuAdapter
|
||||
|
||||
__all__ = [
|
||||
"PlatformAdapter",
|
||||
"BilibiliAdapter",
|
||||
"ZhihuAdapter",
|
||||
]
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>python -c "from kb_workflow.pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor, ArchiveProcessor; from kb_workflow.pipeline.platforms import BilibiliAdapter, ZhihuAdapter; print('processors and platforms OK')"</automated>
|
||||
</verify>
|
||||
<done>processors 和 platforms 已移动，所有处理器和适配器可导入</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: 更新 CLI 入口和测试文件导入</name>
|
||||
<files>
|
||||
kb-workflow/kb_workflow.py,
|
||||
tests/test_pipeline/*.py,
|
||||
tests/test_processors/*.py,
|
||||
tests/test_platforms/*.py,
|
||||
tests/test_cli/*.py,
|
||||
tests/test_integration/*.py
|
||||
</files>
|
||||
<action>
|
||||
1. 更新 kb-workflow/kb_workflow.py:
|
||||
- 删除 sys.path.insert 操作（第 41-43 行）
|
||||
- 更新所有导入:
|
||||
```python
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from kb_workflow.pipeline.pipeline import Pipeline
|
||||
from kb_workflow.pipeline.processors import (
|
||||
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
||||
DownloadProcessor, ParseProcessor
|
||||
)
|
||||
from kb_workflow.pipeline.schemas import PipelineConfig
|
||||
```
|
||||
|
||||
2. 使用 grep 找出所有需要更新的测试文件:
|
||||
```bash
|
||||
grep -rn "from kb_pipeline" tests/ --include="*.py"
|
||||
```
|
||||
|
||||
3. 批量更新测试文件导入（使用 sed 或手动）:
|
||||
- `from kb_pipeline.xxx` → `from kb_workflow.pipeline.xxx`
|
||||
- 例如: `from kb_pipeline.schemas` → `from kb_workflow.pipeline.schemas`
|
||||
- 例如: `from kb_pipeline.processors` → `from kb_workflow.pipeline.processors`
|
||||
- 例如: `from kb_pipeline.platforms` → `from kb_workflow.pipeline.platforms`
|
||||
|
||||
4. 更新 tests/test_zhihu_metadata/ 中的导入（如果有）
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd /home/starmen1265/work/kb-skills/kb-workflow && python kb_workflow.py --help</automated>
|
||||
</verify>
|
||||
<done>CLI 入口可执行，测试文件导入路径已更新</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4: 删除原 kb_pipeline 目录并运行全量测试</name>
|
||||
<files>
|
||||
(删除) kb_pipeline/
|
||||
</files>
|
||||
<action>
|
||||
1. 运行全量测试验证整合成功:
|
||||
```bash
|
||||
pytest tests/ -v --tb=short -x
|
||||
```
|
||||
|
||||
2. 测试通过后，删除原目录:
|
||||
```bash
|
||||
git rm -r kb_pipeline/
|
||||
```
|
||||
|
||||
3. 再次运行测试确认删除不影响功能:
|
||||
```bash
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
4. 验证 CLI 功能:
|
||||
```bash
|
||||
cd kb-workflow && python kb_workflow.py --help
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/ -v --tb=short</automated>
|
||||
</verify>
|
||||
<done>原 kb_pipeline 目录已删除，所有测试通过，CLI 正常工作</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
1. 核心导入验证:
|
||||
- `from kb_workflow.pipeline import Pipeline, get_settings` 成功
|
||||
- `from kb_workflow.pipeline.processors import ParseProcessor` 成功
|
||||
- `from kb_workflow.pipeline.platforms import BilibiliAdapter` 成功
|
||||
2. CLI 验证:
|
||||
- `python kb-workflow/kb_workflow.py --help` 输出帮助信息
|
||||
3. 测试验证:
|
||||
- `pytest tests/ -v` 全部通过
|
||||
4. 清理验证:
|
||||
- `ls kb_pipeline/` 返回 "No such file or directory"
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- [ ] kb-workflow/pipeline/ 目录包含所有 pipeline 模块
|
||||
- [ ] kb-workflow/kb_workflow.py 使用新导入路径
|
||||
- [ ] 所有测试文件使用新导入路径
|
||||
- [ ] 所有测试通过 (pytest tests/ -v)
|
||||
- [ ] CLI 可执行 (python kb_workflow.py --help)
|
||||
- [ ] 原 kb_pipeline 目录已删除
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/06-code-consolidation/06-02-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user