docs(phase-06): research code consolidation domain
This commit is contained in:
@@ -0,0 +1,448 @@
|
||||
# Phase 6: 代码整合 - Research
|
||||
|
||||
**Researched:** 2026-03-16
|
||||
**Domain:** Python 代码重构、模块整合、导入路径迁移
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Summary
|
||||
|
||||
本阶段将三个独立的 Python 模块整合进 kb-workflow 包：
|
||||
1. `kb-bilibili-metadata/` (3 files) - B站元数据提取
|
||||
2. `kb-zhihu-metadata/` (5 files) - 知乎内容提取
|
||||
3. `kb_pipeline/` (20 files) - 工作流管道核心
|
||||
|
||||
整合后的目录结构将使所有代码统一在 `kb-workflow` 包下，便于维护和分发。主要工作是文件移动和导入路径更新，不涉及功能改动。
|
||||
|
||||
**Primary recommendation:** 按照先 metadata 后 pipeline 的顺序，分步整合，每步完成后运行测试验证。
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### 目录结构
|
||||
- **完全扁平化**: kb-workflow/metadata/ 和 kb-workflow/pipeline/ 作为顶级子目录
|
||||
- **元数据目录**: kb-workflow/metadata/bilibili_extractor.py, zhihu_extractor.py 等
|
||||
- **Pipeline 目录**: kb-workflow/pipeline/processors/, kb-workflow/pipeline/platforms/ 等
|
||||
- **保留原文件名**: bilibili_extractor.py 不重命名为 bilibili.py，保持迁移安全
|
||||
|
||||
### 导入策略
|
||||
- **外部导入**: 统一从 kb_workflow 导入
|
||||
- `from kb_workflow.metadata.bilibili_extractor import ...`
|
||||
- `from kb_workflow.pipeline.processors import ...`
|
||||
- **内部导入**: 使用绝对导入
|
||||
- `from kb_workflow.pipeline.processors.base import BaseProcessor`
|
||||
- **更新范围**: 所有测试文件、CLI 入口、处理器间引用
|
||||
|
||||
### 整合顺序
|
||||
- **先 metadata 后 pipeline**:
|
||||
1. 移动 kb-bilibili-metadata → kb-workflow/metadata/
|
||||
2. 移动 kb-zhihu-metadata → kb-workflow/metadata/
|
||||
3. 更新 metadata 相关导入
|
||||
4. 运行测试验证
|
||||
5. 移动 kb_pipeline → kb-workflow/pipeline/
|
||||
6. 更新 pipeline 相关导入
|
||||
7. 运行测试验证
|
||||
|
||||
### 验证策略
|
||||
- **分步验证**: 每个子整合完成后运行测试
|
||||
- **测试命令**: `pytest tests/ -v`
|
||||
- **失败处理**: 测试失败时不继续下一步，先修复问题
|
||||
|
||||
### 清理策略
|
||||
- **立即删除**: 整合完成且测试通过后立即删除原目录
|
||||
- **删除目录**: kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/
|
||||
- **Git 提交**: 每个子整合单独提交，便于回滚
|
||||
|
||||
### Claude's Discretion
|
||||
- 具体导入路径更新时的细节调整
|
||||
- 处理意外的循环导入问题
|
||||
- 测试文件的导入更新方式
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
None — discussion stayed within phase scope
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|-----------------|
|
||||
| MET-01 | kb-bilibili-metadata 整合进 kb-workflow/metadata/ | 移动 bilibili_extractor.py, bilibili_metadata.py, __init__.py |
|
||||
| MET-02 | kb-zhihu-metadata 整合进 kb-workflow/metadata/ | 移动 zhihu_extractor.py, zhihu_metadata.py, zhihu_article_crawler.py, markdown_converter.py, __init__.py |
|
||||
| MET-03 | 删除独立的 kb-bilibili-metadata 目录 | 整合验证后删除 |
|
||||
| MET-04 | 删除独立的 kb-zhihu-metadata 目录 | 整合验证后删除 |
|
||||
| MET-05 | 更新所有导入路径指向新位置 | 见下方导入映射表 |
|
||||
| PIP-01 | kb_pipeline/ 合并进 kb-workflow/ | 移动全部 20 个文件，保持子目录结构 |
|
||||
| PIP-02 | 保持 processors/, platforms/, schemas.py, state.py, config.py 结构 | 直接移动，不改结构 |
|
||||
| PIP-03 | 删除独立的 kb_pipeline 目录 | 整合验证后删除 |
|
||||
| PIP-04 | 更新 CLI 入口导入路径 | 更新 kb_workflow.py 中的导入 |
|
||||
</phase_requirements>
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| Python | 3.12+ | 运行时 | 项目要求 |
|
||||
| pytest | latest | 测试框架 | 已有测试基础设施 |
|
||||
|
||||
### Supporting
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| grep/sed | - | 批量查找替换导入 | 导入路径更新 |
|
||||
|
||||
### Alternatives Considered
|
||||
| Instead of | Could Use | Tradeoff |
|
||||
|------------|-----------|----------|
|
||||
| 手动移动文件 | git mv | git mv 保留历史，推荐使用 |
|
||||
| sed 替换 | IDE 重构 | IDE 更安全但需手动操作每个文件 |
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Target Project Structure
|
||||
```
|
||||
kb-workflow/
|
||||
├── kb_workflow.py # CLI 入口 (已存在)
|
||||
├── metadata/ # 新建
|
||||
│ ├── __init__.py
|
||||
│ ├── bilibili_extractor.py
|
||||
│ ├── bilibili_metadata.py
|
||||
│ ├── zhihu_extractor.py
|
||||
│ ├── zhihu_metadata.py
|
||||
│ ├── zhihu_article_crawler.py
|
||||
│ └── markdown_converter.py
|
||||
└── pipeline/ # 新建
|
||||
├── __init__.py
|
||||
├── pipeline.py
|
||||
├── schemas.py
|
||||
├── config.py
|
||||
├── state.py
|
||||
├── errors.py
|
||||
├── retry.py
|
||||
├── idempotency.py
|
||||
├── link_parser.py
|
||||
├── processors/
|
||||
│ ├── __init__.py
|
||||
│ ├── base.py
|
||||
│ ├── parse.py
|
||||
│ ├── download.py
|
||||
│ ├── asr.py
|
||||
│ ├── analyze.py
|
||||
│ └── archive.py
|
||||
└── platforms/
|
||||
├── __init__.py
|
||||
├── base.py
|
||||
├── bilibili.py
|
||||
└── zhihu.py
|
||||
```
|
||||
|
||||
### Pattern 1: 导入路径迁移
|
||||
**What:** 将 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx`
|
||||
**When to use:** 所有整合后的导入语句
|
||||
|
||||
**旧导入 → 新导入映射表:**
|
||||
|
||||
| 旧路径 | 新路径 |
|
||||
|--------|--------|
|
||||
| `from kb_pipeline.xxx` | `from kb_workflow.pipeline.xxx` |
|
||||
| `from bilibili_extractor import` | `from kb_workflow.metadata.bilibili_extractor import` |
|
||||
| `from zhihu_extractor import` | `from kb_workflow.metadata.zhihu_extractor import` |
|
||||
|
||||
**CLI 入口 (kb_workflow.py) 导入更新:**
|
||||
```python
|
||||
# 旧
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.pipeline import Pipeline
|
||||
from kb_pipeline.processors import (...)
|
||||
from kb_pipeline.schemas import PipelineConfig
|
||||
|
||||
# 新
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from kb_workflow.pipeline.pipeline import Pipeline
|
||||
from kb_workflow.pipeline.processors import (...)
|
||||
from kb_workflow.pipeline.schemas import PipelineConfig
|
||||
```
|
||||
|
||||
**Platform adapters 导入更新 (platforms/bilibili.py):**
|
||||
```python
|
||||
# 旧 - 动态路径导入
|
||||
_BILIBILI_METADATA_PATH = Path(__file__).parent.parent.parent / "kb-bilibili-metadata"
|
||||
from bilibili_extractor import BilibiliExtractor
|
||||
|
||||
# 新 - 标准包导入
|
||||
from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor
|
||||
```
|
||||
|
||||
**Platform adapters 导入更新 (platforms/zhihu.py):**
|
||||
```python
|
||||
# 旧 - 动态路径导入
|
||||
_kb_zhihu_path = Path(__file__).parent.parent.parent / "kb-zhihu-metadata"
|
||||
from zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
|
||||
# 新 - 标准包导入
|
||||
from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
```
|
||||
|
||||
### Pattern 2: 内部相对导入转绝对导入
|
||||
**What:** pipeline 内部模块使用绝对导入
|
||||
**When to use:** pipeline 子模块之间的相互引用
|
||||
|
||||
**示例 (processors/base.py):**
|
||||
```python
|
||||
# 旧
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.schemas import StepInput, StepOutput
|
||||
|
||||
# 新
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from kb_workflow.pipeline.schemas import StepInput, StepOutput
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
- **不要使用相对导入**: `from .xxx import yyy` 在包结构变化后容易出问题
|
||||
- **不要保留 sys.path 操作**: 删除类似 `sys.path.insert(0, ...)` 的动态路径代码
|
||||
- **不要跳过测试验证**: 每步整合后必须运行测试
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| 批量查找导入 | 手动逐文件查找 | `grep -rn "from kb_pipeline"` | 确保不遗漏 |
|
||||
| 保留 Git 历史 | cp + rm | `git mv` | 保留文件历史 |
|
||||
| 验证导入正确性 | 手动检查 | `python -c "from kb_workflow.pipeline import ..."` | 快速验证 |
|
||||
|
||||
**Key insight:** 使用 `git mv` 移动文件可以保留 Git 历史，便于后续追踪变更。
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: 循环导入
|
||||
**What goes wrong:** 整合后模块间可能产生循环依赖
|
||||
**Why it happens:** 包结构变化后，__init__.py 的导入顺序可能引发问题
|
||||
**How to avoid:**
|
||||
1. 在 __init__.py 中只导出必要的符号
|
||||
2. 使用延迟导入 (lazy import) 解决循环依赖
|
||||
**Warning signs:** `ImportError: cannot import name 'xxx' from partially initialized module`
|
||||
|
||||
### Pitfall 2: 遗漏测试文件导入
|
||||
**What goes wrong:** 只更新了源码导入，忘记更新测试文件
|
||||
**Why it happens:** 测试文件数量多，分散在不同目录
|
||||
**How to avoid:** 使用 grep 全面搜索所有 `from kb_` 导入
|
||||
**Warning signs:** `pytest` 运行时报 ImportError
|
||||
|
||||
### Pitfall 3: sys.path 动态操作
|
||||
**What goes wrong:** 旧代码中有 `sys.path.insert()` 操作指向已删除的目录
|
||||
**Why it happens:** 原独立模块使用动态路径相互引用
|
||||
**How to avoid:** 删除所有 `sys.path` 操作，改用标准包导入
|
||||
**Warning signs:** 代码中仍有 `Path(__file__).parent.parent / "kb-xxx"`
|
||||
|
||||
### Pitfall 4: __init__.py 导出不一致
|
||||
**What goes wrong:** __init__.py 中导出的符号与实际模块不匹配
|
||||
**Why it happens:** 移动文件后忘记更新 __init__.py
|
||||
**How to avoid:** 移动后立即更新 __init__.py 的 __all__ 列表
|
||||
**Warning signs:** `from kb_workflow.pipeline import xxx` 失败
|
||||
|
||||
## Code Examples
|
||||
|
||||
### 整合后的 metadata/__init__.py
|
||||
```python
|
||||
"""
|
||||
KB Workflow Metadata Module
|
||||
|
||||
提供 B站和知乎的元数据提取功能
|
||||
"""
|
||||
|
||||
from .bilibili_extractor import BilibiliExtractor
|
||||
from .zhihu_extractor import ZhihuPlaywrightExtractor
|
||||
|
||||
__all__ = [
|
||||
"BilibiliExtractor",
|
||||
"ZhihuPlaywrightExtractor",
|
||||
]
|
||||
```
|
||||
|
||||
### 整合后的 pipeline/__init__.py
|
||||
```python
|
||||
"""KB Pipeline - 工作流管道基础设施
|
||||
|
||||
提供内容处理工作流的核心 Schema 和工具。
|
||||
"""
|
||||
|
||||
# 注意：所有导入路径从 kb_pipeline.xxx 改为 kb_workflow.pipeline.xxx
|
||||
from kb_workflow.pipeline.schemas import (
|
||||
StepInput,
|
||||
StepOutput,
|
||||
StepResult,
|
||||
StepStatus,
|
||||
StepRecord,
|
||||
WorkflowState,
|
||||
StepDefinition,
|
||||
)
|
||||
from kb_workflow.pipeline.state import WorkflowStateManager
|
||||
from kb_workflow.pipeline.errors import (
|
||||
KBError,
|
||||
NetworkError,
|
||||
ValidationError,
|
||||
SystemError,
|
||||
AuthError,
|
||||
ErrorType,
|
||||
)
|
||||
from kb_workflow.pipeline.config import KBSettings, get_settings
|
||||
from kb_workflow.pipeline.retry import retry_on_network_error
|
||||
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
|
||||
from kb_workflow.pipeline.idempotency import IdempotencyChecker
|
||||
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter
|
||||
|
||||
__all__ = [
|
||||
# Schemas
|
||||
"StepInput",
|
||||
"StepOutput",
|
||||
"StepResult",
|
||||
"StepStatus",
|
||||
"StepRecord",
|
||||
"WorkflowState",
|
||||
"StepDefinition",
|
||||
# State
|
||||
"WorkflowStateManager",
|
||||
# Errors
|
||||
"KBError",
|
||||
"NetworkError",
|
||||
"ValidationError",
|
||||
"SystemError",
|
||||
"AuthError",
|
||||
"ErrorType",
|
||||
# Config
|
||||
"KBSettings",
|
||||
"get_settings",
|
||||
# Retry
|
||||
"retry_on_network_error",
|
||||
# Link Parser
|
||||
"LinkParser",
|
||||
"ParsedLink",
|
||||
# Idempotency
|
||||
"IdempotencyChecker",
|
||||
# Platform Adapters
|
||||
"PlatformAdapter",
|
||||
"BilibiliAdapter",
|
||||
"ZhihuAdapter",
|
||||
]
|
||||
|
||||
__version__ = "0.1.0"
|
||||
```
|
||||
|
||||
### 查找所有需要更新的导入
|
||||
```bash
|
||||
# 查找所有 kb_pipeline 导入
|
||||
grep -rn "from kb_pipeline" --include="*.py" /home/starmen1265/work/kb-skills/
|
||||
|
||||
# 查找所有 bilibili_extractor 导入
|
||||
grep -rn "from bilibili_extractor\|import bilibili_extractor" --include="*.py" /home/starmen1265/work/kb-skills/
|
||||
|
||||
# 查找所有 zhihu_extractor 导入
|
||||
grep -rn "from zhihu_extractor\|import zhihu_extractor" --include="*.py" /home/starmen1265/work/kb-skills/
|
||||
```
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| 独立包 + sys.path | 统一包结构 | Phase 6 | 简化导入，便于维护 |
|
||||
|
||||
**Deprecated/outdated:**
|
||||
- 动态路径导入 (`sys.path.insert`): 应删除，改用标准包导入
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **是否需要更新 pytest.ini 的 pythonpath?**
|
||||
- 当前配置 `pythonpath = .` 应该仍然有效
|
||||
- 建议：整合后验证 `pytest tests/` 是否正常工作
|
||||
|
||||
2. **kb-workflow 是否需要 setup.py 或 pyproject.toml?**
|
||||
- 当前项目似乎不使用打包
|
||||
- 建议：本阶段不涉及，保持现状
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | pytest |
|
||||
| Config file | pytest.ini (pythonpath = ., testpaths = tests) |
|
||||
| Quick run command | `pytest tests/ -v --tb=short` |
|
||||
| Full suite command | `pytest tests/ -v` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| MET-01 | bilibili_extractor 可从新位置导入 | import test | `python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor"` | N/A - Wave 0 |
|
||||
| MET-02 | zhihu_extractor 可从新位置导入 | import test | `python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor"` | N/A - Wave 0 |
|
||||
| MET-05 | 所有导入路径更新正确 | import test | `python -c "from kb_workflow.pipeline import Pipeline, get_settings"` | N/A - Wave 0 |
|
||||
| PIP-01 | pipeline 模块可从新位置导入 | unit | `pytest tests/test_pipeline/ -v` | ✅ 已存在 |
|
||||
| PIP-02 | processors 功能正常 | unit | `pytest tests/test_processors/ -v` | ✅ 已存在 |
|
||||
| PIP-04 | CLI 入口可用 | integration | `python kb-workflow/kb_workflow.py --help` | N/A - Wave 0 |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `pytest tests/ -v --tb=short -x` (失败即停)
|
||||
- **Per wave merge:** `pytest tests/ -v`
|
||||
- **Phase gate:** Full suite green before `/gsd:verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `tests/test_imports.py` — 验证所有新导入路径可用的测试
|
||||
- [ ] 导入验证脚本 — 快速检查所有模块可导入
|
||||
|
||||
**建议 Wave 0 任务：**
|
||||
1. 创建 `tests/test_imports.py` 测试文件
|
||||
2. 添加以下测试用例：
|
||||
- `test_import_metadata_bilibili()`: 验证 `from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor`
|
||||
- `test_import_metadata_zhihu()`: 验证 `from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor`
|
||||
- `test_import_pipeline_core()`: 验证 `from kb_workflow.pipeline import Pipeline, get_settings`
|
||||
- `test_import_pipeline_processors()`: 验证所有 processors 可导入
|
||||
- `test_import_pipeline_platforms()`: 验证所有 platforms 可导入
|
||||
|
||||
### 整合验证命令
|
||||
|
||||
**验证 metadata 整合:**
|
||||
```bash
|
||||
# 1. 验证模块可导入
|
||||
python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor; print('bilibili OK')"
|
||||
python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor; print('zhihu OK')"
|
||||
|
||||
# 2. 运行相关测试
|
||||
pytest tests/test_platforms/test_bilibili.py tests/test_platforms/test_zhihu.py -v
|
||||
```
|
||||
|
||||
**验证 pipeline 整合:**
|
||||
```bash
|
||||
# 1. 验证模块可导入
|
||||
python -c "from kb_workflow.pipeline import Pipeline; print('pipeline OK')"
|
||||
python -c "from kb_workflow.pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor, ArchiveProcessor; print('processors OK')"
|
||||
|
||||
# 2. 运行全量测试
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
**验证 CLI 可用:**
|
||||
```bash
|
||||
cd kb-workflow && python kb_workflow.py --help
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- 项目源码分析 - 直接阅读 kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/, kb-workflow/
|
||||
- CONTEXT.md - 用户决策和约束
|
||||
- pytest.ini - 测试配置
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- None required - 本阶段为纯代码迁移，无需外部资料
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
- None
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH - 项目已使用 Python 3.12+ 和 pytest
|
||||
- Architecture: HIGH - 目标结构已在 CONTEXT.md 中明确
|
||||
- Pitfalls: HIGH - 基于对现有代码的完整分析
|
||||
|
||||
**Research date:** 2026-03-16
|
||||
**Valid until:** 30 days (稳定的 Python 迁移模式)
|
||||
Reference in New Issue
Block a user