diff --git a/.planning/phases/06-code-consolidation/06-RESEARCH.md b/.planning/phases/06-code-consolidation/06-RESEARCH.md new file mode 100644 index 0000000..c433617 --- /dev/null +++ b/.planning/phases/06-code-consolidation/06-RESEARCH.md @@ -0,0 +1,448 @@ +# Phase 6: 代码整合 - Research + +**Researched:** 2026-03-16 +**Domain:** Python 代码重构、模块整合、导入路径迁移 +**Confidence:** HIGH + +## Summary + +本阶段将三个独立的 Python 模块整合进 kb-workflow 包: +1. `kb-bilibili-metadata/` (3 files) - B站元数据提取 +2. `kb-zhihu-metadata/` (5 files) - 知乎内容提取 +3. `kb_pipeline/` (20 files) - 工作流管道核心 + +整合后的目录结构将使所有代码统一在 `kb-workflow` 包下,便于维护和分发。主要工作是文件移动和导入路径更新,不涉及功能改动。 + +**Primary recommendation:** 按照先 metadata 后 pipeline 的顺序,分步整合,每步完成后运行测试验证。 + + +## User Constraints (from CONTEXT.md) + +### 目录结构 +- **完全扁平化**: kb-workflow/metadata/ 和 kb-workflow/pipeline/ 作为顶级子目录 +- **元数据目录**: kb-workflow/metadata/bilibili_extractor.py, zhihu_extractor.py 等 +- **Pipeline 目录**: kb-workflow/pipeline/processors/, kb-workflow/pipeline/platforms/ 等 +- **保留原文件名**: bilibili_extractor.py 不重命名为 bilibili.py,保持迁移安全 + +### 导入策略 +- **外部导入**: 统一从 kb_workflow 导入 + - `from kb_workflow.metadata.bilibili_extractor import ...` + - `from kb_workflow.pipeline.processors import ...` +- **内部导入**: 使用绝对导入 + - `from kb_workflow.pipeline.processors.base import BaseProcessor` +- **更新范围**: 所有测试文件、CLI 入口、处理器间引用 + +### 整合顺序 +- **先 metadata 后 pipeline**: + 1. 移动 kb-bilibili-metadata → kb-workflow/metadata/ + 2. 移动 kb-zhihu-metadata → kb-workflow/metadata/ + 3. 更新 metadata 相关导入 + 4. 运行测试验证 + 5. 移动 kb_pipeline → kb-workflow/pipeline/ + 6. 更新 pipeline 相关导入 + 7. 运行测试验证 + +### 验证策略 +- **分步验证**: 每个子整合完成后运行测试 +- **测试命令**: `pytest tests/ -v` +- **失败处理**: 测试失败时不继续下一步,先修复问题 + +### 清理策略 +- **立即删除**: 整合完成且测试通过后立即删除原目录 +- **删除目录**: kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/ +- **Git 提交**: 每个子整合单独提交,便于回滚 + +### Claude's Discretion +- 具体导入路径更新时的细节调整 +- 处理意外的循环导入问题 +- 测试文件的导入更新方式 + +### Deferred Ideas (OUT OF SCOPE) +None — discussion stayed within phase scope + + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| MET-01 | kb-bilibili-metadata 整合进 kb-workflow/metadata/ | 移动 bilibili_extractor.py, bilibili_metadata.py, __init__.py | +| MET-02 | kb-zhihu-metadata 整合进 kb-workflow/metadata/ | 移动 zhihu_extractor.py, zhihu_metadata.py, zhihu_article_crawler.py, markdown_converter.py, __init__.py | +| MET-03 | 删除独立的 kb-bilibili-metadata 目录 | 整合验证后删除 | +| MET-04 | 删除独立的 kb-zhihu-metadata 目录 | 整合验证后删除 | +| MET-05 | 更新所有导入路径指向新位置 | 见下方导入映射表 | +| PIP-01 | kb_pipeline/ 合并进 kb-workflow/ | 移动全部 20 个文件,保持子目录结构 | +| PIP-02 | 保持 processors/, platforms/, schemas.py, state.py, config.py 结构 | 直接移动,不改结构 | +| PIP-03 | 删除独立的 kb_pipeline 目录 | 整合验证后删除 | +| PIP-04 | 更新 CLI 入口导入路径 | 更新 kb_workflow.py 中的导入 | + + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| Python | 3.12+ | 运行时 | 项目要求 | +| pytest | latest | 测试框架 | 已有测试基础设施 | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| grep/sed | - | 批量查找替换导入 | 导入路径更新 | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| 手动移动文件 | git mv | git mv 保留历史,推荐使用 | +| sed 替换 | IDE 重构 | IDE 更安全但需手动操作每个文件 | + +## Architecture Patterns + +### Target Project Structure +``` +kb-workflow/ +├── kb_workflow.py # CLI 入口 (已存在) +├── metadata/ # 新建 +│ ├── __init__.py +│ ├── bilibili_extractor.py +│ ├── bilibili_metadata.py +│ ├── zhihu_extractor.py +│ ├── zhihu_metadata.py +│ ├── zhihu_article_crawler.py +│ └── markdown_converter.py +└── pipeline/ # 新建 + ├── __init__.py + ├── pipeline.py + ├── schemas.py + ├── config.py + ├── state.py + ├── errors.py + ├── retry.py + ├── idempotency.py + ├── link_parser.py + ├── processors/ + │ ├── __init__.py + │ ├── base.py + │ ├── parse.py + │ ├── download.py + │ ├── asr.py + │ ├── analyze.py + │ └── archive.py + └── platforms/ + ├── __init__.py + ├── base.py + ├── bilibili.py + └── zhihu.py +``` + +### Pattern 1: 导入路径迁移 +**What:** 将 `from kb_pipeline.xxx` 改为 `from kb_workflow.pipeline.xxx` +**When to use:** 所有整合后的导入语句 + +**旧导入 → 新导入映射表:** + +| 旧路径 | 新路径 | +|--------|--------| +| `from kb_pipeline.xxx` | `from kb_workflow.pipeline.xxx` | +| `from bilibili_extractor import` | `from kb_workflow.metadata.bilibili_extractor import` | +| `from zhihu_extractor import` | `from kb_workflow.metadata.zhihu_extractor import` | + +**CLI 入口 (kb_workflow.py) 导入更新:** +```python +# 旧 +from kb_pipeline.config import get_settings +from kb_pipeline.pipeline import Pipeline +from kb_pipeline.processors import (...) +from kb_pipeline.schemas import PipelineConfig + +# 新 +from kb_workflow.pipeline.config import get_settings +from kb_workflow.pipeline.pipeline import Pipeline +from kb_workflow.pipeline.processors import (...) +from kb_workflow.pipeline.schemas import PipelineConfig +``` + +**Platform adapters 导入更新 (platforms/bilibili.py):** +```python +# 旧 - 动态路径导入 +_BILIBILI_METADATA_PATH = Path(__file__).parent.parent.parent / "kb-bilibili-metadata" +from bilibili_extractor import BilibiliExtractor + +# 新 - 标准包导入 +from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor +``` + +**Platform adapters 导入更新 (platforms/zhihu.py):** +```python +# 旧 - 动态路径导入 +_kb_zhihu_path = Path(__file__).parent.parent.parent / "kb-zhihu-metadata" +from zhihu_extractor import ZhihuPlaywrightExtractor + +# 新 - 标准包导入 +from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor +``` + +### Pattern 2: 内部相对导入转绝对导入 +**What:** pipeline 内部模块使用绝对导入 +**When to use:** pipeline 子模块之间的相互引用 + +**示例 (processors/base.py):** +```python +# 旧 +from kb_pipeline.config import get_settings +from kb_pipeline.schemas import StepInput, StepOutput + +# 新 +from kb_workflow.pipeline.config import get_settings +from kb_workflow.pipeline.schemas import StepInput, StepOutput +``` + +### Anti-Patterns to Avoid +- **不要使用相对导入**: `from .xxx import yyy` 在包结构变化后容易出问题 +- **不要保留 sys.path 操作**: 删除类似 `sys.path.insert(0, ...)` 的动态路径代码 +- **不要跳过测试验证**: 每步整合后必须运行测试 + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| 批量查找导入 | 手动逐文件查找 | `grep -rn "from kb_pipeline"` | 确保不遗漏 | +| 保留 Git 历史 | cp + rm | `git mv` | 保留文件历史 | +| 验证导入正确性 | 手动检查 | `python -c "from kb_workflow.pipeline import ..."` | 快速验证 | + +**Key insight:** 使用 `git mv` 移动文件可以保留 Git 历史,便于后续追踪变更。 + +## Common Pitfalls + +### Pitfall 1: 循环导入 +**What goes wrong:** 整合后模块间可能产生循环依赖 +**Why it happens:** 包结构变化后,__init__.py 的导入顺序可能引发问题 +**How to avoid:** +1. 在 __init__.py 中只导出必要的符号 +2. 使用延迟导入 (lazy import) 解决循环依赖 +**Warning signs:** `ImportError: cannot import name 'xxx' from partially initialized module` + +### Pitfall 2: 遗漏测试文件导入 +**What goes wrong:** 只更新了源码导入,忘记更新测试文件 +**Why it happens:** 测试文件数量多,分散在不同目录 +**How to avoid:** 使用 grep 全面搜索所有 `from kb_` 导入 +**Warning signs:** `pytest` 运行时报 ImportError + +### Pitfall 3: sys.path 动态操作 +**What goes wrong:** 旧代码中有 `sys.path.insert()` 操作指向已删除的目录 +**Why it happens:** 原独立模块使用动态路径相互引用 +**How to avoid:** 删除所有 `sys.path` 操作,改用标准包导入 +**Warning signs:** 代码中仍有 `Path(__file__).parent.parent / "kb-xxx"` + +### Pitfall 4: __init__.py 导出不一致 +**What goes wrong:** __init__.py 中导出的符号与实际模块不匹配 +**Why it happens:** 移动文件后忘记更新 __init__.py +**How to avoid:** 移动后立即更新 __init__.py 的 __all__ 列表 +**Warning signs:** `from kb_workflow.pipeline import xxx` 失败 + +## Code Examples + +### 整合后的 metadata/__init__.py +```python +""" +KB Workflow Metadata Module + +提供 B站和知乎的元数据提取功能 +""" + +from .bilibili_extractor import BilibiliExtractor +from .zhihu_extractor import ZhihuPlaywrightExtractor + +__all__ = [ + "BilibiliExtractor", + "ZhihuPlaywrightExtractor", +] +``` + +### 整合后的 pipeline/__init__.py +```python +"""KB Pipeline - 工作流管道基础设施 + +提供内容处理工作流的核心 Schema 和工具。 +""" + +# 注意:所有导入路径从 kb_pipeline.xxx 改为 kb_workflow.pipeline.xxx +from kb_workflow.pipeline.schemas import ( + StepInput, + StepOutput, + StepResult, + StepStatus, + StepRecord, + WorkflowState, + StepDefinition, +) +from kb_workflow.pipeline.state import WorkflowStateManager +from kb_workflow.pipeline.errors import ( + KBError, + NetworkError, + ValidationError, + SystemError, + AuthError, + ErrorType, +) +from kb_workflow.pipeline.config import KBSettings, get_settings +from kb_workflow.pipeline.retry import retry_on_network_error +from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink +from kb_workflow.pipeline.idempotency import IdempotencyChecker +from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter + +__all__ = [ + # Schemas + "StepInput", + "StepOutput", + "StepResult", + "StepStatus", + "StepRecord", + "WorkflowState", + "StepDefinition", + # State + "WorkflowStateManager", + # Errors + "KBError", + "NetworkError", + "ValidationError", + "SystemError", + "AuthError", + "ErrorType", + # Config + "KBSettings", + "get_settings", + # Retry + "retry_on_network_error", + # Link Parser + "LinkParser", + "ParsedLink", + # Idempotency + "IdempotencyChecker", + # Platform Adapters + "PlatformAdapter", + "BilibiliAdapter", + "ZhihuAdapter", +] + +__version__ = "0.1.0" +``` + +### 查找所有需要更新的导入 +```bash +# 查找所有 kb_pipeline 导入 +grep -rn "from kb_pipeline" --include="*.py" /home/starmen1265/work/kb-skills/ + +# 查找所有 bilibili_extractor 导入 +grep -rn "from bilibili_extractor\|import bilibili_extractor" --include="*.py" /home/starmen1265/work/kb-skills/ + +# 查找所有 zhihu_extractor 导入 +grep -rn "from zhihu_extractor\|import zhihu_extractor" --include="*.py" /home/starmen1265/work/kb-skills/ +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| 独立包 + sys.path | 统一包结构 | Phase 6 | 简化导入,便于维护 | + +**Deprecated/outdated:** +- 动态路径导入 (`sys.path.insert`): 应删除,改用标准包导入 + +## Open Questions + +1. **是否需要更新 pytest.ini 的 pythonpath?** + - 当前配置 `pythonpath = .` 应该仍然有效 + - 建议:整合后验证 `pytest tests/` 是否正常工作 + +2. **kb-workflow 是否需要 setup.py 或 pyproject.toml?** + - 当前项目似乎不使用打包 + - 建议:本阶段不涉及,保持现状 + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | pytest | +| Config file | pytest.ini (pythonpath = ., testpaths = tests) | +| Quick run command | `pytest tests/ -v --tb=short` | +| Full suite command | `pytest tests/ -v` | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| MET-01 | bilibili_extractor 可从新位置导入 | import test | `python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor"` | N/A - Wave 0 | +| MET-02 | zhihu_extractor 可从新位置导入 | import test | `python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor"` | N/A - Wave 0 | +| MET-05 | 所有导入路径更新正确 | import test | `python -c "from kb_workflow.pipeline import Pipeline, get_settings"` | N/A - Wave 0 | +| PIP-01 | pipeline 模块可从新位置导入 | unit | `pytest tests/test_pipeline/ -v` | ✅ 已存在 | +| PIP-02 | processors 功能正常 | unit | `pytest tests/test_processors/ -v` | ✅ 已存在 | +| PIP-04 | CLI 入口可用 | integration | `python kb-workflow/kb_workflow.py --help` | N/A - Wave 0 | + +### Sampling Rate +- **Per task commit:** `pytest tests/ -v --tb=short -x` (失败即停) +- **Per wave merge:** `pytest tests/ -v` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `tests/test_imports.py` — 验证所有新导入路径可用的测试 +- [ ] 导入验证脚本 — 快速检查所有模块可导入 + +**建议 Wave 0 任务:** +1. 创建 `tests/test_imports.py` 测试文件 +2. 添加以下测试用例: + - `test_import_metadata_bilibili()`: 验证 `from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor` + - `test_import_metadata_zhihu()`: 验证 `from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor` + - `test_import_pipeline_core()`: 验证 `from kb_workflow.pipeline import Pipeline, get_settings` + - `test_import_pipeline_processors()`: 验证所有 processors 可导入 + - `test_import_pipeline_platforms()`: 验证所有 platforms 可导入 + +### 整合验证命令 + +**验证 metadata 整合:** +```bash +# 1. 验证模块可导入 +python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor; print('bilibili OK')" +python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor; print('zhihu OK')" + +# 2. 运行相关测试 +pytest tests/test_platforms/test_bilibili.py tests/test_platforms/test_zhihu.py -v +``` + +**验证 pipeline 整合:** +```bash +# 1. 验证模块可导入 +python -c "from kb_workflow.pipeline import Pipeline; print('pipeline OK')" +python -c "from kb_workflow.pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor, ArchiveProcessor; print('processors OK')" + +# 2. 运行全量测试 +pytest tests/ -v +``` + +**验证 CLI 可用:** +```bash +cd kb-workflow && python kb_workflow.py --help +``` + +## Sources + +### Primary (HIGH confidence) +- 项目源码分析 - 直接阅读 kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/, kb-workflow/ +- CONTEXT.md - 用户决策和约束 +- pytest.ini - 测试配置 + +### Secondary (MEDIUM confidence) +- None required - 本阶段为纯代码迁移,无需外部资料 + +### Tertiary (LOW confidence) +- None + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - 项目已使用 Python 3.12+ 和 pytest +- Architecture: HIGH - 目标结构已在 CONTEXT.md 中明确 +- Pitfalls: HIGH - 基于对现有代码的完整分析 + +**Research date:** 2026-03-16 +**Valid until:** 30 days (稳定的 Python 迁移模式)