starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
33acdf0c2e9f3f0d464774ece5a4345746674595
kb-skills/.planning/phases/06-code-consolidation/06-RESEARCH.md
T

16 KiB
Raw Blame History

Phase 6: 代码整合 - Research

Researched: 2026-03-16 Domain: Python 代码重构、模块整合、导入路径迁移 Confidence: HIGH

Summary

本阶段将三个独立的 Python 模块整合进 kb-workflow 包：

  1. kb-bilibili-metadata/ (3 files) - B站元数据提取
  2. kb-zhihu-metadata/ (5 files) - 知乎内容提取
  3. kb_pipeline/ (20 files) - 工作流管道核心

整合后的目录结构将使所有代码统一在 kb-workflow 包下，便于维护和分发。主要工作是文件移动和导入路径更新，不涉及功能改动。

Primary recommendation: 按照先 metadata 后 pipeline 的顺序，分步整合，每步完成后运行测试验证。

<user_constraints>

User Constraints (from CONTEXT.md)

目录结构

  • 完全扁平化: kb-workflow/metadata/ 和 kb-workflow/pipeline/ 作为顶级子目录
  • 元数据目录: kb-workflow/metadata/bilibili_extractor.py, zhihu_extractor.py 等
  • Pipeline 目录: kb-workflow/pipeline/processors/, kb-workflow/pipeline/platforms/ 等
  • 保留原文件名: bilibili_extractor.py 不重命名为 bilibili.py，保持迁移安全

导入策略

  • 外部导入: 统一从 kb_workflow 导入
    • from kb_workflow.metadata.bilibili_extractor import ...
    • from kb_workflow.pipeline.processors import ...
  • 内部导入: 使用绝对导入
    • from kb_workflow.pipeline.processors.base import BaseProcessor
  • 更新范围: 所有测试文件、CLI 入口、处理器间引用

整合顺序

  • 先 metadata 后 pipeline:
    1. 移动 kb-bilibili-metadata → kb-workflow/metadata/
    2. 移动 kb-zhihu-metadata → kb-workflow/metadata/
    3. 更新 metadata 相关导入
    4. 运行测试验证
    5. 移动 kb_pipeline → kb-workflow/pipeline/
    6. 更新 pipeline 相关导入
    7. 运行测试验证

验证策略

  • 分步验证: 每个子整合完成后运行测试
  • 测试命令: pytest tests/ -v
  • 失败处理: 测试失败时不继续下一步，先修复问题

清理策略

  • 立即删除: 整合完成且测试通过后立即删除原目录
  • 删除目录: kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/
  • Git 提交: 每个子整合单独提交，便于回滚

Claude's Discretion

  • 具体导入路径更新时的细节调整
  • 处理意外的循环导入问题
  • 测试文件的导入更新方式

Deferred Ideas (OUT OF SCOPE)

None — discussion stayed within phase scope </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
MET-01 kb-bilibili-metadata 整合进 kb-workflow/metadata/ 移动 bilibili_extractor.py, bilibili_metadata.py, __init__.py
MET-02 kb-zhihu-metadata 整合进 kb-workflow/metadata/ 移动 zhihu_extractor.py, zhihu_metadata.py, zhihu_article_crawler.py, markdown_converter.py, __init__.py
MET-03 删除独立的 kb-bilibili-metadata 目录 整合验证后删除
MET-04 删除独立的 kb-zhihu-metadata 目录 整合验证后删除
MET-05 更新所有导入路径指向新位置 见下方导入映射表
PIP-01 kb_pipeline/ 合并进 kb-workflow/ 移动全部 20 个文件，保持子目录结构
PIP-02 保持 processors/, platforms/, schemas.py, state.py, config.py 结构 直接移动，不改结构
PIP-03 删除独立的 kb_pipeline 目录 整合验证后删除
PIP-04 更新 CLI 入口导入路径 更新 kb_workflow.py 中的导入
</phase_requirements>

Standard Stack

Core

Library Version Purpose Why Standard
Python 3.12+ 运行时 项目要求
pytest latest 测试框架 已有测试基础设施

Supporting

Library Version Purpose When to Use
grep/sed - 批量查找替换导入 导入路径更新

Alternatives Considered

Instead of Could Use Tradeoff
手动移动文件 git mv git mv 保留历史，推荐使用
sed 替换 IDE 重构 IDE 更安全但需手动操作每个文件

Architecture Patterns

Target Project Structure

kb-workflow/
├── kb_workflow.py        # CLI 入口 (已存在)
├── metadata/             # 新建
│   ├── __init__.py
│   ├── bilibili_extractor.py
│   ├── bilibili_metadata.py
│   ├── zhihu_extractor.py
│   ├── zhihu_metadata.py
│   ├── zhihu_article_crawler.py
│   └── markdown_converter.py
└── pipeline/             # 新建
    ├── __init__.py
    ├── pipeline.py
    ├── schemas.py
    ├── config.py
    ├── state.py
    ├── errors.py
    ├── retry.py
    ├── idempotency.py
    ├── link_parser.py
    ├── processors/
    │   ├── __init__.py
    │   ├── base.py
    │   ├── parse.py
    │   ├── download.py
    │   ├── asr.py
    │   ├── analyze.py
    │   └── archive.py
    └── platforms/
        ├── __init__.py
        ├── base.py
        ├── bilibili.py
        └── zhihu.py

Pattern 1: 导入路径迁移

What: 将 from kb_pipeline.xxx 改为 from kb_workflow.pipeline.xxx When to use: 所有整合后的导入语句

旧导入 → 新导入映射表:

旧路径 新路径
from kb_pipeline.xxx from kb_workflow.pipeline.xxx
from bilibili_extractor import from kb_workflow.metadata.bilibili_extractor import
from zhihu_extractor import from kb_workflow.metadata.zhihu_extractor import

CLI 入口 (kb_workflow.py) 导入更新:

# 旧
from kb_pipeline.config import get_settings
from kb_pipeline.pipeline import Pipeline
from kb_pipeline.processors import (...)
from kb_pipeline.schemas import PipelineConfig

# 新
from kb_workflow.pipeline.config import get_settings
from kb_workflow.pipeline.pipeline import Pipeline
from kb_workflow.pipeline.processors import (...)
from kb_workflow.pipeline.schemas import PipelineConfig

Platform adapters 导入更新 (platforms/bilibili.py):

# 旧 - 动态路径导入
_BILIBILI_METADATA_PATH = Path(__file__).parent.parent.parent / "kb-bilibili-metadata"
from bilibili_extractor import BilibiliExtractor

# 新 - 标准包导入
from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor

Platform adapters 导入更新 (platforms/zhihu.py):

# 旧 - 动态路径导入
_kb_zhihu_path = Path(__file__).parent.parent.parent / "kb-zhihu-metadata"
from zhihu_extractor import ZhihuPlaywrightExtractor

# 新 - 标准包导入
from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor

Pattern 2: 内部相对导入转绝对导入

What: pipeline 内部模块使用绝对导入 When to use: pipeline 子模块之间的相互引用

示例 (processors/base.py):

# 旧
from kb_pipeline.config import get_settings
from kb_pipeline.schemas import StepInput, StepOutput

# 新
from kb_workflow.pipeline.config import get_settings
from kb_workflow.pipeline.schemas import StepInput, StepOutput

Anti-Patterns to Avoid

  • 不要使用相对导入: from .xxx import yyy 在包结构变化后容易出问题
  • 不要保留 sys.path 操作: 删除类似 sys.path.insert(0, ...) 的动态路径代码
  • 不要跳过测试验证: 每步整合后必须运行测试

Don't Hand-Roll

Problem Don't Build Use Instead Why
批量查找导入 手动逐文件查找 grep -rn "from kb_pipeline" 确保不遗漏
保留 Git 历史 cp + rm git mv 保留文件历史
验证导入正确性 手动检查 python -c "from kb_workflow.pipeline import ..." 快速验证

Key insight: 使用 git mv 移动文件可以保留 Git 历史，便于后续追踪变更。

Common Pitfalls

Pitfall 1: 循环导入

What goes wrong: 整合后模块间可能产生循环依赖 Why it happens: 包结构变化后，__init__.py 的导入顺序可能引发问题 How to avoid:

  1. 在 __init__.py 中只导出必要的符号
  2. 使用延迟导入 (lazy import) 解决循环依赖 Warning signs: ImportError: cannot import name 'xxx' from partially initialized module

Pitfall 2: 遗漏测试文件导入

What goes wrong: 只更新了源码导入，忘记更新测试文件 Why it happens: 测试文件数量多，分散在不同目录 How to avoid: 使用 grep 全面搜索所有 from kb_ 导入 Warning signs: pytest 运行时报 ImportError

Pitfall 3: sys.path 动态操作

What goes wrong: 旧代码中有 sys.path.insert() 操作指向已删除的目录 Why it happens: 原独立模块使用动态路径相互引用 How to avoid: 删除所有 sys.path 操作，改用标准包导入 Warning signs: 代码中仍有 Path(__file__).parent.parent / "kb-xxx"

Pitfall 4: __init__.py 导出不一致

What goes wrong: __init__.py 中导出的符号与实际模块不匹配 Why it happens: 移动文件后忘记更新 __init__.py How to avoid: 移动后立即更新 __init__.py 的 all 列表 Warning signs: from kb_workflow.pipeline import xxx 失败

Code Examples

整合后的 metadata/__init__.py

"""
KB Workflow Metadata Module

提供 B站和知乎的元数据提取功能
"""

from .bilibili_extractor import BilibiliExtractor
from .zhihu_extractor import ZhihuPlaywrightExtractor

__all__ = [
    "BilibiliExtractor",
    "ZhihuPlaywrightExtractor",
]

整合后的 pipeline/__init__.py

"""KB Pipeline - 工作流管道基础设施

提供内容处理工作流的核心 Schema 和工具。
"""

# 注意：所有导入路径从 kb_pipeline.xxx 改为 kb_workflow.pipeline.xxx
from kb_workflow.pipeline.schemas import (
    StepInput,
    StepOutput,
    StepResult,
    StepStatus,
    StepRecord,
    WorkflowState,
    StepDefinition,
)
from kb_workflow.pipeline.state import WorkflowStateManager
from kb_workflow.pipeline.errors import (
    KBError,
    NetworkError,
    ValidationError,
    SystemError,
    AuthError,
    ErrorType,
)
from kb_workflow.pipeline.config import KBSettings, get_settings
from kb_workflow.pipeline.retry import retry_on_network_error
from kb_workflow.pipeline.link_parser import LinkParser, ParsedLink
from kb_workflow.pipeline.idempotency import IdempotencyChecker
from kb_workflow.pipeline.platforms import PlatformAdapter, BilibiliAdapter, ZhihuAdapter

__all__ = [
    # Schemas
    "StepInput",
    "StepOutput",
    "StepResult",
    "StepStatus",
    "StepRecord",
    "WorkflowState",
    "StepDefinition",
    # State
    "WorkflowStateManager",
    # Errors
    "KBError",
    "NetworkError",
    "ValidationError",
    "SystemError",
    "AuthError",
    "ErrorType",
    # Config
    "KBSettings",
    "get_settings",
    # Retry
    "retry_on_network_error",
    # Link Parser
    "LinkParser",
    "ParsedLink",
    # Idempotency
    "IdempotencyChecker",
    # Platform Adapters
    "PlatformAdapter",
    "BilibiliAdapter",
    "ZhihuAdapter",
]

__version__ = "0.1.0"

查找所有需要更新的导入

# 查找所有 kb_pipeline 导入
grep -rn "from kb_pipeline" --include="*.py" /home/starmen1265/work/kb-skills/

# 查找所有 bilibili_extractor 导入
grep -rn "from bilibili_extractor\|import bilibili_extractor" --include="*.py" /home/starmen1265/work/kb-skills/

# 查找所有 zhihu_extractor 导入
grep -rn "from zhihu_extractor\|import zhihu_extractor" --include="*.py" /home/starmen1265/work/kb-skills/

State of the Art

Old Approach Current Approach When Changed Impact
独立包 + sys.path 统一包结构 Phase 6 简化导入，便于维护

Deprecated/outdated:

  • 动态路径导入 (sys.path.insert): 应删除，改用标准包导入

Open Questions

  1. 是否需要更新 pytest.ini 的 pythonpath?

    • 当前配置 pythonpath = . 应该仍然有效
    • 建议：整合后验证 pytest tests/ 是否正常工作

  2. kb-workflow 是否需要 setup.py 或 pyproject.toml?

    • 当前项目似乎不使用打包
    • 建议：本阶段不涉及，保持现状

Validation Architecture

Test Framework

Property Value
Framework pytest
Config file pytest.ini (pythonpath = ., testpaths = tests)
Quick run command pytest tests/ -v --tb=short
Full suite command pytest tests/ -v

Phase Requirements → Test Map

Req ID Behavior Test Type Automated Command File Exists?
MET-01 bilibili_extractor 可从新位置导入 import test python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor" N/A - Wave 0
MET-02 zhihu_extractor 可从新位置导入 import test python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor" N/A - Wave 0
MET-05 所有导入路径更新正确 import test python -c "from kb_workflow.pipeline import Pipeline, get_settings" N/A - Wave 0
PIP-01 pipeline 模块可从新位置导入 unit pytest tests/test_pipeline/ -v ✅ 已存在
PIP-02 processors 功能正常 unit pytest tests/test_processors/ -v ✅ 已存在
PIP-04 CLI 入口可用 integration python kb-workflow/kb_workflow.py --help N/A - Wave 0

Sampling Rate

  • Per task commit: pytest tests/ -v --tb=short -x (失败即停)
  • Per wave merge: pytest tests/ -v
  • Phase gate: Full suite green before /gsd:verify-work

Wave 0 Gaps

  • tests/test_imports.py — 验证所有新导入路径可用的测试
  • 导入验证脚本 — 快速检查所有模块可导入

建议 Wave 0 任务：

  1. 创建 tests/test_imports.py 测试文件
  2. 添加以下测试用例：
    • test_import_metadata_bilibili(): 验证 from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor
    • test_import_metadata_zhihu(): 验证 from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor
    • test_import_pipeline_core(): 验证 from kb_workflow.pipeline import Pipeline, get_settings
    • test_import_pipeline_processors(): 验证所有 processors 可导入
    • test_import_pipeline_platforms(): 验证所有 platforms 可导入

整合验证命令

验证 metadata 整合:

# 1. 验证模块可导入
python -c "from kb_workflow.metadata.bilibili_extractor import BilibiliExtractor; print('bilibili OK')"
python -c "from kb_workflow.metadata.zhihu_extractor import ZhihuPlaywrightExtractor; print('zhihu OK')"

# 2. 运行相关测试
pytest tests/test_platforms/test_bilibili.py tests/test_platforms/test_zhihu.py -v

验证 pipeline 整合:

# 1. 验证模块可导入
python -c "from kb_workflow.pipeline import Pipeline; print('pipeline OK')"
python -c "from kb_workflow.pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor, ArchiveProcessor; print('processors OK')"

# 2. 运行全量测试
pytest tests/ -v

验证 CLI 可用:

cd kb-workflow && python kb_workflow.py --help

Sources

Primary (HIGH confidence)

  • 项目源码分析 - 直接阅读 kb-bilibili-metadata/, kb-zhihu-metadata/, kb_pipeline/, kb-workflow/
  • CONTEXT.md - 用户决策和约束
  • pytest.ini - 测试配置

Secondary (MEDIUM confidence)

  • None required - 本阶段为纯代码迁移，无需外部资料

Tertiary (LOW confidence)

  • None

Metadata

Confidence breakdown:

  • Standard stack: HIGH - 项目已使用 Python 3.12+ 和 pytest
  • Architecture: HIGH - 目标结构已在 CONTEXT.md 中明确
  • Pitfalls: HIGH - 基于对现有代码的完整分析

Research date: 2026-03-16 Valid until: 30 days (稳定的 Python 迁移模式)

Reference in New Issue View Git Blame Copy Permalink