diff --git a/.planning/phases/15-output-compat/15-RESEARCH.md b/.planning/phases/15-output-compat/15-RESEARCH.md new file mode 100644 index 0000000..52b4442 --- /dev/null +++ b/.planning/phases/15-output-compat/15-RESEARCH.md @@ -0,0 +1,365 @@ +# Phase 15: 输出优化与兼容性 - Research + +**研究时间:** 2026-03-23 +**技术领域:** 配置管理 (pydantic-settings)、输出格式验证、向后兼容性 +**置信度:** HIGH + +## Summary + +Phase 15 专注于配置开关集成和向后兼容性验证。核心任务是将 `enable_semantic_segmentation` 添加到 KBSettings,并确保配置变更不影响现有功能。研究发现 Phase 13/14 已实现所需的输出格式(`_generate_markdown()` 支持标题和纯文本两种模式),本阶段主要是配置集成和验证工作。 + +**主要建议:** 使用 pydantic-settings 的标准模式添加布尔配置字段,通过环境变量 `KB_ENABLE_SEMANTIC_SEGMENTATION` 控制,默认值为 `True` 以提供开箱即用的语义分段体验。 + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +- **配置开关位置**: 添加到 KBSettings — 将 `enable_semantic_segmentation` 添加到 `kb_workflow/pipeline/config.py` 中的 `KBSettings` 类 +- **配置方式**: 用户可在 `~/.kb_env` 文件中设置 `KB_ENABLE_SEMANTIC_SEGMENTATION=true` +- **环境变量命名**: `KB_ENABLE_SEMANTIC_SEGMENTATION` — 使用全名,清晰明确 +- **风格一致**: 与现有配置变量命名保持一致(KB_ 前缀,全大写下划线分隔) +- **默认行为**: 默认启用 — 当用户未配置时,默认值为 `True`(启用语义分段) +- **API Key 检查保留**: 即使配置启用,如果 AI API key 未配置,代码仍会自动降级到规则分段 + +### Claude's Discretion +- KBSettings 中字段的具体类型定义(bool Field with default) +- 是否需要添加 enable_cache 相关的配置选项到 KBSettings +- 配置变更后的日志记录级别 + +### Deferred Ideas (OUT OF SCOPE) +- 缓存配置选项 — 是否需要将 enable_cache 也添加到 KBSettings(当前只有构造函数参数) +- 分段粒度配置 — 未来可能允许用户配置目标分段数量(当前硬编码为 5-8) +- 环境变量简化 — 是否需要更短的别名(如 KB_LLMSeg)作为备选 + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| OUTPUT-01 | 主题/章节分段输出带标题的 Markdown 结构 | 已在 Phase 13 实现,`_generate_markdown()` 支持 VideoSegment.title → H2 标题 | +| OUTPUT-02 | 论点/观点分段输出纯文本,保持原有风格 | 已实现,title=null 时不输出标题,仅输出 content | +| OUTPUT-03 | 输出与现有 SRTProcessor 兼容,不影响其他平台处理 | 已验证,规则分段继续使用 "## 内容" 标题,AnalyzeProcessor 独立不受影响 | +| CONFIG-01 | LLM 分段功能可通过配置启用/禁用 | 研究提供 pydantic-settings 集成方案 | +| CONFIG-02 | 向后兼容,不破坏现有简单规则分段 | 已验证,enable_semantic_segmentation=False 时使用规则分段器 | +| CONFIG-03 | 知乎/文章平台继续使用 AnalyzeProcessor,不受影响 | 已验证,AnalyzeProcessor 不依赖 SRTProcessor 的分段逻辑 | + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| pydantic-settings | 2.13.1 | 从环境变量和 .kb_env 文件加载配置 | 项目已使用,提供 BaseSettings 基类和 env_prefix 机制 | +| pydantic | 2.12.5 | 配置字段验证和类型安全 | 项目已使用,提供 Field 和 Field 定义 | +| pytest | (现有版本) | 测试框架 | 项目已有完整测试基础设施 | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| unittest.mock | (stdlib) | 测试时模拟环境变量 | 测试配置加载逻辑 | +| pytest fixtures | (现有) | 共享测试资源和设置 | 现有 conftest.py 模式 | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| pydantic-settings | 直接读取 os.getenv | 失去类型验证、自动文档、环境文件支持 | +| bool Field | str Field + 自定义解析 | 增加复杂度,pydantic 已支持布尔字符串解析 | + +**Installation:** +```bash +# 无需安装,项目已包含所有依赖 +# 如需验证安装: +python3 -c "import pydantic_settings; print(pydantic_settings.__version__)" +python3 -c "import pydantic; print(pydantic.__version__)" +``` + +## Architecture Patterns + +### Recommended Project Structure +``` +kb_workflow/ +├── pipeline/ +│ ├── config.py # 添加 enable_semantic_segmentation 字段 +│ └── processors/ +│ ├── srt.py # 读取配置并传递给构造函数 +│ └── analyze.py # 不受影响,独立处理器 +tests/ +├── test_pipeline/ +│ └── test_config.py # 添加新字段测试 +└── test_processors/ + ├── test_srt.py # 添加配置集成测试 + └── conftest.py # 可能需要添加配置相关的 fixtures +``` + +### Pattern 1: Pydantic Settings Field with Default +**What:** 使用 `Field(default=True)` 定义布尔配置字段 +**When to use:** 添加可选的布尔配置项,支持环境变量覆盖 +**Example:** +```python +# Source: https://docs.pydantic.dev/2.0/api/pydantic_settings/ +from pydantic import Field +from pydantic_settings import BaseSettings, SettingsConfigDict + +class KBSettings(BaseSettings): + model_config = SettingsConfigDict( + env_file=str(Path.home() / ".kb_env"), + env_file_encoding="utf-8", + env_prefix="KB_", + extra="ignore", + ) + + # 新增字段 + enable_semantic_segmentation: bool = Field( + default=True, + description="启用 LLM 语义分段(默认启用)" + ) +``` + +**Environment Variable Resolution:** +```bash +# ~/.kb_env 文件 +KB_ENABLE_SEMANTIC_SEGMENTATION=true + +# 或环境变量 +export KB_ENABLE_SEMANTIC_SEGMENTATION=true + +# Pydantic 自动解析以下字符串为布尔 True: +# - "true", "True", "TRUE", "t", "yes", "y", "1" +# - False: "false", "False", "FALSE", "f", "no", "n", "0" +``` + +### Pattern 2: Configuration Integration in Processor +**What:** 在 Processor 构造函数中从 `get_settings()` 读取配置 +**When to use:** 需要将全局配置传递给处理器 +**Example:** +```python +# kb_workflow/pipeline/processors/srt.py (修改示例) +from kb_workflow.pipeline.config import get_settings + +class SRTProcessor(BaseProcessor): + def __init__( + self, + state_manager: Optional[WorkflowStateManager] = None, + enable_correction: bool = True, + enable_semantic_segmentation: Optional[bool] = None, # 允许覆盖 + enable_cache: bool = True + ): + super().__init__(state_manager) + + # 如果未明确指定,从配置读取 + if enable_semantic_segmentation is None: + settings = get_settings() + enable_semantic_segmentation = settings.enable_semantic_segmentation + + self.enable_semantic_segmentation = enable_semantic_segmentation + + # 记录配置状态 + logger.info( + f"SRTProcessor initialized: " + f"semantic_segmentation={'enabled' if self.enable_semantic_segmentation else 'disabled'}" + ) +``` + +### Anti-Patterns to Avoid +- **硬编码默认值在调用处:** 应从 KBSettings 统一读取,而非在 `cli.py` 或 `pipeline.py` 中硬编码 +- **配置验证逻辑分散:** 应依赖 pydantic 的自动验证,而非手写验证代码 +- **忽略环境变量前缀:** 所有配置项必须遵循 `KB_` 前缀约定 + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| 配置文件解析 | 手动读取 ~/.kb_env | pydantic-settings BaseSettings | 自动处理 env 文件、环境变量、类型验证 | +| 布尔字符串解析 | 手动转换 "true"/"false" | pydantic bool Field | 自动支持多种布尔表示,错误处理 | +| 配置缓存 | 单例模式手写 | @lru_cache on get_settings() | 已有实现,线程安全 | + +**Key insight:** pydantic-settings 提供了生产级配置管理的所有必要功能,手写方案只会增加维护负担和出错风险。 + +## Common Pitfalls + +### Pitfall 1: 配置未生效由于缓存 +**What goes wrong:** 修改环境变量或 .kb_env 后,`get_settings()` 返回旧值 +**Why it happens:** `get_settings()` 使用 `@lru_cache`,同一进程内配置只加载一次 +**How to avoid:** 测试时调用 `get_settings.cache_clear()`,或重启进程 +**Warning signs:** 配置变更后行为不变,日志显示旧配置值 + +### Pitfall 2: 布尔字段类型错误 +**What goes wrong:** 环境变量设为 "enabled"/"disabled" 导致 ValidationError +**Why it happens:** Pydantic bool 只接受标准布尔值(true/false, yes/no, 1/0 等) +**How to avoid:** 使用标准布尔值,或在 Field 中添加自定义验证器 +**Warning signs:** 启动时抛出 `ValidationError: true|false expected` + +### Pitfall 3: 配置作用域混淆 +**What goes wrong:** Processor 构造函数参数与环境变量配置冲突 +**Why it happens:** 未明确参数优先级(init_kwargs > env_vars > defaults) +**How to avoid:** 文档化参数优先级,允许显式参数覆盖配置 +**Warning signs:** 用户设置环境变量但行为不符预期 + +## Code Examples + +Verified patterns from official sources: + +### Pydantic Settings Bool Field +```python +# Source: https://docs.pydantic.dev/2.0/api/pydantic_settings/ +from pydantic import Field +from pydantic_settings import BaseSettings + +class KBSettings(BaseSettings): + # 布尔字段,默认 True,支持环境变量 + enable_feature: bool = Field( + default=True, + description="功能开关描述" + ) + +# 使用方式 +# 1. 默认值 +settings = KBSettings() # enable_feature = True + +# 2. 环境变量覆盖 +# export KB_ENABLE_FEATURE=false +settings = KBSettings() # enable_feature = False + +# 3. 构造函数参数覆盖 +settings = KBSettings(enable_feature=False) # enable_feature = False +``` + +### Testing Configuration with Environment Variables +```python +# Source: existing tests/test_pipeline/test_config.py pattern +import os +import pytest +from unittest.mock import patch + +def test_enable_semantic_segmentation_from_env(): + """Test KB_ENABLE_SEMANTIC_SEGMENTATION env var is respected""" + with patch.dict(os.environ, { + "KB_BASE_PATH": "/tmp/kb", + "KB_ARCHIVE_PATH": "/tmp/archive", + "KB_ENABLE_SEMANTIC_SEGMENTATION": "false" + }, clear=False): + get_settings.cache_clear() + settings = get_settings() + assert settings.enable_semantic_segmentation is False + +def test_enable_semantic_segmentation_default(): + """Test default value is True when env var not set""" + with patch.dict(os.environ, { + "KB_BASE_PATH": "/tmp/kb", + "KB_ARCHIVE_PATH": "/tmp/archive" + }, clear=True): + get_settings.cache_clear() + settings = get_settings() + assert settings.enable_semantic_segmentation is True +``` + +### Verifying Backward Compatibility +```python +# Source: existing SRTProcessor code pattern +def test_srt_processor_with_semantic_disabled(): + """Test that disabling semantic segmentation uses rule-based""" + processor = SRTProcessor( + enable_semantic_segmentation=False, + enable_correction=False + ) + + # 验证使用规则分段器 + from kb_workflow.srt.segmenter import SemanticSegmenter + assert isinstance(processor.segmenter, SemanticSegmenter) + assert not isinstance(processor.segmenter, SemanticSegmentationEngine) + +def test_srt_processor_respects_config(): + """Test SRTProcessor reads from KBSettings""" + with patch.dict(os.environ, { + "KB_BASE_PATH": "/tmp/kb", + "KB_ARCHIVE_PATH": "/tmp/archive", + "KB_ENABLE_SEMANTIC_SEGMENTATION": "false" + }): + get_settings.cache_clear() + processor = SRTProcessor() # 应从配置读取 + + # 验证配置被正确应用 + assert processor.enable_semantic_segmentation is False +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| 硬编码配置 | pydantic-settings + .kb_env | 项目初始 | 集中管理、类型安全 | +| 构造函数参数 | 构造函数参数 + 环境变量 | Phase 15 | 用户可通过配置文件控制,无需修改代码 | + +**Deprecated/outdated:** +- 手动解析环境变量(`os.getenv()`)— 应使用 pydantic-settings 自动解析 +- 配置文件自定义格式(JSON/YAML)— .kb_env dotenv 格式为标准 + +## Validation Architecture + +> Nyquist verification enabled (workflow.nyquist_validation: true) + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | pytest (版本未知,项目现有) | +| Config file | pytest.ini (pythonpath = ., testpaths = tests) | +| Quick run command | `pytest tests/test_pipeline/test_config.py -x -v` | +| Full suite command | `pytest tests/ -x` | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| OUTPUT-01 | 带 H2 标题的 Markdown 输出 | integration | `pytest tests/test_srt_processor.py::test_generate_markdown_with_titles -x` | ✅ Phase 13 | +| OUTPUT-02 | 纯文本输出(title=null) | integration | `pytest tests/test_srt_processor.py::test_generate_markdown_without_titles -x` | ✅ Phase 13 | +| OUTPUT-03 | 规则分段输出 "## 内容" | integration | `pytest tests/test_srt_processor.py::test_generate_markdown_rule_based -x` | ✅ Phase 13 | +| CONFIG-01 | KBSettings 新字段加载 | unit | `pytest tests/test_pipeline/test_config.py::test_kbsettings_enable_semantic_segmentation_field -x` | ❌ Wave 0 | +| CONFIG-01 | 环境变量覆盖默认值 | unit | `pytest tests/test_pipeline/test_config.py::test_enable_semantic_segmentation_from_env -x` | ❌ Wave 0 | +| CONFIG-02 | 禁用时使用规则分段 | integration | `pytest tests/test_processors/test_srt.py::test_srt_processor_with_semantic_disabled -x` | ❌ Wave 0 | +| CONFIG-03 | AnalyzeProcessor 不受影响 | integration | `pytest tests/test_processors/test_analyze.py -x` | ✅ 已存在 | + +### Sampling Rate +- **Per task commit:** `pytest tests/test_pipeline/test_config.py -x -v` +- **Per wave merge:** `pytest tests/test_pipeline/test_config.py tests/test_processors/test_srt.py -x` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `tests/test_pipeline/test_config.py` — 添加新字段测试(CONFIG-01) +- [ ] `tests/test_processors/test_srt.py` — 添加配置集成测试(CONFIG-01, CONFIG-02) +- [ ] `tests/test_processors/test_srt.py` — 验证向后兼容性(CONFIG-02, OUTPUT-03) +- [ ] Framework install: 无需安装(现有 pytest 环境) +- [ ] Shared fixtures: `tests/test_processors/conftest.py` 已存在,可能需要添加配置 mock + +## Open Questions + +1. **enable_cache 是否也需要添加到 KBSettings?** + - What we know: 当前 `enable_cache` 只有构造函数参数,未集成到 KBSettings + - What's unclear: 用户是否需要全局控制缓存开关 + - Recommendation: Phase 15 范围内不添加,保留为构造函数参数(符合 Claude's Discretion) + +2. **配置变更日志级别** + - What we know: SRTProcessor 初始化时已有日志记录 + - What's unclear: 应使用 INFO 还是 DEBUG 级别 + - Recommendation: INFO 级别(用户可见,便于调试配置问题) + +## Sources + +### Primary (HIGH confidence) +- [pydantic-settings API Documentation](https://docs.pydantic.dev/2.0/api/pydantic_settings/) - BaseSettings, Field, env_prefix +- [Pydantic Settings Usage](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) - 配置文件和环境变量加载 +- [kb_workflow/pipeline/config.py](file:///home/starmen1265/work/kb-skills/kb_workflow/pipeline/config.py) - 现有 KBSettings 实现 +- [kb_workflow/pipeline/processors/srt.py](file:///home/starmen1265/work/kb-skills/kb_workflow/pipeline/processors/srt.py) - SRTProcessor 构造函数和配置集成 +- [kb_workflow/srt/semantic_engine.py](file:///home/starmen1265/work/kb-skills/kb_workflow/srt/semantic_engine.py) - _generate_markdown() 实现 + +### Secondary (MEDIUM confidence) +- [pytest documentation](https://docs.pytest.org/) - 测试框架标准用法 +- Existing test files: `tests/test_pipeline/test_config.py`, `tests/test_srt_processor.py` - 项目测试模式 + +### Tertiary (LOW confidence) +- None (所有研究基于官方文档或项目代码) + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - 基于项目现有依赖和官方文档 +- Architecture: HIGH - pydantic-settings 是行业标准,项目已采用 +- Pitfalls: HIGH - 基于已知 pydantic 使用陷阱和项目测试经验 +- Output format: HIGH - 基于代码审查,Phase 13 已实现所需功能 + +**Research date:** 2026-03-23 +**Valid until:** 30 days (pydantic-settings 稳定版本,项目配置模式成熟)