# Phase 1: Foundation - Research **Researched:** 2026-03-13 **Domain:** Python workflow infrastructure - state management, schema validation, configuration, error handling **Confidence:** HIGH ## Summary Phase 1 建立工作流基础设施的核心组件:状态管理器、Schema 注册表、错误处理器和配置管理器。这些组件遵循 Python 生态系统的最佳实践,使用 Pydantic v2 进行数据验证,pydantic-settings 进行配置管理,以及原子文件写入模式保证状态持久化的可靠性。 **Primary recommendation:** 使用 Pydantic v2 BaseModel 定义所有 Schema,pydantic-settings 管理配置,复用 gsd-openclaw/state_manager.py 的原子写入模式,定义统一的错误类型层次结构。 ## User Constraints (from CONTEXT.md) ### Locked Decisions #### 状态管理 - **状态粒度**: 步骤级状态 — 符合现有子技能映射 (metadata→content→whisper→analyzer) - **状态关联**: 按内容ID关联 — 如 BV123xxx,重复链接自动识别 - **幂等执行**: 自动跳过已完成步骤 — 重新执行同一链接时从中断点继续 - **备份机制**: 需要备份 — 原子写入 + .bak 轮转,崩溃后可恢复 - **存储位置**: 状态文件存放在内容目录下 — 方便关联查看 - **历史记录**: 只保留最新状态 — 不追加历史执行记录 #### Schema 验证 - **验证时机**: 立即验证 — 步骤完成后立即验证,失败则阻止进入下一步 - **Schema 定义位置**: 集中定义 — 所有步骤的 Schema 集中在一个模块 - **验证范围**: 验证输入和输出 — 步骤执行前验证输入,执行后验证输出 - **Schema 格式**: Pydantic 模型 — 自带验证和序列化,类型安全 #### 错误处理 - **错误分类**: 支持四种错误类型 - 网络错误 (403/404/500/超时) - 验证错误 (Schema 验证失败) - 系统错误 (文件系统/内存/依赖缺失) - 认证/配额错误 (API配额/Cookie过期/Token失效) - **重试策略**: 自动重试 — 网络错误自动重试3次,指数退避 - **错误信息**: 完整上下文 — 步骤名、输入摘要、错误类型、失败原因、可操作建议 - **降级策略**: 定义降级链 — 如官方字幕失败后尝试语音识别 #### 配置管理 - **配置文件位置**: ~/.kb_env — 延续现有习惯 - **验证时机**: 启动时验证 — 缺失必需变量则报错退出 - **敏感信息处理**: 环境变量存敏感 — Cookie/Token 用环境变量,配置文件只存非敏感配置 - **配置库**: pydantic-settings — 自动验证和类型转换 ### Claude's Discretion 无特别指定 — 在锁定决策框架内做技术实现选择。 ### Deferred Ideas (OUT OF SCOPE) None — discussion stayed within phase scope ## Phase Requirements | ID | Description | Research Support | |----|-------------|-----------------| | PIPE-01 | 统一的管道模型 - 每个步骤是独立管道,定义清晰的输入→处理→输出 | Pydantic BaseModel 定义 StepInput/StepOutput | | PIPE-02 | 步骤级状态管理 - 主要步骤完成后记录状态到 workflow_state.json | 原子写入 + 备份轮转模式 (参考 gsd-openclaw/state_manager.py) | | PIPE-03 | 统一 JSON Schema - 所有子技能遵守统一的 StepInput/StepOutput 格式规范 | Pydantic v2 model_dump() / model_validate() | | REL-01 | 幂等执行保证 - 重复执行同一链接不会产生重复文件或重复处理 | WorkflowState.is_step_completed() 检查 | | REL-02 | 错误恢复策略 - 403/超时/网络错误有明确的降级策略和重试机制 | FallbackChain + retry_with_backoff() | | REL-03 | 清晰的流程定义 - 每个步骤的前置条件、输出格式、成功标准明确 | StepDefinition + StepResult Pydantic 模型 | | VAL-01 | Schema 验证机制 - 每个步骤输出通过 Pydantic Schema 验证,失败则阻止进入下一步 | StepOutput 基类 + model_validate() | | MNT-02 | 统一配置管理 - 环境变量集中在 ~/.kb_env,使用 pydantic-settings 管理 | pydantic-settings BaseSettings + env_file | | MNT-03 | 结构化错误处理 - 错误包含上下文信息,便于调试和日志追踪 | KBError 层次结构 + StructuredErrorContext | ## Standard Stack ### Core | Library | Version | Purpose | Why Standard | |---------|---------|---------|--------------| | pydantic | 2.x | 数据验证和 Schema 定义 | Python 生态类型安全验证的事实标准,v2 性能提升 5-50x | | pydantic-settings | 2.x | 配置管理 | 官方配置库,自动环境变量映射和验证 | | python | 3.12+ | 运行时 | 项目约束,支持最新类型语法 | ### Supporting | Library | Version | Purpose | When to Use | |---------|---------|---------|-------------| | pytest | 7.0+ | 测试框架 | 所有单元测试和集成测试 | | tenacity | 8.x | 重试机制 | 网络错误自动重试,指数退避 | ### Alternatives Considered | Instead of | Could Use | Tradeoff | |------------|-----------|----------| | pydantic | dataclasses + 手动验证 | 失去自动验证、序列化、JSON Schema 生成 | | pydantic-settings | dynaconf | dynaconf 更复杂,pydantic-settings 更轻量且与 Pydantic 无缝集成 | | tenacity | 手写重试逻辑 | tenacity 提供成熟的指数退避和钩子机制 | **Installation:** ```bash pip install pydantic>=2.0 pydantic-settings>=2.0 pytest>=7.0 tenacity>=8.0 ``` ## Architecture Patterns ### Recommended Project Structure ``` kb-skills/ ├── kb_workflow/ # 现有工作流工具 │ └── kb_workflow.py ├── kb_pipeline/ # 新建:核心管道基础设施 │ ├── __init__.py │ ├── state.py # WorkflowStateManager │ ├── schemas.py # StepInput, StepOutput, StepResult │ ├── errors.py # KBError 层次结构 │ ├── config.py # KBSettings (pydantic-settings) │ └── retry.py # retry_with_backoff └── tests/ └── test_pipeline/ ├── test_state.py ├── test_schemas.py ├── test_errors.py └── test_config.py ``` ### Pattern 1: Atomic State Persistence **What:** 原子写入状态文件,防止崩溃时数据损坏 **When to use:** 所有 workflow_state.json 写入操作 **Example:** ```python # Source: gsd-openclaw/src/gsd_openclaw/state_manager.py (已验证) import json import os import tempfile from pathlib import Path def save_state_atomically(state: dict, state_path: Path) -> None: """Atomic state save with backup rotation.""" backup_path = state_path.with_suffix('.json.bak') # 1. Create parent directories if missing state_path.parent.mkdir(parents=True, exist_ok=True) # 2. Rotate backup: if state.json exists, move it to state.json.bak if state_path.exists(): if backup_path.exists(): backup_path.unlink() state_path.rename(backup_path) # 3. Write to temp file in same directory (for atomic rename) fd, temp_path = tempfile.mkstemp(dir=state_path.parent, suffix='.tmp') try: with os.fdopen(fd, 'w', encoding='utf-8') as f: json.dump(state, f, indent=2, ensure_ascii=False) # 4. Atomic replace os.replace(temp_path, state_path) except Exception: if os.path.exists(temp_path): os.unlink(temp_path) raise ``` ### Pattern 2: Pydantic Schema Registry **What:** 集中定义所有步骤的输入输出 Schema **When to use:** 定义新步骤或验证步骤输出 **Example:** ```python # Source: https://docs.pydantic.dev/latest/concepts/models/ from pydantic import BaseModel, Field from typing import Optional, Literal from datetime import datetime class StepInput(BaseModel): """所有步骤输入的基类""" content_id: str = Field(..., description="内容唯一标识,如 BV号") platform: Literal["bilibili", "zhihu"] = Field(..., description="平台名称") class StepOutput(BaseModel): """所有步骤输出的基类""" success: bool = Field(..., description="步骤是否成功") error_message: Optional[str] = Field(None, description="错误信息") completed_at: datetime = Field(default_factory=datetime.now) class Config: # Pydantic v2 配置 json_schema_extra = { "example": { "success": True, "error_message": None, "completed_at": "2026-03-13T10:00:00" } } class MetadataStepOutput(StepOutput): """元数据步骤输出""" title: Optional[str] = None author: Optional[str] = None duration_seconds: Optional[int] = None ``` ### Pattern 3: pydantic-settings Configuration **What:** 使用 pydantic-settings 管理环境变量配置 **When to use:** 应用启动时加载配置 **Example:** ```python # Source: https://docs.pydantic.dev/latest/concepts/pydantic_settings/ from pydantic import Field, SecretStr from pydantic_settings import BaseSettings, SettingsConfigDict class KBSettings(BaseSettings): """KB 工作流配置""" model_config = SettingsConfigDict( env_file='~/.kb_env', env_file_encoding='utf-8', env_prefix='KB_', extra='ignore' # 兼容旧配置文件 ) # 必需配置 base_path: str = Field(..., description="知识库根目录") archive_path: str = Field(..., description="归档目录") # 可选配置 git_root: str = Field(default="", description="Git 仓库根目录") git_user: str = Field(default="", description="Git 用户名") git_email: str = Field(default="", description="Git 邮箱") # 敏感信息(从环境变量读取,不存配置文件) hf_token: SecretStr = Field(default="", description="HuggingFace Token") zhihu_cookie: SecretStr = Field(default="", description="知乎 Cookie") # 使用 settings = KBSettings() # 自动从 ~/.kb_env 和环境变量加载 ``` ### Pattern 4: Structured Error Handling **What:** 结构化错误类型,包含完整上下文 **When to use:** 所有步骤失败时抛出 **Example:** ```python from enum import Enum from typing import Optional, Any from pydantic import BaseModel class ErrorType(str, Enum): NETWORK = "network" VALIDATION = "validation" SYSTEM = "system" AUTH = "auth" class KBError(Exception): """KB 工作流基础错误""" def __init__( self, message: str, error_type: ErrorType, step_name: str, input_summary: str, suggestion: str, cause: Optional[Exception] = None ): super().__init__(message) self.error_type = error_type self.step_name = step_name self.input_summary = input_summary self.suggestion = suggestion self.cause = cause def to_dict(self) -> dict: return { "error_type": self.error_type.value, "step_name": self.step_name, "input_summary": self.input_summary, "message": str(self), "suggestion": self.suggestion } class NetworkError(KBError): """网络错误 (403/404/500/超时)""" def __init__(self, step_name: str, input_summary: str, status_code: int = None, **kwargs): suggestion = f"检查网络连接,状态码: {status_code}" if status_code else "检查网络连接" super().__init__( message=f"网络请求失败: {status_code or 'unknown'}", error_type=ErrorType.NETWORK, step_name=step_name, input_summary=input_summary, suggestion=suggestion, **kwargs ) self.status_code = status_code ``` ### Anti-Patterns to Avoid - **直接写入状态文件无原子保证**: 崩溃时可能导致文件损坏或内容丢失 - **散落的配置读取**: `os.environ.get()` 分散在多个文件,难以追踪和测试 - **裸 Exception 捕获**: 丢失错误类型信息,无法实现针对性恢复 - **Schema 定义分散**: 每个子技能自己定义格式,难以统一验证 ## Don't Hand-Roll | Problem | Don't Build | Use Instead | Why | |---------|-------------|-------------|-----| | 配置管理 | 手写 env 文件解析 | pydantic-settings | 自动类型转换、验证、环境变量映射 | | 数据验证 | 手写 if-else 检查 | Pydantic BaseModel | 类型安全、自动序列化、JSON Schema 生成 | | 重试逻辑 | while 循环 + sleep | tenacity | 成熟的指数退避、钩子、停止条件 | | 原子写入 | 手写临时文件逻辑 | 参考 state_manager.py 模式 | 边界情况处理(权限、磁盘满、崩溃) | **Key insight:** 配置和验证是 Python 生态中已解决的问题,使用 Pydantic 生态可以获得类型安全、IDE 支持和自动文档生成。 ## Common Pitfalls ### Pitfall 1: 状态文件损坏 **What goes wrong:** 写入状态文件时崩溃,导致文件内容不完整或为空 **Why it happens:** 直接写入目标文件,没有使用原子替换 **How to avoid:** 使用 write-to-temp-then-rename 模式,确保写入是原子的 **Warning signs:** workflow_state.json 为空或包含截断的 JSON ### Pitfall 2: 配置分散难以调试 **What goes wrong:** 环境变量散落在多个文件,不知道哪个变量在哪使用 **Why it happens:** 每个模块直接调用 `os.environ.get()` **How to avoid:** 使用 pydantic-settings 集中定义所有配置,启动时一次性验证 **Warning signs:** 同一变量在多处读取,默认值不一致 ### Pitfall 3: Schema 演进导致兼容性问题 **What goes wrong:** 添加新字段后旧的状态文件无法加载 **Why it happens:** Schema 变更没有向后兼容性考虑 **How to avoid:** - 使用 Optional 类型和默认值 - 使用 schema_version 字段标记版本 - load_state 时处理旧版本数据 ### Pitfall 4: 错误信息缺少上下文 **What goes wrong:** 错误只有 "Network error",不知道是哪个步骤、哪个输入 **Why it happens:** 只抛出简单的 Exception 或使用通用错误消息 **How to avoid:** 使用结构化错误类,强制包含 step_name, input_summary, suggestion ## Code Examples ### WorkflowState 完整定义 ```python # Source: 基于 gsd-openclaw/state_manager.py 和 CONTEXT.md 决策 from pydantic import BaseModel, Field from typing import Optional, Dict, List from datetime import datetime from enum import Enum class StepStatus(str, Enum): PENDING = "pending" RUNNING = "running" COMPLETED = "completed" FAILED = "failed" SKIPPED = "skipped" class StepRecord(BaseModel): """单个步骤的执行记录""" step_name: str status: StepStatus started_at: Optional[datetime] = None completed_at: Optional[datetime] = None error_message: Optional[str] = None output_summary: Optional[str] = None class WorkflowState(BaseModel): """工作流状态 - 存储在 workflow_state.json""" schema_version: str = Field(default="1.0") content_id: str = Field(..., description="内容唯一标识") platform: str = Field(..., description="平台名称") source_url: str = Field(..., description="原始链接") created_at: datetime = Field(default_factory=datetime.now) updated_at: datetime = Field(default_factory=datetime.now) steps: Dict[str, StepRecord] = Field(default_factory=dict) def is_step_completed(self, step_name: str) -> bool: """检查步骤是否已完成""" record = self.steps.get(step_name) return record is not None and record.status == StepStatus.COMPLETED def mark_step_started(self, step_name: str) -> None: """标记步骤开始""" self.steps[step_name] = StepRecord( step_name=step_name, status=StepStatus.RUNNING, started_at=datetime.now() ) self.updated_at = datetime.now() def mark_step_completed(self, step_name: str, output_summary: str = None) -> None: """标记步骤完成""" if step_name in self.steps: self.steps[step_name].status = StepStatus.COMPLETED self.steps[step_name].completed_at = datetime.now() self.steps[step_name].output_summary = output_summary else: self.steps[step_name] = StepRecord( step_name=step_name, status=StepStatus.COMPLETED, completed_at=datetime.now(), output_summary=output_summary ) self.updated_at = datetime.now() def mark_step_failed(self, step_name: str, error_message: str) -> None: """标记步骤失败""" if step_name in self.steps: self.steps[step_name].status = StepStatus.FAILED self.steps[step_name].error_message = error_message self.steps[step_name].completed_at = datetime.now() else: self.steps[step_name] = StepRecord( step_name=step_name, status=StepStatus.FAILED, error_message=error_message, completed_at=datetime.now() ) self.updated_at = datetime.now() ``` ### WorkflowStateManager ```python # Source: 基于 gsd-openclaw/state_manager.py import json from pathlib import Path from typing import Optional from .schemas import WorkflowState class WorkflowStateManager: """工作流状态管理器""" STATE_FILENAME = "workflow_state.json" def __init__(self, content_dir: Path): """ Args: content_dir: 内容目录路径,状态文件将存放在此目录下 """ self.content_dir = Path(content_dir) self.state_path = self.content_dir / self.STATE_FILENAME def load(self) -> Optional[WorkflowState]: """加载状态,如果不存在返回 None""" if not self.state_path.exists(): # 尝试加载备份 backup_path = self.state_path.with_suffix('.json.bak') if backup_path.exists(): try: with open(backup_path, 'r', encoding='utf-8') as f: data = json.load(f) return WorkflowState.model_validate(data) except (json.JSONDecodeError, ValueError): pass return None try: with open(self.state_path, 'r', encoding='utf-8') as f: data = json.load(f) return WorkflowState.model_validate(data) except (json.JSONDecodeError, ValueError) as e: # 主文件损坏,尝试备份 backup_path = self.state_path.with_suffix('.json.bak') if backup_path.exists(): with open(backup_path, 'r', encoding='utf-8') as f: data = json.load(f) return WorkflowState.model_validate(data) raise def save(self, state: WorkflowState) -> None: """原子保存状态""" import os import tempfile backup_path = self.state_path.with_suffix('.json.bak') # 确保目录存在 self.content_dir.mkdir(parents=True, exist_ok=True) # 轮转备份 if self.state_path.exists(): if backup_path.exists(): backup_path.unlink() self.state_path.rename(backup_path) # 原子写入 fd, temp_path = tempfile.mkstemp(dir=self.content_dir, suffix='.tmp') try: with os.fdopen(fd, 'w', encoding='utf-8') as f: f.write(state.model_dump_json(indent=2)) os.replace(temp_path, self.state_path) except Exception: if os.path.exists(temp_path): os.unlink(temp_path) raise def get_or_create(self, content_id: str, platform: str, source_url: str) -> WorkflowState: """获取现有状态或创建新状态""" state = self.load() if state is None: state = WorkflowState( content_id=content_id, platform=platform, source_url=source_url ) return state ``` ### Retry with Exponential Backoff ```python # Source: tenacity 最佳实践 from tenacity import ( retry, stop_after_attempt, wait_exponential, retry_if_exception_type, before_sleep_log ) import logging from .errors import NetworkError logger = logging.getLogger(__name__) retry_on_network_error = retry( stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10), retry=retry_if_exception_type(NetworkError), before_sleep=before_sleep_log(logger, logging.WARNING), reraise=True ) # 使用 @retry_on_network_error def fetch_with_retry(url: str) -> bytes: """带重试的网络请求""" import requests try: response = requests.get(url, timeout=30) response.raise_for_status() return response.content except requests.RequestException as e: raise NetworkError( step_name="fetch", input_summary=f"url={url}", status_code=getattr(e.response, 'status_code', None) if hasattr(e, 'response') else None, cause=e ) ``` ## State of the Art | Old Approach | Current Approach | When Changed | Impact | |--------------|------------------|--------------|--------| | Pydantic v1 Config class | Pydantic v2 model_config | Pydantic v2 (2023) | 更清晰的配置语法,性能提升 | | 手写 env 文件解析 | pydantic-settings | pydantic-settings 2.0 (2023) | 自动验证、类型转换、嵌套模型支持 | | 手写重试逻辑 | tenacity 库 | 成熟实践 | 指数退避、钩子、条件重试 | | dataclass + 手动验证 | Pydantic BaseModel | Pydantic v2 (2023) | 性能提升 5-50x,更丰富的验证器 | **Deprecated/outdated:** - Pydantic v1 `class Config:` 语法: 使用 `model_config = ConfigDict(...)` 替代 - python-dotenv 单独使用: pydantic-settings 已内置 dotenv 支持 ## Open Questions 1. **降级链如何配置化?** - What we know: 需要支持 "官方字幕 → Whisper → FunASR" 这样的降级链 - What's unclear: 降级链是硬编码还是可配置?配置格式是什么? - Recommendation: Phase 1 实现简单的 FallbackChain 类,降级顺序通过代码定义;Phase 2+ 考虑配置文件支持 2. **状态文件的并发访问?** - What we know: 当前设计假设单进程执行 - What's unclear: 是否需要支持多进程同时处理不同内容? - Recommendation: Phase 1 不处理并发,使用文件锁是后续优化点 ## Validation Architecture ### Test Framework | Property | Value | |----------|-------| | Framework | pytest 7.x | | Config file | pyproject.toml (参考 gsd-openclaw) | | Quick run command | `pytest tests/test_pipeline/ -v --tb=short` | | Full suite command | `pytest tests/ -v --tb=short` | ### Phase Requirements → Test Map | Req ID | Behavior | Test Type | Automated Command | File Exists? | |--------|----------|-----------|-------------------|-------------| | PIPE-01 | StepInput/StepOutput Schema 定义 | unit | `pytest tests/test_pipeline/test_schemas.py -v` | Wave 0 | | PIPE-02 | 原子状态保存和备份轮转 | unit | `pytest tests/test_pipeline/test_state.py::TestAtomicWrite -v` | Wave 0 | | PIPE-03 | JSON Schema 导出和验证 | unit | `pytest tests/test_pipeline/test_schemas.py::TestJSONSchema -v` | Wave 0 | | REL-01 | is_step_completed 幂等检查 | unit | `pytest tests/test_pipeline/test_state.py::TestIdempotency -v` | Wave 0 | | REL-02 | 网络错误重试 | unit | `pytest tests/test_pipeline/test_retry.py -v` | Wave 0 | | REL-03 | StepDefinition 验证 | unit | `pytest tests/test_pipeline/test_schemas.py::TestStepDefinition -v` | Wave 0 | | VAL-01 | Pydantic Schema 验证 | unit | `pytest tests/test_pipeline/test_schemas.py::TestValidation -v` | Wave 0 | | MNT-02 | pydantic-settings 配置加载 | unit | `pytest tests/test_pipeline/test_config.py -v` | Wave 0 | | MNT-03 | 结构化错误上下文 | unit | `pytest tests/test_pipeline/test_errors.py -v` | Wave 0 | ### Sampling Rate - **Per task commit:** `pytest tests/test_pipeline/ -v --tb=short -x` - **Per wave merge:** `pytest tests/ -v --tb=short` - **Phase gate:** Full suite green before `/gsd:verify-work` ### Wave 0 Gaps - [ ] `tests/test_pipeline/test_state.py` — 测试 WorkflowStateManager - [ ] `tests/test_pipeline/test_schemas.py` — 测试 StepInput/StepOutput/StepResult - [ ] `tests/test_pipeline/test_errors.py` — 测试 KBError 层次结构 - [ ] `tests/test_pipeline/test_config.py` — 测试 KBSettings - [ ] `tests/test_pipeline/test_retry.py` — 测试重试逻辑 - [ ] `tests/test_pipeline/conftest.py` — 共享 fixtures - [ ] `kb_pipeline/__init__.py` — 包初始化 ## Sources ### Primary (HIGH confidence) - [Pydantic Models Documentation](https://docs.pydantic.dev/latest/concepts/models/) - BaseModel, validation, serialization - [Pydantic Settings Documentation](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) - BaseSettings, env_file, secrets - [gsd-openclaw/state_manager.py](/home/starmen1265/work/kb-skills/gsd-openclaw/src/gsd_openclaw/state_manager.py) - 原子写入、备份轮转参考实现 - [gsd-openclaw/tests/test_state_manager.py](/home/starmen1265/work/kb-skills/gsd-openclaw/tests/test_state_manager.py) - 状态管理测试模式参考 ### Secondary (MEDIUM confidence) - [kb-workflow/kb_workflow.py](/home/starmen1265/work/kb-skills/kb-workflow/kb_workflow.py) - 现有目录管理和查重逻辑 ### Tertiary (LOW confidence) - 无 — 所有关键信息已从 Primary 源验证 ## Metadata **Confidence breakdown:** - Standard stack: HIGH - Pydantic 是 Python 生态成熟方案,文档完整 - Architecture: HIGH - 复用 gsd-openclaw 已验证的模式,符合项目决策 - Pitfalls: HIGH - 基于实际代码审查和已知问题 **Research date:** 2026-03-13 **Valid until:** 2026-04-13 (30 days - Pydantic 生态稳定)