starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete project research

Browse Source
This commit is contained in:
starmen1265
2026-03-13 15:45:47 +08:00
parent 431732179c
commit 408a02e615
5 changed files with 1289 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/research/ARCHITECTURE.md
+427
View File
@@ -0,0 +1,427 @@
# Architecture Research
**Domain:** Skill-based Workflow Orchestration System
**Researched:** 2026-03-13
**Confidence:** HIGH
## Standard Architecture
### System Overview
技能型工作流编排系统采用分层架构，将处理逻辑、状态管理和执行协调分离。基于当前研究和行业最佳实践，推荐以下四层架构：
```
+---------------------------------------------------------------+
| Orchestration Layer |
| [Workflow Coordinator] [State Manager] [Error Handler] |
+---------------------------------------------------------------+
| Skill Definition Layer |
| [SKILL.md specs] [Schema Definitions] [Trigger Rules] |
+---------------------------------------------------------------+
| Execution Layer |
| [Platform Skills] [Processing Skills] [Analysis Skills] |
+---------------------------------------------------------------+
| Infrastructure Layer |
| [File System] [External APIs] [AI Services] [State Store] |
+---------------------------------------------------------------+
```
### Component Responsibilities
| Component | Responsibility | Typical Implementation |
|-----------|----------------|------------------------|
| Workflow Coordinator | 协调多个技能的执行顺序、依赖管理 | kb-workflow 主控模块 |
| State Manager | 步骤级状态持久化、断点续传支持 | workflow_state.json + 原子写入 |
| Error Handler | 错误分类、降级策略、重试逻辑 | 统一错误处理模块 |
| Skill Registry | 技能发现、能力声明、Schema 注册 | SKILL.md + JSON Schema |
| Platform Skills | 平台特定内容获取（B站/知乎） | kb-bilibili-*, kb-zhihu-* |
| Processing Skills | 内容转换（语音识别/OCR） | kb-whisper, kb-funasr, kb-ocr |
| Analysis Skills | AI 深度分析、报告生成 | kb-deep-analyzer |
## Recommended Project Structure
```
kb-skills/
+-- skills/ # 技能模块目录
| +-- core/ # 核心基础设施
| | +-- state_manager.py # 状态管理
| | +-- schema_registry.py # Schema 注册
| | +-- pipeline.py # 管道执行引擎
| | +-- error_handler.py # 错误处理
| |
| +-- platforms/ # 平台技能
| | +-- bilibili/ # B站技能组
| | | +-- SKILL.md
| | | +-- metadata.py
| | | +-- content.py
| | +-- zhihu/ # 知乎技能组
| | +-- SKILL.md
| | +-- metadata.py
| | +-- content.py
| |
| +-- processing/ # 处理技能
| | +-- whisper/ # 语音识别
| | +-- funasr/ # 中文语音识别
| | +-- ocr/ # OCR 处理
| |
| +-- analysis/ # 分析技能
| +-- deep_analyzer/ # 深度分析
|
+-- schemas/ # JSON Schema 定义
| +-- workflow_state.schema.json
| +-- skill_output.schema.json
| +-- platform_metadata.schema.json
|
+-- workflows/ # 工作流定义
| +-- video_pipeline.yaml # 视频处理流程
| +-- article_pipeline.yaml # 文章处理流程
|
+-- cli.py # 统一 CLI 入口
+-- config.py # 配置管理
```
### Structure Rationale
- **skills/core/:** 核心基础设施，所有技能共享的状态管理、Schema 验证和管道执行
- **skills/platforms/:** 按平台组织，每个平台有自己的元数据提取和内容下载技能
- **skills/processing/:** 内容转换技能，独立于平台
- **skills/analysis/:** AI 分析技能，作为管道的最后阶段
- **schemas/:** 统一的 JSON Schema 定义，确保接口一致性
- **workflows/:** 声明式工作流定义，指定技能组合和执行顺序
## Architectural Patterns
### Pattern 1: Pipeline Orchestration (管道编排)
**What:** 将处理流程建模为一系列串联的步骤，每个步骤接收输入、执行处理、产生输出。
**When to use:** 可预测的、重复性的任务，如内容处理管道。
**Trade-offs:**
- 优点：确定性执行、易于调试、支持断点续传
- 缺点：灵活性有限，不适合需要动态决策的场景
**Example:**
```python
# Pipeline definition
class ContentPipeline:
steps = [
("parse", LinkParserSkill),
("metadata", MetadataSkill), # depends on parse
("download", ContentSkill), # depends on metadata
("transcribe", TranscribeSkill), # conditional: only for video
("analyze", AnalyzeSkill), # depends on download + transcribe
]
def run(self, url: str, state: WorkflowState):
for step_name, skill in self.steps:
if state.is_completed(step_name):
continue # Skip completed steps
result = skill.execute(state.get_step_input(step_name))
state.mark_completed(step_name, result)
if not result.success:
return self.handle_error(result.error)
```
### Pattern 2: Step-Level State Management (步骤级状态管理)
**What:** 每个主要步骤完成后立即持久化状态，支持从任意断点恢复执行。
**When to use:** 长时间运行的批处理任务，可能因网络、API 限制等原因中断。
**Trade-offs:**
- 优点：断点续传、执行追踪、幂等性保证
- 缺点：状态管理开销、需要设计状态 Schema
**Example:**
```python
# workflow_state.json
{
"url": "https://www.bilibili.com/video/BV1xx411c7mD",
"platform": "bilibili",
"content_type": "video",
"created_at": "2026-03-13T10:00:00Z",
"updated_at": "2026-03-13T10:15:00Z",
"steps": {
"parse": {
"status": "completed",
"output": {"video_id": "BV1xx411c7mD"},
"completed_at": "2026-03-13T10:00:05Z"
},
"metadata": {
"status": "completed",
"output": {"title": "...", "duration": 1800},
"completed_at": "2026-03-13T10:00:30Z"
},
"download": {
"status": "in_progress",
"output": null,
"started_at": "2026-03-13T10:01:00Z"
}
}
}
```
### Pattern 3: Schema-First Interface Design (Schema 优先接口设计)
**What:** 先定义输入输出的 JSON Schema，再实现技能逻辑。所有技能遵守统一的接口规范。
**When to use:** 多人协作、需要接口稳定性和可测试性的项目。
**Trade-offs:**
- 优点：接口清晰、易于测试、支持自动验证
- 缺点：前期设计成本、Schema 维护负担
**Example:**
```json
// schemas/skill_output.schema.json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["success", "data"],
"properties": {
"success": {"type": "boolean"},
"data": {"type": "object"},
"error": {
"type": "object",
"properties": {
"code": {"type": "string"},
"message": {"type": "string"},
"detail": {"type": "string"}
}
},
"metadata": {
"type": "object",
"properties": {
"duration_ms": {"type": "number"},
"tokens_used": {"type": "number"}
}
}
}
}
```
### Pattern 4: Idempotent Execution (幂等执行)
**What:** 相同输入重复执行产生相同结果，不会产生副作用（如重复文件）。
**When to use:** 任何可能需要重试的处理步骤。
**Trade-offs:**
- 优点：安全重试、简化错误恢复
- 缺点：需要设计去重逻辑
**Example:**
```python
class ContentSkill:
def execute(self, input: dict) -> SkillOutput:
# Generate deterministic filename from input
content_hash = hashlib.sha256(input['url'].encode()).hexdigest()[:12]
output_path = self.get_output_dir() / f"{content_hash}.mp4"
# Check if already exists
if output_path.exists():
return SkillOutput(success=True, data={"path": str(output_path), "cached": True})
# Download content
self.download_content(input['url'], output_path)
return SkillOutput(success=True, data={"path": str(output_path), "cached": False})
```
## Data Flow
### Content Processing Pipeline
```
[URL Input]
|
v
+-------------+ +----------------+
| Link Parser | --> | Platform/Type |
+-------------+ | Detection |
+----------------+
|
+---------------+---------------+
| |
v v
+-------------+ +-------------+
| Bilibili | | Zhihu |
| Pipeline | | Pipeline |
+-------------+ +-------------+
| |
+------+------+ +-------+-------+
| | | |
v v v v
[Metadata] [Download] [Metadata] [Download]
| | | |
+------+------+ +-------+-------+
| |
v v
+-------------+ +-------------+
| Video? | | Article |
| - Whisper | | - Direct |
| - FunASR | | Extract |
| - OCR | +-------------+
+-------------+ |
| |
+---------------+--------------+
|
v
+-----------------+
| Deep Analyzer |
| (AI Analysis) |
+-----------------+
|
v
+-----------------+
| Archive to |
| Knowledge Base |
+-----------------+
```
### State Management Flow
```
[Workflow Start]
|
v
+------------------+
| Load/Init State | <-- workflow_state.json
+------------------+
|
v
+------------------+ +------------------+
| For Each Step | --> | Check Status |
+------------------+ | - completed? skip|
| | - failed? retry |
v | - pending? run |
+------------------+ +------------------+
| Execute Step | |
+------------------+ v
| +------------------+
v | Mark Completed |
+------------------+ | Save State |
| Validate Output | -->| (atomic write) |
+------------------+ +------------------+
|
v
[Next Step / Complete]
```
### Key Data Flows
1. **URL to Metadata:** URL -> Link Parser -> (platform, type, id) -> Platform Metadata Skill -> Metadata JSON
2. **Metadata to Content:** Metadata JSON -> Content Skill -> (video file / article text)
3. **Content to Transcript:** Video File -> Transcribe Skill (Whisper/FunASR) -> Transcript JSON
4. **Content + Transcript to Analysis:** Content + Transcript -> Deep Analyzer -> Analysis Report
## Scaling Considerations
| Scale | Architecture Adjustments |
|-------|--------------------------|
| 1-100 items/day | 当前架构足够，单进程执行 |
| 100-1000 items/day | 添加任务队列，并行处理独立任务 |
| 1000+ items/day | 分布式任务调度，API 请求限流，缓存层 |
### Scaling Priorities
1. **First bottleneck:** 外部 API 限流（B站/知乎）。解决方案：请求队列 + 指数退避重试
2. **Second bottleneck:** AI 分析耗时。解决方案：批处理 + 并行分析
## Anti-Patterns
### Anti-Pattern 1: Global State Mutation
**What people do:** 在多个技能间共享可变全局状态
**Why it's wrong:** 难以追踪状态变化、难以测试、难以恢复
**Do this instead:** 每个步骤从 state 读取输入，将输出写回 state，状态变更通过 state manager 集中管理
### Anti-Pattern 2: Implicit Dependencies
**What people do:** 技能之间通过隐式约定传递数据（如固定文件路径）
**Why it's wrong:** 脆弱的耦合，修改一个技能可能破坏其他技能
**Do this instead:** 在 workflow_state.json 中显式声明依赖关系，通过 state manager 传递数据
### Anti-Pattern 3: Silent Failure
**What people do:** 捕获异常后只打印日志，继续执行
**Why it's wrong:** 下游步骤收到无效输入，产生难以调试的错误
**Do this instead:** 明确的失败处理策略 - 要么终止并记录，要么降级并提供默认值
### Anti-Pattern 4: Non-Idempotent Operations
**What people do:** 每次执行都创建新文件（如 `output_1.mp4`, `output_2.mp4`）
**Why it's wrong:** 重试产生垃圾文件，磁盘占用增长
**Do this instead:** 基于输入生成确定性文件名，已存在则跳过
## Integration Points
### External Services
| Service | Integration Pattern | Notes |
|---------|---------------------|-------|
| Bilibili API | HTTP + Cookie Auth | 需要 BILIBILI_COOKIE 环境变量 |
| Zhihu API | HTTP + Cookie Auth | 需要 ZHIHU_COOKIE 环境变量 |
| AI Provider | OpenAI-compatible API | 需要 AI_API_KEY 环境变量 |
| HuggingFace | API + Local Cache | 需要 HF_TOKEN 环境变量 |
| Git | Command Line | 需要 KB_GIT_USER/EMAIL 配置 |
### Internal Boundaries
| Boundary | Communication | Notes |
|----------|---------------|-------|
| Orchestrator <-> Skills | JSON via stdin/stdout | 标准化输入输出格式 |
| Skills <-> State Manager | File I/O | workflow_state.json |
| Skills <-> External APIs | HTTP | 统一错误处理和重试 |
## Build Order Implications
基于组件依赖关系，建议的构建顺序：
### Phase 1: Foundation (基础设施)
1. **State Manager** - 所有后续组件的基础
2. **Schema Registry** - 定义统一的输入输出格式
3. **Error Handler** - 统一错误处理模式
### Phase 2: Core Skills (核心技能)
4. **Link Parser** - 入口点，无依赖
5. **Platform Metadata Skills** - 依赖 Link Parser 输出
6. **Platform Content Skills** - 依赖 Metadata 输出
### Phase 3: Processing Skills (处理技能)
7. **Whisper/FunASR** - 依赖视频文件
8. **OCR Processor** - 依赖视频截图
### Phase 4: Analysis Skills (分析技能)
9. **Deep Analyzer** - 依赖内容和转录文本
### Phase 5: Orchestration (编排层)
10. **Pipeline Engine** - 组合所有技能
11. **Workflow Definitions** - 声明式流程定义
## Sources
- [AI Architecture Patterns 101: Workflows, Agents, MCPs, and A2A Systems](https://aipmguru.substack.com/p/ai-architecture-patterns-101-workflows) - HIGH confidence, comprehensive pattern overview
- [Data Pipeline Architecture: 5 Design Patterns with Examples - Dagster](https://dagster.io/guides/data-pipeline-architecture-5-design-patterns-with-examples) - HIGH confidence, pipeline patterns
- [Agentic Workflows in 2026: The ultimate guide - Vellum](https://www.vellum.ai/blog/agentic-workflows-emerging-architectures-and-design-patterns) - HIGH confidence, agent architecture patterns
- [State of Open Source Workflow Orchestration Systems 2025](https://www.pracdata.io/p/state-of-workflow-orchestration-ecosystem-2025) - MEDIUM confidence, ecosystem overview
- [ML Pipeline Architecture Design Patterns - Neptune.ai](https://neptune.ai/blog/ml-pipeline-architecture-design-patterns) - MEDIUM confidence, ML-specific patterns
---
*Architecture research for: Skill-based Workflow Orchestration System*
*Researched: 2026-03-13*
.planning/research/FEATURES.md
+200
View File
@@ -0,0 +1,200 @@
# Feature Research
**Domain:** Skill-based Workflow Orchestration (Content Processing Pipelines)
**Researched:** 2026-03-13
**Confidence:** HIGH
## Feature Landscape
### Table Stakes (Users Expect These)
Features users assume exist. Missing these = product feels incomplete.
| Feature | Why Expected | Complexity | Notes |
|---------|--------------|------------|-------|
| **Pipeline/Sequential Orchestration** | Linear processing is the fundamental model - users expect steps to execute in order | LOW | Already in PROJECT.md as core requirement |
| **Idempotent Execution** | Re-running same input should produce same output without duplicates | MEDIUM | Critical for reliability; prevents duplicate files on retry |
| **Checkpoint/Resume from Failure** | Long-running tasks can fail; users expect to continue from where they left off | MEDIUM | workflow_state.json is planned; checkpointing is industry standard |
| **State Management** | Every pipeline needs to track progress - what's done, what's pending | MEDIUM | Step-level state (not task-level) per PROJECT.md |
| **Structured Error Handling** | Failures should be captured, reported, and recoverable | LOW | JSON error format with success/error fields already exists |
| **Input/Output Schema Validation** | Data contracts ensure reliability; JSON Schema is standard | MEDIUM | PIPE-07 in requirements |
| **Clear Step Boundaries** | Users need to understand what each step does, its inputs/outputs | LOW | PIPE-06 requirement |
| **Platform-Specific Content Handlers** | Bilibili/Zhihu support is core functionality | MEDIUM | Already implemented for metadata/content extraction |
| **AI-Powered Analysis** | Deep analysis is the product's value proposition | HIGH | kb-deep-analyzer exists |
### Differentiators (Competitive Advantage)
Features that set the product apart. Not required, but valuable.
| Feature | Value Proposition | Complexity | Notes |
|---------|-------------------|------------|-------|
| **Skill-Based Modular Architecture** | Each skill is independent, testable, and reusable; easier maintenance than monolithic pipelines | MEDIUM | SKILL.md specification pattern; already in architecture |
| **Multi-Modal Content Processing** | Video (ASR/OCR) + Text unified pipeline; handles diverse content types in one flow | HIGH | Whisper, FunASR, OCR already implemented |
| **Tmux-Based Session Persistence** | Long-running operations survive terminal disconnect; GSD-OpenClaw integration | MEDIUM | Unique to this system; TmuxSessionManager exists |
| **Declarative Skill Definition** | SKILL.md with YAML frontmatter defines capabilities; AI agents can discover and execute | LOW | Already in place; differentiates from code-only orchestration |
| **Unified JSON Output Format** | Consistent success/error structure across all skills; easy programmatic consumption | LOW | Already standardized |
| **Git-Integrated Archival** | Processed content versioned and archived automatically | LOW | KB_GIT_ROOT integration |
| **Content Type Detection** | Auto-detect video vs article vs QA; route to appropriate processing path | MEDIUM | Link parser handles this |
### Anti-Features (Commonly Requested, Often Problematic)
Features that seem good but create problems.
| Feature | Why Requested | Why Problematic | Alternative |
|---------|---------------|-----------------|-------------|
| **Real-Time/Streaming Processing** | Users want instant results | Adds significant complexity; batch processing sufficient for content analysis use case; real-time requires different architecture (WebSocket, event streams) | Keep batch processing; add optional polling/notifications |
| **Concurrent/Parallel Agent Execution** | Faster processing | Context sharing between parallel agents is complex; resource contention (API rate limits, model quotas); conflict resolution when results contradict | Sequential orchestration with clear dependencies is more reliable for this use case |
| **Web UI/Dashboard** | Visual workflow management | Significant development effort; CLI is sufficient for current use case; UI maintenance overhead | CLI-first; consider status output in structured formats for future visualization |
| **Multi-Tenant/User Management** | SaaS-like deployment | Adds auth, isolation, billing complexity; single-user tool currently | Keep single-tenant; future: separate deployment per user |
| **Plugin/Extension System** | Third-party integrations | Security concerns; API stability requirements; documentation burden | Keep monolithic but modular skill architecture; internal extensions only |
| **Workflow Visualization** | See pipeline graphically | Nice-to-have but not core value; generates maintenance burden | State file (JSON) is machine-readable; can generate viz later |
| **Automatic Retry with Backoff** | Handle transient failures | Can mask underlying issues; retry logic is complex; may cause rate limiting | Explicit error reporting; manual retry with checkpoint resume |
| **Complex Routing/Handoff Patterns** | Dynamic task delegation | Overkill for linear content processing; adds nondeterminism; harder to debug | Sequential orchestration with explicit steps |
## Feature Dependencies
```
[URL Input]
└──requires──> [Link Parser Skill]
[Link Parser]
└──enables──> [Platform-Specific Metadata Extraction]
[Metadata Extraction]
└──enables──> [Content Download]
[Content Download (Video)]
└──requires──> [ASR/Transcription]
└──requires──> [Whisper OR FunASR]
[Content Download (Video with Text)]
└──requires──> [OCR Processing]
[ASR/OCR/Text Content]
└──enables──> [Deep Analysis]
[Deep Analysis]
└──enables──> [Archive to Git]
[State Management]
└──enables──> [Checkpoint Resume]
└──enables──> [Idempotent Execution]
[Schema Validation]
└──enables──> [Error Detection]
└──enables──> [Consistent Output Format]
```
### Dependency Notes
- **Checkpoint Resume requires State Management:** Cannot resume without tracking what completed; workflow_state.json is the mechanism
- **Idempotent Execution requires State Management:** Need to know if step already completed successfully to skip re-execution
- **Schema Validation enables Error Detection:** Catch malformed outputs early before they cascade to downstream steps
- **Platform-Specific Skills require Link Parser:** Cannot extract metadata without knowing platform and content type
- **Deep Analysis requires Text Content:** ASR/OCR must complete before AI analysis can process
## MVP Definition
### Launch With (v1)
Minimum viable product — what's needed to validate the concept.
- [x] **Sequential Pipeline Orchestration** — Core value proposition; linear processing from URL to analysis
- [x] **Step-Level State Management (workflow_state.json)** — Enables reliability; checkpoint/resume
- [x] **Idempotent Execution** — Same URL processed twice produces same result; no duplicates
- [x] **JSON Schema Validation** — Data contracts between steps; error detection
- [x] **Structured Error Handling** — JSON error format; graceful degradation
- [x] **Bilibili/Zhihu Platform Support** — Core platforms; metadata + content extraction
- [x] **AI Deep Analysis** — Product's value output
### Add After Validation (v1.x)
Features to add once core is working reliably.
- [ ] **Enhanced Error Recovery Strategies** — 403/timeout specific handling; fallback options
- [ ] **OCR for Video Text Extraction** — Currently implemented but integration may need refinement
- [ ] **Speaker Diarization** — Already implemented; may need pipeline integration
- [ ] **Multiple ASR Backend Support** — Whisper + FunASR selection logic
- [ ] **Better Progress Visibility** — Step completion logging/indicators
### Future Consideration (v2+)
Features to defer until product-market fit is established.
- [ ] **Additional Platforms (YouTube, etc.)** — Out of scope per PROJECT.md currently
- [ ] **Concurrent Processing for Independent Steps** — Only if latency becomes critical issue
- [ ] **Web UI for Monitoring** — Consider only if non-technical users need access
- [ ] **API/Webhook Notifications** — If integration with external systems needed
## Feature Prioritization Matrix
| Feature | User Value | Implementation Cost | Priority |
|---------|------------|---------------------|----------|
| Step-Level State Management | HIGH | MEDIUM | P1 |
| Idempotent Execution | HIGH | MEDIUM | P1 |
| JSON Schema Validation | HIGH | LOW | P1 |
| Sequential Pipeline Model | HIGH | LOW | P1 |
| Structured Error Handling | HIGH | LOW | P1 |
| Clear Step Boundaries/Docs | MEDIUM | LOW | P1 |
| Checkpoint/Resume | HIGH | MEDIUM | P2 |
| Platform-Specific Skills | HIGH | MEDIUM | P1 (exists) |
| AI Deep Analysis | HIGH | HIGH | P1 (exists) |
| ASR/OCR Integration | MEDIUM | HIGH | P2 (exists) |
| Error Recovery Strategies | MEDIUM | MEDIUM | P2 |
| Progress Visibility | LOW | LOW | P3 |
| Concurrent Execution | LOW | HIGH | P3 |
| Web UI | LOW | HIGH | P3 |
**Priority key:**
- P1: Must have for launch (reliability core)
- P2: Should have, add when possible (robustness)
- P3: Nice to have, future consideration (convenience)
## Competitor Feature Analysis
| Feature | Apache Airflow | Dagster | Prefect | Our Approach |
|---------|----------------|---------|---------|--------------|
| **Orchestration Model** | DAG-based | Asset-based | Flow-based | Sequential pipeline (simpler) |
| **State Management** | Database-backed | Built-in | Built-in | File-based (workflow_state.json) |
| **Checkpoint/Resume** | Task-level | Asset-level | Result caching | Step-level |
| **Idempotency** | Manual | Built-in checks | Decorator-based | Explicit state check |
| **Schema Validation** | Optional | Built-in | Optional | JSON Schema (planned) |
| **Observability** | UI Dashboard | UI + Lineage | UI + Logging | JSON output + console |
| **Learning Curve** | High | Medium | Low | Low (SKILL.md specs) |
**Key differentiation:** Our skill-based architecture with SKILL.md specifications makes the system more accessible to AI agents and easier to extend than traditional workflow engines. The focus on content processing (not general ETL) allows specialized optimizations.
## Industry Standard Features (2025-2026)
Based on research, modern data pipeline and workflow orchestration systems have these table stakes:
1. **Exactly-Once Processing (E1P)** — Data integrity guarantee; each event processed exactly once
2. **Fault-Tolerant Architecture** — Automatic failover and recovery
3. **Data Lineage/Tracking** — Know where data came from and how it was transformed
4. **Monitoring & Logging** — Real-time visibility into pipeline execution
5. **Scalable Architecture** — Handle varying loads
6. **Self-Service Management** — Easy pipeline creation and modification
7. **Multi-Format Support** — Structured, semi-structured, unstructured data
**Our coverage:**
- E1P: Achieved via idempotent execution
- Fault-Tolerance: Checkpoint/resume covers this
- Lineage: State tracking provides basic lineage
- Monitoring: JSON output + console logging
- Scalability: Not a concern (batch processing, single user)
- Self-Service: SKILL.md specifications enable easy understanding
- Multi-Format: Video + text + images supported
## Sources
- [Striim: 7 Must-Have Features of Modern Data Pipelines](https://www.striim.com/blog/what-is-a-data-pipeline-and-must-have-features-of-modern-data-pipelines/) - HIGH confidence
- [Dagster: Data Pipeline Frameworks Key Features 2025](https://dagster.io/guides/data-pipeline-frameworks-key-features-10-tools-to-know-in-2025) - HIGH confidence
- [Microsoft Azure: AI Agent Orchestration Patterns](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns) - HIGH confidence
- [Medium: Idempotency Guide for Data Pipelines](https://medium.com/@QuarkAndCode/idempotency-guide-iac-apis-data-pipelines-temporal-workflows-8d85dc280613) - MEDIUM confidence
- [Fivetran: Data Pipeline State Management](https://www.fivetran.com/blog/data-pipeline-state-management-an-underappreciated-challenge) - MEDIUM confidence
- PROJECT.md and ARCHITECTURE.md from codebase analysis - HIGH confidence
---
*Feature research for: Skill-based Workflow Orchestration (Content Processing)*
*Researched: 2026-03-13*
.planning/research/PITFALLS.md
+310
View File
@@ -0,0 +1,310 @@
# Pitfalls Research
**Domain:** Skill-based Workflow Orchestration (LLM Agent Pipeline)
**Researched:** 2026-03-13
**Confidence:** HIGH
---
## Critical Pitfalls
### Pitfall 1: Statelessness = Restart from Scratch
**What goes wrong:**
系统没有持久化状态，任何失败都需要从头重新执行整个流程。Whisper 识别了 20 分钟，后续步骤失败？重新识别。下载了 2GB 视频，分析失败？重新下载。
**Why it happens:**
开发者低估了多步骤流程的失败概率。假设每步都成功，或者认为"重新执行一次很快"。实际上，Whisper 识别可能需要 30 分钟，视频下载可能需要 10 分钟，重复执行成本极高。
**How to avoid:**
- 每个主要步骤完成后，将状态写入 `workflow_state.json`
- 状态包含：步骤名、输入参数、输出路径、完成时间戳
- 重新执行时，检查状态文件，跳过已完成的步骤
- 使用幂等设计：重复执行同一链接不会产生重复文件
**Warning signs:**
- 没有状态文件或状态文件只在流程结束时写入
- 重复执行同一任务产生重复输出文件
- 用户抱怨"又要重新跑一遍"
**Phase to address:**
Phase 1 (PIPE-02: 步骤级状态管理, PIPE-04: 幂等执行保证)
---
### Pitfall 2: Error Propagation Without Context
**What goes wrong:**
错误在步骤链中传播时丢失了上下文。步骤 6 失败时报告 "AI API 调用失败"，但不知道是哪个内容、哪个步骤、什么输入导致的。调试需要从头重现。
**Why it happens:**
错误处理只关注"失败了"，没有保留足够上下文。LLM 调用失败时只记录 "API error"，没有记录请求内容、模型参数、重试次数等。
**How to avoid:**
- 每个步骤输出结构化 JSON，包含 `success`、`error`、`context` 字段
- 错误消息必须包含：步骤名、输入摘要、失败原因、可操作建议
- 使用 correlation ID 追踪整个流程
- 区分可重试错误（超时、503）和不可重试错误（403、验证失败）
**Warning signs:**
- 错误消息只有 "Error" 或 "Failed"
- 需要问用户"你输入的是什么？"来调试
- 同样的错误重复出现但不知道根本原因
**Phase to address:**
Phase 1 (PIPE-05: 错误恢复策略, PIPE-07: Schema 验证机制)
---
### Pitfall 3: Assuming Model "Understands Context"
**What goes wrong:**
主代理期望子代理"理解"整个流程的上下文，但实际上每个子代理只看到自己步骤的输入。kb-deep-analyzer 不知道之前的 Whisper 识别质量如何，只能假设输入是可靠的。
**Why it happens:**
人类可以自然地维持上下文，开发者误以为 LLM 也能做到。但每次子代理调用都是独立的会话，没有跨步骤的"记忆"。
**How to avoid:**
- 显式传递所有需要的上下文，不要假设子代理"知道"
- 每个步骤的输入 JSON 必须包含该步骤需要的所有信息
- 使用结构化的输入格式（Schema），而不是自然语言描述
- 在状态文件中记录关键决策和上下文
**Warning signs:**
- 子代理行为不符合预期，"它应该知道..."
- 需要阅读多个文档才能理解一个步骤的输入要求
- 步骤之间的"隐式约定"没有文档化
**Phase to address:**
Phase 1 (PIPE-03: 统一 JSON Schema, PIPE-06: 清晰的流程定义)
---
### Pitfall 4: Parallel Chaos (Uncoordinated Parallel Execution)
**What goes wrong:**
多个子代理同时执行但没有协调机制。结果：文件被覆盖、状态不一致、资源竞争。例如：同时下载和 OCR 处理同一视频导致文件损坏。
**Why it happens:**
并行看起来更"高效"，开发者没有考虑共享资源（文件、状态、API 配额）的并发访问问题。
**How to avoid:**
- 对于这个项目：**串行执行**。每个步骤完成后才开始下一个
- 如果未来需要并行：使用文件锁、状态锁、或任务队列
- 明确定义哪些资源是共享的，哪些步骤可以并行
**Warning signs:**
- 偶发的"文件不存在"或"文件损坏"错误
- 状态文件内容不符合预期
- 同一时间大量 API 调用导致限流
**Phase to address:**
Phase 1 (设计文档中明确：串行执行模式)
---
### Pitfall 5: "Automating Bad Processes"
**What goes wrong:**
手动流程本身有问题（不清晰、有冗余、有错误），自动化只是让错误发生得更快。例如：元数据提取逻辑有 bug，自动化后每个视频都提取错误信息。
**Why it happens:**
急于自动化，没有先验证和优化手动流程。"先写代码再说"的心态。
**How to avoid:**
- 先手动执行完整流程一次，记录每一步
- 验证每个步骤的输出是否符合预期
- 修复流程问题后再自动化
- 每个步骤有明确的成功标准，不只是"没有报错"
**Warning signs:**
- 自动化后发现输出质量不如手动
- 需要频繁修改代码来"修复"流程
- 用户仍然需要手动检查和修正输出
**Phase to address:**
Phase 1 之前：手动验证流程 (PROJECT.md 中已验证的技能列表)
---
### Pitfall 6: Environment Variable Sprawl
**What goes wrong:**
环境变量分散在多个地方：`~/.kb_env`、子技能自己的配置、HF_TOKEN 在子章节、ZHIHU_COOKIE 在另一个文件。配置遗漏或冲突时，错误难以定位。
**Why it happens:**
每个技能独立开发，各自管理配置。没有统一的配置策略。
**How to avoid:**
- 所有环境变量集中在一个文件 (`~/.kb_env`)
- 工作流启动时检查所有必需变量，缺失则报错退出
- 文档中明确列出所有变量及其用途
- 可选变量有明确的默认值和回退策略
**Warning signs:**
- "在我的机器上能跑"问题
- 需要检查多个文件才能找到配置
- 同一变量在不同地方有不同值
**Phase to address:**
Phase 1 (环境变量检查已在 SKILL.md 中定义，需要统一执行)
---
### Pitfall 7: Graceful Degradation That Isn't
**What goes wrong:**
声称有"降级策略"，但实际上降级到无意义的内容。例如：深度分析失败时返回一个模板文本，看起来"成功"了但没有实际价值。
**Why it happens:**
开发者不想让流程"失败"，认为返回一些内容比报错好。但这掩盖了问题，用户得到的是无用的输出。
**How to avoid:**
- **关键步骤必须成功，否则整个流程失败** (PROJECT.md 已定义)
- 区分"必须有"和"最好有"的功能
- 失败时明确报告失败原因，不要假装成功
- 降级策略只用于非关键功能（如：OCR 失败时跳过图片分析）
**Warning signs:**
- 输出看起来"正常"但没有实际内容
- 用户发现分析报告是模板文本
- 成功率统计很高但用户不满意
**Phase to address:**
Phase 1 (SKILL.md 中的"严格失败处理原则"需要严格执行)
---
## Technical Debt Patterns
Shortcuts that seem reasonable but create long-term problems.
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|----------|-------------------|----------------|-----------------|
| 跳过 Schema 验证 | 快速开发，减少代码 | 后续调试困难，接口不兼容问题频发 | Never |
| 状态只在内存中 | 简单实现 | 无法恢复，每次失败重头开始 | 仅限一次性任务 |
| 硬编码路径 | 不用配置 | 迁移困难，环境差异问题 | Never |
| 自然语言描述步骤输入 | 灵活，文档友好 | LLM 理解不一致，调试困难 | 仅限人类阅读的文档 |
| 跳过错误处理 | 节省开发时间 | 生产环境频繁崩溃 | 仅限原型/POC |
| 不记录中间结果 | 节省存储 | 无法断点续传，无法审计 | 仅限幂等操作 |
---
## Integration Gotchas
Common mistakes when connecting to external services.
| Integration | Common Mistake | Correct Approach |
|-------------|----------------|------------------|
| B站 API | 没有处理 403/超时 | 实现重试机制，区分可重试和不可重试错误 |
| Whisper | 假设输出完美 | 后处理清理（数字格式、标点、分段） |
| AI API (GLM-5) | 单次调用失败即退出 | 指数退避重试 (1s → 2s → 4s → 8s) |
| HuggingFace | 没有 HF_TOKEN 仍尝试说话者分离 | 检测 token 存在，无则降级到普通识别 |
| 知乎 | Cookie 过期导致失败 | 明确错误消息，提示用户更新 Cookie |
| Git 归档 | 没有配置用户信息 | 检查 KB_GIT_USER/EMAIL，缺失则跳过 Git |
---
## Performance Traps
Patterns that work at small scale but fail as usage grows.
| Trap | Symptoms | Prevention | When It Breaks |
|------|----------|------------|----------------|
| 内存中累积所有中间结果 | 内存占用持续增长 | 每步完成后写入文件，释放内存 | 处理 1 小时以上视频 |
| 同步等待 Whisper 完成 | 用户长时间无反馈 | 定时汇报进度（每 2-3 分钟） | 超过 10 分钟的任务 |
| 单次下载整个视频 | 网络超时，下载失败 | 分块下载，支持断点续传 | 大于 500MB 的视频 |
| 无限制重试 | 无限循环，资源浪费 | 最大重试次数 + 总超时时间 | 任何外部服务调用 |
---
## Security Mistakes
Domain-specific security issues beyond general web security.
| Mistake | Risk | Prevention |
|---------|------|------------|
| API Key 硬编码 | 泄露到 Git 仓库 | 使用环境变量，.gitignore 排除配置文件 |
| Cookie 明文存储 | 账号被盗风险 | 提醒用户 Cookie 敏感性，不记录到日志 |
| 用户输入直接拼接到命令 | 命令注入 | 验证输入格式，使用参数化调用 |
| 日志记录敏感信息 | 信息泄露 | 过滤 API Key、Cookie 等敏感字段 |
---
## "Looks Done But Isn't" Checklist
Things that appear complete but are missing critical pieces.
- [ ] **State Management:** Often missing `workflow_state.json` — verify 状态文件在每个步骤后更新
- [ ] **Error Handling:** Often missing 可重试/不可重试区分 — verify 错误消息包含可操作建议
- [ ] **Schema Validation:** Often missing 输出验证 — verify 每个步骤输出符合定义的 Schema
- [ ] **Environment Check:** Often missing 启动前检查 — verify 工作流开始时验证所有必需变量
- [ ] **Success Criteria:** Often missing 明确标准 — verify 每个步骤有可验证的成功条件
- [ ] **Resume Capability:** Often missing 断点续传 — verify 中断后可以从最后完成步骤继续
- [ ] **Idempotency:** Often missing 幂等保证 — verify 重复执行不产生重复文件
---
## Recovery Strategies
When pitfalls occur despite prevention, how to recover.
| Pitfall | Recovery Cost | Recovery Steps |
|---------|---------------|----------------|
| 无状态导致重新执行 | HIGH | 1. 识别已完成的步骤（通过输出文件） 2. 手动创建状态文件 3. 从断点继续 |
| 错误上下文丢失 | MEDIUM | 1. 检查所有输出文件 2. 重新执行失败的步骤（带详细日志） 3. 对比成功和失败案例 |
| 配置分散导致问题 | LOW | 1. 创建统一的 ~/.kb_env 2. 迁移所有配置 3. 验证每个技能 |
| 并发问题 | MEDIUM | 1. 停止所有执行 2. 清理临时文件 3. 串行重新执行 |
| 降级处理不当 | LOW | 1. 删除无效输出 2. 修复根本问题 3. 重新执行 |
---
## Pitfall-to-Phase Mapping
How roadmap phases should address these pitfalls.
| Pitfall | Prevention Phase | Verification |
|---------|------------------|--------------|
| Statelessness = Restart from Scratch | Phase 1 (PIPE-02, PIPE-04) | 中断执行后可以从断点继续 |
| Error Propagation Without Context | Phase 1 (PIPE-05, PIPE-07) | 错误消息包含完整上下文和可操作建议 |
| Assuming Model "Understands Context" | Phase 1 (PIPE-03, PIPE-06) | 每个步骤输入 JSON 自包含所有需要信息 |
| Parallel Chaos | Phase 1 (设计文档) | 明确文档：串行执行，无并发 |
| "Automating Bad Processes" | Phase 0 (验证阶段) | 手动执行完整流程，验证每步输出 |
| Environment Variable Sprawl | Phase 1 (环境检查) | 启动时验证所有必需变量，缺失则报错 |
| Graceful Degradation That Isn't | Phase 1 (失败处理原则) | 关键步骤失败 = 整体失败，不假装成功 |
---
## Phase-Specific Warnings
| Phase Topic | Likely Pitfall | Mitigation |
|-------------|---------------|------------|
| 管道模型定义 | 步骤边界模糊 | 明确定义每个步骤的输入、输出、前置条件 |
| 状态管理实现 | 状态格式不一致 | 使用统一的 JSON Schema |
| Schema 设计 | 过于复杂或过于简单 | 从现有输出反推，迭代优化 |
| 幂等实现 | 判断"已存在"的逻辑有漏洞 | 使用内容哈希或唯一标识符 |
| 错误恢复 | 重试导致无限循环 | 设置最大重试次数和总超时 |
| 代码重构 | 破坏现有功能 | 先写测试，小步重构 |
---
## Sources
- [AI Agent Anti-Patterns: Architectural Pitfalls](https://achan2013.medium.com/ai-agent-anti-patterns-part-1-architectural-pitfalls-that-break-enterprise-agents-before-they-32d211dded43) - MEDIUM confidence
- [Demystifying Failures in Platform-Orchestrated Workflows](https://arxiv.org/html/2509.23735v2) - HIGH confidence
- [10 Reasons Multi-Agent Workflows Fail](https://www.infoq.com/presentations/multi-agent-workflow/) - MEDIUM confidence
- [How Tool Chaining Fails in Production LLM Agents](https://futureagi.substack.com/p/how-tool-chaining-fails-in-production) - MEDIUM confidence
- [Why Multi-Agent LLM Systems Fail](https://orq.ai/blog/why-do-multi-agent-llm-systems-fail) - MEDIUM confidence
- [Multi-Step LLM Chains: Best Practices](https://deepchecks.com/orchestrating-multi-step-llm-chains-best-practices/) - HIGH confidence
- [Data Pipeline State Management: An Underappreciated Challenge](https://www.fivetran.com/blog/data-pipeline-state-management-an-underappreciated-challenge) - HIGH confidence
- [9 Things I Hate About Environment Variables](https://prefab.cloud/blog/9-problems-with-env-vars/) - HIGH confidence
- [Mastering Retry Logic Agents: 2025 Best Practices](https://sparkco.ai/blog/mastering-retry-logic-agents-a-deep-dive-into-2025-best-practices) - MEDIUM confidence
- [Structured Handling of Exceptions in LLM-Driven Agentic Workflows](https://arxiv.org/pdf/2508.07935) - HIGH confidence
- [Subagent orchestration: The complete 2025 guide](https://www.eesel.ai/blog/subagent-orchestration) - MEDIUM confidence
- [How Idempotency Keeps Your API Out of Trouble](https://medium.com/@shreyagupta3011/how-idempotency-keeps-your-api-out-of-trouble-7f7848188d39) - HIGH confidence
- [AI Agent 工作流设计的5 个反模式](https://www.moltbook.com/post/f826b987-641d-4bb5-9938-b825ac5a6b13) - MEDIUM confidence
- [Workflow Process Automation Gone Wrong](https://www.flowwright.com/workflow-process-automation-gone-wrong-lessons-learned) - MEDIUM confidence
---
*Pitfalls research for: Skill-based Workflow Orchestration*
*Researched: 2026-03-13*
.planning/research/STACK.md
+164
View File
@@ -0,0 +1,164 @@
# Stack Research
**Domain:** Skill-based Workflow Orchestration System (Python)
**Researched:** 2026-03-13
**Confidence:** HIGH
## Recommended Stack
### Core Technologies
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| **Python** | 3.12+ | Runtime | 项目已使用，3.12 提供更好的性能和错误消息；类型提示增强；f-string 改进 |
| **Pydantic** | 2.11+ | 数据验证 & Schema | v2.11+ 提供更快的解析速度、更低的内存占用；原生支持 JSON Schema 生成；与项目 PIPE-07 需求完美匹配 |
| **pydantic-settings** | 2.8+ | 配置管理 | Pydantic 官方配置方案；支持环境变量、.env 文件、类型安全；解决现有 ~/.kb_env 分散配置问题 |
| **transitions** | 0.9.3+ | 状态机 | 轻量级、面向对象的状态机实现；支持回调、条件转换、嵌套状态；无需数据库，适合步骤级状态管理 |
| **Typer** | 0.15+ | CLI 框架 | 基于 Click 的现代 CLI 库；利用 Python 类型提示自动生成参数；比 argparse 更简洁，比 Click 更现代 |
| **Rich** | 14.1+ | 终端输出美化 | 提供彩色输出、进度条、表格；与 Typer 无缝集成；改善 CLI 用户体验 |
### Supporting Libraries
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| **httpx** | 0.28+ | 异步 HTTP 客户端 | 现有项目已使用；替代 requests 用于需要异步的场景 |
| **tenacity** | 8.5+ | 重试机制 | 实现错误恢复策略 (PIPE-05)；支持指数退避、条件重试 |
| **python-dotenv** | 1.0+ | .env 文件加载 | 与 pydantic-settings 配合；可选，pydantic-settings 已内置支持 |
| **loguru** | 0.7+ | 日志 | 替代 stdlib logging；更简洁的 API；内置旋转、压缩、格式化 |
### Development Tools
| Tool | Purpose | Notes |
|------|---------|-------|
| **uv** | 依赖管理 | 10-100x 快于 pip；兼容 pyproject.toml；支持 lockfile |
| **pytest** | 8.3+ | 测试框架 | 现有项目已使用；配合 pytest-cov 做覆盖率 |
| **ruff** | 0.9+ | Linter + Formatter | 替代 flake8 + black + isort；速度快 |
## Installation
```bash
# Core dependencies
uv add pydantic pydantic-settings transitions typer rich
# Supporting dependencies
uv add httpx tenacity loguru
# Dev dependencies
uv add --dev pytest pytest-cov ruff
```
## Alternatives Considered
| Recommended | Alternative | When to Use Alternative |
|-------------|-------------|-------------------------|
| **transitions** | Prefect 3.0 | 当需要分布式执行、复杂 DAG、Web UI 监控时。但对于本项目的轻量级单机管道，Prefect 过于重量级 |
| **transitions** | Dagster | 当需要数据资产血缘追踪、强类型管道定义时。学习曲线较陡 |
| **transitions** | 纯 JSON 状态 | 当状态极其简单且不需要状态机特性时。但会失去转换验证、回调等安全特性 |
| **Typer** | Click | 当需要更底层的控制或 Typer 的类型推断有问题时。Typer 底层就是 Click |
| **Pydantic** | jsonschema 库 | 当需要验证外部 JSON Schema 且不关心 Python 对象映射时。Pydantic 更 Pythonic |
| **uv** | Poetry | 当团队已熟悉 Poetry 工作流时。但 uv 更快且兼容性更好 |
## What NOT to Use
| Avoid | Why | Use Instead |
|-------|-----|-------------|
| **Airflow** | 需要数据库后端；DAG 定义繁琐；过度工程化 | transitions + 简单状态文件 |
| **Celery** | 需要 message broker；面向任务队列而非工作流编排 | transitions（状态机模式） |
| **argparse** | 样板代码多；无自动帮助生成 | Typer |
| **dataclasses** | 无运行时验证；无 JSON Schema 支持 | Pydantic BaseModel |
| **requests** | 同步阻塞；httpx 提供同步+异步统一 API | httpx |
| **logging (stdlib)** | 配置繁琐；API 不友好 | loguru |
## Stack Patterns by Variant
**If 需要持久化工作流状态到数据库:**
- 添加 SQLAlchemy 2.0+ 作为 ORM
- 配合 alembic 做迁移
- 因为: transitions 状态可以序列化到数据库
**If 需要并发处理多个链接:**
- 使用 asyncio + httpx 异步模式
- 配合 transitions 的 AsyncMachine 扩展
- 因为: 提高吞吐量而不增加复杂度
**If 需要可视化工作流状态:**
- 使用 transitions 的 GraphMachine 扩展
- 生成 Mermaid 或 Graphviz 图表
- 因为: 内置支持，无需额外工具
## Version Compatibility
| Package A | Compatible With | Notes |
|----------|-----------------|-------|
| pydantic 2.11+ | pydantic-settings 2.8+ | 同一组织维护，版本同步更新 |
| typer 0.15+ | rich 14.1+ | Typer 内置 Rich 集成 |
| transitions 0.9.3+ | Python 3.12+ | 支持 Python 3.13 |
## Example Architecture
### 工作流状态 Schema
```python
from pydantic import BaseModel
from enum import Enum
from datetime import datetime
class StepStatus(str, Enum):
PENDING = "pending"
RUNNING = "running"
SUCCESS = "success"
SKIPPED = "skipped"
FAILED = "failed"
class WorkflowState(BaseModel):
link: str
current_step: str
step_status: dict[str, StepStatus] # 步骤名 -> 状态
artifacts: dict[str, str] # 步骤名 -> 输出文件路径
errors: list[str] # 错误历史
created_at: datetime
updated_at: datetime
```
### 管道步骤 Schema 设计
```python
from pydantic import BaseModel
from typing import Literal
class StepInput(BaseModel):
link: str
work_dir: str
# 通用字段...
class StepOutput(BaseModel):
status: Literal["success", "skipped", "failed"]
artifacts: dict[str, str] # 输出文件路径
error: str | None = None
```
### CLI 入口设计
```python
import typer
from rich.console import Console
app = typer.Typer()
console = Console()
@app.command()
def process(link: str, resume: bool = False):
"""处理单个链接"""
# transitions 状态机驱动
```
## Sources
- [Pydantic v2.11 Release](https://pydantic.dev/articles/pydantic-v2-11-release) — 版本特性确认 (HIGH confidence)
- [Pydantic JSON Schema Docs](https://docs.pydantic.dev/latest/concepts/json_schema/) — JSON Schema 生成能力 (HIGH confidence)
- [transitions 0.9.3 on PyPI](https://pypi.org/project/transitions/) — 状态机库详情 (HIGH confidence)
- [Prefect vs Dagster Comparison](https://www.prefect.io/compare/dagster) — 编排工具对比 (MEDIUM confidence)
- [Python CLI Tools Comparison 2025](https://dasroot.net/posts/2025/12/building-cli-tools-python-click-typer-argparse/) — CLI 框架对比 (MEDIUM confidence)
- [uv Official Docs](https://docs.astral.sh/uv/) — 包管理器最佳实践 (HIGH confidence)
- [pydantic-settings Guide](https://levelup.gitconnected.com/pydantic-settings-2025-a-clean-way-to-handle-configs-f1c432030085) — 配置管理方案 (MEDIUM confidence)
---
*Stack research for: Skill-based Workflow Orchestration System*
*Researched: 2026-03-13*
.planning/research/SUMMARY.md
+188
View File
@@ -0,0 +1,188 @@
# Project Research Summary
**Project:** kb-skills (Skill-based Workflow Orchestration System)
**Domain:** Content Processing Pipeline with LLM Agent Orchestration
**Researched:** 2026-03-13
**Confidence:** HIGH
## Executive Summary
这是一个技能型工作流编排系统，专门用于内容处理管道（视频/文章的元数据提取、ASR识别、AI深度分析）。专家构建此类系统时，通常采用分层架构将编排层、技能定义层、执行层和基础设施层分离，并使用步骤级状态管理实现断点续传和幂等执行。
基于研究，推荐采用**串行管道编排模式**，使用 `transitions` 状态机管理步骤状态，`Pydantic` 定义 Schema 并实现验证。核心原则是：每个步骤完成后立即持久化状态，失败时可以从断点恢复，重复执行不会产生重复文件。这与 Airflow/Prefect 等重量级编排工具不同——本项目专注于单机批处理场景，无需分布式执行。
关键风险在于**无状态执行**和**错误上下文丢失**。长时间运行的任务（Whisper识别可能需要30分钟）如果失败后需要从头开始，成本极高。缓解措施是在 Phase 1 就实现 `workflow_state.json` 状态文件，并确保每个步骤输出结构化错误信息。
## Key Findings
### Recommended Stack
推荐使用 Python 3.12+ 作为运行时，配合 Pydantic 2.11+ 进行数据验证和 JSON Schema 生成，transitions 0.9.3+ 实现步骤状态机，Typer + Rich 构建现代化 CLI。这套组合轻量、Pythonic，与项目的 SKILL.md 规范模式完美契合。
**Core technologies:**
- **Pydantic 2.11+**: 数据验证 & Schema 生成 — v2.11+ 解析速度更快，原生支持 JSON Schema，与 PIPE-07 需求匹配
- **transitions 0.9.3+**: 状态机 — 轻量级、支持回调和嵌套状态，无需数据库
- **pydantic-settings 2.8+**: 配置管理 — 解决现有 `~/.kb_env` 分散配置问题
- **Typer 0.15+**: CLI 框架 — 基于 Click 的现代 CLI，利用类型提示自动生成参数
- **tenacity 8.5+**: 重试机制 — 实现指数退避重试，处理 API 调用失败
**避免使用:** Airflow/Celery（过重）、argparse（样板代码多）、requests（同步阻塞）
### Expected Features
**Must have (table stakes):**
- **步骤级状态管理** — 用户期望能从断点恢复，不重新执行已完成的步骤
- **幂等执行** — 相同输入重复执行产生相同结果，无重复文件
- **JSON Schema 验证** — 数据契约确保步骤间接口一致性
- **结构化错误处理** — 错误消息包含上下文和可操作建议
- **平台特定内容处理** — B站/知乎元数据和内容提取
**Should have (competitive):**
- **SKILL.md 声明式技能定义** — AI 代理可发现和执行技能
- **多模态内容处理** — 视频(ASR/OCR)+文本统一管道
- **Tmux 会话持久化** — 长时间运行操作可断开终端
**Defer (v2+):**
- **并发执行** — 当前串行模式足够，并发增加复杂度
- **Web UI** — CLI 优先，可视化需求低
- **多租户** — 单用户工具
### Architecture Approach
推荐四层架构：编排层（Workflow Coordinator、State Manager、Error Handler）、技能定义层（SKILL.md specs、Schema Definitions）、执行层（Platform/Processing/Analysis Skills）、基础设施层（File System、External APIs、AI Services）。
**Major components:**
1. **State Manager** — 管理 `workflow_state.json`，实现步骤级状态持久化和原子写入
2. **Schema Registry** — 统一的输入输出 JSON Schema 定义和验证
3. **Pipeline Engine** — 串行执行技能，检查状态跳过已完成步骤
4. **Platform Skills** — B站/知乎的元数据提取和内容下载
5. **Processing Skills** — Whisper/FunASR 语音识别、OCR 处理
6. **Analysis Skills** — kb-deep-analyzer AI 深度分析
### Critical Pitfalls
1. **无状态执行 = 从头重来** — Whisper 识别 20 分钟后失败，没有状态文件则需要重新识别。避免：每个步骤完成后写入 `workflow_state.json`
2. **错误上下文丢失** — 步骤 6 失败只报告 "API error"，不知道是哪个内容、什么输入。避免：结构化 JSON 错误，包含步骤名、输入摘要、失败原因、可操作建议
3. **假设模型"理解上下文"** — 主代理期望子代理知道之前发生了什么，但每次调用是独立会话。避免：显式传递所有需要的上下文，输入 JSON 自包含
4. **非幂等操作** — 每次执行创建 `output_1.mp4`、`output_2.mp4`。避免：基于输入生成确定性文件名，已存在则跳过
5. **环境变量分散** — `~/.kb_env`、子技能配置、HF_TOKEN 分散各处。避免：统一配置文件，启动时检查所有必需变量
## Implications for Roadmap
Based on research, suggested phase structure:
### Phase 1: Foundation (基础设施层)
**Rationale:** 状态管理和 Schema 验证是所有后续组件的基础，必须首先实现。没有状态文件就无法实现断点续传和幂等执行。
**Delivers:** State Manager、Schema Registry、Error Handler、统一配置管理
**Addresses:** 步骤级状态管理、幂等执行、JSON Schema 验证、结构化错误处理
**Avoids:** 无状态执行、错误上下文丢失、环境变量分散
**Components:**
- `skills/core/state_manager.py` — workflow_state.json 管理
- `skills/core/schema_registry.py` — JSON Schema 定义和验证
- `skills/core/error_handler.py` — 统一错误处理
- `config.py` — pydantic-settings 配置管理
### Phase 2: Core Skills (核心技能集成)
**Rationale:** 基于现有技能进行集成，将分散的技能统一到管道模型中。Link Parser 是入口点，无依赖；Platform Skills 依赖其输出。
**Delivers:** Link Parser 集成、B站/知乎 Metadata 和 Content 技能适配
**Uses:** Pydantic (Schema)、transitions (状态)、Typer (CLI)
**Implements:** Pipeline Engine 原型
**Components:**
- Link Parser 技能适配（已有 kb-link-parser）
- B站/知乎技能适配（已有 kb-bilibili-*、kb-zhihu-*）
- 管道执行引擎原型
### Phase 3: Processing Skills (处理技能集成)
**Rationale:** ASR 和 OCR 依赖视频文件，必须在内容下载后执行。这些技能已有实现，需要集成到管道。
**Delivers:** Whisper/FunASR 集成、OCR 集成、处理技能状态管理
**Addresses:** 多模态内容处理
**Avoids:** 假设模型理解上下文 — 显式传递转录结果给分析步骤
**Components:**
- Whisper/FunASR 技能适配
- OCR 技能适配
- 处理结果 Schema 定义
### Phase 4: Analysis & Orchestration (分析技能与编排)
**Rationale:** 深度分析是管道的最终输出，依赖所有前置步骤。编排层整合所有技能形成完整工作流。
**Delivers:** Deep Analyzer 集成、完整管道编排、工作流定义文件
**Uses:** 所有前置组件
**Implements:** 声明式工作流定义 (YAML)
**Components:**
- Deep Analyzer 技能适配
- Pipeline Engine 完善
- `workflows/video_pipeline.yaml`
- `workflows/article_pipeline.yaml`
### Phase 5: Polish & Testing (完善与测试)
**Rationale:** 核心功能完成后，补充测试、文档和边缘情况处理。
**Delivers:** 单元测试、集成测试、错误恢复策略完善、用户文档
**Addresses:** 错误恢复策略、进度可见性
### Phase Ordering Rationale
- **依赖关系决定顺序:** State Manager 是所有组件的基础，必须 Phase 1；Analysis 依赖 Processing，Processing 依赖 Platform Skills
- **架构模式指导分组:** Foundation -> Skills -> Orchestration 符合分层架构
- **避免陷阱:** Phase 1 就实现状态文件和 Schema，防止无状态执行和接口不一致
### Research Flags
Phases likely needing deeper research during planning:
- **Phase 3:** ASR/OCR 集成可能需要研究 FunASR vs Whisper 选择逻辑，以及 HuggingFace token 缺失时的降级策略
- **Phase 4:** AI API 调用的重试策略和超时配置可能需要根据实际使用调整
Phases with standard patterns (skip research-phase):
- **Phase 1:** 状态管理和 Schema 验证是成熟模式，Pydantic/transitions 文档充分
- **Phase 2:** 现有技能适配，无需额外研究
- **Phase 5:** 测试和文档是标准实践
## Confidence Assessment
| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | Pydantic/transitions/Typer 都是成熟库，官方文档完善，与项目需求匹配度高 |
| Features | HIGH | 基于现有代码库分析和行业标准，功能边界清晰 |
| Architecture | HIGH | 四层架构是工作流系统的标准模式，管道编排模式适合内容处理场景 |
| Pitfalls | HIGH | 基于多篇研究论文和行业实践，坑点识别充分 |
**Overall confidence:** HIGH
### Gaps to Address
- **ASR 后端选择:** Whisper vs FunASR 的选择逻辑需要根据实际性能测试确定，目前建议支持配置切换
- **错误恢复策略细节:** 403/超时等具体错误类型的处理策略需要在实现时细化
- **进度汇报机制:** 长时间运行的 Whisper 任务需要进度汇报，具体实现方式待定
## Sources
### Primary (HIGH confidence)
- Pydantic v2.11 Release & JSON Schema Docs — 技术选型验证
- transitions 0.9.3 on PyPI — 状态机库详情
- Dagster: Data Pipeline Architecture Design Patterns — 管道模式
- AI Architecture Patterns 101: Workflows, Agents — 架构模式
- Demystifying Failures in Platform-Orchestrated Workflows (arxiv) — 失败模式研究
### Secondary (MEDIUM confidence)
- Prefect vs Dagster Comparison — 编排工具对比
- Multi-Step LLM Chains: Best Practices — LLM 链最佳实践
- Data Pipeline State Management (Fivetran) — 状态管理挑战
- 9 Things I Hate About Environment Variables — 配置管理问题
### Tertiary (LOW confidence)
- AI Agent Anti-Patterns (Medium) — 反模式识别，需要结合实践验证
---
*Research completed: 2026-03-13*
*Ready for roadmap: yes*