# Stack Research

**Domain:** Skill-based Workflow Orchestration System (Python)
**Researched:** 2026-03-18 (Updated for v1.3 Anti-Crawler)
**Confidence:** HIGH

## Recommended Stack

### Core Technologies

| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| **Python** | 3.12+ | Runtime | 项目已使用，3.12 提供更好的性能和错误消息；类型提示增强；f-string 改进 |
| **Pydantic** | 2.11+ | 数据验证 & Schema | v2.11+ 提供更快的解析速度、更低的内存占用；原生支持 JSON Schema 生成；与项目 PIPE-07 需求完美匹配 |
| **pydantic-settings** | 2.8+ | 配置管理 | Pydantic 官方配置方案；支持环境变量、.env 文件、类型安全；解决现有 ~/.kb_env 分散配置问题 |
| **transitions** | 0.9.3+ | 状态机 | 轻量级、面向对象的状态机实现；支持回调、条件转换、嵌套状态；无需数据库，适合步骤级状态管理 |
| **Typer** | 0.15+ | CLI 框架 | 基于 Click 的现代 CLI 库；利用 Python 类型提示自动生成参数；比 argparse 更简洁，比 Click 更现代 |
| **Rich** | 14.1+ | 终端输出美化 | 提供彩色输出、进度条、表格；与 Typer 无缝集成；改善 CLI 用户体验 |

### v1.3 Anti-Crawler Libraries (NEW)

| Library | Version | Purpose | Why Recommended |
|---------|---------|---------|-----------------|
| **aiolimiter** | 1.2.1 | asyncio 限流 | 实现 Leaky Bucket 算法；与现有 asyncio 架构完美配合；零依赖；比手动 sleep 更精确控制请求速率 |
| **playwright-stealth** | 2.0.2 | 浏览器指纹伪装 | 隐藏 Playwright 自动化特征；防止 webdriver 检测；简单易用（一行代码集成）；与现有 Playwright 代码兼容 |
| **camoufox** | 0.4.11 | 高级反指纹 (Phase 2) | 基于 Firefox 的反检测浏览器；比 playwright-stealth 更强；适合知乎/B站高安全场景；可选依赖 |

### Supporting Libraries

| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| **httpx** | 0.28+ | 异步 HTTP 客户端 | 现有项目已使用；替代 requests 用于需要异步的场景 |
| **tenacity** | 8.5+ | 重试机制 | 实现错误恢复策略 (PIPE-05)；支持指数退避、条件重试；v1.3 增强：添加 wait_random 抖动 |
| **python-dotenv** | 1.0+ | .env 文件加载 | 与 pydantic-settings 配合；可选，pydantic-settings 已内置支持 |
| **loguru** | 0.7+ | 日志 | 替代 stdlib logging；更简洁的 API；内置旋转、压缩、格式化 |
| **Playwright** | 1.50+ | 浏览器自动化 | 现有项目已使用；支持代理、API 拦截、隐身模式 |

### Development Tools

| Tool | Purpose | Notes |
|------|---------|-------|
| **uv** | 依赖管理 | 10-100x 快于 pip；兼容 pyproject.toml；支持 lockfile |
| **pytest** | 8.3+ | 测试框架 | 现有项目已使用；配合 pytest-cov 做覆盖率 |
| **ruff** | 0.9+ | Linter + Formatter | 替代 flake8 + black + isort；速度快 |

## Installation

```bash
# Core dependencies
uv add pydantic pydantic-settings transitions typer rich

# Supporting dependencies
uv add httpx tenacity loguru

# v1.3 Anti-Crawler (NEW)
uv add aiolimiter playwright-stealth

# v1.3 Optional: Advanced anti-fingerprinting (Phase 2)
uv add camoufox  # 仅在 playwright-stealth 不足时使用

# Dev dependencies
uv add --dev pytest pytest-cov ruff
```

## Alternatives Considered

### Anti-Crawler Alternatives

| Recommended | Alternative | Why Not |
|-------------|-------------|---------|
| **aiolimiter** | asyncio.sleep | 手动 sleep 不精确；无法应对并发场景；无速率保证 |
| **aiolimiter** | ratelimit 库 | 仅支持同步；不支持 asyncio |
| **aiolimiter** | aiometer | 功能过多（并发限制、批处理）；项目只需速率限制 |
| **playwright-stealth** | undetected-chromedriver | 仅支持 Selenium；需要 Chrome；与 Playwright 不兼容 |
| **playwright-stealth** | puppeteer-stealth | JavaScript 生态；不适用于 Python |
| **camoufox** | Selenium Wire | 过于底层；需要手动配置代理和请求修改 |
| **自定义 ProxyPool** | rotating-proxy 库 | 代码质量差；维护不活跃；不支持自定义代理源 |
| **自定义 ProxyPool** | scrapy-rotating-proxies | 与 Scrapy 绑定；不适用于 Playwright |
| **CAPTCHA 检测 + 告警** | 2captcha / YesCaptcha | PROJECT.md 明确排除付费打码服务 |
| **CAPTCHA 检测 + 告警** | ddddocr | OCR 识别率不稳定；验证码类型多变 |

### General Alternatives

| Recommended | Alternative | When to Use Alternative |
|-------------|-------------|-------------------------|
| **transitions** | Prefect 3.0 | 当需要分布式执行、复杂 DAG、Web UI 监控时。但对于本项目的轻量级单机管道，Prefect 过于重量级 |
| **transitions** | Dagster | 当需要数据资产血缘追踪、强类型管道定义时。学习曲线较陡 |
| **transitions** | 纯 JSON 状态 | 当状态极其简单且不需要状态机特性时。但会失去转换验证、回调等安全特性 |
| **Typer** | Click | 当需要更底层的控制或 Typer 的类型推断有问题时。Typer 底层就是 Click |
| **Pydantic** | jsonschema 库 | 当需要验证外部 JSON Schema 且不关心 Python 对象映射时。Pydantic 更 Pythonic |
| **uv** | Poetry | 当团队已熟悉 Poetry 工作流时。但 uv 更快且兼容性更好 |

## What NOT to Use

| Avoid | Why | Use Instead |
|-------|-----|-------------|
| **Airflow** | 需要数据库后端；DAG 定义繁琐；过度工程化 | transitions + 简单状态文件 |
| **Celery** | 需要 message broker；面向任务队列而非工作流编排 | transitions（状态机模式） |
| **argparse** | 样板代码多；无自动帮助生成 | Typer |
| **dataclasses** | 无运行时验证；无 JSON Schema 支持 | Pydantic BaseModel |
| **requests** | 同步阻塞；httpx 提供同步+异步统一 API | httpx |
| **logging (stdlib)** | 配置繁琐；API 不友好 | loguru |
| **rotating-proxy** | 代码质量差；2020年后无更新；不支持自定义源 | 自定义 ProxyPoolConfig |
| **scrapy-rotating-proxies** | 与 Scrapy 紧密绑定 | 自定义 ProxyPoolConfig |
| **2captcha/YesCaptcha** | 付费服务；PROJECT.md 明确排除 | CAPTCHA 检测 + 告警 + 暂停 |
| **ddddocr** | 识别率不稳定；维护不活跃 | CAPTCHA 检测 + 告警 |

## Stack Patterns by Variant

**If 需要持久化工作流状态到数据库:**
- 添加 SQLAlchemy 2.0+ 作为 ORM
- 配合 alembic 做迁移
- 因为: transitions 状态可以序列化到数据库

**If 需要并发处理多个链接:**
- 使用 asyncio + httpx 异步模式
- 配合 transitions 的 AsyncMachine 扩展
- 因为: 提高吞吐量而不增加复杂度

**If 需要可视化工作流状态:**
- 使用 transitions 的 GraphMachine 扩展
- 生成 Mermaid 或 Graphviz 图表
- 因为: 内置支持，无需额外工具

**If playwright-stealth 被检测 (Phase 2):**
- 切换到 camoufox
- 配合现有的代理配置
- 因为: Firefox 内核更难被指纹识别

## Version Compatibility

| Package A | Compatible With | Notes |
|----------|-----------------|-------|
| pydantic 2.11+ | pydantic-settings 2.8+ | 同一组织维护，版本同步更新 |
| typer 0.15+ | rich 14.1+ | Typer 内置 Rich 集成 |
| transitions 0.9.3+ | Python 3.12+ | 支持 Python 3.13 |
| aiolimiter 1.2.1 | asyncio | 零外部依赖，纯 Python |
| playwright-stealth 2.0.2 | Playwright 1.40+ | 已测试兼容 |
| camoufox 0.4.11 | Python 3.10+ | 支持 Python 3.12+ |

## Example Architecture

### 工作流状态 Schema
```python
from pydantic import BaseModel
from enum import Enum
from datetime import datetime

class StepStatus(str, Enum):
    PENDING = "pending"
    RUNNING = "running"
    SUCCESS = "success"
    SKIPPED = "skipped"
    FAILED = "failed"

class WorkflowState(BaseModel):
    link: str
    current_step: str
    step_status: dict[str, StepStatus]  # 步骤名 -> 状态
    artifacts: dict[str, str]           # 步骤名 -> 输出文件路径
    errors: list[str]                   # 错误历史
    created_at: datetime
    updated_at: datetime
```

### 管道步骤 Schema 设计
```python
from pydantic import BaseModel
from typing import Literal

class StepInput(BaseModel):
    link: str
    work_dir: str
    # 通用字段...

class StepOutput(BaseModel):
    status: Literal["success", "skipped", "failed"]
    artifacts: dict[str, str]  # 输出文件路径
    error: str | None = None
```

### CLI 入口设计
```python
import typer
from rich.console import Console

app = typer.Typer()
console = Console()

@app.command()
def process(link: str, resume: bool = False):
    """处理单个链接"""
    # transitions 状态机驱动
```

### v1.3 Anti-Crawler 集成示例

#### aiolimiter 限流
```python
from aiolimiter import AsyncLimiter
import asyncio

# 10 请求/分钟 = 1 请求/6秒
limiter = AsyncLimiter(10, 60)

async def fetch_with_rate_limit(url: str):
    async with limiter:
        # 请求会被自动限流
        return await httpx_client.get(url)
```

#### playwright-stealth 集成
```python
from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # 应用隐身补丁
    stealth_sync(page)

    # 正常使用，检测不到 webdriver
    page.goto("https://www.zhihu.com")
```

#### tenacity 抖动增强
```python
from tenacity import retry, stop_after_attempt, wait_exponential, wait_random

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10) + wait_random(0, 1),
    reraise=True
)
async def fetch_with_jitter(url: str):
    # 指数退避 + 随机抖动，避免请求同步
    return await httpx_client.get(url)
```

#### 代理池配置
```python
from pydantic import BaseModel
from typing import Optional
import random

class ProxyPoolConfig(BaseModel):
    """代理池配置"""
    proxies: list[str]  # ["http://ip:port", ...]
    strategy: str = "round_robin"  # round_robin | random

    def get_next_proxy(self) -> Optional[str]:
        if not self.proxies:
            return None
        if self.strategy == "random":
            return random.choice(self.proxies)
        # round_robin
        proxy = self.proxies.pop(0)
        self.proxies.append(proxy)
        return proxy
```

#### CAPTCHA 检测
```python
from dataclasses import dataclass
from enum import Enum

class CaptchaType(str, Enum):
    SLIDER = "slider"      # 滑块验证码
    CLICK = "click"        # 点选验证码
    TEXT = "text"          # 文字验证码
    UNKNOWN = "unknown"

@dataclass
class CaptchaDetection:
    detected: bool
    captcha_type: CaptchaType
    confidence: float  # 0.0 - 1.0

def detect_captcha(page_content: str) -> CaptchaDetection:
    """检测页面是否包含验证码"""
    captcha_indicators = [
        ("滑块", CaptchaType.SLIDER),
        ("验证码", CaptchaType.UNKNOWN),
        ("geetest", CaptchaType.SLIDER),
        ("点击", CaptchaType.CLICK),
    ]

    for indicator, captcha_type in captcha_indicators:
        if indicator in page_content:
            return CaptchaDetection(
                detected=True,
                captcha_type=captcha_type,
                confidence=0.8
            )

    return CaptchaDetection(detected=False, captcha_type=CaptchaType.UNKNOWN, confidence=0.0)
```

## Sources

### Core Stack (HIGH confidence)
- [Pydantic v2.11 Release](https://pydantic.dev/articles/pydantic-v2-11-release) — 版本特性确认
- [Pydantic JSON Schema Docs](https://docs.pydantic.dev/latest/concepts/json_schema/) — JSON Schema 生成能力
- [transitions 0.9.3 on PyPI](https://pypi.org/project/transitions/) — 状态机库详情
- [uv Official Docs](https://docs.astral.sh/uv/) — 包管理器最佳实践

### Anti-Crawler (HIGH confidence)
- [aiolimiter on PyPI](https://pypi.org/project/aiolimiter/) — asyncio 限流库
- [aiolimiter GitHub](https://github.com/mjpieters/aiolimiter) — 文档和示例
- [playwright-stealth on PyPI](https://pypi.org/project/playwright-stealth/) — 浏览器隐身库
- [playwright-stealth GitHub](https://github.com/AtuboLabs/playwright_stealth) — 源码和使用指南
- [camoufox on PyPI](https://pypi.org/project/camoufox/) — Firefox 反检测浏览器
- [camoufox GitHub](https://github.com/daijro/camoufox) — 文档和配置选项

### Comparisons (MEDIUM confidence)
- [Prefect vs Dagster Comparison](https://www.prefect.io/compare/dagster) — 编排工具对比
- [Python CLI Tools Comparison 2025](https://dasroot.net/posts/2025/12/building-cli-tools-python-click-typer-argparse/) — CLI 框架对比
- [pydantic-settings Guide](https://levelup.gitconnected.com/pydantic-settings-2025-a-clean-way-to-handle-configs-f1c432030085) — 配置管理方案
- [tenacity wait_random documentation](https://tenacity.readthedocs.io/) — 抖动策略

---
*Stack research for: Skill-based Workflow Orchestration System*
*Researched: 2026-03-13, Updated: 2026-03-18 for v1.3 Anti-Crawler*