# Architecture Research

**Domain:** Skill-based Workflow Orchestration System
**Researched:** 2026-03-13 (updated 2026-03-18 for v1.3)
**Confidence:** HIGH

---

## v1.3 补充：反爬虫优化架构

**Milestone:** Anti-Crawler Optimization
**Researched:** 2026-03-18
**Focus:** 代理池、行为伪装、验证码检测、错误恢复

---

## Executive Summary

本文档研究如何将反爬虫优化功能集成到现有的 kb-skills 架构中。基于对现有代码结构和反爬虫最佳实践的分析，推荐采用 **模块化的中间件/装饰器模式**，将反爬虫功能作为独立组件注入到现有的 Pipeline 和 Crawler 中。

关键架构决策：
1. **代理池管理** — 新建 `kb_workflow/crawler/proxy_pool.py` 模块，作为独立服务
2. **行为伪装** — 通过装饰器/中间件模式集成到 Playwright 调用
3. **验证码处理** — 检测-暂停机制，不自动解决（避免付费服务依赖）
4. **错误恢复** — 增强现有 `retry.py`，添加 jitter 和状态感知

---

## Current Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────────┐
│                              CLI Layer                                   │
│  kb_workflow/cli.py - process/login subcommands                          │
├─────────────────────────────────────────────────────────────────────────┤
│                            Pipeline Layer                                │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐         │
│  │ ParseProcessor  │→ │DownloadProcessor│→ │  ASRProcessor   │ → ...   │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘         │
│           │                    │                    │                   │
│           ↓                    ↓                    ↓                   │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │              Platform Adapters (BilibiliAdapter, ZhihuAdapter)   │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                            Crawler Layer                                 │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuCrawler (Playwright) - API Interception + Cookie Auth     │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                          Infrastructure                                  │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐              │
│  │ CrawlerConfig │  │   KBSettings  │  │  WorkflowState│              │
│  └───────────────┘  └───────────────┘  └───────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Recommended New Architecture (v1.3)

```
┌─────────────────────────────────────────────────────────────────────────┐
│                              CLI Layer                                   │
│  kb_workflow/cli.py - process/login subcommands                          │
├─────────────────────────────────────────────────────────────────────────┤
│                            Pipeline Layer                                │
│  [Existing processors - unchanged]                                       │
├─────────────────────────────────────────────────────────────────────────┤
│                     Platform Adapters Layer                              │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuAdapter / BilibiliAdapter                                   │   │
│  │  (uses @with_anti_detection decorator)                           │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                       Anti-Crawler Layer (NEW)                           │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐         │
│  │   ProxyPool     │  │BehaviorCamouflag│  │ CaptchaDetector │         │
│  │   Manager       │  │     e           │  │                 │         │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘         │
│           │                    │                    │                   │
│           └────────────────────┴────────────────────┘                   │
│                                │                                         │
│                    ┌───────────┴───────────┐                            │
│                    │  @with_anti_detection │                            │
│                    │     (decorator)       │                            │
│                    └───────────────────────┘                            │
├─────────────────────────────────────────────────────────────────────────┤
│                            Crawler Layer                                 │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuCrawler (enhanced with anti-detection hooks)              │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                          Infrastructure                                  │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐              │
│  │CrawlerConfig  │  │AntiCrawlerConf│  │  KBSettings   │              │
│  │  (enhanced)   │  │    (NEW)      │  │  (enhanced)   │              │
│  └───────────────┘  └───────────────┘  └───────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Component Design

### 1. Proxy Pool Manager

**Location:** `kb_workflow/crawler/proxy_pool.py`

**Rationale:** 独立模块，与现有 `CrawlerConfig` 分离，因为：
- 代理池是状态性的（需要跟踪健康状态、使用次数）
- 可能被多个 Crawler 共享
- 需要独立的配置源（环境变量、文件、API）

**Design:**

```python
# kb_workflow/crawler/proxy_pool.py

from typing import Optional, List, Dict, Any
from pydantic import BaseModel
import random
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class ProxyInfo:
    """代理信息"""
    url: str
    protocol: str = "http"  # http, https, socks5
    username: Optional[str] = None
    password: Optional[str] = None
    last_used: Optional[datetime] = None
    fail_count: int = 0
    success_count: int = 0
    is_healthy: bool = True

    def to_playwright_format(self) -> Dict[str, Any]:
        config = {"server": self.url}
        if self.username:
            config["username"] = self.username
        if self.password:
            config["password"] = self.password
        return config


class ProxyPoolConfig(BaseModel):
    """代理池配置"""
    enabled: bool = False
    # 代理列表（支持多个代理）
    proxies: List[str] = []
    # 轮换策略: round_robin, random, least_used
    rotation_strategy: str = "random"
    # 健康检查
    health_check_enabled: bool = True
    max_fail_count: int = 3  # 连续失败 N 次后标记为不健康
    # 冷却时间（秒）
    cooldown_seconds: float = 0  # 同一代理两次使用之间的最小间隔

    @classmethod
    def from_env(cls) -> "ProxyPoolConfig":
        """从环境变量加载代理池配置"""
        import os
        proxies_str = os.environ.get("KB_PROXY_POOL", "")
        proxies = [p.strip() for p in proxies_str.split(",") if p.strip()]
        return cls(
            enabled=len(proxies) > 0,
            proxies=proxies,
            rotation_strategy=os.environ.get("KB_PROXY_STRATEGY", "random"),
        )


class ProxyPoolManager:
    """代理池管理器

    职责：
    - 管理代理列表
    - 轮换策略
    - 健康状态跟踪
    """

    def __init__(self, config: Optional[ProxyPoolConfig] = None):
        self.config = config or ProxyPoolConfig.from_env()
        self._proxies: List[ProxyInfo] = []
        self._current_index = 0
        self._init_proxies()

    def _init_proxies(self):
        """初始化代理列表"""
        for proxy_url in self.config.proxies:
            self._proxies.append(ProxyInfo(url=proxy_url))

    def get_proxy(self) -> Optional[ProxyInfo]:
        """获取下一个可用代理"""
        healthy_proxies = [p for p in self._proxies if p.is_healthy]
        if not healthy_proxies:
            return None

        if self.config.rotation_strategy == "round_robin":
            proxy = healthy_proxies[self._current_index % len(healthy_proxies)]
            self._current_index += 1
        else:  # random
            proxy = random.choice(healthy_proxies)

        proxy.last_used = datetime.now()
        return proxy

    def report_success(self, proxy_url: str):
        """报告代理使用成功"""
        for p in self._proxies:
            if p.url == proxy_url:
                p.success_count += 1
                p.fail_count = 0  # 重置失败计数
                break

    def report_failure(self, proxy_url: str):
        """报告代理使用失败"""
        for p in self._proxies:
            if p.url == proxy_url:
                p.fail_count += 1
                if p.fail_count >= self.config.max_fail_count:
                    p.is_healthy = False
                break


# Module-level singleton
_proxy_pool: Optional[ProxyPoolManager] = None

def get_proxy_pool() -> ProxyPoolManager:
    """获取代理池管理器单例"""
    global _proxy_pool
    if _proxy_pool is None:
        _proxy_pool = ProxyPoolManager()
    return _proxy_pool
```

---

### 2. Behavior Camouflage

**Location:** `kb_workflow/crawler/behavior.py`

**Rationale:** 行为伪装逻辑应该：
- 可以独立测试
- 可以被装饰器或直接调用
- 支持可配置的随机性

**Design:**

```python
# kb_workflow/crawler/behavior.py

import random
import time
from typing import Optional, Callable
from functools import wraps
from pydantic import BaseModel, Field


class BehaviorConfig(BaseModel):
    """行为伪装配置"""
    # 请求间隔
    min_delay: float = Field(default=1.0, description="最小延迟（秒）")
    max_delay: float = Field(default=3.0, description="最大延迟（秒）")

    # Jitter 配置（防止模式检测）
    jitter_enabled: bool = True
    jitter_range: float = Field(default=0.3, description="抖动范围（+/- 比例）")

    # 人类模拟
    mouse_movement_enabled: bool = False  # Playwright 场景下可选
    scroll_simulation_enabled: bool = False

    # User-Agent 随机化
    user_agent_rotation: bool = True


# 常用 User-Agent 池
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
]


class BehaviorCamouflage:
    """行为伪装管理器"""

    def __init__(self, config: Optional[BehaviorConfig] = None):
        self.config = config or BehaviorConfig()

    def get_random_delay(self) -> float:
        """获取随机延迟时间（带 jitter）"""
        base_delay = random.uniform(self.config.min_delay, self.config.max_delay)

        if self.config.jitter_enabled:
            jitter = base_delay * self.config.jitter_range * random.uniform(-1, 1)
            base_delay += jitter

        return max(0.1, base_delay)  # 确保非负

    def apply_delay(self):
        """应用随机延迟"""
        delay = self.get_random_delay()
        time.sleep(delay)
        return delay

    def get_random_user_agent(self) -> str:
        """获取随机 User-Agent"""
        if self.config.user_agent_rotation:
            return random.choice(USER_AGENTS)
        return USER_AGENTS[0]

    def get_playwright_context_options(self) -> dict:
        """获取 Playwright context 选项（包含反检测设置）"""
        options = {
            "user_agent": self.get_random_user_agent(),
        }
        return options


def with_behavior_camouflage(config: Optional[BehaviorConfig] = None):
    """装饰器：为函数添加行为伪装

    用于同步函数，在调用前添加随机延迟。

    Example:
        @with_behavior_camouflage()
        def fetch_page(url):
            return requests.get(url)
    """
    camouflage = BehaviorCamouflage(config)

    def decorator(func: Callable):
        @wraps(func)
        def wrapper(*args, **kwargs):
            camouflage.apply_delay()
            return func(*args, **kwargs)
        return wrapper

    return decorator
```

---

### 3. CAPTCHA Detection

**Location:** `kb_workflow/crawler/captcha.py`

**Rationale:**
- **不实现自动解决** — 根据项目约束（Out of Scope: 付费打码服务）
- **检测+暂停机制** — 检测到验证码时暂停并通知用户

**Design:**

```python
# kb_workflow/crawler/captcha.py

import re
from typing import Optional, Tuple
from enum import Enum
from dataclasses import dataclass
from pydantic import BaseModel

from kb_workflow.pipeline.errors import KBError, ErrorType


class CaptchaType(str, Enum):
    """验证码类型"""
    RECAPTCHA_V2 = "recaptcha_v2"
    RECAPTCHA_V3 = "recaptcha_v3"
    HCAPTCHA = "hcaptcha"
    CLOUDFLARE = "cloudflare"
    IMAGE_CAPTCHA = "image_captcha"
    SLIDER = "slider"
    UNKNOWN = "unknown"


@dataclass
class CaptchaDetectionResult:
    """验证码检测结果"""
    detected: bool
    captcha_type: Optional[CaptchaType] = None
    confidence: float = 0.0
    indicators: list = None

    def __post_init__(self):
        if self.indicators is None:
            self.indicators = []


class CaptchaDetector:
    """验证码检测器

    职责：
    - 检测页面是否包含验证码
    - 识别验证码类型
    - 不负责解决验证码（需要人工介入）
    """

    # 验证码指示器（页面元素/URL/内容）
    CAPTCHA_INDICATORS = {
        CaptchaType.RECAPTCHA_V2: [
            "g-recaptcha",
            "recaptcha-checkbox",
            "google.com/recaptcha",
        ],
        CaptchaType.RECAPTCHA_V3: [
            "grecaptcha.execute",
            "recaptcha-v3",
        ],
        CaptchaType.HCAPTCHA: [
            "h-captcha",
            "hcaptcha.com",
        ],
        CaptchaType.CLOUDFLARE: [
            "cloudflare",
            "cf-browser-verification",
            "challenge-platform",
        ],
        CaptchaType.IMAGE_CAPTCHA: [
            "captcha.png",
            "verifycode",
            "img captcha",
        ],
        CaptchaType.SLIDER: [
            "slider",
            "slide-verify",
            "drag verify",
        ],
    }

    def detect_from_html(self, html_content: str) -> CaptchaDetectionResult:
        """从 HTML 内容检测验证码"""
        html_lower = html_content.lower()
        indicators = []

        for captcha_type, patterns in self.CAPTCHA_INDICATORS.items():
            for pattern in patterns:
                if pattern.lower() in html_lower:
                    indicators.append(pattern)
                    return CaptchaDetectionResult(
                        detected=True,
                        captcha_type=captcha_type,
                        confidence=0.8,
                        indicators=indicators,
                    )

        return CaptchaDetectionResult(detected=False)

    def detect_from_response(
        self,
        status_code: int,
        headers: dict,
        url: str
    ) -> CaptchaDetectionResult:
        """从 HTTP 响应检测验证码"""
        # 403 通常表示被拦截或需要验证
        if status_code == 403:
            return CaptchaDetectionResult(
                detected=True,
                captcha_type=CaptchaType.UNKNOWN,
                confidence=0.6,
                indicators=["403 Forbidden"],
            )

        # 检查 URL 重定向
        if "captcha" in url.lower() or "verify" in url.lower():
            return CaptchaDetectionResult(
                detected=True,
                captcha_type=CaptchaType.UNKNOWN,
                confidence=0.7,
                indicators=[f"URL redirect: {url}"],
            )

        return CaptchaDetectionResult(detected=False)


class CaptchaError(KBError):
    """验证码错误

    当检测到验证码时抛出，需要人工介入。
    """

    def __init__(
        self,
        step_name: str,
        input_summary: str,
        captcha_type: CaptchaType,
        suggestion: str = "请手动完成验证码后重试"
    ):
        super().__init__(
            message=f"检测到验证码: {captcha_type.value}",
            error_type=ErrorType.AUTH,
            step_name=step_name,
            input_summary=input_summary,
            suggestion=suggestion,
        )
        self.captcha_type = captcha_type
```

---

### 4. Enhanced Retry Mechanism

**Location:** `kb_workflow/pipeline/retry.py` (enhance existing)

**Rationale:**
- 现有实现使用 tenacity，已经很好
- 增强点：添加 jitter、状态感知、验证码感知

**Enhanced Design:**

```python
# kb_workflow/pipeline/retry.py (enhancements)

import random
import logging
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type,
    before_sleep_log,
    wait_combine,
    wait_random,
)

from .errors import NetworkError, AuthError

logger = logging.getLogger(__name__)


def wait_exponential_jitter(
    multiplier: float = 1,
    min: float = 2,
    max: float = 10,
    jitter: float = 0.5
):
    """指数退避 + Jitter

    Args:
        multiplier: 退避乘数
        min: 最小等待时间
        max: 最大等待时间
        jitter: 随机抖动范围（0-1）
    """
    return wait_combine(
        wait_exponential(multiplier=multiplier, min=min, max=max),
        wait_random(0, jitter)
    )


# 增强版重试装饰器（带 jitter）
retry_on_network_error_with_jitter = retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
    retry=retry_if_exception_type(NetworkError),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
"""带 Jitter 的网络错误重试装饰器

在指数退避基础上增加随机抖动，避免多个请求同时重试的"惊群效应"。
"""


# 验证码感知重试（遇到 CaptchaError 不重试）
retry_on_transient_error = retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
    retry=(
        retry_if_exception_type(NetworkError) &
        ~retry_if_exception_type(AuthError)  # AuthError 不重试
    ),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
"""瞬态错误重试装饰器

只重试网络错误，不重试认证错误（包括验证码）。
"""
```

---

### 5. Unified Anti-Detection Decorator

**Location:** `kb_workflow/crawler/anti_detection.py`

**Rationale:**
- 统一入口点，组合所有反爬虫功能
- 便于在 PlatformAdapter 中使用

**Design:**

```python
# kb_workflow/crawler/anti_detection.py

from typing import Optional, Callable, Dict, Any
from functools import wraps
from dataclasses import dataclass

from .proxy_pool import ProxyPoolManager, get_proxy_pool, ProxyInfo
from .behavior import BehaviorCamouflage, BehaviorConfig
from .captcha import CaptchaDetector, CaptchaError, CaptchaType


@dataclass
class AntiDetectionConfig:
    """反检测配置"""
    proxy_enabled: bool = True
    behavior_camouflage_enabled: bool = True
    captcha_detection_enabled: bool = True
    behavior_config: Optional[BehaviorConfig] = None


class AntiDetectionContext:
    """反检测上下文

    在一次爬取过程中传递的状态信息。
    """
    def __init__(self):
        self.proxy: Optional[ProxyInfo] = None
        self.delays_applied: list = []
        self.captcha_detected: bool = False


def with_anti_detection(config: Optional[AntiDetectionConfig] = None):
    """装饰器：集成所有反检测功能

    组合使用：
    1. 代理池轮换
    2. 行为伪装
    3. 验证码检测

    Example:
        @with_anti_detection()
        async def fetch_content(self, url: str, context: AntiDetectionContext = None):
            # context.proxy 可用于 Playwright
            # 延迟已自动应用
            ...
    """
    _config = config or AntiDetectionConfig()
    behavior = BehaviorCamouflage(_config.behavior_config)
    proxy_pool = get_proxy_pool() if _config.proxy_enabled else None
    captcha_detector = CaptchaDetector() if _config.captcha_detection_enabled else None

    def decorator(func: Callable):
        @wraps(func)
        async def async_wrapper(*args, **kwargs):
            context = AntiDetectionContext()

            # 1. 获取代理
            if proxy_pool and proxy_pool.config.enabled:
                context.proxy = proxy_pool.get_proxy()

            # 2. 应用行为伪装延迟
            if _config.behavior_camouflage_enabled:
                delay = behavior.apply_delay()
                context.delays_applied.append(delay)

            # 3. 执行原始函数
            try:
                result = await func(*args, context=context, **kwargs)

                # 报告代理成功
                if context.proxy and proxy_pool:
                    proxy_pool.report_success(context.proxy.url)

                return result

            except Exception as e:
                # 报告代理失败
                if context.proxy and proxy_pool:
                    proxy_pool.report_failure(context.proxy.url)
                raise

        @wraps(func)
        def sync_wrapper(*args, **kwargs):
            context = AntiDetectionContext()

            # 1. 获取代理
            if proxy_pool and proxy_pool.config.enabled:
                context.proxy = proxy_pool.get_proxy()

            # 2. 应用行为伪装延迟
            if _config.behavior_camouflage_enabled:
                delay = behavior.apply_delay()
                context.delays_applied.append(delay)

            # 3. 执行原始函数
            try:
                result = func(*args, context=context, **kwargs)

                if context.proxy and proxy_pool:
                    proxy_pool.report_success(context.proxy.url)

                return result

            except Exception as e:
                if context.proxy and proxy_pool:
                    proxy_pool.report_failure(context.proxy.url)
                raise

        # 根据函数类型选择包装器
        import asyncio
        if asyncio.iscoroutinefunction(func):
            return async_wrapper
        return sync_wrapper

    return decorator
```

---

## Integration Points

### Integration with ZhihuCrawler

**Modified File:** `kb_workflow/crawler/zhihu_crawler.py`

**Changes:**

```python
# In ZhihuCrawler.__init__:
from .anti_detection import AntiDetectionConfig, get_proxy_pool

def __init__(
    self,
    headless: bool = True,
    timeout: int = 30,
    config: Optional[CrawlerConfig] = None,
    anti_detection_config: Optional[AntiDetectionConfig] = None
):
    self.headless = headless
    self.timeout = timeout
    self.config = config or get_crawler_config()
    self.anti_detection_config = anti_detection_config or AntiDetectionConfig()
    # ... rest of init

# In _start_browser_with_proxy:
def _start_browser_with_proxy(self, cookie: str) -> None:
    # Use proxy pool if enabled
    if self.anti_detection_config.proxy_enabled:
        proxy_pool = get_proxy_pool()
        if proxy_pool.config.enabled:
            proxy = proxy_pool.get_proxy()
            if proxy:
                self.config.proxy = ProxyConfig(
                    server=proxy.url,
                    username=proxy.username,
                    password=proxy.password
                )
    # ... rest of method
```

### Integration with ZhihuAdapter

**Modified File:** `kb_workflow/pipeline/platforms/zhihu.py`

**Changes:**

```python
from kb_workflow.crawler.anti_detection import (
    with_anti_detection,
    AntiDetectionContext,
    AntiDetectionConfig
)

class ZhihuAdapter(PlatformAdapter):
    def __init__(
        self,
        headless: bool = True,
        timeout: int = 30,
        output_dir: Optional[Path] = None,
        idempotency_checker: Optional[IdempotencyChecker] = None,
        anti_detection_config: Optional[AntiDetectionConfig] = None
    ):
        # ... existing init
        self.anti_detection_config = anti_detection_config or AntiDetectionConfig()

    @with_anti_detection()
    async def fetch_metadata(
        self,
        content_id: str,
        content_type: str,
        source_url: str = None,
        context: AntiDetectionContext = None  # 注入的上下文
    ) -> Dict[str, Any]:
        # Use context.proxy if needed
        # ...
```

### Integration with Configuration

**Modified File:** `kb_workflow/pipeline/config.py`

**Changes:**

```python
class KBSettings(BaseSettings):
    # ... existing fields

    # Anti-crawler settings
    proxy_pool: str = Field(
        default="",
        description="代理池配置（逗号分隔的代理URL）"
    )
    anti_detection_enabled: bool = Field(
        default=True,
        description="是否启用反检测功能"
    )
```

---

## Data Flow

### Request Flow with Anti-Detection

```
[CLI] process command
    |
    v
[Pipeline] DownloadProcessor.process()
    |
    v
[ZhihuAdapter] @with_anti_detection decorator
    |
    +-> [ProxyPool] get_proxy() -> returns available proxy
    |
    +-> [BehaviorCamouflage] apply_delay() -> wait random time
    |
    +-> [ZhihuCrawler] crawl_answer(url, proxy=...)
            |
            +-> [Playwright] start browser with proxy
            |
            +-> [CaptchaDetector] detect_from_html() -> check for captcha
            |       |
            |       +-> If detected -> raise CaptchaError
            |
            +-> Return CrawlResult
    |
    v
[ProxyPool] report_success() / report_failure()
    |
    v
[Pipeline] continue to next processor
```

### Error Recovery Flow

```
[Request fails]
    |
    v
[Retry Logic] retry_on_transient_error
    |
    +-> Is NetworkError? -> Retry with exponential backoff + jitter
    |
    +-> Is CaptchaError? -> Do NOT retry, surface to user
    |
    +-> Is AuthError? -> Do NOT retry, trigger QR login
    |
    v
[After max retries] -> Log error, update state, raise to caller
```

---

## Recommended Build Order

基于依赖关系，建议按以下顺序实现：

### Phase 1: Foundation (No external dependencies)
1. **`kb_workflow/crawler/behavior.py`** — 行为伪装模块
   - 独立模块，无外部依赖
   - 可独立测试

2. **`kb_workflow/crawler/captcha.py`** — 验证码检测模块
   - 独立模块，只需要 HTML 解析
   - 定义 CaptchaError 错误类型

### Phase 2: Integration
3. **`kb_workflow/crawler/proxy_pool.py`** — 代理池管理
   - 依赖配置系统
   - 需要与 CrawlerConfig 集成

4. **`kb_workflow/pipeline/retry.py` (enhance)** — 增强重试机制
   - 添加 jitter 支持
   - 添加验证码感知

### Phase 3: Unified API
5. **`kb_workflow/crawler/anti_detection.py`** — 统一装饰器
   - 组合所有组件
   - 提供 `@with_anti_detection` 装饰器

### Phase 4: Platform Integration
6. **`kb_workflow/crawler/zhihu_crawler.py` (modify)** — 集成到 ZhihuCrawler
7. **`kb_workflow/pipeline/platforms/zhihu.py` (modify)** — 集成到 ZhihuAdapter
8. **`kb_workflow/pipeline/config.py` (modify)** — 添加配置项

### Phase 5: Bilibili Support (if needed)
9. **`kb_workflow/pipeline/platforms/bilibili.py` (modify)** — B站反爬适配

---

## Anti-Patterns (v1.3 Specific)

### Anti-Pattern 1: Hardcoding Proxy Settings in Crawler

**What people do:** 在 Crawler 类中硬编码代理配置

```python
# BAD
class ZhihuCrawler:
    def __init__(self):
        self.proxy = "http://127.0.0.1:7890"  # Hardcoded
```

**Why it's wrong:** 无法动态切换代理，无法支持代理池

**Do this instead:** 使用独立的 ProxyPoolManager

---

### Anti-Pattern 2: Retry All Errors Blindly

**What people do:** 对所有错误都进行重试

```python
# BAD
@retry(stop=stop_after_attempt(3))
def crawl():
    ...
```

**Why it's wrong:** 验证码错误重试会立即再次触发验证码，造成无限循环

**Do this instead:** 使用错误类型过滤

```python
# GOOD
@retry(retry=retry_if_exception_type(NetworkError))
def crawl():
    ...
```

---

### Anti-Pattern 3: Fixed Delay Intervals

**What people do:** 使用固定延迟

```python
# BAD
time.sleep(2)  # Fixed 2 second delay
```

**Why it's wrong:** 固定延迟容易被模式检测识别

**Do this instead:** 使用随机延迟 + jitter

```python
# GOOD
delay = random.uniform(1, 3) + random.uniform(-0.5, 0.5)
time.sleep(delay)
```

---

### Anti-Pattern 4: Captcha Auto-Solve Without Fallback

**What people do:** 依赖第三方验证码解决服务，没有降级方案

**Why it's wrong:** 增加成本，服务不可用时无法工作

**Do this instead:** 检测 + 暂停 + 通知用户（本项目约束）

---

## Scaling Considerations (v1.3)

| Scale | Architecture Adjustments |
|-------|--------------------------|
| 个人使用（当前） | 单进程，内存代理池，本地配置 |
| 小规模（10-100 URL/天） | 添加代理持久化，简单的健康检查 |
| 中规模（100-1000 URL/天） | 考虑使用代理池服务 API，分布式状态存储 |

### Scaling Priorities

1. **First bottleneck:** 代理被封 — 通过代理池轮换解决
2. **Second bottleneck:** 账号被封 — 通过行为伪装和降低频率解决
3. **Third bottleneck:** 验证码频繁 — 考虑暂停机制 + 人工介入

---

## v1.3 Sources

- [ZenRows - How to Avoid Bot Detection with Playwright](https://www.zenrows.com/blog/avoid-playwright-bot-detection) - HIGH confidence
- [Zyte - Best Proxy Providers for Web Scraping 2026](https://www.zyte.com/learn/best-proxy-providers-for-web-scraping/) - HIGH confidence
- [Rebrowser - Python Requests Retry Guide](https://rebrowser.net/blog/python-requests-retry-the-ultimate-guide-to-handling-failed-http-requests-in-python) - HIGH confidence
- [GitHub - backoff Python library](https://github.com/litl/backoff) - HIGH confidence
- [Stack Overflow - How to rotate proxies in Python](https://stackoverflow.com/questions/55872164/how-to-rotate-proxies-on-a-python-requests) - HIGH confidence
- [Medium - Exponential Backoff Decorator](https://medium.com/@suryasekhar/implementing-exponential-backoff-decorator-in-python-26ddf783aea0) - MEDIUM confidence
- [Trickster.dev - Custom Scrapy Middleware for Proxy Pool](https://www.trickster.dev/post/writing-custom-scrapy-middleware-for-proxy-pool-integration/) - HIGH confidence

---

## Original Architecture Research (v1.0)

*Below is the original architecture research from v1.0, preserved for reference.*

---

## Standard Architecture

### System Overview

技能型工作流编排系统采用分层架构，将处理逻辑、状态管理和执行协调分离。基于当前研究和行业最佳实践，推荐以下四层架构：

```
+---------------------------------------------------------------+
|                    Orchestration Layer                        |
|    [Workflow Coordinator] [State Manager] [Error Handler]     |
+---------------------------------------------------------------+
|                    Skill Definition Layer                     |
|    [SKILL.md specs] [Schema Definitions] [Trigger Rules]      |
+---------------------------------------------------------------+
|                    Execution Layer                            |
|    [Platform Skills] [Processing Skills] [Analysis Skills]    |
+---------------------------------------------------------------+
|                    Infrastructure Layer                       |
|    [File System] [External APIs] [AI Services] [State Store]  |
+---------------------------------------------------------------+
```

### Component Responsibilities

| Component | Responsibility | Typical Implementation |
|-----------|----------------|------------------------|
| Workflow Coordinator | 协调多个技能的执行顺序、依赖管理 | kb-workflow 主控模块 |
| State Manager | 步骤级状态持久化、断点续传支持 | workflow_state.json + 原子写入 |
| Error Handler | 错误分类、降级策略、重试逻辑 | 统一错误处理模块 |
| Skill Registry | 技能发现、能力声明、Schema 注册 | SKILL.md + JSON Schema |
| Platform Skills | 平台特定内容获取（B站/知乎） | kb-bilibili-*, kb-zhihu-* |
| Processing Skills | 内容转换（语音识别/OCR） | kb-whisper, kb-funasr, kb-ocr |
| Analysis Skills | AI 深度分析、报告生成 | kb-deep-analyzer |

## Recommended Project Structure

```
kb-skills/
+-- skills/                      # 技能模块目录
|   +-- core/                    # 核心基础设施
|   |   +-- state_manager.py     # 状态管理
|   |   +-- schema_registry.py   # Schema 注册
|   |   +-- pipeline.py          # 管道执行引擎
|   |   +-- error_handler.py     # 错误处理
|   |
|   +-- platforms/               # 平台技能
|   |   +-- bilibili/            # B站技能组
|   |   |   +-- SKILL.md
|   |   |   +-- metadata.py
|   |   |   +-- content.py
|   |   +-- zhihu/               # 知乎技能组
|   |       +-- SKILL.md
|   |       +-- metadata.py
|   |       +-- content.py
|   |
|   +-- processing/              # 处理技能
|   |   +-- whisper/             # 语音识别
|   |   +-- funasr/              # 中文语音识别
|   |   +-- ocr/                 # OCR 处理
|   |
|   +-- analysis/                # 分析技能
|       +-- deep_analyzer/       # 深度分析
|
+-- schemas/                     # JSON Schema 定义
|   +-- workflow_state.schema.json
|   +-- skill_output.schema.json
|   +-- platform_metadata.schema.json
|
+-- workflows/                   # 工作流定义
|   +-- video_pipeline.yaml      # 视频处理流程
|   +-- article_pipeline.yaml    # 文章处理流程
|
+-- cli.py                       # 统一 CLI 入口
+-- config.py                    # 配置管理
```

### Structure Rationale

- **skills/core/:** 核心基础设施，所有技能共享的状态管理、Schema 验证和管道执行
- **skills/platforms/:** 按平台组织，每个平台有自己的元数据提取和内容下载技能
- **skills/processing/:** 内容转换技能，独立于平台
- **skills/analysis/:** AI 分析技能，作为管道的最后阶段
- **schemas/:** 统一的 JSON Schema 定义，确保接口一致性
- **workflows/:** 声明式工作流定义，指定技能组合和执行顺序

## Architectural Patterns

### Pattern 1: Pipeline Orchestration (管道编排)

**What:** 将处理流程建模为一系列串联的步骤，每个步骤接收输入、执行处理、产生输出。

**When to use:** 可预测的、重复性的任务，如内容处理管道。

**Trade-offs:**
- 优点：确定性执行、易于调试、支持断点续传
- 缺点：灵活性有限，不适合需要动态决策的场景

**Example:**

```python
# Pipeline definition
class ContentPipeline:
    steps = [
        ("parse", LinkParserSkill),
        ("metadata", MetadataSkill),      # depends on parse
        ("download", ContentSkill),       # depends on metadata
        ("transcribe", TranscribeSkill),  # conditional: only for video
        ("analyze", AnalyzeSkill),        # depends on download + transcribe
    ]

    def run(self, url: str, state: WorkflowState):
        for step_name, skill in self.steps:
            if state.is_completed(step_name):
                continue  # Skip completed steps

            result = skill.execute(state.get_step_input(step_name))
            state.mark_completed(step_name, result)

            if not result.success:
                return self.handle_error(result.error)
```

### Pattern 2: Step-Level State Management (步骤级状态管理)

**What:** 每个主要步骤完成后立即持久化状态，支持从任意断点恢复执行。

**When to use:** 长时间运行的批处理任务，可能因网络、API 限制等原因中断。

**Trade-offs:**
- 优点：断点续传、执行追踪、幂等性保证
- 缺点：状态管理开销、需要设计状态 Schema

**Example:**

```python
# workflow_state.json
{
    "url": "https://www.bilibili.com/video/BV1xx411c7mD",
    "platform": "bilibili",
    "content_type": "video",
    "created_at": "2026-03-13T10:00:00Z",
    "updated_at": "2026-03-13T10:15:00Z",
    "steps": {
        "parse": {
            "status": "completed",
            "output": {"video_id": "BV1xx411c7mD"},
            "completed_at": "2026-03-13T10:00:05Z"
        },
        "metadata": {
            "status": "completed",
            "output": {"title": "...", "duration": 1800},
            "completed_at": "2026-03-13T10:00:30Z"
        },
        "download": {
            "status": "in_progress",
            "output": null,
            "started_at": "2026-03-13T10:01:00Z"
        }
    }
}
```

### Pattern 3: Schema-First Interface Design (Schema 优先接口设计)

**What:** 先定义输入输出的 JSON Schema，再实现技能逻辑。所有技能遵守统一的接口规范。

**When to use:** 多人协作、需要接口稳定性和可测试性的项目。

**Trade-offs:**
- 优点：接口清晰、易于测试、支持自动验证
- 缺点：前期设计成本、Schema 维护负担

**Example:**

```json
// schemas/skill_output.schema.json
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "required": ["success", "data"],
    "properties": {
        "success": {"type": "boolean"},
        "data": {"type": "object"},
        "error": {
            "type": "object",
            "properties": {
                "code": {"type": "string"},
                "message": {"type": "string"},
                "detail": {"type": "string"}
            }
        },
        "metadata": {
            "type": "object",
            "properties": {
                "duration_ms": {"type": "number"},
                "tokens_used": {"type": "number"}
            }
        }
    }
}
```

### Pattern 4: Idempotent Execution (幂等执行)

**What:** 相同输入重复执行产生相同结果，不会产生副作用（如重复文件）。

**When to use:** 任何可能需要重试的处理步骤。

**Trade-offs:**
- 优点：安全重试、简化错误恢复
- 缺点：需要设计去重逻辑

**Example:**

```python
class ContentSkill:
    def execute(self, input: dict) -> SkillOutput:
        # Generate deterministic filename from input
        content_hash = hashlib.sha256(input['url'].encode()).hexdigest()[:12]
        output_path = self.get_output_dir() / f"{content_hash}.mp4"

        # Check if already exists
        if output_path.exists():
            return SkillOutput(success=True, data={"path": str(output_path), "cached": True})

        # Download content
        self.download_content(input['url'], output_path)
        return SkillOutput(success=True, data={"path": str(output_path), "cached": False})
```

## Data Flow

### Content Processing Pipeline

```
[URL Input]
    |
    v
+-------------+     +----------------+
| Link Parser | --> | Platform/Type  |
+-------------+     | Detection      |
                    +----------------+
                           |
           +---------------+---------------+
           |                               |
           v                               v
    +-------------+                 +-------------+
    | Bilibili    |                 | Zhihu       |
    | Pipeline    |                 | Pipeline    |
    +-------------+                 +-------------+
           |                               |
    +------+------+                +-------+-------+
    |             |                |               |
    v             v                v               v
[Metadata]   [Download]       [Metadata]     [Download]
    |             |                |               |
    +------+------+                +-------+-------+
           |                               |
           v                               v
    +-------------+                 +-------------+
    | Video?      |                 | Article     |
    | - Whisper   |                 | - Direct    |
    | - FunASR    |                 |   Extract   |
    | - OCR       |                 +-------------+
    +-------------+                       |
           |                              |
           +---------------+--------------+
                           |
                           v
                  +-----------------+
                  | Deep Analyzer   |
                  | (AI Analysis)   |
                  +-----------------+
                           |
                           v
                  +-----------------+
                  | Archive to      |
                  | Knowledge Base  |
                  +-----------------+
```

### State Management Flow

```
[Workflow Start]
       |
       v
+------------------+
| Load/Init State  | <-- workflow_state.json
+------------------+
       |
       v
+------------------+     +------------------+
| For Each Step    | --> | Check Status     |
+------------------+     | - completed? skip|
       |                 | - failed? retry  |
       v                 | - pending? run   |
+------------------+     +------------------+
| Execute Step     |            |
+------------------+            v
       |               +------------------+
       v               | Mark Completed   |
+------------------+    | Save State       |
| Validate Output  | -->| (atomic write)   |
+------------------+    +------------------+
       |
       v
[Next Step / Complete]
```

### Key Data Flows

1. **URL to Metadata:** URL -> Link Parser -> (platform, type, id) -> Platform Metadata Skill -> Metadata JSON
2. **Metadata to Content:** Metadata JSON -> Content Skill -> (video file / article text)
3. **Content to Transcript:** Video File -> Transcribe Skill (Whisper/FunASR) -> Transcript JSON
4. **Content + Transcript to Analysis:** Content + Transcript -> Deep Analyzer -> Analysis Report

## Scaling Considerations

| Scale | Architecture Adjustments |
|-------|--------------------------|
| 1-100 items/day | 当前架构足够，单进程执行 |
| 100-1000 items/day | 添加任务队列，并行处理独立任务 |
| 1000+ items/day | 分布式任务调度，API 请求限流，缓存层 |

### Scaling Priorities

1. **First bottleneck:** 外部 API 限流（B站/知乎）。解决方案：请求队列 + 指数退避重试
2. **Second bottleneck:** AI 分析耗时。解决方案：批处理 + 并行分析

## Anti-Patterns

### Anti-Pattern 1: Global State Mutation

**What people do:** 在多个技能间共享可变全局状态

**Why it's wrong:** 难以追踪状态变化、难以测试、难以恢复

**Do this instead:** 每个步骤从 state 读取输入，将输出写回 state，状态变更通过 state manager 集中管理

### Anti-Pattern 2: Implicit Dependencies

**What people do:** 技能之间通过隐式约定传递数据（如固定文件路径）

**Why it's wrong:** 脆弱的耦合，修改一个技能可能破坏其他技能

**Do this instead:** 在 workflow_state.json 中显式声明依赖关系，通过 state manager 传递数据

### Anti-Pattern 3: Silent Failure

**What people do:** 捕获异常后只打印日志，继续执行

**Why it's wrong:** 下游步骤收到无效输入，产生难以调试的错误

**Do this instead:** 明确的失败处理策略 - 要么终止并记录，要么降级并提供默认值

### Anti-Pattern 4: Non-Idempotent Operations

**What people do:** 每次执行都创建新文件（如 `output_1.mp4`, `output_2.mp4`）

**Why it's wrong:** 重试产生垃圾文件，磁盘占用增长

**Do this instead:** 基于输入生成确定性文件名，已存在则跳过

## Integration Points

### External Services

| Service | Integration Pattern | Notes |
|---------|---------------------|-------|
| Bilibili API | HTTP + Cookie Auth | 需要 BILIBILI_COOKIE 环境变量 |
| Zhihu API | HTTP + Cookie Auth | 需要 ZHIHU_COOKIE 环境变量 |
| AI Provider | OpenAI-compatible API | 需要 AI_API_KEY 环境变量 |
| HuggingFace | API + Local Cache | 需要 HF_TOKEN 环境变量 |
| Git | Command Line | 需要 KB_GIT_USER/EMAIL 配置 |

### Internal Boundaries

| Boundary | Communication | Notes |
|----------|---------------|-------|
| Orchestrator <-> Skills | JSON via stdin/stdout | 标准化输入输出格式 |
| Skills <-> State Manager | File I/O | workflow_state.json |
| Skills <-> External APIs | HTTP | 统一错误处理和重试 |

## Build Order Implications

基于组件依赖关系，建议的构建顺序：

### Phase 1: Foundation (基础设施)
1. **State Manager** - 所有后续组件的基础
2. **Schema Registry** - 定义统一的输入输出格式
3. **Error Handler** - 统一错误处理模式

### Phase 2: Core Skills (核心技能)
4. **Link Parser** - 入口点，无依赖
5. **Platform Metadata Skills** - 依赖 Link Parser 输出
6. **Platform Content Skills** - 依赖 Metadata 输出

### Phase 3: Processing Skills (处理技能)
7. **Whisper/FunASR** - 依赖视频文件
8. **OCR Processor** - 依赖视频截图

### Phase 4: Analysis Skills (分析技能)
9. **Deep Analyzer** - 依赖内容和转录文本

### Phase 5: Orchestration (编排层)
10. **Pipeline Engine** - 组合所有技能
11. **Workflow Definitions** - 声明式流程定义

## Sources

- [AI Architecture Patterns 101: Workflows, Agents, MCPs, and A2A Systems](https://aipmguru.substack.com/p/ai-architecture-patterns-101-workflows) - HIGH confidence, comprehensive pattern overview
- [Data Pipeline Architecture: 5 Design Patterns with Examples - Dagster](https://dagster.io/guides/data-pipeline-architecture-5-design-patterns-with-examples) - HIGH confidence, pipeline patterns
- [Agentic Workflows in 2026: The ultimate guide - Vellum](https://www.vellum.ai/blog/agentic-workflows-emerging-architectures-and-design-patterns) - HIGH confidence, agent architecture patterns
- [State of Open Source Workflow Orchestration Systems 2025](https://www.pracdata.io/p/state-of-workflow-orchestration-ecosystem-2025) - MEDIUM confidence, ecosystem overview
- [ML Pipeline Architecture Design Patterns - Neptune.ai](https://neptune.ai/blog/ml-pipeline-architecture-design-patterns) - MEDIUM confidence, ML-specific patterns

---

*Architecture research for: Skill-based Workflow Orchestration System*
*Original research: 2026-03-13*
*Updated for v1.3 Anti-Crawler: 2026-03-18*

---

## v1.6 语义分段架构研究

**Milestone:** LLM-based Semantic Segmentation
**Researched:** 2026-03-21
**Focus:** 使用 LLM 替代基于规则的语义分段，实现智能章节识别和标题生成

---

### Executive Summary

本文档研究如何将 LLM 语义分段集成到现有的 SRT 处理架构中。基于对现有代码（`kb_workflow/srt/` 模块和 `SRTProcessor`）的分析，推荐采用 **渐进式迁移模式**，保留现有 RuleBasedSegmenter 作为 fallback，新增 LLMBasedSegmenter 作为主分段器。

关键架构决策：
1. **新增 SemanticSegmenter 组件** — 替代/增强现有的 `segmenter.py`
2. **单次 LLM 调用策略** — 完整上下文分析，避免边界拼接问题
3. **三层降级策略** — LLM 失败 → 响应解析失败 → AI 不可用 → RuleBasedSegmenter
4. **渐进式集成** — 不破坏现有功能，新组件作为可选增强

---

### Current Architecture (v1.5)

```
┌─────────────────────────────────────────────────────────────┐
│                    SRTProcessor                              │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐        │
│  │ parser.py   │→ │segmenter.py  │→ │corrector.py │        │
│  │ (SRT解析)   │  │ (规则分段)    │  │ (ASR纠错)   │        │
│  └─────────────┘  └──────────────┘  └─────────────┘        │
│         │                  │                  │              │
│         ↓                  ↓                  ↓              │
│   List[Segment]    List[str]         List[str]              │
│   (时间戳+文本)      (段落)            (纠错后)              │
└─────────────────────────────────────────────────────────────┘
```

**现有组件：**
- `parser.py`: 解析 SRT → Segment(id, start_time, end_time, text)
- `segmenter.py`: SemanticSegmenter (基于时间间隔3s + 长度500字符规则)
- `corrector.py`: ASRCorrector (使用 SimpleAIClient 纠错)

---

### Recommended Architecture (v1.6)

```
┌─────────────────────────────────────────────────────────────────┐
│                         SRTProcessor                             │
│  ┌─────────────┐                                                │
│  │ parser.py   │                                                │
│  │ (SRT解析)   │                                                │
│  └──────┬──────┘                                                │
│         │                                                       │
│         ↓ List[Segment] (id, start_time, end_time, text)        │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │         semantic_segmenter.py (NEW)                       │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 1. 构建 Prompt (视频信息 + 字幕内容)               │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 2. 调用 LLM (SimpleAIClient.generate)              │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 3. 解析响应 (## 标题 + 内容)                       │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 4. 返回 List[SemanticParagraph]                     │  │   │
│  │  │    - title: str (章节标题)                          │  │   │
│  │  │    - content: str (段落内容)                        │  │   │
│  │  │    - time_range: (start, end) (可选)               │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  └─────────────────────┼─────────────────────────────────────┘   │
│                        ↓                                         │
│            List[SemanticParagraph]                               │
│         (标题 + 结构化内容)                                       │
│                        ↓                                         │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    Fallback Strategy                     │   │
│  │  LLM 失败 → 响应解析失败 → AI 不可用                      │   │
│  │      ↓                                                    │   │
│  │  segmenter.py (RuleBasedSegmenter) - 保留作为 fallback   │   │
│  └────────────────────┬─────────────────────────────────────┘   │
│                       ↓                                          │
│            List[str] (向后兼容)                                  │
│                       ↓                                          │
│  ┌─────────────┐  ┌──────────────┐                              │
│  │corrector.py │  │ Markdown生成 │                              │
│  │ (ASR纠错)   │  │ (标题+内容)   │                              │
│  └─────────────┘  └──────────────┘                              │
└─────────────────────────────────────────────────────────────────┘
```

---

### Component Boundaries

| Component | Responsibility | Communicates With |
|-----------|---------------|-------------------|
| **parser.py** | 解析 SRT 格式 → Segment 对象 | 无（输入层） |
| **semantic_segmenter.py** (NEW) | LLM 语义分析 → 结构化段落 | parser (输入), corrector (输出), SimpleAIClient |
| **segmenter.py** (EXIST) | 基于规则的分段（fallback） | semantic_segmenter (降级) |
| **corrector.py** | ASR 纠错 | semantic_segmenter (输入), SimpleAIClient |
| **SRTProcessor** | 编排处理流程 | 所有模块 |

---

### Data Flow Changes

**旧流程 (v1.5):**
```
SRT File → parser.parse_srt_file() → List[Segment]
    ↓
segmenter.segment() (基于规则)
    ↓
List[str] (段落)
    ↓
corrector.correct() (逐段落纠错)
    ↓
生成 Markdown
```

**新流程 (v1.6):**
```
SRT File → parser.parse_srt_file() → List[Segment]
    ↓
[NEW] semantic_segmenter.segment()
    ├─ 构建 Prompt (视频信息 + 字幕文本)
    ├─ 调用 LLM (SimpleAIClient)
    ├─ 解析响应 (## 标题 + 内容)
    └─ 返回 List[SemanticParagraph]
    ├─ title: str
    ├─ content: str
    └─ time_range: (float, float) (可选)
    ↓
[FALLBACK] segmenter.segment() (LLM 失败时)
    ↓
List[SemanticParagraph] → flatten → List[str]
    ↓
corrector.correct() (逐段落纠错)
    ↓
生成 Markdown (标题 + 内容结构)
```

---

### Response Parsing Strategy

**LLM 响应格式要求:**
```
## 第一部分：机器学习基础概念
段落内容1...
段落内容2...

## 第二部分：深度学习原理
段落内容3...
段落内容4...
```

**解析方法:**
```python
@dataclass
class SemanticParagraph:
    """语义分段结果"""
    title: str           # 章节标题
    content: str         # 段落内容
    start_time: float    # 开始时间（可选）
    end_time: float      # 结束时间（可选）

def parse_llm_response(response: str) -> List[SemanticParagraph]:
    """解析 LLM 响应为结构化段落"""
    paragraphs = []
    current_title = "无标题"
    current_content = []

    for line in response.split('\n'):
        if line.startswith('##'):
            # 保存上一个段落
            if current_content:
                content = '\n'.join(current_content).strip()
                if content:
                    paragraphs.append(SemanticParagraph(
                        title=current_title,
                        content=content
                    ))
            # 开始新段落
            current_title = line.lstrip('#').strip()
            current_content = []
        else:
            current_content.append(line)

    # 保存最后一个段落
    if current_content:
        content = '\n'.join(current_content).strip()
        if content:
            paragraphs.append(SemanticParagraph(
                title=current_title,
                content=content
            ))

    return paragraphs
```

---

### LLM Call Strategy

**推荐：单次调用**

**原因:**
1. **上下文完整性** — LLM 理解完整视频内容，分段更准确
2. **避免边界问题** — 多次调用的拼接处容易出现语义不连贯
3. **简化实现** — 单次调用更容易测试和调试

**Prompt 设计:**
```python
SEGMENTATION_PROMPT = """你是一个视频内容分析师。请将以下字幕内容按语义分段，并为每个段落生成标题。

【视频信息】
标题：{video_title}
时长：{duration}

【字幕内容】
{concatenated_segments}

【输出要求】
1. 按语义完整性分段（保持话题/论点完整）
2. 识别主题变化边界
3. 为每个段落生成简洁标题（不超过20字）
4. 输出格式：

## 第一部分：{标题}
{段落内容}

## 第二部分：{标题}
{段落内容}

...（以此类推）

注意：
- 段落长度建议 200-800 字符
- 在自然句号处分段
- 标题要概括该段核心内容
"""
```

**输入处理选项:**

**选项 1: 纯文本（推荐）**
```python
# 合并所有 Segment.text
full_text = '\n'.join(seg.text for seg in segments)
```

**选项 2: 带时间戳标记（可选，用于辅助理解）**
```python
# 添加时间戳辅助 LLM 理解结构
marked_text = '\n'.join(
    f"[{format_time(seg.start_time)}] {seg.text}"
    for seg in segments
)
```

**替代方案：流式调用**
- 用于超长视频（>1小时字幕）
- 分块调用 LLM，每块 10-15 分钟字幕
- 需要额外的边界拼接逻辑
- **建议：** 初期使用单次调用，后续根据实际需求优化

---

### Fallback Strategy

**三层降级策略:**

```python
class LLMBasedSegmenter:
    def __init__(self, ai_client: SimpleAIClient):
        self.ai_client = ai_client
        self.rule_based = SemanticSegmenter()  # Fallback

    def segment(self, segments: List[Segment]) -> List[SemanticParagraph]:
        # Level 3: AI 客户端不可用
        if not self.ai_client or not self.ai_client.api_key:
            logger.info("AI client not available, using rule-based")
            return self._to_semantic_paragraphs(
                self.rule_based.segment(segments)
            )

        try:
            # Level 1: LLM 调用
            prompt = self._build_prompt(segments)
            response = self.ai_client.generate(prompt)

            # Level 2: 响应解析
            try:
                paragraphs = self._parse_llm_response(response)
                if not paragraphs:
                    raise ValueError("Empty parsed result")
                return paragraphs
            except Exception as e:
                logger.warning(f"Failed to parse LLM response: {e}")
                return self._to_semantic_paragraphs(
                    self.rule_based.segment(segments)
                )

        except Exception as e:
            logger.warning(f"LLM call failed: {e}, using rule-based")
            return self._to_semantic_paragraphs(
                self.rule_based.segment(segments)
            )
```

---

### Suggested Build Order

**Phase 1: 基础组件（不影响现有功能）**
1. 创建 `kb_workflow/srt/semantic_segmenter.py`
2. 实现 `SemanticParagraph` 数据类
3. 实现 `parse_llm_response()` 方法（独立于 LLM 调用）
4. 编写单元测试（mock LLM 响应，测试解析逻辑）

**Phase 2: LLM 集成**
5. 实现 `_build_prompt()` 方法
6. 集成 SimpleAIClient
7. 实现 `segment()` 主方法
8. 实现降级策略（fallback）
9. 编写集成测试（使用真实 LLM 或 mock）

**Phase 3: 处理器集成**
10. 修改 `SRTProcessor.__init__()`：
    - 添加 `use_llm_segmenter: bool = False` 参数
    - 条件性初始化 `llm_segmenter`
11. 修改 `SRTProcessor.process()`：
    - 调用新分段器或保留旧分段器
    - 处理 `SemanticParagraph` 输出
12. 更新 `_generate_markdown()`：
    - 处理标题和内容结构
    - 生成带层级的 Markdown
13. 保留 RuleBasedSegmenter 作为 fallback

**Phase 4: 输出优化**
14. 优化 Markdown 格式（标题层级 `###`）
15. 添加时间戳（可选，在段落中标注时间范围）
16. 更新文档和测试

---

### Schema Changes

**新增数据类:**
```python
@dataclass
class SemanticParagraph:
    """语义分段结果"""
    title: str           # 章节标题
    content: str         # 段落内容
    start_time: Optional[float] = None  # 开始时间（可选）
    end_time: Optional[float] = None    # 结束时间（可选）
```

**修改 SRTOutput (向后兼容):**
```python
class SRTOutput(BaseModel):
    success: bool
    output_file: str
    paragraphs: List[str]                    # 保留向后兼容
    semantic_paragraphs: Optional[List[SemanticParagraph]] = None  # 新增
    tags: List[str]
    corrected: bool
```

---

### Patterns to Follow

#### Pattern 1: 渐进式迁移
**What:** 保留现有组件，新组件作为可选增强
**When:** 重构关键路径时
**Example:**
```python
class SRTProcessor:
    def __init__(self, use_llm_segmenter: bool = False):
        self.rule_based_segmenter = SemanticSegmenter()  # 保留
        if use_llm_segmenter:
            self.llm_segmenter = LLMBasedSegmenter()     # 新增

    async def process(self, input_data):
        # 优先使用 LLM，失败时降级
        if hasattr(self, 'llm_segmenter'):
            try:
                paragraphs = self.llm_segmenter.segment(segments)
            except Exception:
                paragraphs = self.rule_based_segmenter.segment(segments)
        else:
            paragraphs = self.rule_based_segmenter.segment(segments)
```

#### Pattern 2: 协议兼容性
**What:** 新旧组件共享相同接口
**When:** 替换组件时
**Example:**
```python
# 两个分段器实现相同接口
class Segmenter(Protocol):
    def segment(self, segments: List[Segment]) -> List[str]:
        ...

class RuleBasedSegmenter:
    def segment(self, segments: List[Segment]) -> List[str]:
        # 基于规则
        ...

class LLMBasedSegmenter:
    def segment(self, segments: List[Segment]) -> List[str]:
        # 基于 LLM
        # 内部使用 SemanticParagraph，输出时合并标题和内容
        return [
            f"## {p.title}\n\n{p.content}"
            for p in self._segment_internal(segments)
        ]
```

---

### Anti-Patterns to Avoid

#### Anti-Pattern 1: 破坏性替换
**What:** 直接删除 RuleBasedSegmenter
**Why bad:** LLM 可能失败，失去降级选项
**Instead:** 保留旧组件作为 fallback

#### Anti-Pattern 2: 紧耦合 Prompt 逻辑
**What:** 将 prompt 构建硬编码在 segmenter 中
**Why bad:** 难以测试和优化
**Instead:** 提取为独立方法或模板，便于测试

#### Anti-Pattern 3: 忽略时间戳信息
**What:** 仅传递文本给 LLM，忽略时间信息
**Why bad:** LLM 无法理解视频结构（停顿、章节）
**Instead:** 在 prompt 中包含时间戳标记或视频时长信息

---

### Scalability Considerations

| Concern | At 100 videos | At 10K videos | At 1M videos |
|---------|--------------|--------------|-------------|
| **LLM 调用延迟** | 可接受（~10-30s/视频） | 需要并发处理 | 需要队列+批处理 |
| **LLM 成本** | 可忽略 | 需要监控 | 需要缓存+降级策略 |
| **错误率** | 低，手动处理 | 需要自动重试 | 需要监控告警 |
| **响应解析** | 简单 | 需要容错增强 | 需要多模型支持 |

**建议:**
- 初期：单线程处理，完整错误日志
- 中期：添加并发处理（asyncio.Semaphore 限制并发数）
- 长期：分布式队列（Celery/RQ）+ 结果缓存

---

### Integration Points Summary

**新增集成点:**
1. `SRTProcessor.__init__()` → 条件性初始化 `LLMBasedSegmenter`
2. `SRTProcessor.process()` → 调用新分段器或旧分段器
3. `_generate_markdown()` → 处理 `SemanticParagraph` 结构

**保留集成点:**
1. `parser.parse_srt_file()` — 不变
2. `corrector.correct()` — 不变（处理 `List[str]`）
3. `WorkflowStateManager` — 不变

**修改点:**
1. `kb_workflow/srt/__init__.py` — 导出 `SemanticParagraph`
2. `kb_workflow/pipeline/schemas.py` — 更新 `SRTOutput`

---

### v1.6 Sources

- 现有架构分析: `kb_workflow/srt/` 模块
- SRTProcessor 实现: `kb_workflow/pipeline/processors/srt.py`
- SimpleAIClient 实现: `kb_workflow/deep_analyzer/__init__.py`
- 测试用例: `tests/test_srt_processor.py`
- 项目上下文: `.planning/PROJECT.md`

---

*Architecture research for: LLM-based Semantic Segmentation*
*Research date: 2026-03-21*