Archived

This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

starmen1265 440c885448 docs: complete v1.6 semantic segmentation research

2026-03-21 23:53:55 +08:00

69 KiB

Raw Blame History

Architecture Research

Domain: Skill-based Workflow Orchestration System Researched: 2026-03-13 (updated 2026-03-18 for v1.3) Confidence: HIGH

v1.3 补充：反爬虫优化架构

Milestone: Anti-Crawler Optimization Researched: 2026-03-18 Focus: 代理池、行为伪装、验证码检测、错误恢复

Executive Summary

本文档研究如何将反爬虫优化功能集成到现有的 kb-skills 架构中。基于对现有代码结构和反爬虫最佳实践的分析，推荐采用 模块化的中间件/装饰器模式，将反爬虫功能作为独立组件注入到现有的 Pipeline 和 Crawler 中。

关键架构决策：

代理池管理 — 新建 kb_workflow/crawler/proxy_pool.py 模块，作为独立服务
行为伪装 — 通过装饰器/中间件模式集成到 Playwright 调用
验证码处理 — 检测-暂停机制，不自动解决（避免付费服务依赖）
错误恢复 — 增强现有 retry.py，添加 jitter 和状态感知

Current Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                              CLI Layer                                   │
│  kb_workflow/cli.py - process/login subcommands                          │
├─────────────────────────────────────────────────────────────────────────┤
│                            Pipeline Layer                                │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐         │
│  │ ParseProcessor  │→ │DownloadProcessor│→ │  ASRProcessor   │ → ...   │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘         │
│           │                    │                    │                   │
│           ↓                    ↓                    ↓                   │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │              Platform Adapters (BilibiliAdapter, ZhihuAdapter)   │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                            Crawler Layer                                 │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuCrawler (Playwright) - API Interception + Cookie Auth     │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                          Infrastructure                                  │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐              │
│  │ CrawlerConfig │  │   KBSettings  │  │  WorkflowState│              │
│  └───────────────┘  └───────────────┘  └───────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘

Recommended New Architecture (v1.3)

┌─────────────────────────────────────────────────────────────────────────┐
│                              CLI Layer                                   │
│  kb_workflow/cli.py - process/login subcommands                          │
├─────────────────────────────────────────────────────────────────────────┤
│                            Pipeline Layer                                │
│  [Existing processors - unchanged]                                       │
├─────────────────────────────────────────────────────────────────────────┤
│                     Platform Adapters Layer                              │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuAdapter / BilibiliAdapter                                   │   │
│  │  (uses @with_anti_detection decorator)                           │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                       Anti-Crawler Layer (NEW)                           │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐         │
│  │   ProxyPool     │  │BehaviorCamouflag│  │ CaptchaDetector │         │
│  │   Manager       │  │     e           │  │                 │         │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘         │
│           │                    │                    │                   │
│           └────────────────────┴────────────────────┘                   │
│                                │                                         │
│                    ┌───────────┴───────────┐                            │
│                    │  @with_anti_detection │                            │
│                    │     (decorator)       │                            │
│                    └───────────────────────┘                            │
├─────────────────────────────────────────────────────────────────────────┤
│                            Crawler Layer                                 │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ZhihuCrawler (enhanced with anti-detection hooks)              │   │
│  └─────────────────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────────────┤
│                          Infrastructure                                  │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐              │
│  │CrawlerConfig  │  │AntiCrawlerConf│  │  KBSettings   │              │
│  │  (enhanced)   │  │    (NEW)      │  │  (enhanced)   │              │
│  └───────────────┘  └───────────────┘  └───────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘

Component Design

1. Proxy Pool Manager

Location: kb_workflow/crawler/proxy_pool.py

Rationale: 独立模块，与现有 CrawlerConfig 分离，因为：

代理池是状态性的（需要跟踪健康状态、使用次数）
可能被多个 Crawler 共享
需要独立的配置源（环境变量、文件、API）

Design:

# kb_workflow/crawler/proxy_pool.py

from typing import Optional, List, Dict, Any
from pydantic import BaseModel
import random
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class ProxyInfo:
    """代理信息"""
    url: str
    protocol: str = "http"  # http, https, socks5
    username: Optional[str] = None
    password: Optional[str] = None
    last_used: Optional[datetime] = None
    fail_count: int = 0
    success_count: int = 0
    is_healthy: bool = True

    def to_playwright_format(self) -> Dict[str, Any]:
        config = {"server": self.url}
        if self.username:
            config["username"] = self.username
        if self.password:
            config["password"] = self.password
        return config


class ProxyPoolConfig(BaseModel):
    """代理池配置"""
    enabled: bool = False
    # 代理列表（支持多个代理）
    proxies: List[str] = []
    # 轮换策略: round_robin, random, least_used
    rotation_strategy: str = "random"
    # 健康检查
    health_check_enabled: bool = True
    max_fail_count: int = 3  # 连续失败 N 次后标记为不健康
    # 冷却时间（秒）
    cooldown_seconds: float = 0  # 同一代理两次使用之间的最小间隔

    @classmethod
    def from_env(cls) -> "ProxyPoolConfig":
        """从环境变量加载代理池配置"""
        import os
        proxies_str = os.environ.get("KB_PROXY_POOL", "")
        proxies = [p.strip() for p in proxies_str.split(",") if p.strip()]
        return cls(
            enabled=len(proxies) > 0,
            proxies=proxies,
            rotation_strategy=os.environ.get("KB_PROXY_STRATEGY", "random"),
        )


class ProxyPoolManager:
    """代理池管理器

    职责：
    - 管理代理列表
    - 轮换策略
    - 健康状态跟踪
    """

    def __init__(self, config: Optional[ProxyPoolConfig] = None):
        self.config = config or ProxyPoolConfig.from_env()
        self._proxies: List[ProxyInfo] = []
        self._current_index = 0
        self._init_proxies()

    def _init_proxies(self):
        """初始化代理列表"""
        for proxy_url in self.config.proxies:
            self._proxies.append(ProxyInfo(url=proxy_url))

    def get_proxy(self) -> Optional[ProxyInfo]:
        """获取下一个可用代理"""
        healthy_proxies = [p for p in self._proxies if p.is_healthy]
        if not healthy_proxies:
            return None

        if self.config.rotation_strategy == "round_robin":
            proxy = healthy_proxies[self._current_index % len(healthy_proxies)]
            self._current_index += 1
        else:  # random
            proxy = random.choice(healthy_proxies)

        proxy.last_used = datetime.now()
        return proxy

    def report_success(self, proxy_url: str):
        """报告代理使用成功"""
        for p in self._proxies:
            if p.url == proxy_url:
                p.success_count += 1
                p.fail_count = 0  # 重置失败计数
                break

    def report_failure(self, proxy_url: str):
        """报告代理使用失败"""
        for p in self._proxies:
            if p.url == proxy_url:
                p.fail_count += 1
                if p.fail_count >= self.config.max_fail_count:
                    p.is_healthy = False
                break


# Module-level singleton
_proxy_pool: Optional[ProxyPoolManager] = None

def get_proxy_pool() -> ProxyPoolManager:
    """获取代理池管理器单例"""
    global _proxy_pool
    if _proxy_pool is None:
        _proxy_pool = ProxyPoolManager()
    return _proxy_pool

2. Behavior Camouflage

Location: kb_workflow/crawler/behavior.py

Rationale: 行为伪装逻辑应该：

可以独立测试
可以被装饰器或直接调用
支持可配置的随机性

Design:

# kb_workflow/crawler/behavior.py

import random
import time
from typing import Optional, Callable
from functools import wraps
from pydantic import BaseModel, Field


class BehaviorConfig(BaseModel):
    """行为伪装配置"""
    # 请求间隔
    min_delay: float = Field(default=1.0, description="最小延迟（秒）")
    max_delay: float = Field(default=3.0, description="最大延迟（秒）")

    # Jitter 配置（防止模式检测）
    jitter_enabled: bool = True
    jitter_range: float = Field(default=0.3, description="抖动范围（+/- 比例）")

    # 人类模拟
    mouse_movement_enabled: bool = False  # Playwright 场景下可选
    scroll_simulation_enabled: bool = False

    # User-Agent 随机化
    user_agent_rotation: bool = True


# 常用 User-Agent 池
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
]


class BehaviorCamouflage:
    """行为伪装管理器"""

    def __init__(self, config: Optional[BehaviorConfig] = None):
        self.config = config or BehaviorConfig()

    def get_random_delay(self) -> float:
        """获取随机延迟时间（带 jitter）"""
        base_delay = random.uniform(self.config.min_delay, self.config.max_delay)

        if self.config.jitter_enabled:
            jitter = base_delay * self.config.jitter_range * random.uniform(-1, 1)
            base_delay += jitter

        return max(0.1, base_delay)  # 确保非负

    def apply_delay(self):
        """应用随机延迟"""
        delay = self.get_random_delay()
        time.sleep(delay)
        return delay

    def get_random_user_agent(self) -> str:
        """获取随机 User-Agent"""
        if self.config.user_agent_rotation:
            return random.choice(USER_AGENTS)
        return USER_AGENTS[0]

    def get_playwright_context_options(self) -> dict:
        """获取 Playwright context 选项（包含反检测设置）"""
        options = {
            "user_agent": self.get_random_user_agent(),
        }
        return options


def with_behavior_camouflage(config: Optional[BehaviorConfig] = None):
    """装饰器：为函数添加行为伪装

    用于同步函数，在调用前添加随机延迟。

    Example:
        @with_behavior_camouflage()
        def fetch_page(url):
            return requests.get(url)
    """
    camouflage = BehaviorCamouflage(config)

    def decorator(func: Callable):
        @wraps(func)
        def wrapper(*args, **kwargs):
            camouflage.apply_delay()
            return func(*args, **kwargs)
        return wrapper

    return decorator

3. CAPTCHA Detection

Location: kb_workflow/crawler/captcha.py

Rationale:

不实现自动解决 — 根据项目约束（Out of Scope: 付费打码服务）
检测+暂停机制 — 检测到验证码时暂停并通知用户

Design:

# kb_workflow/crawler/captcha.py

import re
from typing import Optional, Tuple
from enum import Enum
from dataclasses import dataclass
from pydantic import BaseModel

from kb_workflow.pipeline.errors import KBError, ErrorType


class CaptchaType(str, Enum):
    """验证码类型"""
    RECAPTCHA_V2 = "recaptcha_v2"
    RECAPTCHA_V3 = "recaptcha_v3"
    HCAPTCHA = "hcaptcha"
    CLOUDFLARE = "cloudflare"
    IMAGE_CAPTCHA = "image_captcha"
    SLIDER = "slider"
    UNKNOWN = "unknown"


@dataclass
class CaptchaDetectionResult:
    """验证码检测结果"""
    detected: bool
    captcha_type: Optional[CaptchaType] = None
    confidence: float = 0.0
    indicators: list = None

    def __post_init__(self):
        if self.indicators is None:
            self.indicators = []


class CaptchaDetector:
    """验证码检测器

    职责：
    - 检测页面是否包含验证码
    - 识别验证码类型
    - 不负责解决验证码（需要人工介入）
    """

    # 验证码指示器（页面元素/URL/内容）
    CAPTCHA_INDICATORS = {
        CaptchaType.RECAPTCHA_V2: [
            "g-recaptcha",
            "recaptcha-checkbox",
            "google.com/recaptcha",
        ],
        CaptchaType.RECAPTCHA_V3: [
            "grecaptcha.execute",
            "recaptcha-v3",
        ],
        CaptchaType.HCAPTCHA: [
            "h-captcha",
            "hcaptcha.com",
        ],
        CaptchaType.CLOUDFLARE: [
            "cloudflare",
            "cf-browser-verification",
            "challenge-platform",
        ],
        CaptchaType.IMAGE_CAPTCHA: [
            "captcha.png",
            "verifycode",
            "img captcha",
        ],
        CaptchaType.SLIDER: [
            "slider",
            "slide-verify",
            "drag verify",
        ],
    }

    def detect_from_html(self, html_content: str) -> CaptchaDetectionResult:
        """从 HTML 内容检测验证码"""
        html_lower = html_content.lower()
        indicators = []

        for captcha_type, patterns in self.CAPTCHA_INDICATORS.items():
            for pattern in patterns:
                if pattern.lower() in html_lower:
                    indicators.append(pattern)
                    return CaptchaDetectionResult(
                        detected=True,
                        captcha_type=captcha_type,
                        confidence=0.8,
                        indicators=indicators,
                    )

        return CaptchaDetectionResult(detected=False)

    def detect_from_response(
        self,
        status_code: int,
        headers: dict,
        url: str
    ) -> CaptchaDetectionResult:
        """从 HTTP 响应检测验证码"""
        # 403 通常表示被拦截或需要验证
        if status_code == 403:
            return CaptchaDetectionResult(
                detected=True,
                captcha_type=CaptchaType.UNKNOWN,
                confidence=0.6,
                indicators=["403 Forbidden"],
            )

        # 检查 URL 重定向
        if "captcha" in url.lower() or "verify" in url.lower():
            return CaptchaDetectionResult(
                detected=True,
                captcha_type=CaptchaType.UNKNOWN,
                confidence=0.7,
                indicators=[f"URL redirect: {url}"],
            )

        return CaptchaDetectionResult(detected=False)


class CaptchaError(KBError):
    """验证码错误

    当检测到验证码时抛出，需要人工介入。
    """

    def __init__(
        self,
        step_name: str,
        input_summary: str,
        captcha_type: CaptchaType,
        suggestion: str = "请手动完成验证码后重试"
    ):
        super().__init__(
            message=f"检测到验证码: {captcha_type.value}",
            error_type=ErrorType.AUTH,
            step_name=step_name,
            input_summary=input_summary,
            suggestion=suggestion,
        )
        self.captcha_type = captcha_type

4. Enhanced Retry Mechanism

Location: kb_workflow/pipeline/retry.py (enhance existing)

Rationale:

现有实现使用 tenacity，已经很好
增强点：添加 jitter、状态感知、验证码感知

Enhanced Design:

# kb_workflow/pipeline/retry.py (enhancements)

import random
import logging
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type,
    before_sleep_log,
    wait_combine,
    wait_random,
)

from .errors import NetworkError, AuthError

logger = logging.getLogger(__name__)


def wait_exponential_jitter(
    multiplier: float = 1,
    min: float = 2,
    max: float = 10,
    jitter: float = 0.5
):
    """指数退避 + Jitter

    Args:
        multiplier: 退避乘数
        min: 最小等待时间
        max: 最大等待时间
        jitter: 随机抖动范围（0-1）
    """
    return wait_combine(
        wait_exponential(multiplier=multiplier, min=min, max=max),
        wait_random(0, jitter)
    )


# 增强版重试装饰器（带 jitter）
retry_on_network_error_with_jitter = retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
    retry=retry_if_exception_type(NetworkError),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
"""带 Jitter 的网络错误重试装饰器

在指数退避基础上增加随机抖动，避免多个请求同时重试的"惊群效应"。
"""


# 验证码感知重试（遇到 CaptchaError 不重试）
retry_on_transient_error = retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
    retry=(
        retry_if_exception_type(NetworkError) &
        ~retry_if_exception_type(AuthError)  # AuthError 不重试
    ),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
"""瞬态错误重试装饰器

只重试网络错误，不重试认证错误（包括验证码）。
"""

5. Unified Anti-Detection Decorator

Location: kb_workflow/crawler/anti_detection.py

Rationale:

统一入口点，组合所有反爬虫功能
便于在 PlatformAdapter 中使用

Design:

# kb_workflow/crawler/anti_detection.py

from typing import Optional, Callable, Dict, Any
from functools import wraps
from dataclasses import dataclass

from .proxy_pool import ProxyPoolManager, get_proxy_pool, ProxyInfo
from .behavior import BehaviorCamouflage, BehaviorConfig
from .captcha import CaptchaDetector, CaptchaError, CaptchaType


@dataclass
class AntiDetectionConfig:
    """反检测配置"""
    proxy_enabled: bool = True
    behavior_camouflage_enabled: bool = True
    captcha_detection_enabled: bool = True
    behavior_config: Optional[BehaviorConfig] = None


class AntiDetectionContext:
    """反检测上下文

    在一次爬取过程中传递的状态信息。
    """
    def __init__(self):
        self.proxy: Optional[ProxyInfo] = None
        self.delays_applied: list = []
        self.captcha_detected: bool = False


def with_anti_detection(config: Optional[AntiDetectionConfig] = None):
    """装饰器：集成所有反检测功能

    组合使用：
    1. 代理池轮换
    2. 行为伪装
    3. 验证码检测

    Example:
        @with_anti_detection()
        async def fetch_content(self, url: str, context: AntiDetectionContext = None):
            # context.proxy 可用于 Playwright
            # 延迟已自动应用
            ...
    """
    _config = config or AntiDetectionConfig()
    behavior = BehaviorCamouflage(_config.behavior_config)
    proxy_pool = get_proxy_pool() if _config.proxy_enabled else None
    captcha_detector = CaptchaDetector() if _config.captcha_detection_enabled else None

    def decorator(func: Callable):
        @wraps(func)
        async def async_wrapper(*args, **kwargs):
            context = AntiDetectionContext()

            # 1. 获取代理
            if proxy_pool and proxy_pool.config.enabled:
                context.proxy = proxy_pool.get_proxy()

            # 2. 应用行为伪装延迟
            if _config.behavior_camouflage_enabled:
                delay = behavior.apply_delay()
                context.delays_applied.append(delay)

            # 3. 执行原始函数
            try:
                result = await func(*args, context=context, **kwargs)

                # 报告代理成功
                if context.proxy and proxy_pool:
                    proxy_pool.report_success(context.proxy.url)

                return result

            except Exception as e:
                # 报告代理失败
                if context.proxy and proxy_pool:
                    proxy_pool.report_failure(context.proxy.url)
                raise

        @wraps(func)
        def sync_wrapper(*args, **kwargs):
            context = AntiDetectionContext()

            # 1. 获取代理
            if proxy_pool and proxy_pool.config.enabled:
                context.proxy = proxy_pool.get_proxy()

            # 2. 应用行为伪装延迟
            if _config.behavior_camouflage_enabled:
                delay = behavior.apply_delay()
                context.delays_applied.append(delay)

            # 3. 执行原始函数
            try:
                result = func(*args, context=context, **kwargs)

                if context.proxy and proxy_pool:
                    proxy_pool.report_success(context.proxy.url)

                return result

            except Exception as e:
                if context.proxy and proxy_pool:
                    proxy_pool.report_failure(context.proxy.url)
                raise

        # 根据函数类型选择包装器
        import asyncio
        if asyncio.iscoroutinefunction(func):
            return async_wrapper
        return sync_wrapper

    return decorator

Integration Points

Integration with ZhihuCrawler

Modified File: kb_workflow/crawler/zhihu_crawler.py

Changes:

# In ZhihuCrawler.__init__:
from .anti_detection import AntiDetectionConfig, get_proxy_pool

def __init__(
    self,
    headless: bool = True,
    timeout: int = 30,
    config: Optional[CrawlerConfig] = None,
    anti_detection_config: Optional[AntiDetectionConfig] = None
):
    self.headless = headless
    self.timeout = timeout
    self.config = config or get_crawler_config()
    self.anti_detection_config = anti_detection_config or AntiDetectionConfig()
    # ... rest of init

# In _start_browser_with_proxy:
def _start_browser_with_proxy(self, cookie: str) -> None:
    # Use proxy pool if enabled
    if self.anti_detection_config.proxy_enabled:
        proxy_pool = get_proxy_pool()
        if proxy_pool.config.enabled:
            proxy = proxy_pool.get_proxy()
            if proxy:
                self.config.proxy = ProxyConfig(
                    server=proxy.url,
                    username=proxy.username,
                    password=proxy.password
                )
    # ... rest of method

Integration with ZhihuAdapter

Modified File: kb_workflow/pipeline/platforms/zhihu.py

Changes:

from kb_workflow.crawler.anti_detection import (
    with_anti_detection,
    AntiDetectionContext,
    AntiDetectionConfig
)

class ZhihuAdapter(PlatformAdapter):
    def __init__(
        self,
        headless: bool = True,
        timeout: int = 30,
        output_dir: Optional[Path] = None,
        idempotency_checker: Optional[IdempotencyChecker] = None,
        anti_detection_config: Optional[AntiDetectionConfig] = None
    ):
        # ... existing init
        self.anti_detection_config = anti_detection_config or AntiDetectionConfig()

    @with_anti_detection()
    async def fetch_metadata(
        self,
        content_id: str,
        content_type: str,
        source_url: str = None,
        context: AntiDetectionContext = None  # 注入的上下文
    ) -> Dict[str, Any]:
        # Use context.proxy if needed
        # ...

Integration with Configuration

Modified File: kb_workflow/pipeline/config.py

Changes:

class KBSettings(BaseSettings):
    # ... existing fields

    # Anti-crawler settings
    proxy_pool: str = Field(
        default="",
        description="代理池配置（逗号分隔的代理URL）"
    )
    anti_detection_enabled: bool = Field(
        default=True,
        description="是否启用反检测功能"
    )

Data Flow

Request Flow with Anti-Detection

[CLI] process command
    |
    v
[Pipeline] DownloadProcessor.process()
    |
    v
[ZhihuAdapter] @with_anti_detection decorator
    |
    +-> [ProxyPool] get_proxy() -> returns available proxy
    |
    +-> [BehaviorCamouflage] apply_delay() -> wait random time
    |
    +-> [ZhihuCrawler] crawl_answer(url, proxy=...)
            |
            +-> [Playwright] start browser with proxy
            |
            +-> [CaptchaDetector] detect_from_html() -> check for captcha
            |       |
            |       +-> If detected -> raise CaptchaError
            |
            +-> Return CrawlResult
    |
    v
[ProxyPool] report_success() / report_failure()
    |
    v
[Pipeline] continue to next processor

Error Recovery Flow

[Request fails]
    |
    v
[Retry Logic] retry_on_transient_error
    |
    +-> Is NetworkError? -> Retry with exponential backoff + jitter
    |
    +-> Is CaptchaError? -> Do NOT retry, surface to user
    |
    +-> Is AuthError? -> Do NOT retry, trigger QR login
    |
    v
[After max retries] -> Log error, update state, raise to caller

Recommended Build Order

基于依赖关系，建议按以下顺序实现：

Phase 1: Foundation (No external dependencies)

kb_workflow/crawler/behavior.py — 行为伪装模块
- 独立模块，无外部依赖
- 可独立测试
kb_workflow/crawler/captcha.py — 验证码检测模块
- 独立模块，只需要 HTML 解析
- 定义 CaptchaError 错误类型

Phase 2: Integration

kb_workflow/crawler/proxy_pool.py — 代理池管理
- 依赖配置系统
- 需要与 CrawlerConfig 集成
kb_workflow/pipeline/retry.py (enhance) — 增强重试机制
- 添加 jitter 支持
- 添加验证码感知

Phase 3: Unified API

kb_workflow/crawler/anti_detection.py — 统一装饰器
- 组合所有组件
- 提供 @with_anti_detection 装饰器

Phase 4: Platform Integration

kb_workflow/crawler/zhihu_crawler.py (modify) — 集成到 ZhihuCrawler
kb_workflow/pipeline/platforms/zhihu.py (modify) — 集成到 ZhihuAdapter
kb_workflow/pipeline/config.py (modify) — 添加配置项

Phase 5: Bilibili Support (if needed)

kb_workflow/pipeline/platforms/bilibili.py (modify) — B站反爬适配

Anti-Patterns (v1.3 Specific)

Anti-Pattern 1: Hardcoding Proxy Settings in Crawler

What people do: 在 Crawler 类中硬编码代理配置

# BAD
class ZhihuCrawler:
    def __init__(self):
        self.proxy = "http://127.0.0.1:7890"  # Hardcoded

Why it's wrong: 无法动态切换代理，无法支持代理池

Do this instead: 使用独立的 ProxyPoolManager

Anti-Pattern 2: Retry All Errors Blindly

What people do: 对所有错误都进行重试

# BAD
@retry(stop=stop_after_attempt(3))
def crawl():
    ...

Why it's wrong: 验证码错误重试会立即再次触发验证码，造成无限循环

Do this instead: 使用错误类型过滤

# GOOD
@retry(retry=retry_if_exception_type(NetworkError))
def crawl():
    ...

Anti-Pattern 3: Fixed Delay Intervals

What people do: 使用固定延迟

# BAD
time.sleep(2)  # Fixed 2 second delay

Why it's wrong: 固定延迟容易被模式检测识别

Do this instead: 使用随机延迟 + jitter

# GOOD
delay = random.uniform(1, 3) + random.uniform(-0.5, 0.5)
time.sleep(delay)

Anti-Pattern 4: Captcha Auto-Solve Without Fallback

What people do: 依赖第三方验证码解决服务，没有降级方案

Why it's wrong: 增加成本，服务不可用时无法工作

Do this instead: 检测 + 暂停 + 通知用户（本项目约束）

Scaling Considerations (v1.3)

Scale	Architecture Adjustments
个人使用（当前）	单进程，内存代理池，本地配置
小规模（10-100 URL/天）	添加代理持久化，简单的健康检查
中规模（100-1000 URL/天）	考虑使用代理池服务 API，分布式状态存储

Scaling Priorities

First bottleneck: 代理被封 — 通过代理池轮换解决
Second bottleneck: 账号被封 — 通过行为伪装和降低频率解决
Third bottleneck: 验证码频繁 — 考虑暂停机制 + 人工介入

v1.3 Sources

ZenRows - How to Avoid Bot Detection with Playwright - HIGH confidence
Zyte - Best Proxy Providers for Web Scraping 2026 - HIGH confidence
Rebrowser - Python Requests Retry Guide - HIGH confidence
GitHub - backoff Python library - HIGH confidence
Stack Overflow - How to rotate proxies in Python - HIGH confidence
Medium - Exponential Backoff Decorator - MEDIUM confidence
Trickster.dev - Custom Scrapy Middleware for Proxy Pool - HIGH confidence

Original Architecture Research (v1.0)

Below is the original architecture research from v1.0, preserved for reference.

Standard Architecture

System Overview

技能型工作流编排系统采用分层架构，将处理逻辑、状态管理和执行协调分离。基于当前研究和行业最佳实践，推荐以下四层架构：

+---------------------------------------------------------------+
|                    Orchestration Layer                        |
|    [Workflow Coordinator] [State Manager] [Error Handler]     |
+---------------------------------------------------------------+
|                    Skill Definition Layer                     |
|    [SKILL.md specs] [Schema Definitions] [Trigger Rules]      |
+---------------------------------------------------------------+
|                    Execution Layer                            |
|    [Platform Skills] [Processing Skills] [Analysis Skills]    |
+---------------------------------------------------------------+
|                    Infrastructure Layer                       |
|    [File System] [External APIs] [AI Services] [State Store]  |
+---------------------------------------------------------------+

Component Responsibilities

Component	Responsibility	Typical Implementation
Workflow Coordinator	协调多个技能的执行顺序、依赖管理	kb-workflow 主控模块
State Manager	步骤级状态持久化、断点续传支持	workflow_state.json + 原子写入
Error Handler	错误分类、降级策略、重试逻辑	统一错误处理模块
Skill Registry	技能发现、能力声明、Schema 注册	SKILL.md + JSON Schema
Platform Skills	平台特定内容获取（B站/知乎）	kb-bilibili-, kb-zhihu-
Processing Skills	内容转换（语音识别/OCR）	kb-whisper, kb-funasr, kb-ocr
Analysis Skills	AI 深度分析、报告生成	kb-deep-analyzer

Recommended Project Structure

kb-skills/
+-- skills/                      # 技能模块目录
|   +-- core/                    # 核心基础设施
|   |   +-- state_manager.py     # 状态管理
|   |   +-- schema_registry.py   # Schema 注册
|   |   +-- pipeline.py          # 管道执行引擎
|   |   +-- error_handler.py     # 错误处理
|   |
|   +-- platforms/               # 平台技能
|   |   +-- bilibili/            # B站技能组
|   |   |   +-- SKILL.md
|   |   |   +-- metadata.py
|   |   |   +-- content.py
|   |   +-- zhihu/               # 知乎技能组
|   |       +-- SKILL.md
|   |       +-- metadata.py
|   |       +-- content.py
|   |
|   +-- processing/              # 处理技能
|   |   +-- whisper/             # 语音识别
|   |   +-- funasr/              # 中文语音识别
|   |   +-- ocr/                 # OCR 处理
|   |
|   +-- analysis/                # 分析技能
|       +-- deep_analyzer/       # 深度分析
|
+-- schemas/                     # JSON Schema 定义
|   +-- workflow_state.schema.json
|   +-- skill_output.schema.json
|   +-- platform_metadata.schema.json
|
+-- workflows/                   # 工作流定义
|   +-- video_pipeline.yaml      # 视频处理流程
|   +-- article_pipeline.yaml    # 文章处理流程
|
+-- cli.py                       # 统一 CLI 入口
+-- config.py                    # 配置管理

Structure Rationale

skills/core/: 核心基础设施，所有技能共享的状态管理、Schema 验证和管道执行
skills/platforms/: 按平台组织，每个平台有自己的元数据提取和内容下载技能
skills/processing/: 内容转换技能，独立于平台
skills/analysis/: AI 分析技能，作为管道的最后阶段
schemas/: 统一的 JSON Schema 定义，确保接口一致性
workflows/: 声明式工作流定义，指定技能组合和执行顺序

Architectural Patterns

Pattern 1: Pipeline Orchestration (管道编排)

What: 将处理流程建模为一系列串联的步骤，每个步骤接收输入、执行处理、产生输出。

When to use: 可预测的、重复性的任务，如内容处理管道。

Trade-offs:

优点：确定性执行、易于调试、支持断点续传
缺点：灵活性有限，不适合需要动态决策的场景

Example:

# Pipeline definition
class ContentPipeline:
    steps = [
        ("parse", LinkParserSkill),
        ("metadata", MetadataSkill),      # depends on parse
        ("download", ContentSkill),       # depends on metadata
        ("transcribe", TranscribeSkill),  # conditional: only for video
        ("analyze", AnalyzeSkill),        # depends on download + transcribe
    ]

    def run(self, url: str, state: WorkflowState):
        for step_name, skill in self.steps:
            if state.is_completed(step_name):
                continue  # Skip completed steps

            result = skill.execute(state.get_step_input(step_name))
            state.mark_completed(step_name, result)

            if not result.success:
                return self.handle_error(result.error)

Pattern 2: Step-Level State Management (步骤级状态管理)

What: 每个主要步骤完成后立即持久化状态，支持从任意断点恢复执行。

When to use: 长时间运行的批处理任务，可能因网络、API 限制等原因中断。

Trade-offs:

优点：断点续传、执行追踪、幂等性保证
缺点：状态管理开销、需要设计状态 Schema

Example:

# workflow_state.json
{
    "url": "https://www.bilibili.com/video/BV1xx411c7mD",
    "platform": "bilibili",
    "content_type": "video",
    "created_at": "2026-03-13T10:00:00Z",
    "updated_at": "2026-03-13T10:15:00Z",
    "steps": {
        "parse": {
            "status": "completed",
            "output": {"video_id": "BV1xx411c7mD"},
            "completed_at": "2026-03-13T10:00:05Z"
        },
        "metadata": {
            "status": "completed",
            "output": {"title": "...", "duration": 1800},
            "completed_at": "2026-03-13T10:00:30Z"
        },
        "download": {
            "status": "in_progress",
            "output": null,
            "started_at": "2026-03-13T10:01:00Z"
        }
    }
}

Pattern 3: Schema-First Interface Design (Schema 优先接口设计)

What: 先定义输入输出的 JSON Schema，再实现技能逻辑。所有技能遵守统一的接口规范。

When to use: 多人协作、需要接口稳定性和可测试性的项目。

Trade-offs:

优点：接口清晰、易于测试、支持自动验证
缺点：前期设计成本、Schema 维护负担

Example:

// schemas/skill_output.schema.json
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "required": ["success", "data"],
    "properties": {
        "success": {"type": "boolean"},
        "data": {"type": "object"},
        "error": {
            "type": "object",
            "properties": {
                "code": {"type": "string"},
                "message": {"type": "string"},
                "detail": {"type": "string"}
            }
        },
        "metadata": {
            "type": "object",
            "properties": {
                "duration_ms": {"type": "number"},
                "tokens_used": {"type": "number"}
            }
        }
    }
}

Pattern 4: Idempotent Execution (幂等执行)

What: 相同输入重复执行产生相同结果，不会产生副作用（如重复文件）。

When to use: 任何可能需要重试的处理步骤。

Trade-offs:

优点：安全重试、简化错误恢复
缺点：需要设计去重逻辑

Example:

class ContentSkill:
    def execute(self, input: dict) -> SkillOutput:
        # Generate deterministic filename from input
        content_hash = hashlib.sha256(input['url'].encode()).hexdigest()[:12]
        output_path = self.get_output_dir() / f"{content_hash}.mp4"

        # Check if already exists
        if output_path.exists():
            return SkillOutput(success=True, data={"path": str(output_path), "cached": True})

        # Download content
        self.download_content(input['url'], output_path)
        return SkillOutput(success=True, data={"path": str(output_path), "cached": False})

Data Flow

Content Processing Pipeline

[URL Input]
    |
    v
+-------------+     +----------------+
| Link Parser | --> | Platform/Type  |
+-------------+     | Detection      |
                    +----------------+
                           |
           +---------------+---------------+
           |                               |
           v                               v
    +-------------+                 +-------------+
    | Bilibili    |                 | Zhihu       |
    | Pipeline    |                 | Pipeline    |
    +-------------+                 +-------------+
           |                               |
    +------+------+                +-------+-------+
    |             |                |               |
    v             v                v               v
[Metadata]   [Download]       [Metadata]     [Download]
    |             |                |               |
    +------+------+                +-------+-------+
           |                               |
           v                               v
    +-------------+                 +-------------+
    | Video?      |                 | Article     |
    | - Whisper   |                 | - Direct    |
    | - FunASR    |                 |   Extract   |
    | - OCR       |                 +-------------+
    +-------------+                       |
           |                              |
           +---------------+--------------+
                           |
                           v
                  +-----------------+
                  | Deep Analyzer   |
                  | (AI Analysis)   |
                  +-----------------+
                           |
                           v
                  +-----------------+
                  | Archive to      |
                  | Knowledge Base  |
                  +-----------------+

State Management Flow

[Workflow Start]
       |
       v
+------------------+
| Load/Init State  | <-- workflow_state.json
+------------------+
       |
       v
+------------------+     +------------------+
| For Each Step    | --> | Check Status     |
+------------------+     | - completed? skip|
       |                 | - failed? retry  |
       v                 | - pending? run   |
+------------------+     +------------------+
| Execute Step     |            |
+------------------+            v
       |               +------------------+
       v               | Mark Completed   |
+------------------+    | Save State       |
| Validate Output  | -->| (atomic write)   |
+------------------+    +------------------+
       |
       v
[Next Step / Complete]

Key Data Flows

URL to Metadata: URL -> Link Parser -> (platform, type, id) -> Platform Metadata Skill -> Metadata JSON
Metadata to Content: Metadata JSON -> Content Skill -> (video file / article text)
Content to Transcript: Video File -> Transcribe Skill (Whisper/FunASR) -> Transcript JSON
Content + Transcript to Analysis: Content + Transcript -> Deep Analyzer -> Analysis Report

Scaling Considerations

Scale	Architecture Adjustments
1-100 items/day	当前架构足够，单进程执行
100-1000 items/day	添加任务队列，并行处理独立任务
1000+ items/day	分布式任务调度，API 请求限流，缓存层

Scaling Priorities

First bottleneck: 外部 API 限流（B站/知乎）。解决方案：请求队列 + 指数退避重试
Second bottleneck: AI 分析耗时。解决方案：批处理 + 并行分析

Anti-Patterns

Anti-Pattern 1: Global State Mutation

What people do: 在多个技能间共享可变全局状态

Why it's wrong: 难以追踪状态变化、难以测试、难以恢复

Do this instead: 每个步骤从 state 读取输入，将输出写回 state，状态变更通过 state manager 集中管理

Anti-Pattern 2: Implicit Dependencies

What people do: 技能之间通过隐式约定传递数据（如固定文件路径）

Why it's wrong: 脆弱的耦合，修改一个技能可能破坏其他技能

Do this instead: 在 workflow_state.json 中显式声明依赖关系，通过 state manager 传递数据

Anti-Pattern 3: Silent Failure

What people do: 捕获异常后只打印日志，继续执行

Why it's wrong: 下游步骤收到无效输入，产生难以调试的错误

Do this instead: 明确的失败处理策略 - 要么终止并记录，要么降级并提供默认值

Anti-Pattern 4: Non-Idempotent Operations

What people do: 每次执行都创建新文件（如 output_1.mp4, output_2.mp4）

Why it's wrong: 重试产生垃圾文件，磁盘占用增长

Do this instead: 基于输入生成确定性文件名，已存在则跳过

Integration Points

External Services

Service	Integration Pattern	Notes
Bilibili API	HTTP + Cookie Auth	需要 BILIBILI_COOKIE 环境变量
Zhihu API	HTTP + Cookie Auth	需要 ZHIHU_COOKIE 环境变量
AI Provider	OpenAI-compatible API	需要 AI_API_KEY 环境变量
HuggingFace	API + Local Cache	需要 HF_TOKEN 环境变量
Git	Command Line	需要 KB_GIT_USER/EMAIL 配置

Internal Boundaries

Boundary	Communication	Notes
Orchestrator <-> Skills	JSON via stdin/stdout	标准化输入输出格式
Skills <-> State Manager	File I/O	workflow_state.json
Skills <-> External APIs	HTTP	统一错误处理和重试

Build Order Implications

基于组件依赖关系，建议的构建顺序：

Phase 1: Foundation (基础设施)

State Manager - 所有后续组件的基础
Schema Registry - 定义统一的输入输出格式
Error Handler - 统一错误处理模式

Phase 2: Core Skills (核心技能)

Link Parser - 入口点，无依赖
Platform Metadata Skills - 依赖 Link Parser 输出
Platform Content Skills - 依赖 Metadata 输出

Phase 3: Processing Skills (处理技能)

Whisper/FunASR - 依赖视频文件
OCR Processor - 依赖视频截图

Phase 4: Analysis Skills (分析技能)

Deep Analyzer - 依赖内容和转录文本

Phase 5: Orchestration (编排层)

Pipeline Engine - 组合所有技能
Workflow Definitions - 声明式流程定义

Sources

AI Architecture Patterns 101: Workflows, Agents, MCPs, and A2A Systems - HIGH confidence, comprehensive pattern overview
Data Pipeline Architecture: 5 Design Patterns with Examples - Dagster - HIGH confidence, pipeline patterns
Agentic Workflows in 2026: The ultimate guide - Vellum - HIGH confidence, agent architecture patterns
State of Open Source Workflow Orchestration Systems 2025 - MEDIUM confidence, ecosystem overview
ML Pipeline Architecture Design Patterns - Neptune.ai - MEDIUM confidence, ML-specific patterns

Architecture research for: Skill-based Workflow Orchestration System Original research: 2026-03-13 Updated for v1.3 Anti-Crawler: 2026-03-18

v1.6 语义分段架构研究

Milestone: LLM-based Semantic Segmentation Researched: 2026-03-21 Focus: 使用 LLM 替代基于规则的语义分段，实现智能章节识别和标题生成

Executive Summary

本文档研究如何将 LLM 语义分段集成到现有的 SRT 处理架构中。基于对现有代码（kb_workflow/srt/ 模块和 SRTProcessor）的分析，推荐采用 渐进式迁移模式，保留现有 RuleBasedSegmenter 作为 fallback，新增 LLMBasedSegmenter 作为主分段器。

关键架构决策：

新增 SemanticSegmenter 组件 — 替代/增强现有的 segmenter.py
单次 LLM 调用策略 — 完整上下文分析，避免边界拼接问题
三层降级策略 — LLM 失败 → 响应解析失败 → AI 不可用 → RuleBasedSegmenter
渐进式集成 — 不破坏现有功能，新组件作为可选增强

Current Architecture (v1.5)

┌─────────────────────────────────────────────────────────────┐
│                    SRTProcessor                              │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐        │
│  │ parser.py   │→ │segmenter.py  │→ │corrector.py │        │
│  │ (SRT解析)   │  │ (规则分段)    │  │ (ASR纠错)   │        │
│  └─────────────┘  └──────────────┘  └─────────────┘        │
│         │                  │                  │              │
│         ↓                  ↓                  ↓              │
│   List[Segment]    List[str]         List[str]              │
│   (时间戳+文本)      (段落)            (纠错后)              │
└─────────────────────────────────────────────────────────────┘

现有组件：

parser.py: 解析 SRT → Segment(id, start_time, end_time, text)
segmenter.py: SemanticSegmenter (基于时间间隔3s + 长度500字符规则)
corrector.py: ASRCorrector (使用 SimpleAIClient 纠错)

Recommended Architecture (v1.6)

┌─────────────────────────────────────────────────────────────────┐
│                         SRTProcessor                             │
│  ┌─────────────┐                                                │
│  │ parser.py   │                                                │
│  │ (SRT解析)   │                                                │
│  └──────┬──────┘                                                │
│         │                                                       │
│         ↓ List[Segment] (id, start_time, end_time, text)        │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │         semantic_segmenter.py (NEW)                       │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 1. 构建 Prompt (视频信息 + 字幕内容)               │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 2. 调用 LLM (SimpleAIClient.generate)              │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 3. 解析响应 (## 标题 + 内容)                       │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  │                     ↓                                     │   │
│  │  ┌────────────────────────────────────────────────────┐  │   │
│  │  │ 4. 返回 List[SemanticParagraph]                     │  │   │
│  │  │    - title: str (章节标题)                          │  │   │
│  │  │    - content: str (段落内容)                        │  │   │
│  │  │    - time_range: (start, end) (可选)               │  │   │
│  │  └──────────────────┬─────────────────────────────────┘  │   │
│  └─────────────────────┼─────────────────────────────────────┘   │
│                        ↓                                         │
│            List[SemanticParagraph]                               │
│         (标题 + 结构化内容)                                       │
│                        ↓                                         │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    Fallback Strategy                     │   │
│  │  LLM 失败 → 响应解析失败 → AI 不可用                      │   │
│  │      ↓                                                    │   │
│  │  segmenter.py (RuleBasedSegmenter) - 保留作为 fallback   │   │
│  └────────────────────┬─────────────────────────────────────┘   │
│                       ↓                                          │
│            List[str] (向后兼容)                                  │
│                       ↓                                          │
│  ┌─────────────┐  ┌──────────────┐                              │
│  │corrector.py │  │ Markdown生成 │                              │
│  │ (ASR纠错)   │  │ (标题+内容)   │                              │
│  └─────────────┘  └──────────────┘                              │
└─────────────────────────────────────────────────────────────────┘

Component Boundaries

Component	Responsibility	Communicates With
parser.py	解析 SRT 格式 → Segment 对象	无（输入层）
semantic_segmenter.py (NEW)	LLM 语义分析 → 结构化段落	parser (输入), corrector (输出), SimpleAIClient
segmenter.py (EXIST)	基于规则的分段（fallback）	semantic_segmenter (降级)
corrector.py	ASR 纠错	semantic_segmenter (输入), SimpleAIClient
SRTProcessor	编排处理流程	所有模块

Data Flow Changes

旧流程 (v1.5):

SRT File → parser.parse_srt_file() → List[Segment]
    ↓
segmenter.segment() (基于规则)
    ↓
List[str] (段落)
    ↓
corrector.correct() (逐段落纠错)
    ↓
生成 Markdown

新流程 (v1.6):

SRT File → parser.parse_srt_file() → List[Segment]
    ↓
[NEW] semantic_segmenter.segment()
    ├─ 构建 Prompt (视频信息 + 字幕文本)
    ├─ 调用 LLM (SimpleAIClient)
    ├─ 解析响应 (## 标题 + 内容)
    └─ 返回 List[SemanticParagraph]
    ├─ title: str
    ├─ content: str
    └─ time_range: (float, float) (可选)
    ↓
[FALLBACK] segmenter.segment() (LLM 失败时)
    ↓
List[SemanticParagraph] → flatten → List[str]
    ↓
corrector.correct() (逐段落纠错)
    ↓
生成 Markdown (标题 + 内容结构)

Response Parsing Strategy

LLM 响应格式要求:

## 第一部分：机器学习基础概念
段落内容1...
段落内容2...

## 第二部分：深度学习原理
段落内容3...
段落内容4...

解析方法:

@dataclass
class SemanticParagraph:
    """语义分段结果"""
    title: str           # 章节标题
    content: str         # 段落内容
    start_time: float    # 开始时间（可选）
    end_time: float      # 结束时间（可选）

def parse_llm_response(response: str) -> List[SemanticParagraph]:
    """解析 LLM 响应为结构化段落"""
    paragraphs = []
    current_title = "无标题"
    current_content = []

    for line in response.split('\n'):
        if line.startswith('##'):
            # 保存上一个段落
            if current_content:
                content = '\n'.join(current_content).strip()
                if content:
                    paragraphs.append(SemanticParagraph(
                        title=current_title,
                        content=content
                    ))
            # 开始新段落
            current_title = line.lstrip('#').strip()
            current_content = []
        else:
            current_content.append(line)

    # 保存最后一个段落
    if current_content:
        content = '\n'.join(current_content).strip()
        if content:
            paragraphs.append(SemanticParagraph(
                title=current_title,
                content=content
            ))

    return paragraphs

LLM Call Strategy

推荐：单次调用

原因:

上下文完整性 — LLM 理解完整视频内容，分段更准确
避免边界问题 — 多次调用的拼接处容易出现语义不连贯
简化实现 — 单次调用更容易测试和调试

Prompt 设计:

SEGMENTATION_PROMPT = """你是一个视频内容分析师。请将以下字幕内容按语义分段，并为每个段落生成标题。

【视频信息】
标题：{video_title}
时长：{duration}

【字幕内容】
{concatenated_segments}

【输出要求】
1. 按语义完整性分段（保持话题/论点完整）
2. 识别主题变化边界
3. 为每个段落生成简洁标题（不超过20字）
4. 输出格式：

## 第一部分：{标题}
{段落内容}

## 第二部分：{标题}
{段落内容}

...（以此类推）

注意：
- 段落长度建议 200-800 字符
- 在自然句号处分段
- 标题要概括该段核心内容
"""

输入处理选项:

选项 1: 纯文本（推荐）

# 合并所有 Segment.text
full_text = '\n'.join(seg.text for seg in segments)

选项 2: 带时间戳标记（可选，用于辅助理解）

# 添加时间戳辅助 LLM 理解结构
marked_text = '\n'.join(
    f"[{format_time(seg.start_time)}] {seg.text}"
    for seg in segments
)

替代方案：流式调用

用于超长视频（>1小时字幕）
分块调用 LLM，每块 10-15 分钟字幕
需要额外的边界拼接逻辑
建议： 初期使用单次调用，后续根据实际需求优化

Fallback Strategy

三层降级策略:

class LLMBasedSegmenter:
    def __init__(self, ai_client: SimpleAIClient):
        self.ai_client = ai_client
        self.rule_based = SemanticSegmenter()  # Fallback

    def segment(self, segments: List[Segment]) -> List[SemanticParagraph]:
        # Level 3: AI 客户端不可用
        if not self.ai_client or not self.ai_client.api_key:
            logger.info("AI client not available, using rule-based")
            return self._to_semantic_paragraphs(
                self.rule_based.segment(segments)
            )

        try:
            # Level 1: LLM 调用
            prompt = self._build_prompt(segments)
            response = self.ai_client.generate(prompt)

            # Level 2: 响应解析
            try:
                paragraphs = self._parse_llm_response(response)
                if not paragraphs:
                    raise ValueError("Empty parsed result")
                return paragraphs
            except Exception as e:
                logger.warning(f"Failed to parse LLM response: {e}")
                return self._to_semantic_paragraphs(
                    self.rule_based.segment(segments)
                )

        except Exception as e:
            logger.warning(f"LLM call failed: {e}, using rule-based")
            return self._to_semantic_paragraphs(
                self.rule_based.segment(segments)
            )

Suggested Build Order

Phase 1: 基础组件（不影响现有功能）

创建 kb_workflow/srt/semantic_segmenter.py
实现 SemanticParagraph 数据类
实现 parse_llm_response() 方法（独立于 LLM 调用）
编写单元测试（mock LLM 响应，测试解析逻辑）

Phase 2: LLM 集成 5. 实现 _build_prompt() 方法 6. 集成 SimpleAIClient 7. 实现 segment() 主方法 8. 实现降级策略（fallback） 9. 编写集成测试（使用真实 LLM 或 mock）

Phase 3: 处理器集成 10. 修改 SRTProcessor.__init__()： - 添加 use_llm_segmenter: bool = False 参数 - 条件性初始化 llm_segmenter 11. 修改 SRTProcessor.process()： - 调用新分段器或保留旧分段器 - 处理 SemanticParagraph 输出 12. 更新 _generate_markdown()： - 处理标题和内容结构 - 生成带层级的 Markdown 13. 保留 RuleBasedSegmenter 作为 fallback

Phase 4: 输出优化 14. 优化 Markdown 格式（标题层级 ###） 15. 添加时间戳（可选，在段落中标注时间范围） 16. 更新文档和测试

Schema Changes

新增数据类:

@dataclass
class SemanticParagraph:
    """语义分段结果"""
    title: str           # 章节标题
    content: str         # 段落内容
    start_time: Optional[float] = None  # 开始时间（可选）
    end_time: Optional[float] = None    # 结束时间（可选）

修改 SRTOutput (向后兼容):

class SRTOutput(BaseModel):
    success: bool
    output_file: str
    paragraphs: List[str]                    # 保留向后兼容
    semantic_paragraphs: Optional[List[SemanticParagraph]] = None  # 新增
    tags: List[str]
    corrected: bool

Patterns to Follow

Pattern 1: 渐进式迁移

What: 保留现有组件，新组件作为可选增强 When: 重构关键路径时 Example:

class SRTProcessor:
    def __init__(self, use_llm_segmenter: bool = False):
        self.rule_based_segmenter = SemanticSegmenter()  # 保留
        if use_llm_segmenter:
            self.llm_segmenter = LLMBasedSegmenter()     # 新增

    async def process(self, input_data):
        # 优先使用 LLM，失败时降级
        if hasattr(self, 'llm_segmenter'):
            try:
                paragraphs = self.llm_segmenter.segment(segments)
            except Exception:
                paragraphs = self.rule_based_segmenter.segment(segments)
        else:
            paragraphs = self.rule_based_segmenter.segment(segments)

Pattern 2: 协议兼容性

What: 新旧组件共享相同接口 When: 替换组件时 Example:

# 两个分段器实现相同接口
class Segmenter(Protocol):
    def segment(self, segments: List[Segment]) -> List[str]:
        ...

class RuleBasedSegmenter:
    def segment(self, segments: List[Segment]) -> List[str]:
        # 基于规则
        ...

class LLMBasedSegmenter:
    def segment(self, segments: List[Segment]) -> List[str]:
        # 基于 LLM
        # 内部使用 SemanticParagraph，输出时合并标题和内容
        return [
            f"## {p.title}\n\n{p.content}"
            for p in self._segment_internal(segments)
        ]

Anti-Patterns to Avoid

Anti-Pattern 1: 破坏性替换

What: 直接删除 RuleBasedSegmenter Why bad: LLM 可能失败，失去降级选项 Instead: 保留旧组件作为 fallback

Anti-Pattern 2: 紧耦合 Prompt 逻辑

What: 将 prompt 构建硬编码在 segmenter 中 Why bad: 难以测试和优化 Instead: 提取为独立方法或模板，便于测试

Anti-Pattern 3: 忽略时间戳信息

What: 仅传递文本给 LLM，忽略时间信息 Why bad: LLM 无法理解视频结构（停顿、章节） Instead: 在 prompt 中包含时间戳标记或视频时长信息

Scalability Considerations

Concern	At 100 videos	At 10K videos	At 1M videos
LLM 调用延迟	可接受（~10-30s/视频）	需要并发处理	需要队列+批处理
LLM 成本	可忽略	需要监控	需要缓存+降级策略
错误率	低，手动处理	需要自动重试	需要监控告警
响应解析	简单	需要容错增强	需要多模型支持

建议:

初期：单线程处理，完整错误日志
中期：添加并发处理（asyncio.Semaphore 限制并发数）
长期：分布式队列（Celery/RQ）+ 结果缓存

Integration Points Summary

新增集成点:

SRTProcessor.__init__() → 条件性初始化 LLMBasedSegmenter
SRTProcessor.process() → 调用新分段器或旧分段器
_generate_markdown() → 处理 SemanticParagraph 结构

保留集成点:

parser.parse_srt_file() — 不变
corrector.correct() — 不变（处理 List[str]）
WorkflowStateManager — 不变

修改点:

kb_workflow/srt/__init__.py — 导出 SemanticParagraph
kb_workflow/pipeline/schemas.py — 更新 SRTOutput

v1.6 Sources

现有架构分析: kb_workflow/srt/ 模块
SRTProcessor 实现: kb_workflow/pipeline/processors/srt.py
SimpleAIClient 实现: kb_workflow/deep_analyzer/__init__.py
测试用例: tests/test_srt_processor.py
项目上下文: .planning/PROJECT.md

Architecture research for: LLM-based Semantic Segmentation Research date: 2026-03-21

69 KiB Raw Blame History Unescape Escape

Architecture Research

v1.3 补充：反爬虫优化架构

Executive Summary

Current Architecture Overview

Recommended New Architecture (v1.3)

Component Design

1. Proxy Pool Manager

2. Behavior Camouflage

3. CAPTCHA Detection

4. Enhanced Retry Mechanism

5. Unified Anti-Detection Decorator

Integration Points

Integration with ZhihuCrawler

Integration with ZhihuAdapter

Integration with Configuration

Data Flow

Request Flow with Anti-Detection

Error Recovery Flow

Recommended Build Order

Phase 1: Foundation (No external dependencies)

Phase 2: Integration

Phase 3: Unified API

Phase 4: Platform Integration

Phase 5: Bilibili Support (if needed)

Anti-Patterns (v1.3 Specific)

Anti-Pattern 1: Hardcoding Proxy Settings in Crawler

Anti-Pattern 2: Retry All Errors Blindly

Anti-Pattern 3: Fixed Delay Intervals

Anti-Pattern 4: Captcha Auto-Solve Without Fallback

Scaling Considerations (v1.3)

Scaling Priorities

v1.3 Sources

Original Architecture Research (v1.0)

Standard Architecture

System Overview

Component Responsibilities

Recommended Project Structure

Structure Rationale

Architectural Patterns

Pattern 1: Pipeline Orchestration (管道编排)

Pattern 2: Step-Level State Management (步骤级状态管理)

Pattern 3: Schema-First Interface Design (Schema 优先接口设计)

Pattern 4: Idempotent Execution (幂等执行)

Data Flow

Content Processing Pipeline

State Management Flow

Key Data Flows

Scaling Considerations

Scaling Priorities

Anti-Patterns

Anti-Pattern 1: Global State Mutation

Anti-Pattern 2: Implicit Dependencies

Anti-Pattern 3: Silent Failure

Anti-Pattern 4: Non-Idempotent Operations

Integration Points

External Services

Internal Boundaries

Build Order Implications

Phase 1: Foundation (基础设施)

Phase 2: Core Skills (核心技能)

Phase 3: Processing Skills (处理技能)

Phase 4: Analysis Skills (分析技能)

Phase 5: Orchestration (编排层)

Sources

v1.6 语义分段架构研究

Executive Summary

Current Architecture (v1.5)

Recommended Architecture (v1.6)

Component Boundaries

Data Flow Changes

Response Parsing Strategy

LLM Call Strategy

Fallback Strategy

Suggested Build Order

Schema Changes

Patterns to Follow

Pattern 1: 渐进式迁移

Pattern 2: 协议兼容性

Anti-Patterns to Avoid

69 KiB

Raw Blame History