1936 lines
69 KiB
Markdown
1936 lines
69 KiB
Markdown
# Architecture Research
|
||
|
||
**Domain:** Skill-based Workflow Orchestration System
|
||
**Researched:** 2026-03-13 (updated 2026-03-18 for v1.3)
|
||
**Confidence:** HIGH
|
||
|
||
---
|
||
|
||
## v1.3 补充：反爬虫优化架构
|
||
|
||
**Milestone:** Anti-Crawler Optimization
|
||
**Researched:** 2026-03-18
|
||
**Focus:** 代理池、行为伪装、验证码检测、错误恢复
|
||
|
||
---
|
||
|
||
## Executive Summary
|
||
|
||
本文档研究如何将反爬虫优化功能集成到现有的 kb-skills 架构中。基于对现有代码结构和反爬虫最佳实践的分析，推荐采用 **模块化的中间件/装饰器模式**，将反爬虫功能作为独立组件注入到现有的 Pipeline 和 Crawler 中。
|
||
|
||
关键架构决策：
|
||
1. **代理池管理** — 新建 `kb_workflow/crawler/proxy_pool.py` 模块，作为独立服务
|
||
2. **行为伪装** — 通过装饰器/中间件模式集成到 Playwright 调用
|
||
3. **验证码处理** — 检测-暂停机制，不自动解决（避免付费服务依赖）
|
||
4. **错误恢复** — 增强现有 `retry.py`，添加 jitter 和状态感知
|
||
|
||
---
|
||
|
||
## Current Architecture Overview
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ CLI Layer │
|
||
│ kb_workflow/cli.py - process/login subcommands │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Pipeline Layer │
|
||
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
|
||
│ │ ParseProcessor │→ │DownloadProcessor│→ │ ASRProcessor │ → ... │
|
||
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
|
||
│ │ │ │ │
|
||
│ ↓ ↓ ↓ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Platform Adapters (BilibiliAdapter, ZhihuAdapter) │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Crawler Layer │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ ZhihuCrawler (Playwright) - API Interception + Cookie Auth │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Infrastructure │
|
||
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
|
||
│ │ CrawlerConfig │ │ KBSettings │ │ WorkflowState│ │
|
||
│ └───────────────┘ └───────────────┘ └───────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Recommended New Architecture (v1.3)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ CLI Layer │
|
||
│ kb_workflow/cli.py - process/login subcommands │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Pipeline Layer │
|
||
│ [Existing processors - unchanged] │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Platform Adapters Layer │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ ZhihuAdapter / BilibiliAdapter │ │
|
||
│ │ (uses @with_anti_detection decorator) │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Anti-Crawler Layer (NEW) │
|
||
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
|
||
│ │ ProxyPool │ │BehaviorCamouflag│ │ CaptchaDetector │ │
|
||
│ │ Manager │ │ e │ │ │ │
|
||
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
|
||
│ │ │ │ │
|
||
│ └────────────────────┴────────────────────┘ │
|
||
│ │ │
|
||
│ ┌───────────┴───────────┐ │
|
||
│ │ @with_anti_detection │ │
|
||
│ │ (decorator) │ │
|
||
│ └───────────────────────┘ │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Crawler Layer │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ ZhihuCrawler (enhanced with anti-detection hooks) │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ Infrastructure │
|
||
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
|
||
│ │CrawlerConfig │ │AntiCrawlerConf│ │ KBSettings │ │
|
||
│ │ (enhanced) │ │ (NEW) │ │ (enhanced) │ │
|
||
│ └───────────────┘ └───────────────┘ └───────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Component Design
|
||
|
||
### 1. Proxy Pool Manager
|
||
|
||
**Location:** `kb_workflow/crawler/proxy_pool.py`
|
||
|
||
**Rationale:** 独立模块，与现有 `CrawlerConfig` 分离，因为：
|
||
- 代理池是状态性的（需要跟踪健康状态、使用次数）
|
||
- 可能被多个 Crawler 共享
|
||
- 需要独立的配置源（环境变量、文件、API）
|
||
|
||
**Design:**
|
||
|
||
```python
|
||
# kb_workflow/crawler/proxy_pool.py
|
||
|
||
from typing import Optional, List, Dict, Any
|
||
from pydantic import BaseModel
|
||
import random
|
||
from dataclasses import dataclass, field
|
||
from datetime import datetime
|
||
|
||
@dataclass
|
||
class ProxyInfo:
|
||
"""代理信息"""
|
||
url: str
|
||
protocol: str = "http" # http, https, socks5
|
||
username: Optional[str] = None
|
||
password: Optional[str] = None
|
||
last_used: Optional[datetime] = None
|
||
fail_count: int = 0
|
||
success_count: int = 0
|
||
is_healthy: bool = True
|
||
|
||
def to_playwright_format(self) -> Dict[str, Any]:
|
||
config = {"server": self.url}
|
||
if self.username:
|
||
config["username"] = self.username
|
||
if self.password:
|
||
config["password"] = self.password
|
||
return config
|
||
|
||
|
||
class ProxyPoolConfig(BaseModel):
|
||
"""代理池配置"""
|
||
enabled: bool = False
|
||
# 代理列表（支持多个代理）
|
||
proxies: List[str] = []
|
||
# 轮换策略: round_robin, random, least_used
|
||
rotation_strategy: str = "random"
|
||
# 健康检查
|
||
health_check_enabled: bool = True
|
||
max_fail_count: int = 3 # 连续失败 N 次后标记为不健康
|
||
# 冷却时间（秒）
|
||
cooldown_seconds: float = 0 # 同一代理两次使用之间的最小间隔
|
||
|
||
@classmethod
|
||
def from_env(cls) -> "ProxyPoolConfig":
|
||
"""从环境变量加载代理池配置"""
|
||
import os
|
||
proxies_str = os.environ.get("KB_PROXY_POOL", "")
|
||
proxies = [p.strip() for p in proxies_str.split(",") if p.strip()]
|
||
return cls(
|
||
enabled=len(proxies) > 0,
|
||
proxies=proxies,
|
||
rotation_strategy=os.environ.get("KB_PROXY_STRATEGY", "random"),
|
||
)
|
||
|
||
|
||
class ProxyPoolManager:
|
||
"""代理池管理器
|
||
|
||
职责：
|
||
- 管理代理列表
|
||
- 轮换策略
|
||
- 健康状态跟踪
|
||
"""
|
||
|
||
def __init__(self, config: Optional[ProxyPoolConfig] = None):
|
||
self.config = config or ProxyPoolConfig.from_env()
|
||
self._proxies: List[ProxyInfo] = []
|
||
self._current_index = 0
|
||
self._init_proxies()
|
||
|
||
def _init_proxies(self):
|
||
"""初始化代理列表"""
|
||
for proxy_url in self.config.proxies:
|
||
self._proxies.append(ProxyInfo(url=proxy_url))
|
||
|
||
def get_proxy(self) -> Optional[ProxyInfo]:
|
||
"""获取下一个可用代理"""
|
||
healthy_proxies = [p for p in self._proxies if p.is_healthy]
|
||
if not healthy_proxies:
|
||
return None
|
||
|
||
if self.config.rotation_strategy == "round_robin":
|
||
proxy = healthy_proxies[self._current_index % len(healthy_proxies)]
|
||
self._current_index += 1
|
||
else: # random
|
||
proxy = random.choice(healthy_proxies)
|
||
|
||
proxy.last_used = datetime.now()
|
||
return proxy
|
||
|
||
def report_success(self, proxy_url: str):
|
||
"""报告代理使用成功"""
|
||
for p in self._proxies:
|
||
if p.url == proxy_url:
|
||
p.success_count += 1
|
||
p.fail_count = 0 # 重置失败计数
|
||
break
|
||
|
||
def report_failure(self, proxy_url: str):
|
||
"""报告代理使用失败"""
|
||
for p in self._proxies:
|
||
if p.url == proxy_url:
|
||
p.fail_count += 1
|
||
if p.fail_count >= self.config.max_fail_count:
|
||
p.is_healthy = False
|
||
break
|
||
|
||
|
||
# Module-level singleton
|
||
_proxy_pool: Optional[ProxyPoolManager] = None
|
||
|
||
def get_proxy_pool() -> ProxyPoolManager:
|
||
"""获取代理池管理器单例"""
|
||
global _proxy_pool
|
||
if _proxy_pool is None:
|
||
_proxy_pool = ProxyPoolManager()
|
||
return _proxy_pool
|
||
```
|
||
|
||
---
|
||
|
||
### 2. Behavior Camouflage
|
||
|
||
**Location:** `kb_workflow/crawler/behavior.py`
|
||
|
||
**Rationale:** 行为伪装逻辑应该：
|
||
- 可以独立测试
|
||
- 可以被装饰器或直接调用
|
||
- 支持可配置的随机性
|
||
|
||
**Design:**
|
||
|
||
```python
|
||
# kb_workflow/crawler/behavior.py
|
||
|
||
import random
|
||
import time
|
||
from typing import Optional, Callable
|
||
from functools import wraps
|
||
from pydantic import BaseModel, Field
|
||
|
||
|
||
class BehaviorConfig(BaseModel):
|
||
"""行为伪装配置"""
|
||
# 请求间隔
|
||
min_delay: float = Field(default=1.0, description="最小延迟（秒）")
|
||
max_delay: float = Field(default=3.0, description="最大延迟（秒）")
|
||
|
||
# Jitter 配置（防止模式检测）
|
||
jitter_enabled: bool = True
|
||
jitter_range: float = Field(default=0.3, description="抖动范围（+/- 比例）")
|
||
|
||
# 人类模拟
|
||
mouse_movement_enabled: bool = False # Playwright 场景下可选
|
||
scroll_simulation_enabled: bool = False
|
||
|
||
# User-Agent 随机化
|
||
user_agent_rotation: bool = True
|
||
|
||
|
||
# 常用 User-Agent 池
|
||
USER_AGENTS = [
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
|
||
]
|
||
|
||
|
||
class BehaviorCamouflage:
|
||
"""行为伪装管理器"""
|
||
|
||
def __init__(self, config: Optional[BehaviorConfig] = None):
|
||
self.config = config or BehaviorConfig()
|
||
|
||
def get_random_delay(self) -> float:
|
||
"""获取随机延迟时间（带 jitter）"""
|
||
base_delay = random.uniform(self.config.min_delay, self.config.max_delay)
|
||
|
||
if self.config.jitter_enabled:
|
||
jitter = base_delay * self.config.jitter_range * random.uniform(-1, 1)
|
||
base_delay += jitter
|
||
|
||
return max(0.1, base_delay) # 确保非负
|
||
|
||
def apply_delay(self):
|
||
"""应用随机延迟"""
|
||
delay = self.get_random_delay()
|
||
time.sleep(delay)
|
||
return delay
|
||
|
||
def get_random_user_agent(self) -> str:
|
||
"""获取随机 User-Agent"""
|
||
if self.config.user_agent_rotation:
|
||
return random.choice(USER_AGENTS)
|
||
return USER_AGENTS[0]
|
||
|
||
def get_playwright_context_options(self) -> dict:
|
||
"""获取 Playwright context 选项（包含反检测设置）"""
|
||
options = {
|
||
"user_agent": self.get_random_user_agent(),
|
||
}
|
||
return options
|
||
|
||
|
||
def with_behavior_camouflage(config: Optional[BehaviorConfig] = None):
|
||
"""装饰器：为函数添加行为伪装
|
||
|
||
用于同步函数，在调用前添加随机延迟。
|
||
|
||
Example:
|
||
@with_behavior_camouflage()
|
||
def fetch_page(url):
|
||
return requests.get(url)
|
||
"""
|
||
camouflage = BehaviorCamouflage(config)
|
||
|
||
def decorator(func: Callable):
|
||
@wraps(func)
|
||
def wrapper(*args, **kwargs):
|
||
camouflage.apply_delay()
|
||
return func(*args, **kwargs)
|
||
return wrapper
|
||
|
||
return decorator
|
||
```
|
||
|
||
---
|
||
|
||
### 3. CAPTCHA Detection
|
||
|
||
**Location:** `kb_workflow/crawler/captcha.py`
|
||
|
||
**Rationale:**
|
||
- **不实现自动解决** — 根据项目约束（Out of Scope: 付费打码服务）
|
||
- **检测+暂停机制** — 检测到验证码时暂停并通知用户
|
||
|
||
**Design:**
|
||
|
||
```python
|
||
# kb_workflow/crawler/captcha.py
|
||
|
||
import re
|
||
from typing import Optional, Tuple
|
||
from enum import Enum
|
||
from dataclasses import dataclass
|
||
from pydantic import BaseModel
|
||
|
||
from kb_workflow.pipeline.errors import KBError, ErrorType
|
||
|
||
|
||
class CaptchaType(str, Enum):
|
||
"""验证码类型"""
|
||
RECAPTCHA_V2 = "recaptcha_v2"
|
||
RECAPTCHA_V3 = "recaptcha_v3"
|
||
HCAPTCHA = "hcaptcha"
|
||
CLOUDFLARE = "cloudflare"
|
||
IMAGE_CAPTCHA = "image_captcha"
|
||
SLIDER = "slider"
|
||
UNKNOWN = "unknown"
|
||
|
||
|
||
@dataclass
|
||
class CaptchaDetectionResult:
|
||
"""验证码检测结果"""
|
||
detected: bool
|
||
captcha_type: Optional[CaptchaType] = None
|
||
confidence: float = 0.0
|
||
indicators: list = None
|
||
|
||
def __post_init__(self):
|
||
if self.indicators is None:
|
||
self.indicators = []
|
||
|
||
|
||
class CaptchaDetector:
|
||
"""验证码检测器
|
||
|
||
职责：
|
||
- 检测页面是否包含验证码
|
||
- 识别验证码类型
|
||
- 不负责解决验证码（需要人工介入）
|
||
"""
|
||
|
||
# 验证码指示器（页面元素/URL/内容）
|
||
CAPTCHA_INDICATORS = {
|
||
CaptchaType.RECAPTCHA_V2: [
|
||
"g-recaptcha",
|
||
"recaptcha-checkbox",
|
||
"google.com/recaptcha",
|
||
],
|
||
CaptchaType.RECAPTCHA_V3: [
|
||
"grecaptcha.execute",
|
||
"recaptcha-v3",
|
||
],
|
||
CaptchaType.HCAPTCHA: [
|
||
"h-captcha",
|
||
"hcaptcha.com",
|
||
],
|
||
CaptchaType.CLOUDFLARE: [
|
||
"cloudflare",
|
||
"cf-browser-verification",
|
||
"challenge-platform",
|
||
],
|
||
CaptchaType.IMAGE_CAPTCHA: [
|
||
"captcha.png",
|
||
"verifycode",
|
||
"img captcha",
|
||
],
|
||
CaptchaType.SLIDER: [
|
||
"slider",
|
||
"slide-verify",
|
||
"drag verify",
|
||
],
|
||
}
|
||
|
||
def detect_from_html(self, html_content: str) -> CaptchaDetectionResult:
|
||
"""从 HTML 内容检测验证码"""
|
||
html_lower = html_content.lower()
|
||
indicators = []
|
||
|
||
for captcha_type, patterns in self.CAPTCHA_INDICATORS.items():
|
||
for pattern in patterns:
|
||
if pattern.lower() in html_lower:
|
||
indicators.append(pattern)
|
||
return CaptchaDetectionResult(
|
||
detected=True,
|
||
captcha_type=captcha_type,
|
||
confidence=0.8,
|
||
indicators=indicators,
|
||
)
|
||
|
||
return CaptchaDetectionResult(detected=False)
|
||
|
||
def detect_from_response(
|
||
self,
|
||
status_code: int,
|
||
headers: dict,
|
||
url: str
|
||
) -> CaptchaDetectionResult:
|
||
"""从 HTTP 响应检测验证码"""
|
||
# 403 通常表示被拦截或需要验证
|
||
if status_code == 403:
|
||
return CaptchaDetectionResult(
|
||
detected=True,
|
||
captcha_type=CaptchaType.UNKNOWN,
|
||
confidence=0.6,
|
||
indicators=["403 Forbidden"],
|
||
)
|
||
|
||
# 检查 URL 重定向
|
||
if "captcha" in url.lower() or "verify" in url.lower():
|
||
return CaptchaDetectionResult(
|
||
detected=True,
|
||
captcha_type=CaptchaType.UNKNOWN,
|
||
confidence=0.7,
|
||
indicators=[f"URL redirect: {url}"],
|
||
)
|
||
|
||
return CaptchaDetectionResult(detected=False)
|
||
|
||
|
||
class CaptchaError(KBError):
|
||
"""验证码错误
|
||
|
||
当检测到验证码时抛出，需要人工介入。
|
||
"""
|
||
|
||
def __init__(
|
||
self,
|
||
step_name: str,
|
||
input_summary: str,
|
||
captcha_type: CaptchaType,
|
||
suggestion: str = "请手动完成验证码后重试"
|
||
):
|
||
super().__init__(
|
||
message=f"检测到验证码: {captcha_type.value}",
|
||
error_type=ErrorType.AUTH,
|
||
step_name=step_name,
|
||
input_summary=input_summary,
|
||
suggestion=suggestion,
|
||
)
|
||
self.captcha_type = captcha_type
|
||
```
|
||
|
||
---
|
||
|
||
### 4. Enhanced Retry Mechanism
|
||
|
||
**Location:** `kb_workflow/pipeline/retry.py` (enhance existing)
|
||
|
||
**Rationale:**
|
||
- 现有实现使用 tenacity，已经很好
|
||
- 增强点：添加 jitter、状态感知、验证码感知
|
||
|
||
**Enhanced Design:**
|
||
|
||
```python
|
||
# kb_workflow/pipeline/retry.py (enhancements)
|
||
|
||
import random
|
||
import logging
|
||
from tenacity import (
|
||
retry,
|
||
stop_after_attempt,
|
||
wait_exponential,
|
||
retry_if_exception_type,
|
||
before_sleep_log,
|
||
wait_combine,
|
||
wait_random,
|
||
)
|
||
|
||
from .errors import NetworkError, AuthError
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
|
||
def wait_exponential_jitter(
|
||
multiplier: float = 1,
|
||
min: float = 2,
|
||
max: float = 10,
|
||
jitter: float = 0.5
|
||
):
|
||
"""指数退避 + Jitter
|
||
|
||
Args:
|
||
multiplier: 退避乘数
|
||
min: 最小等待时间
|
||
max: 最大等待时间
|
||
jitter: 随机抖动范围（0-1）
|
||
"""
|
||
return wait_combine(
|
||
wait_exponential(multiplier=multiplier, min=min, max=max),
|
||
wait_random(0, jitter)
|
||
)
|
||
|
||
|
||
# 增强版重试装饰器（带 jitter）
|
||
retry_on_network_error_with_jitter = retry(
|
||
stop=stop_after_attempt(3),
|
||
wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
|
||
retry=retry_if_exception_type(NetworkError),
|
||
before_sleep=before_sleep_log(logger, logging.WARNING),
|
||
reraise=True,
|
||
)
|
||
"""带 Jitter 的网络错误重试装饰器
|
||
|
||
在指数退避基础上增加随机抖动，避免多个请求同时重试的"惊群效应"。
|
||
"""
|
||
|
||
|
||
# 验证码感知重试（遇到 CaptchaError 不重试）
|
||
retry_on_transient_error = retry(
|
||
stop=stop_after_attempt(3),
|
||
wait=wait_exponential_jitter(multiplier=1, min=2, max=10, jitter=0.5),
|
||
retry=(
|
||
retry_if_exception_type(NetworkError) &
|
||
~retry_if_exception_type(AuthError) # AuthError 不重试
|
||
),
|
||
before_sleep=before_sleep_log(logger, logging.WARNING),
|
||
reraise=True,
|
||
)
|
||
"""瞬态错误重试装饰器
|
||
|
||
只重试网络错误，不重试认证错误（包括验证码）。
|
||
"""
|
||
```
|
||
|
||
---
|
||
|
||
### 5. Unified Anti-Detection Decorator
|
||
|
||
**Location:** `kb_workflow/crawler/anti_detection.py`
|
||
|
||
**Rationale:**
|
||
- 统一入口点，组合所有反爬虫功能
|
||
- 便于在 PlatformAdapter 中使用
|
||
|
||
**Design:**
|
||
|
||
```python
|
||
# kb_workflow/crawler/anti_detection.py
|
||
|
||
from typing import Optional, Callable, Dict, Any
|
||
from functools import wraps
|
||
from dataclasses import dataclass
|
||
|
||
from .proxy_pool import ProxyPoolManager, get_proxy_pool, ProxyInfo
|
||
from .behavior import BehaviorCamouflage, BehaviorConfig
|
||
from .captcha import CaptchaDetector, CaptchaError, CaptchaType
|
||
|
||
|
||
@dataclass
|
||
class AntiDetectionConfig:
|
||
"""反检测配置"""
|
||
proxy_enabled: bool = True
|
||
behavior_camouflage_enabled: bool = True
|
||
captcha_detection_enabled: bool = True
|
||
behavior_config: Optional[BehaviorConfig] = None
|
||
|
||
|
||
class AntiDetectionContext:
|
||
"""反检测上下文
|
||
|
||
在一次爬取过程中传递的状态信息。
|
||
"""
|
||
def __init__(self):
|
||
self.proxy: Optional[ProxyInfo] = None
|
||
self.delays_applied: list = []
|
||
self.captcha_detected: bool = False
|
||
|
||
|
||
def with_anti_detection(config: Optional[AntiDetectionConfig] = None):
|
||
"""装饰器：集成所有反检测功能
|
||
|
||
组合使用：
|
||
1. 代理池轮换
|
||
2. 行为伪装
|
||
3. 验证码检测
|
||
|
||
Example:
|
||
@with_anti_detection()
|
||
async def fetch_content(self, url: str, context: AntiDetectionContext = None):
|
||
# context.proxy 可用于 Playwright
|
||
# 延迟已自动应用
|
||
...
|
||
"""
|
||
_config = config or AntiDetectionConfig()
|
||
behavior = BehaviorCamouflage(_config.behavior_config)
|
||
proxy_pool = get_proxy_pool() if _config.proxy_enabled else None
|
||
captcha_detector = CaptchaDetector() if _config.captcha_detection_enabled else None
|
||
|
||
def decorator(func: Callable):
|
||
@wraps(func)
|
||
async def async_wrapper(*args, **kwargs):
|
||
context = AntiDetectionContext()
|
||
|
||
# 1. 获取代理
|
||
if proxy_pool and proxy_pool.config.enabled:
|
||
context.proxy = proxy_pool.get_proxy()
|
||
|
||
# 2. 应用行为伪装延迟
|
||
if _config.behavior_camouflage_enabled:
|
||
delay = behavior.apply_delay()
|
||
context.delays_applied.append(delay)
|
||
|
||
# 3. 执行原始函数
|
||
try:
|
||
result = await func(*args, context=context, **kwargs)
|
||
|
||
# 报告代理成功
|
||
if context.proxy and proxy_pool:
|
||
proxy_pool.report_success(context.proxy.url)
|
||
|
||
return result
|
||
|
||
except Exception as e:
|
||
# 报告代理失败
|
||
if context.proxy and proxy_pool:
|
||
proxy_pool.report_failure(context.proxy.url)
|
||
raise
|
||
|
||
@wraps(func)
|
||
def sync_wrapper(*args, **kwargs):
|
||
context = AntiDetectionContext()
|
||
|
||
# 1. 获取代理
|
||
if proxy_pool and proxy_pool.config.enabled:
|
||
context.proxy = proxy_pool.get_proxy()
|
||
|
||
# 2. 应用行为伪装延迟
|
||
if _config.behavior_camouflage_enabled:
|
||
delay = behavior.apply_delay()
|
||
context.delays_applied.append(delay)
|
||
|
||
# 3. 执行原始函数
|
||
try:
|
||
result = func(*args, context=context, **kwargs)
|
||
|
||
if context.proxy and proxy_pool:
|
||
proxy_pool.report_success(context.proxy.url)
|
||
|
||
return result
|
||
|
||
except Exception as e:
|
||
if context.proxy and proxy_pool:
|
||
proxy_pool.report_failure(context.proxy.url)
|
||
raise
|
||
|
||
# 根据函数类型选择包装器
|
||
import asyncio
|
||
if asyncio.iscoroutinefunction(func):
|
||
return async_wrapper
|
||
return sync_wrapper
|
||
|
||
return decorator
|
||
```
|
||
|
||
---
|
||
|
||
## Integration Points
|
||
|
||
### Integration with ZhihuCrawler
|
||
|
||
**Modified File:** `kb_workflow/crawler/zhihu_crawler.py`
|
||
|
||
**Changes:**
|
||
|
||
```python
|
||
# In ZhihuCrawler.__init__:
|
||
from .anti_detection import AntiDetectionConfig, get_proxy_pool
|
||
|
||
def __init__(
|
||
self,
|
||
headless: bool = True,
|
||
timeout: int = 30,
|
||
config: Optional[CrawlerConfig] = None,
|
||
anti_detection_config: Optional[AntiDetectionConfig] = None
|
||
):
|
||
self.headless = headless
|
||
self.timeout = timeout
|
||
self.config = config or get_crawler_config()
|
||
self.anti_detection_config = anti_detection_config or AntiDetectionConfig()
|
||
# ... rest of init
|
||
|
||
# In _start_browser_with_proxy:
|
||
def _start_browser_with_proxy(self, cookie: str) -> None:
|
||
# Use proxy pool if enabled
|
||
if self.anti_detection_config.proxy_enabled:
|
||
proxy_pool = get_proxy_pool()
|
||
if proxy_pool.config.enabled:
|
||
proxy = proxy_pool.get_proxy()
|
||
if proxy:
|
||
self.config.proxy = ProxyConfig(
|
||
server=proxy.url,
|
||
username=proxy.username,
|
||
password=proxy.password
|
||
)
|
||
# ... rest of method
|
||
```
|
||
|
||
### Integration with ZhihuAdapter
|
||
|
||
**Modified File:** `kb_workflow/pipeline/platforms/zhihu.py`
|
||
|
||
**Changes:**
|
||
|
||
```python
|
||
from kb_workflow.crawler.anti_detection import (
|
||
with_anti_detection,
|
||
AntiDetectionContext,
|
||
AntiDetectionConfig
|
||
)
|
||
|
||
class ZhihuAdapter(PlatformAdapter):
|
||
def __init__(
|
||
self,
|
||
headless: bool = True,
|
||
timeout: int = 30,
|
||
output_dir: Optional[Path] = None,
|
||
idempotency_checker: Optional[IdempotencyChecker] = None,
|
||
anti_detection_config: Optional[AntiDetectionConfig] = None
|
||
):
|
||
# ... existing init
|
||
self.anti_detection_config = anti_detection_config or AntiDetectionConfig()
|
||
|
||
@with_anti_detection()
|
||
async def fetch_metadata(
|
||
self,
|
||
content_id: str,
|
||
content_type: str,
|
||
source_url: str = None,
|
||
context: AntiDetectionContext = None # 注入的上下文
|
||
) -> Dict[str, Any]:
|
||
# Use context.proxy if needed
|
||
# ...
|
||
```
|
||
|
||
### Integration with Configuration
|
||
|
||
**Modified File:** `kb_workflow/pipeline/config.py`
|
||
|
||
**Changes:**
|
||
|
||
```python
|
||
class KBSettings(BaseSettings):
|
||
# ... existing fields
|
||
|
||
# Anti-crawler settings
|
||
proxy_pool: str = Field(
|
||
default="",
|
||
description="代理池配置（逗号分隔的代理URL）"
|
||
)
|
||
anti_detection_enabled: bool = Field(
|
||
default=True,
|
||
description="是否启用反检测功能"
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
### Request Flow with Anti-Detection
|
||
|
||
```
|
||
[CLI] process command
|
||
|
|
||
v
|
||
[Pipeline] DownloadProcessor.process()
|
||
|
|
||
v
|
||
[ZhihuAdapter] @with_anti_detection decorator
|
||
|
|
||
+-> [ProxyPool] get_proxy() -> returns available proxy
|
||
|
|
||
+-> [BehaviorCamouflage] apply_delay() -> wait random time
|
||
|
|
||
+-> [ZhihuCrawler] crawl_answer(url, proxy=...)
|
||
|
|
||
+-> [Playwright] start browser with proxy
|
||
|
|
||
+-> [CaptchaDetector] detect_from_html() -> check for captcha
|
||
| |
|
||
| +-> If detected -> raise CaptchaError
|
||
|
|
||
+-> Return CrawlResult
|
||
|
|
||
v
|
||
[ProxyPool] report_success() / report_failure()
|
||
|
|
||
v
|
||
[Pipeline] continue to next processor
|
||
```
|
||
|
||
### Error Recovery Flow
|
||
|
||
```
|
||
[Request fails]
|
||
|
|
||
v
|
||
[Retry Logic] retry_on_transient_error
|
||
|
|
||
+-> Is NetworkError? -> Retry with exponential backoff + jitter
|
||
|
|
||
+-> Is CaptchaError? -> Do NOT retry, surface to user
|
||
|
|
||
+-> Is AuthError? -> Do NOT retry, trigger QR login
|
||
|
|
||
v
|
||
[After max retries] -> Log error, update state, raise to caller
|
||
```
|
||
|
||
---
|
||
|
||
## Recommended Build Order
|
||
|
||
基于依赖关系，建议按以下顺序实现：
|
||
|
||
### Phase 1: Foundation (No external dependencies)
|
||
1. **`kb_workflow/crawler/behavior.py`** — 行为伪装模块
|
||
- 独立模块，无外部依赖
|
||
- 可独立测试
|
||
|
||
2. **`kb_workflow/crawler/captcha.py`** — 验证码检测模块
|
||
- 独立模块，只需要 HTML 解析
|
||
- 定义 CaptchaError 错误类型
|
||
|
||
### Phase 2: Integration
|
||
3. **`kb_workflow/crawler/proxy_pool.py`** — 代理池管理
|
||
- 依赖配置系统
|
||
- 需要与 CrawlerConfig 集成
|
||
|
||
4. **`kb_workflow/pipeline/retry.py` (enhance)** — 增强重试机制
|
||
- 添加 jitter 支持
|
||
- 添加验证码感知
|
||
|
||
### Phase 3: Unified API
|
||
5. **`kb_workflow/crawler/anti_detection.py`** — 统一装饰器
|
||
- 组合所有组件
|
||
- 提供 `@with_anti_detection` 装饰器
|
||
|
||
### Phase 4: Platform Integration
|
||
6. **`kb_workflow/crawler/zhihu_crawler.py` (modify)** — 集成到 ZhihuCrawler
|
||
7. **`kb_workflow/pipeline/platforms/zhihu.py` (modify)** — 集成到 ZhihuAdapter
|
||
8. **`kb_workflow/pipeline/config.py` (modify)** — 添加配置项
|
||
|
||
### Phase 5: Bilibili Support (if needed)
|
||
9. **`kb_workflow/pipeline/platforms/bilibili.py` (modify)** — B站反爬适配
|
||
|
||
---
|
||
|
||
## Anti-Patterns (v1.3 Specific)
|
||
|
||
### Anti-Pattern 1: Hardcoding Proxy Settings in Crawler
|
||
|
||
**What people do:** 在 Crawler 类中硬编码代理配置
|
||
|
||
```python
|
||
# BAD
|
||
class ZhihuCrawler:
|
||
def __init__(self):
|
||
self.proxy = "http://127.0.0.1:7890" # Hardcoded
|
||
```
|
||
|
||
**Why it's wrong:** 无法动态切换代理，无法支持代理池
|
||
|
||
**Do this instead:** 使用独立的 ProxyPoolManager
|
||
|
||
---
|
||
|
||
### Anti-Pattern 2: Retry All Errors Blindly
|
||
|
||
**What people do:** 对所有错误都进行重试
|
||
|
||
```python
|
||
# BAD
|
||
@retry(stop=stop_after_attempt(3))
|
||
def crawl():
|
||
...
|
||
```
|
||
|
||
**Why it's wrong:** 验证码错误重试会立即再次触发验证码，造成无限循环
|
||
|
||
**Do this instead:** 使用错误类型过滤
|
||
|
||
```python
|
||
# GOOD
|
||
@retry(retry=retry_if_exception_type(NetworkError))
|
||
def crawl():
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
### Anti-Pattern 3: Fixed Delay Intervals
|
||
|
||
**What people do:** 使用固定延迟
|
||
|
||
```python
|
||
# BAD
|
||
time.sleep(2) # Fixed 2 second delay
|
||
```
|
||
|
||
**Why it's wrong:** 固定延迟容易被模式检测识别
|
||
|
||
**Do this instead:** 使用随机延迟 + jitter
|
||
|
||
```python
|
||
# GOOD
|
||
delay = random.uniform(1, 3) + random.uniform(-0.5, 0.5)
|
||
time.sleep(delay)
|
||
```
|
||
|
||
---
|
||
|
||
### Anti-Pattern 4: Captcha Auto-Solve Without Fallback
|
||
|
||
**What people do:** 依赖第三方验证码解决服务，没有降级方案
|
||
|
||
**Why it's wrong:** 增加成本，服务不可用时无法工作
|
||
|
||
**Do this instead:** 检测 + 暂停 + 通知用户（本项目约束）
|
||
|
||
---
|
||
|
||
## Scaling Considerations (v1.3)
|
||
|
||
| Scale | Architecture Adjustments |
|
||
|-------|--------------------------|
|
||
| 个人使用（当前） | 单进程，内存代理池，本地配置 |
|
||
| 小规模（10-100 URL/天） | 添加代理持久化，简单的健康检查 |
|
||
| 中规模（100-1000 URL/天） | 考虑使用代理池服务 API，分布式状态存储 |
|
||
|
||
### Scaling Priorities
|
||
|
||
1. **First bottleneck:** 代理被封 — 通过代理池轮换解决
|
||
2. **Second bottleneck:** 账号被封 — 通过行为伪装和降低频率解决
|
||
3. **Third bottleneck:** 验证码频繁 — 考虑暂停机制 + 人工介入
|
||
|
||
---
|
||
|
||
## v1.3 Sources
|
||
|
||
- [ZenRows - How to Avoid Bot Detection with Playwright](https://www.zenrows.com/blog/avoid-playwright-bot-detection) - HIGH confidence
|
||
- [Zyte - Best Proxy Providers for Web Scraping 2026](https://www.zyte.com/learn/best-proxy-providers-for-web-scraping/) - HIGH confidence
|
||
- [Rebrowser - Python Requests Retry Guide](https://rebrowser.net/blog/python-requests-retry-the-ultimate-guide-to-handling-failed-http-requests-in-python) - HIGH confidence
|
||
- [GitHub - backoff Python library](https://github.com/litl/backoff) - HIGH confidence
|
||
- [Stack Overflow - How to rotate proxies in Python](https://stackoverflow.com/questions/55872164/how-to-rotate-proxies-on-a-python-requests) - HIGH confidence
|
||
- [Medium - Exponential Backoff Decorator](https://medium.com/@suryasekhar/implementing-exponential-backoff-decorator-in-python-26ddf783aea0) - MEDIUM confidence
|
||
- [Trickster.dev - Custom Scrapy Middleware for Proxy Pool](https://www.trickster.dev/post/writing-custom-scrapy-middleware-for-proxy-pool-integration/) - HIGH confidence
|
||
|
||
---
|
||
|
||
## Original Architecture Research (v1.0)
|
||
|
||
*Below is the original architecture research from v1.0, preserved for reference.*
|
||
|
||
---
|
||
|
||
## Standard Architecture
|
||
|
||
### System Overview
|
||
|
||
技能型工作流编排系统采用分层架构，将处理逻辑、状态管理和执行协调分离。基于当前研究和行业最佳实践，推荐以下四层架构：
|
||
|
||
```
|
||
+---------------------------------------------------------------+
|
||
| Orchestration Layer |
|
||
| [Workflow Coordinator] [State Manager] [Error Handler] |
|
||
+---------------------------------------------------------------+
|
||
| Skill Definition Layer |
|
||
| [SKILL.md specs] [Schema Definitions] [Trigger Rules] |
|
||
+---------------------------------------------------------------+
|
||
| Execution Layer |
|
||
| [Platform Skills] [Processing Skills] [Analysis Skills] |
|
||
+---------------------------------------------------------------+
|
||
| Infrastructure Layer |
|
||
| [File System] [External APIs] [AI Services] [State Store] |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
### Component Responsibilities
|
||
|
||
| Component | Responsibility | Typical Implementation |
|
||
|-----------|----------------|------------------------|
|
||
| Workflow Coordinator | 协调多个技能的执行顺序、依赖管理 | kb-workflow 主控模块 |
|
||
| State Manager | 步骤级状态持久化、断点续传支持 | workflow_state.json + 原子写入 |
|
||
| Error Handler | 错误分类、降级策略、重试逻辑 | 统一错误处理模块 |
|
||
| Skill Registry | 技能发现、能力声明、Schema 注册 | SKILL.md + JSON Schema |
|
||
| Platform Skills | 平台特定内容获取（B站/知乎） | kb-bilibili-*, kb-zhihu-* |
|
||
| Processing Skills | 内容转换（语音识别/OCR） | kb-whisper, kb-funasr, kb-ocr |
|
||
| Analysis Skills | AI 深度分析、报告生成 | kb-deep-analyzer |
|
||
|
||
## Recommended Project Structure
|
||
|
||
```
|
||
kb-skills/
|
||
+-- skills/ # 技能模块目录
|
||
| +-- core/ # 核心基础设施
|
||
| | +-- state_manager.py # 状态管理
|
||
| | +-- schema_registry.py # Schema 注册
|
||
| | +-- pipeline.py # 管道执行引擎
|
||
| | +-- error_handler.py # 错误处理
|
||
| |
|
||
| +-- platforms/ # 平台技能
|
||
| | +-- bilibili/ # B站技能组
|
||
| | | +-- SKILL.md
|
||
| | | +-- metadata.py
|
||
| | | +-- content.py
|
||
| | +-- zhihu/ # 知乎技能组
|
||
| | +-- SKILL.md
|
||
| | +-- metadata.py
|
||
| | +-- content.py
|
||
| |
|
||
| +-- processing/ # 处理技能
|
||
| | +-- whisper/ # 语音识别
|
||
| | +-- funasr/ # 中文语音识别
|
||
| | +-- ocr/ # OCR 处理
|
||
| |
|
||
| +-- analysis/ # 分析技能
|
||
| +-- deep_analyzer/ # 深度分析
|
||
|
|
||
+-- schemas/ # JSON Schema 定义
|
||
| +-- workflow_state.schema.json
|
||
| +-- skill_output.schema.json
|
||
| +-- platform_metadata.schema.json
|
||
|
|
||
+-- workflows/ # 工作流定义
|
||
| +-- video_pipeline.yaml # 视频处理流程
|
||
| +-- article_pipeline.yaml # 文章处理流程
|
||
|
|
||
+-- cli.py # 统一 CLI 入口
|
||
+-- config.py # 配置管理
|
||
```
|
||
|
||
### Structure Rationale
|
||
|
||
- **skills/core/:** 核心基础设施，所有技能共享的状态管理、Schema 验证和管道执行
|
||
- **skills/platforms/:** 按平台组织，每个平台有自己的元数据提取和内容下载技能
|
||
- **skills/processing/:** 内容转换技能，独立于平台
|
||
- **skills/analysis/:** AI 分析技能，作为管道的最后阶段
|
||
- **schemas/:** 统一的 JSON Schema 定义，确保接口一致性
|
||
- **workflows/:** 声明式工作流定义，指定技能组合和执行顺序
|
||
|
||
## Architectural Patterns
|
||
|
||
### Pattern 1: Pipeline Orchestration (管道编排)
|
||
|
||
**What:** 将处理流程建模为一系列串联的步骤，每个步骤接收输入、执行处理、产生输出。
|
||
|
||
**When to use:** 可预测的、重复性的任务，如内容处理管道。
|
||
|
||
**Trade-offs:**
|
||
- 优点：确定性执行、易于调试、支持断点续传
|
||
- 缺点：灵活性有限，不适合需要动态决策的场景
|
||
|
||
**Example:**
|
||
|
||
```python
|
||
# Pipeline definition
|
||
class ContentPipeline:
|
||
steps = [
|
||
("parse", LinkParserSkill),
|
||
("metadata", MetadataSkill), # depends on parse
|
||
("download", ContentSkill), # depends on metadata
|
||
("transcribe", TranscribeSkill), # conditional: only for video
|
||
("analyze", AnalyzeSkill), # depends on download + transcribe
|
||
]
|
||
|
||
def run(self, url: str, state: WorkflowState):
|
||
for step_name, skill in self.steps:
|
||
if state.is_completed(step_name):
|
||
continue # Skip completed steps
|
||
|
||
result = skill.execute(state.get_step_input(step_name))
|
||
state.mark_completed(step_name, result)
|
||
|
||
if not result.success:
|
||
return self.handle_error(result.error)
|
||
```
|
||
|
||
### Pattern 2: Step-Level State Management (步骤级状态管理)
|
||
|
||
**What:** 每个主要步骤完成后立即持久化状态，支持从任意断点恢复执行。
|
||
|
||
**When to use:** 长时间运行的批处理任务，可能因网络、API 限制等原因中断。
|
||
|
||
**Trade-offs:**
|
||
- 优点：断点续传、执行追踪、幂等性保证
|
||
- 缺点：状态管理开销、需要设计状态 Schema
|
||
|
||
**Example:**
|
||
|
||
```python
|
||
# workflow_state.json
|
||
{
|
||
"url": "https://www.bilibili.com/video/BV1xx411c7mD",
|
||
"platform": "bilibili",
|
||
"content_type": "video",
|
||
"created_at": "2026-03-13T10:00:00Z",
|
||
"updated_at": "2026-03-13T10:15:00Z",
|
||
"steps": {
|
||
"parse": {
|
||
"status": "completed",
|
||
"output": {"video_id": "BV1xx411c7mD"},
|
||
"completed_at": "2026-03-13T10:00:05Z"
|
||
},
|
||
"metadata": {
|
||
"status": "completed",
|
||
"output": {"title": "...", "duration": 1800},
|
||
"completed_at": "2026-03-13T10:00:30Z"
|
||
},
|
||
"download": {
|
||
"status": "in_progress",
|
||
"output": null,
|
||
"started_at": "2026-03-13T10:01:00Z"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### Pattern 3: Schema-First Interface Design (Schema 优先接口设计)
|
||
|
||
**What:** 先定义输入输出的 JSON Schema，再实现技能逻辑。所有技能遵守统一的接口规范。
|
||
|
||
**When to use:** 多人协作、需要接口稳定性和可测试性的项目。
|
||
|
||
**Trade-offs:**
|
||
- 优点：接口清晰、易于测试、支持自动验证
|
||
- 缺点：前期设计成本、Schema 维护负担
|
||
|
||
**Example:**
|
||
|
||
```json
|
||
// schemas/skill_output.schema.json
|
||
{
|
||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||
"type": "object",
|
||
"required": ["success", "data"],
|
||
"properties": {
|
||
"success": {"type": "boolean"},
|
||
"data": {"type": "object"},
|
||
"error": {
|
||
"type": "object",
|
||
"properties": {
|
||
"code": {"type": "string"},
|
||
"message": {"type": "string"},
|
||
"detail": {"type": "string"}
|
||
}
|
||
},
|
||
"metadata": {
|
||
"type": "object",
|
||
"properties": {
|
||
"duration_ms": {"type": "number"},
|
||
"tokens_used": {"type": "number"}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### Pattern 4: Idempotent Execution (幂等执行)
|
||
|
||
**What:** 相同输入重复执行产生相同结果，不会产生副作用（如重复文件）。
|
||
|
||
**When to use:** 任何可能需要重试的处理步骤。
|
||
|
||
**Trade-offs:**
|
||
- 优点：安全重试、简化错误恢复
|
||
- 缺点：需要设计去重逻辑
|
||
|
||
**Example:**
|
||
|
||
```python
|
||
class ContentSkill:
|
||
def execute(self, input: dict) -> SkillOutput:
|
||
# Generate deterministic filename from input
|
||
content_hash = hashlib.sha256(input['url'].encode()).hexdigest()[:12]
|
||
output_path = self.get_output_dir() / f"{content_hash}.mp4"
|
||
|
||
# Check if already exists
|
||
if output_path.exists():
|
||
return SkillOutput(success=True, data={"path": str(output_path), "cached": True})
|
||
|
||
# Download content
|
||
self.download_content(input['url'], output_path)
|
||
return SkillOutput(success=True, data={"path": str(output_path), "cached": False})
|
||
```
|
||
|
||
## Data Flow
|
||
|
||
### Content Processing Pipeline
|
||
|
||
```
|
||
[URL Input]
|
||
|
|
||
v
|
||
+-------------+ +----------------+
|
||
| Link Parser | --> | Platform/Type |
|
||
+-------------+ | Detection |
|
||
+----------------+
|
||
|
|
||
+---------------+---------------+
|
||
| |
|
||
v v
|
||
+-------------+ +-------------+
|
||
| Bilibili | | Zhihu |
|
||
| Pipeline | | Pipeline |
|
||
+-------------+ +-------------+
|
||
| |
|
||
+------+------+ +-------+-------+
|
||
| | | |
|
||
v v v v
|
||
[Metadata] [Download] [Metadata] [Download]
|
||
| | | |
|
||
+------+------+ +-------+-------+
|
||
| |
|
||
v v
|
||
+-------------+ +-------------+
|
||
| Video? | | Article |
|
||
| - Whisper | | - Direct |
|
||
| - FunASR | | Extract |
|
||
| - OCR | +-------------+
|
||
+-------------+ |
|
||
| |
|
||
+---------------+--------------+
|
||
|
|
||
v
|
||
+-----------------+
|
||
| Deep Analyzer |
|
||
| (AI Analysis) |
|
||
+-----------------+
|
||
|
|
||
v
|
||
+-----------------+
|
||
| Archive to |
|
||
| Knowledge Base |
|
||
+-----------------+
|
||
```
|
||
|
||
### State Management Flow
|
||
|
||
```
|
||
[Workflow Start]
|
||
|
|
||
v
|
||
+------------------+
|
||
| Load/Init State | <-- workflow_state.json
|
||
+------------------+
|
||
|
|
||
v
|
||
+------------------+ +------------------+
|
||
| For Each Step | --> | Check Status |
|
||
+------------------+ | - completed? skip|
|
||
| | - failed? retry |
|
||
v | - pending? run |
|
||
+------------------+ +------------------+
|
||
| Execute Step | |
|
||
+------------------+ v
|
||
| +------------------+
|
||
v | Mark Completed |
|
||
+------------------+ | Save State |
|
||
| Validate Output | -->| (atomic write) |
|
||
+------------------+ +------------------+
|
||
|
|
||
v
|
||
[Next Step / Complete]
|
||
```
|
||
|
||
### Key Data Flows
|
||
|
||
1. **URL to Metadata:** URL -> Link Parser -> (platform, type, id) -> Platform Metadata Skill -> Metadata JSON
|
||
2. **Metadata to Content:** Metadata JSON -> Content Skill -> (video file / article text)
|
||
3. **Content to Transcript:** Video File -> Transcribe Skill (Whisper/FunASR) -> Transcript JSON
|
||
4. **Content + Transcript to Analysis:** Content + Transcript -> Deep Analyzer -> Analysis Report
|
||
|
||
## Scaling Considerations
|
||
|
||
| Scale | Architecture Adjustments |
|
||
|-------|--------------------------|
|
||
| 1-100 items/day | 当前架构足够，单进程执行 |
|
||
| 100-1000 items/day | 添加任务队列，并行处理独立任务 |
|
||
| 1000+ items/day | 分布式任务调度，API 请求限流，缓存层 |
|
||
|
||
### Scaling Priorities
|
||
|
||
1. **First bottleneck:** 外部 API 限流（B站/知乎）。解决方案：请求队列 + 指数退避重试
|
||
2. **Second bottleneck:** AI 分析耗时。解决方案：批处理 + 并行分析
|
||
|
||
## Anti-Patterns
|
||
|
||
### Anti-Pattern 1: Global State Mutation
|
||
|
||
**What people do:** 在多个技能间共享可变全局状态
|
||
|
||
**Why it's wrong:** 难以追踪状态变化、难以测试、难以恢复
|
||
|
||
**Do this instead:** 每个步骤从 state 读取输入，将输出写回 state，状态变更通过 state manager 集中管理
|
||
|
||
### Anti-Pattern 2: Implicit Dependencies
|
||
|
||
**What people do:** 技能之间通过隐式约定传递数据（如固定文件路径）
|
||
|
||
**Why it's wrong:** 脆弱的耦合，修改一个技能可能破坏其他技能
|
||
|
||
**Do this instead:** 在 workflow_state.json 中显式声明依赖关系，通过 state manager 传递数据
|
||
|
||
### Anti-Pattern 3: Silent Failure
|
||
|
||
**What people do:** 捕获异常后只打印日志，继续执行
|
||
|
||
**Why it's wrong:** 下游步骤收到无效输入，产生难以调试的错误
|
||
|
||
**Do this instead:** 明确的失败处理策略 - 要么终止并记录，要么降级并提供默认值
|
||
|
||
### Anti-Pattern 4: Non-Idempotent Operations
|
||
|
||
**What people do:** 每次执行都创建新文件（如 `output_1.mp4`, `output_2.mp4`）
|
||
|
||
**Why it's wrong:** 重试产生垃圾文件，磁盘占用增长
|
||
|
||
**Do this instead:** 基于输入生成确定性文件名，已存在则跳过
|
||
|
||
## Integration Points
|
||
|
||
### External Services
|
||
|
||
| Service | Integration Pattern | Notes |
|
||
|---------|---------------------|-------|
|
||
| Bilibili API | HTTP + Cookie Auth | 需要 BILIBILI_COOKIE 环境变量 |
|
||
| Zhihu API | HTTP + Cookie Auth | 需要 ZHIHU_COOKIE 环境变量 |
|
||
| AI Provider | OpenAI-compatible API | 需要 AI_API_KEY 环境变量 |
|
||
| HuggingFace | API + Local Cache | 需要 HF_TOKEN 环境变量 |
|
||
| Git | Command Line | 需要 KB_GIT_USER/EMAIL 配置 |
|
||
|
||
### Internal Boundaries
|
||
|
||
| Boundary | Communication | Notes |
|
||
|----------|---------------|-------|
|
||
| Orchestrator <-> Skills | JSON via stdin/stdout | 标准化输入输出格式 |
|
||
| Skills <-> State Manager | File I/O | workflow_state.json |
|
||
| Skills <-> External APIs | HTTP | 统一错误处理和重试 |
|
||
|
||
## Build Order Implications
|
||
|
||
基于组件依赖关系，建议的构建顺序：
|
||
|
||
### Phase 1: Foundation (基础设施)
|
||
1. **State Manager** - 所有后续组件的基础
|
||
2. **Schema Registry** - 定义统一的输入输出格式
|
||
3. **Error Handler** - 统一错误处理模式
|
||
|
||
### Phase 2: Core Skills (核心技能)
|
||
4. **Link Parser** - 入口点，无依赖
|
||
5. **Platform Metadata Skills** - 依赖 Link Parser 输出
|
||
6. **Platform Content Skills** - 依赖 Metadata 输出
|
||
|
||
### Phase 3: Processing Skills (处理技能)
|
||
7. **Whisper/FunASR** - 依赖视频文件
|
||
8. **OCR Processor** - 依赖视频截图
|
||
|
||
### Phase 4: Analysis Skills (分析技能)
|
||
9. **Deep Analyzer** - 依赖内容和转录文本
|
||
|
||
### Phase 5: Orchestration (编排层)
|
||
10. **Pipeline Engine** - 组合所有技能
|
||
11. **Workflow Definitions** - 声明式流程定义
|
||
|
||
## Sources
|
||
|
||
- [AI Architecture Patterns 101: Workflows, Agents, MCPs, and A2A Systems](https://aipmguru.substack.com/p/ai-architecture-patterns-101-workflows) - HIGH confidence, comprehensive pattern overview
|
||
- [Data Pipeline Architecture: 5 Design Patterns with Examples - Dagster](https://dagster.io/guides/data-pipeline-architecture-5-design-patterns-with-examples) - HIGH confidence, pipeline patterns
|
||
- [Agentic Workflows in 2026: The ultimate guide - Vellum](https://www.vellum.ai/blog/agentic-workflows-emerging-architectures-and-design-patterns) - HIGH confidence, agent architecture patterns
|
||
- [State of Open Source Workflow Orchestration Systems 2025](https://www.pracdata.io/p/state-of-workflow-orchestration-ecosystem-2025) - MEDIUM confidence, ecosystem overview
|
||
- [ML Pipeline Architecture Design Patterns - Neptune.ai](https://neptune.ai/blog/ml-pipeline-architecture-design-patterns) - MEDIUM confidence, ML-specific patterns
|
||
|
||
---
|
||
|
||
*Architecture research for: Skill-based Workflow Orchestration System*
|
||
*Original research: 2026-03-13*
|
||
*Updated for v1.3 Anti-Crawler: 2026-03-18*
|
||
|
||
---
|
||
|
||
## v1.6 语义分段架构研究
|
||
|
||
**Milestone:** LLM-based Semantic Segmentation
|
||
**Researched:** 2026-03-21
|
||
**Focus:** 使用 LLM 替代基于规则的语义分段，实现智能章节识别和标题生成
|
||
|
||
---
|
||
|
||
### Executive Summary
|
||
|
||
本文档研究如何将 LLM 语义分段集成到现有的 SRT 处理架构中。基于对现有代码（`kb_workflow/srt/` 模块和 `SRTProcessor`）的分析，推荐采用 **渐进式迁移模式**，保留现有 RuleBasedSegmenter 作为 fallback，新增 LLMBasedSegmenter 作为主分段器。
|
||
|
||
关键架构决策：
|
||
1. **新增 SemanticSegmenter 组件** — 替代/增强现有的 `segmenter.py`
|
||
2. **单次 LLM 调用策略** — 完整上下文分析，避免边界拼接问题
|
||
3. **三层降级策略** — LLM 失败 → 响应解析失败 → AI 不可用 → RuleBasedSegmenter
|
||
4. **渐进式集成** — 不破坏现有功能，新组件作为可选增强
|
||
|
||
---
|
||
|
||
### Current Architecture (v1.5)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ SRTProcessor │
|
||
│ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │
|
||
│ │ parser.py │→ │segmenter.py │→ │corrector.py │ │
|
||
│ │ (SRT解析) │ │ (规则分段) │ │ (ASR纠错) │ │
|
||
│ └─────────────┘ └──────────────┘ └─────────────┘ │
|
||
│ │ │ │ │
|
||
│ ↓ ↓ ↓ │
|
||
│ List[Segment] List[str] List[str] │
|
||
│ (时间戳+文本) (段落) (纠错后) │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**现有组件：**
|
||
- `parser.py`: 解析 SRT → Segment(id, start_time, end_time, text)
|
||
- `segmenter.py`: SemanticSegmenter (基于时间间隔3s + 长度500字符规则)
|
||
- `corrector.py`: ASRCorrector (使用 SimpleAIClient 纠错)
|
||
|
||
---
|
||
|
||
### Recommended Architecture (v1.6)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ SRTProcessor │
|
||
│ ┌─────────────┐ │
|
||
│ │ parser.py │ │
|
||
│ │ (SRT解析) │ │
|
||
│ └──────┬──────┘ │
|
||
│ │ │
|
||
│ ↓ List[Segment] (id, start_time, end_time, text) │
|
||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||
│ │ semantic_segmenter.py (NEW) │ │
|
||
│ │ ┌────────────────────────────────────────────────────┐ │ │
|
||
│ │ │ 1. 构建 Prompt (视频信息 + 字幕内容) │ │ │
|
||
│ │ └──────────────────┬─────────────────────────────────┘ │ │
|
||
│ │ ↓ │ │
|
||
│ │ ┌────────────────────────────────────────────────────┐ │ │
|
||
│ │ │ 2. 调用 LLM (SimpleAIClient.generate) │ │ │
|
||
│ │ └──────────────────┬─────────────────────────────────┘ │ │
|
||
│ │ ↓ │ │
|
||
│ │ ┌────────────────────────────────────────────────────┐ │ │
|
||
│ │ │ 3. 解析响应 (## 标题 + 内容) │ │ │
|
||
│ │ └──────────────────┬─────────────────────────────────┘ │ │
|
||
│ │ ↓ │ │
|
||
│ │ ┌────────────────────────────────────────────────────┐ │ │
|
||
│ │ │ 4. 返回 List[SemanticParagraph] │ │ │
|
||
│ │ │ - title: str (章节标题) │ │ │
|
||
│ │ │ - content: str (段落内容) │ │ │
|
||
│ │ │ - time_range: (start, end) (可选) │ │ │
|
||
│ │ └──────────────────┬─────────────────────────────────┘ │ │
|
||
│ └─────────────────────┼─────────────────────────────────────┘ │
|
||
│ ↓ │
|
||
│ List[SemanticParagraph] │
|
||
│ (标题 + 结构化内容) │
|
||
│ ↓ │
|
||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||
│ │ Fallback Strategy │ │
|
||
│ │ LLM 失败 → 响应解析失败 → AI 不可用 │ │
|
||
│ │ ↓ │ │
|
||
│ │ segmenter.py (RuleBasedSegmenter) - 保留作为 fallback │ │
|
||
│ └────────────────────┬─────────────────────────────────────┘ │
|
||
│ ↓ │
|
||
│ List[str] (向后兼容) │
|
||
│ ↓ │
|
||
│ ┌─────────────┐ ┌──────────────┐ │
|
||
│ │corrector.py │ │ Markdown生成 │ │
|
||
│ │ (ASR纠错) │ │ (标题+内容) │ │
|
||
│ └─────────────┘ └──────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
### Component Boundaries
|
||
|
||
| Component | Responsibility | Communicates With |
|
||
|-----------|---------------|-------------------|
|
||
| **parser.py** | 解析 SRT 格式 → Segment 对象 | 无（输入层） |
|
||
| **semantic_segmenter.py** (NEW) | LLM 语义分析 → 结构化段落 | parser (输入), corrector (输出), SimpleAIClient |
|
||
| **segmenter.py** (EXIST) | 基于规则的分段（fallback） | semantic_segmenter (降级) |
|
||
| **corrector.py** | ASR 纠错 | semantic_segmenter (输入), SimpleAIClient |
|
||
| **SRTProcessor** | 编排处理流程 | 所有模块 |
|
||
|
||
---
|
||
|
||
### Data Flow Changes
|
||
|
||
**旧流程 (v1.5):**
|
||
```
|
||
SRT File → parser.parse_srt_file() → List[Segment]
|
||
↓
|
||
segmenter.segment() (基于规则)
|
||
↓
|
||
List[str] (段落)
|
||
↓
|
||
corrector.correct() (逐段落纠错)
|
||
↓
|
||
生成 Markdown
|
||
```
|
||
|
||
**新流程 (v1.6):**
|
||
```
|
||
SRT File → parser.parse_srt_file() → List[Segment]
|
||
↓
|
||
[NEW] semantic_segmenter.segment()
|
||
├─ 构建 Prompt (视频信息 + 字幕文本)
|
||
├─ 调用 LLM (SimpleAIClient)
|
||
├─ 解析响应 (## 标题 + 内容)
|
||
└─ 返回 List[SemanticParagraph]
|
||
├─ title: str
|
||
├─ content: str
|
||
└─ time_range: (float, float) (可选)
|
||
↓
|
||
[FALLBACK] segmenter.segment() (LLM 失败时)
|
||
↓
|
||
List[SemanticParagraph] → flatten → List[str]
|
||
↓
|
||
corrector.correct() (逐段落纠错)
|
||
↓
|
||
生成 Markdown (标题 + 内容结构)
|
||
```
|
||
|
||
---
|
||
|
||
### Response Parsing Strategy
|
||
|
||
**LLM 响应格式要求:**
|
||
```
|
||
## 第一部分：机器学习基础概念
|
||
段落内容1...
|
||
段落内容2...
|
||
|
||
## 第二部分：深度学习原理
|
||
段落内容3...
|
||
段落内容4...
|
||
```
|
||
|
||
**解析方法:**
|
||
```python
|
||
@dataclass
|
||
class SemanticParagraph:
|
||
"""语义分段结果"""
|
||
title: str # 章节标题
|
||
content: str # 段落内容
|
||
start_time: float # 开始时间（可选）
|
||
end_time: float # 结束时间（可选）
|
||
|
||
def parse_llm_response(response: str) -> List[SemanticParagraph]:
|
||
"""解析 LLM 响应为结构化段落"""
|
||
paragraphs = []
|
||
current_title = "无标题"
|
||
current_content = []
|
||
|
||
for line in response.split('\n'):
|
||
if line.startswith('##'):
|
||
# 保存上一个段落
|
||
if current_content:
|
||
content = '\n'.join(current_content).strip()
|
||
if content:
|
||
paragraphs.append(SemanticParagraph(
|
||
title=current_title,
|
||
content=content
|
||
))
|
||
# 开始新段落
|
||
current_title = line.lstrip('#').strip()
|
||
current_content = []
|
||
else:
|
||
current_content.append(line)
|
||
|
||
# 保存最后一个段落
|
||
if current_content:
|
||
content = '\n'.join(current_content).strip()
|
||
if content:
|
||
paragraphs.append(SemanticParagraph(
|
||
title=current_title,
|
||
content=content
|
||
))
|
||
|
||
return paragraphs
|
||
```
|
||
|
||
---
|
||
|
||
### LLM Call Strategy
|
||
|
||
**推荐：单次调用**
|
||
|
||
**原因:**
|
||
1. **上下文完整性** — LLM 理解完整视频内容，分段更准确
|
||
2. **避免边界问题** — 多次调用的拼接处容易出现语义不连贯
|
||
3. **简化实现** — 单次调用更容易测试和调试
|
||
|
||
**Prompt 设计:**
|
||
```python
|
||
SEGMENTATION_PROMPT = """你是一个视频内容分析师。请将以下字幕内容按语义分段，并为每个段落生成标题。
|
||
|
||
【视频信息】
|
||
标题：{video_title}
|
||
时长：{duration}
|
||
|
||
【字幕内容】
|
||
{concatenated_segments}
|
||
|
||
【输出要求】
|
||
1. 按语义完整性分段（保持话题/论点完整）
|
||
2. 识别主题变化边界
|
||
3. 为每个段落生成简洁标题（不超过20字）
|
||
4. 输出格式：
|
||
|
||
## 第一部分：{标题}
|
||
{段落内容}
|
||
|
||
## 第二部分：{标题}
|
||
{段落内容}
|
||
|
||
...（以此类推）
|
||
|
||
注意：
|
||
- 段落长度建议 200-800 字符
|
||
- 在自然句号处分段
|
||
- 标题要概括该段核心内容
|
||
"""
|
||
```
|
||
|
||
**输入处理选项:**
|
||
|
||
**选项 1: 纯文本（推荐）**
|
||
```python
|
||
# 合并所有 Segment.text
|
||
full_text = '\n'.join(seg.text for seg in segments)
|
||
```
|
||
|
||
**选项 2: 带时间戳标记（可选，用于辅助理解）**
|
||
```python
|
||
# 添加时间戳辅助 LLM 理解结构
|
||
marked_text = '\n'.join(
|
||
f"[{format_time(seg.start_time)}] {seg.text}"
|
||
for seg in segments
|
||
)
|
||
```
|
||
|
||
**替代方案：流式调用**
|
||
- 用于超长视频（>1小时字幕）
|
||
- 分块调用 LLM，每块 10-15 分钟字幕
|
||
- 需要额外的边界拼接逻辑
|
||
- **建议：** 初期使用单次调用，后续根据实际需求优化
|
||
|
||
---
|
||
|
||
### Fallback Strategy
|
||
|
||
**三层降级策略:**
|
||
|
||
```python
|
||
class LLMBasedSegmenter:
|
||
def __init__(self, ai_client: SimpleAIClient):
|
||
self.ai_client = ai_client
|
||
self.rule_based = SemanticSegmenter() # Fallback
|
||
|
||
def segment(self, segments: List[Segment]) -> List[SemanticParagraph]:
|
||
# Level 3: AI 客户端不可用
|
||
if not self.ai_client or not self.ai_client.api_key:
|
||
logger.info("AI client not available, using rule-based")
|
||
return self._to_semantic_paragraphs(
|
||
self.rule_based.segment(segments)
|
||
)
|
||
|
||
try:
|
||
# Level 1: LLM 调用
|
||
prompt = self._build_prompt(segments)
|
||
response = self.ai_client.generate(prompt)
|
||
|
||
# Level 2: 响应解析
|
||
try:
|
||
paragraphs = self._parse_llm_response(response)
|
||
if not paragraphs:
|
||
raise ValueError("Empty parsed result")
|
||
return paragraphs
|
||
except Exception as e:
|
||
logger.warning(f"Failed to parse LLM response: {e}")
|
||
return self._to_semantic_paragraphs(
|
||
self.rule_based.segment(segments)
|
||
)
|
||
|
||
except Exception as e:
|
||
logger.warning(f"LLM call failed: {e}, using rule-based")
|
||
return self._to_semantic_paragraphs(
|
||
self.rule_based.segment(segments)
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
### Suggested Build Order
|
||
|
||
**Phase 1: 基础组件（不影响现有功能）**
|
||
1. 创建 `kb_workflow/srt/semantic_segmenter.py`
|
||
2. 实现 `SemanticParagraph` 数据类
|
||
3. 实现 `parse_llm_response()` 方法（独立于 LLM 调用）
|
||
4. 编写单元测试（mock LLM 响应，测试解析逻辑）
|
||
|
||
**Phase 2: LLM 集成**
|
||
5. 实现 `_build_prompt()` 方法
|
||
6. 集成 SimpleAIClient
|
||
7. 实现 `segment()` 主方法
|
||
8. 实现降级策略（fallback）
|
||
9. 编写集成测试（使用真实 LLM 或 mock）
|
||
|
||
**Phase 3: 处理器集成**
|
||
10. 修改 `SRTProcessor.__init__()`：
|
||
- 添加 `use_llm_segmenter: bool = False` 参数
|
||
- 条件性初始化 `llm_segmenter`
|
||
11. 修改 `SRTProcessor.process()`：
|
||
- 调用新分段器或保留旧分段器
|
||
- 处理 `SemanticParagraph` 输出
|
||
12. 更新 `_generate_markdown()`：
|
||
- 处理标题和内容结构
|
||
- 生成带层级的 Markdown
|
||
13. 保留 RuleBasedSegmenter 作为 fallback
|
||
|
||
**Phase 4: 输出优化**
|
||
14. 优化 Markdown 格式（标题层级 `###`）
|
||
15. 添加时间戳（可选，在段落中标注时间范围）
|
||
16. 更新文档和测试
|
||
|
||
---
|
||
|
||
### Schema Changes
|
||
|
||
**新增数据类:**
|
||
```python
|
||
@dataclass
|
||
class SemanticParagraph:
|
||
"""语义分段结果"""
|
||
title: str # 章节标题
|
||
content: str # 段落内容
|
||
start_time: Optional[float] = None # 开始时间（可选）
|
||
end_time: Optional[float] = None # 结束时间（可选）
|
||
```
|
||
|
||
**修改 SRTOutput (向后兼容):**
|
||
```python
|
||
class SRTOutput(BaseModel):
|
||
success: bool
|
||
output_file: str
|
||
paragraphs: List[str] # 保留向后兼容
|
||
semantic_paragraphs: Optional[List[SemanticParagraph]] = None # 新增
|
||
tags: List[str]
|
||
corrected: bool
|
||
```
|
||
|
||
---
|
||
|
||
### Patterns to Follow
|
||
|
||
#### Pattern 1: 渐进式迁移
|
||
**What:** 保留现有组件，新组件作为可选增强
|
||
**When:** 重构关键路径时
|
||
**Example:**
|
||
```python
|
||
class SRTProcessor:
|
||
def __init__(self, use_llm_segmenter: bool = False):
|
||
self.rule_based_segmenter = SemanticSegmenter() # 保留
|
||
if use_llm_segmenter:
|
||
self.llm_segmenter = LLMBasedSegmenter() # 新增
|
||
|
||
async def process(self, input_data):
|
||
# 优先使用 LLM，失败时降级
|
||
if hasattr(self, 'llm_segmenter'):
|
||
try:
|
||
paragraphs = self.llm_segmenter.segment(segments)
|
||
except Exception:
|
||
paragraphs = self.rule_based_segmenter.segment(segments)
|
||
else:
|
||
paragraphs = self.rule_based_segmenter.segment(segments)
|
||
```
|
||
|
||
#### Pattern 2: 协议兼容性
|
||
**What:** 新旧组件共享相同接口
|
||
**When:** 替换组件时
|
||
**Example:**
|
||
```python
|
||
# 两个分段器实现相同接口
|
||
class Segmenter(Protocol):
|
||
def segment(self, segments: List[Segment]) -> List[str]:
|
||
...
|
||
|
||
class RuleBasedSegmenter:
|
||
def segment(self, segments: List[Segment]) -> List[str]:
|
||
# 基于规则
|
||
...
|
||
|
||
class LLMBasedSegmenter:
|
||
def segment(self, segments: List[Segment]) -> List[str]:
|
||
# 基于 LLM
|
||
# 内部使用 SemanticParagraph，输出时合并标题和内容
|
||
return [
|
||
f"## {p.title}\n\n{p.content}"
|
||
for p in self._segment_internal(segments)
|
||
]
|
||
```
|
||
|
||
---
|
||
|
||
### Anti-Patterns to Avoid
|
||
|
||
#### Anti-Pattern 1: 破坏性替换
|
||
**What:** 直接删除 RuleBasedSegmenter
|
||
**Why bad:** LLM 可能失败，失去降级选项
|
||
**Instead:** 保留旧组件作为 fallback
|
||
|
||
#### Anti-Pattern 2: 紧耦合 Prompt 逻辑
|
||
**What:** 将 prompt 构建硬编码在 segmenter 中
|
||
**Why bad:** 难以测试和优化
|
||
**Instead:** 提取为独立方法或模板，便于测试
|
||
|
||
#### Anti-Pattern 3: 忽略时间戳信息
|
||
**What:** 仅传递文本给 LLM，忽略时间信息
|
||
**Why bad:** LLM 无法理解视频结构（停顿、章节）
|
||
**Instead:** 在 prompt 中包含时间戳标记或视频时长信息
|
||
|
||
---
|
||
|
||
### Scalability Considerations
|
||
|
||
| Concern | At 100 videos | At 10K videos | At 1M videos |
|
||
|---------|--------------|--------------|-------------|
|
||
| **LLM 调用延迟** | 可接受（~10-30s/视频） | 需要并发处理 | 需要队列+批处理 |
|
||
| **LLM 成本** | 可忽略 | 需要监控 | 需要缓存+降级策略 |
|
||
| **错误率** | 低，手动处理 | 需要自动重试 | 需要监控告警 |
|
||
| **响应解析** | 简单 | 需要容错增强 | 需要多模型支持 |
|
||
|
||
**建议:**
|
||
- 初期：单线程处理，完整错误日志
|
||
- 中期：添加并发处理（asyncio.Semaphore 限制并发数）
|
||
- 长期：分布式队列（Celery/RQ）+ 结果缓存
|
||
|
||
---
|
||
|
||
### Integration Points Summary
|
||
|
||
**新增集成点:**
|
||
1. `SRTProcessor.__init__()` → 条件性初始化 `LLMBasedSegmenter`
|
||
2. `SRTProcessor.process()` → 调用新分段器或旧分段器
|
||
3. `_generate_markdown()` → 处理 `SemanticParagraph` 结构
|
||
|
||
**保留集成点:**
|
||
1. `parser.parse_srt_file()` — 不变
|
||
2. `corrector.correct()` — 不变（处理 `List[str]`）
|
||
3. `WorkflowStateManager` — 不变
|
||
|
||
**修改点:**
|
||
1. `kb_workflow/srt/__init__.py` — 导出 `SemanticParagraph`
|
||
2. `kb_workflow/pipeline/schemas.py` — 更新 `SRTOutput`
|
||
|
||
---
|
||
|
||
### v1.6 Sources
|
||
|
||
- 现有架构分析: `kb_workflow/srt/` 模块
|
||
- SRTProcessor 实现: `kb_workflow/pipeline/processors/srt.py`
|
||
- SimpleAIClient 实现: `kb_workflow/deep_analyzer/__init__.py`
|
||
- 测试用例: `tests/test_srt_processor.py`
|
||
- 项目上下文: `.planning/PROJECT.md`
|
||
|
||
---
|
||
|
||
*Architecture research for: LLM-based Semantic Segmentation*
|
||
*Research date: 2026-03-21*
|