1885 lines
72 KiB
Markdown
1885 lines
72 KiB
Markdown
# Pitfalls Research
|
||
|
||
**Domain:** Skill-based Workflow Orchestration (LLM Agent Pipeline)
|
||
**Researched:** 2026-03-13 (Original), 2026-03-18 (Anti-Crawler Update), 2026-03-21 (LLM Semantic Segmentation)
|
||
**Confidence:** HIGH (Pipeline), MEDIUM (Anti-Crawler), HIGH (LLM Segmentation)
|
||
|
||
---
|
||
|
||
## Part 1: Pipeline Architecture Pitfalls (v1.0-1.2)
|
||
|
||
> 以下陷阱适用于管道架构本身，已在 v1.0-1.2 中解决。
|
||
|
||
### Pitfall 1: Statelessness = Restart from Scratch
|
||
|
||
**What goes wrong:**
|
||
系统没有持久化状态，任何失败都需要从头重新执行整个流程。Whisper 识别了 20 分钟，后续步骤失败？重新识别。下载了 2GB 视频，分析失败？重新下载。
|
||
|
||
**Why it happens:**
|
||
开发者低估了多步骤流程的失败概率。假设每步都成功，或者认为"重新执行一次很快"。实际上，Whisper 识别可能需要 30 分钟，视频下载可能需要 10 分钟，重复执行成本极高。
|
||
|
||
**How to avoid:**
|
||
- 每个主要步骤完成后，将状态写入 `workflow_state.json`
|
||
- 状态包含：步骤名、输入参数、输出路径、完成时间戳
|
||
- 重新执行时，检查状态文件，跳过已完成的步骤
|
||
- 使用幂等设计：重复执行同一链接不会产生重复文件
|
||
|
||
**Warning signs:**
|
||
- 没有状态文件或状态文件只在流程结束时写入
|
||
- 重复执行同一任务产生重复输出文件
|
||
- 用户抱怨"又要重新跑一遍"
|
||
|
||
**Phase to address:**
|
||
Phase 1 (PIPE-02: 步骤级状态管理, PIPE-04: 幂等执行保证)
|
||
|
||
---
|
||
|
||
### Pitfall 2: Error Propagation Without Context
|
||
|
||
**What goes wrong:**
|
||
错误在步骤链中传播时丢失了上下文。步骤 6 失败时报告 "AI API 调用失败"，但不知道是哪个内容、哪个步骤、什么输入导致的。调试需要从头重现。
|
||
|
||
**Why it happens:**
|
||
错误处理只关注"失败了"，没有保留足够上下文。LLM 调用失败时只记录 "API error"，没有记录请求内容、模型参数、重试次数等。
|
||
|
||
**How to avoid:**
|
||
- 每个步骤输出结构化 JSON，包含 `success`、`error`、`context` 字段
|
||
- 错误消息必须包含：步骤名、输入摘要、失败原因、可操作建议
|
||
- 使用 correlation ID 追踪整个流程
|
||
- 区分可重试错误（超时、503）和不可重试错误（403、验证失败）
|
||
|
||
**Warning signs:**
|
||
- 错误消息只有 "Error" 或 "Failed"
|
||
- 需要问用户"你输入的是什么？"来调试
|
||
- 同样的错误重复出现但不知道根本原因
|
||
|
||
**Phase to address:**
|
||
Phase 1 (PIPE-05: 错误恢复策略, PIPE-07: Schema 验证机制)
|
||
|
||
---
|
||
|
||
### Pitfall 3: Assuming Model "Understands Context"
|
||
|
||
**What goes wrong:**
|
||
主代理期望子代理"理解"整个流程的上下文，但实际上每个子代理只看到自己步骤的输入。kb-deep-analyzer 不知道之前的 Whisper 识别质量如何，只能假设输入是可靠的。
|
||
|
||
**Why it happens:**
|
||
人类可以自然地维持上下文，开发者误以为 LLM 也能做到。但每次子代理调用都是独立的会话，没有跨步骤的"记忆"。
|
||
|
||
**How to avoid:**
|
||
- 显式传递所有需要的上下文，不要假设子代理"知道"
|
||
- 每个步骤的输入 JSON 必须包含该步骤需要的所有信息
|
||
- 使用结构化的输入格式（Schema），而不是自然语言描述
|
||
- 在状态文件中记录关键决策和上下文
|
||
|
||
**Warning signs:**
|
||
- 子代理行为不符合预期，"它应该知道..."
|
||
- 需要阅读多个文档才能理解一个步骤的输入要求
|
||
- 步骤之间的"隐式约定"没有文档化
|
||
|
||
**Phase to address:**
|
||
Phase 1 (PIPE-03: 统一 JSON Schema, PIPE-06: 清晰的流程定义)
|
||
|
||
---
|
||
|
||
### Pitfall 4: Parallel Chaos (Uncoordinated Parallel Execution)
|
||
|
||
**What goes wrong:**
|
||
多个子代理同时执行但没有协调机制。结果：文件被覆盖、状态不一致、资源竞争。例如：同时下载和 OCR 处理同一视频导致文件损坏。
|
||
|
||
**Why it happens:**
|
||
并行看起来更"高效"，开发者没有考虑共享资源（文件、状态、API 配额）的并发访问问题。
|
||
|
||
**How to avoid:**
|
||
- 对于这个项目：**串行执行**。每个步骤完成后才开始下一个
|
||
- 如果未来需要并行：使用文件锁、状态锁、或任务队列
|
||
|
||
**Phase to address:**
|
||
Phase 1 (设计决策：串行执行模式)
|
||
|
||
---
|
||
|
||
### Pitfall 5: "Automating Bad Processes"
|
||
|
||
**What goes wrong:**
|
||
在流程本身有问题时，自动化只是让错误发生得更快。例如：Whisper 识别质量差（没有后处理），自动化后产出大量低质量文本。
|
||
|
||
**Why it happens:**
|
||
急于自动化，没有先验证手动流程的正确性。"先跑起来再说"的心态。
|
||
|
||
**How to avoid:**
|
||
- **先手动执行完整流程，确保每步输出符合预期**
|
||
- 验证流程正确后再自动化
|
||
- 每个步骤有明确的成功标准
|
||
|
||
**Warning signs:**
|
||
- 自动化后输出质量下降
|
||
- 需要"后处理"来修复自动化输出
|
||
|
||
**Phase to address:**
|
||
Phase 0 (验证阶段) - 在自动化之前验证
|
||
|
||
---
|
||
|
||
### Pitfall 6: Environment Variable Sprawl
|
||
|
||
**What goes wrong:**
|
||
配置分散在多个地方：`~/.kb_env`、`HF_TOKEN`、`ZHIHU_COOKIE`、环境变量。用户不知道需要配置什么，"在我的机器上能跑"问题频发。
|
||
|
||
**Why it happens:**
|
||
不同功能由不同时间添加，没有统一的配置管理。
|
||
|
||
**How to avoid:**
|
||
- 工作流启动时检查所有必需变量，缺失则报错退出
|
||
- 文档中明确列出所有变量及其用途
|
||
- 可选变量有明确的默认值和回退策略
|
||
|
||
**Warning signs:**
|
||
- "在我的机器上能跑"问题
|
||
- 需要检查多个文件才能找到配置
|
||
- 同一变量在不同地方有不同值
|
||
|
||
**Phase to address:**
|
||
Phase 1 (环境变量检查已在 SKILL.md 中定义，需要统一执行)
|
||
|
||
---
|
||
|
||
### Pitfall 7: Graceful Degradation That Isn't
|
||
|
||
**What goes wrong:**
|
||
声称有"降级策略"，但实际上降级到无意义的内容。例如：深度分析失败时返回一个模板文本，看起来"成功"了但没有实际价值。
|
||
|
||
**Why it happens:**
|
||
开发者不想让流程"失败"，认为返回一些内容比报错好。但这掩盖了问题，用户得到的是无用的输出。
|
||
|
||
**How to avoid:**
|
||
- **关键步骤必须成功，否则整个流程失败** (PROJECT.md 已定义)
|
||
- 区分"必须有"和"最好有"的功能
|
||
- 失败时明确报告失败原因，不要假装成功
|
||
- 降级策略只用于非关键功能（如：OCR 失败时跳过图片分析）
|
||
|
||
**Warning signs:**
|
||
- 输出看起来"正常"但没有实际内容
|
||
- 用户发现分析报告是模板文本
|
||
- 成功率统计很高但用户不满意
|
||
|
||
**Phase to address:**
|
||
Phase 1 (SKILL.md 中的"严格失败处理原则"需要严格执行)
|
||
|
||
---
|
||
|
||
## Part 2: Anti-Crawler Optimization Pitfalls (v1.3)
|
||
|
||
> 以下陷阱适用于 v1.3 反爬虫优化里程碑，重点关注代理轮换、行为伪装、验证码处理、错误恢复等。
|
||
|
||
### Pitfall AC-1: IP 泄漏 - 代理配置不完整导致真实 IP 暴露
|
||
|
||
**What goes wrong:**
|
||
使用了代理但仍然暴露真实 IP，导致被目标平台封禁。常见于：
|
||
- 只配置了 HTTP 代理但未配置 HTTPS 代理
|
||
- Session 对象未正确绑定代理，部分请求绕过代理
|
||
- DNS 泄漏：DNS 查询未通过代理直接解析
|
||
- WebSocket 连接未走代理
|
||
|
||
**Why it happens:**
|
||
开发者通常只关注主要请求的代理配置，忽略了：
|
||
1. 系统环境变量可能覆盖代码中的代理设置
|
||
2. `requests.Session` 默认 `trust_env=True`，可能使用系统代理而非指定代理
|
||
3. Playwright 的 `proxy` 配置需要同时覆盖 HTTP/HTTPS
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Playwright 正确配置
|
||
context_options = {
|
||
"proxy": {
|
||
"server": "http://proxy.example.com:8080",
|
||
# 确保同时处理 HTTPS
|
||
}
|
||
}
|
||
|
||
# requests Session 正确配置
|
||
session = requests.Session()
|
||
session.trust_env = False # 关键：禁用系统代理干扰
|
||
session.proxies = {
|
||
"http": proxy_url,
|
||
"https": proxy_url, # 必须同时配置
|
||
}
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 使用代理后仍然收到 403 错误
|
||
- 目标平台显示你的真实地理位置
|
||
- IP 检测 API 返回真实 IP 而非代理 IP
|
||
|
||
**Phase to address:** Phase 1 (AC-01: 代理池管理)
|
||
|
||
---
|
||
|
||
### Pitfall AC-2: 代理轮换模式过于规律被识别
|
||
|
||
**What goes wrong:**
|
||
虽然使用了代理轮换，但因为轮换间隔固定、顺序可预测，被平台识别为自动化行为。表现为：
|
||
- 每次请求后立即切换 IP
|
||
- 代理使用顺序固定 (A→B→C→A→B→C)
|
||
- 所有代理来自同一 IP 段或同一数据中心
|
||
|
||
**Why it happens:**
|
||
开发者追求简单实现，使用 `itertools.cycle()` 或简单的取模轮换，忽略了反爬系统会检测：
|
||
1. IP 变化模式
|
||
2. 请求时间分布
|
||
3. 代理 IP 的 ASN (自治系统号) 分布
|
||
|
||
**How to avoid:**
|
||
```python
|
||
import random
|
||
|
||
class ProxyPool:
|
||
def get_proxy(self) -> str:
|
||
# 随机选择而非顺序轮换
|
||
return random.choice(self.healthy_proxies)
|
||
|
||
def should_rotate(self) -> bool:
|
||
# 随机化轮换时机，而非每次请求都轮换
|
||
return random.random() < self.rotation_probability
|
||
```
|
||
|
||
**Better approach:**
|
||
- 使用加权随机选择，健康度高的代理获得更多请求
|
||
- 同一会话内保持 IP 稳定，仅在检测到风险或达到请求阈值时轮换
|
||
- 混合使用住宅代理和数据中心代理
|
||
|
||
**Warning signs:**
|
||
- 新代理 IP 很快被封
|
||
- 收到大量 CAPTCHA 挑战
|
||
- 平台返回"请求过于频繁"错误
|
||
|
||
**Phase to address:** Phase 1 (AC-01: 代理池管理)
|
||
|
||
---
|
||
|
||
### Pitfall AC-3: 验证码检测缺失导致无限阻塞
|
||
|
||
**What goes wrong:**
|
||
爬虫遇到验证码页面后：
|
||
- 继续发送请求，被持续拒绝
|
||
- 无限重试，消耗所有代理资源
|
||
- 没有任何告警，任务静默失败
|
||
|
||
**Why it happens:**
|
||
开发者假设请求成功返回 200 就代表正常，忽略了：
|
||
1. 验证码页面可能返回 200 状态码
|
||
2. Cloudflare/DataDome 等服务返回"假"的正常响应
|
||
3. 平台使用 JavaScript 渲染验证码，API 拦截无法检测
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# 验证码检测逻辑
|
||
CAPTCHA_INDICATORS = [
|
||
"captcha", "verify", "验证", "人机验证",
|
||
"cf-browser-verification", "challenge-platform",
|
||
]
|
||
|
||
def detect_captcha(html_content: str, url: str) -> bool:
|
||
content_lower = html_content.lower()
|
||
return any(indicator in content_lower for indicator in CAPTCHA_INDICATORS)
|
||
|
||
# Playwright 中检测
|
||
def check_for_captcha(page) -> bool:
|
||
captcha_selectors = [
|
||
"iframe[src*='captcha']",
|
||
".captcha-container",
|
||
"#captcha",
|
||
"div[class*='verify']",
|
||
]
|
||
for selector in captcha_selectors:
|
||
if page.locator(selector).count() > 0:
|
||
return True
|
||
return False
|
||
```
|
||
|
||
**Response strategy when detected:**
|
||
1. 立即停止当前任务
|
||
2. 记录触发验证码的 URL 和代理
|
||
3. 将该代理标记为"风险"
|
||
4. 通知用户（对于个人知识库场景，等待人工介入是合理的）
|
||
|
||
**Warning signs:**
|
||
- 响应内容突然变短或为空
|
||
- 响应 HTML 包含验证码相关关键词
|
||
- 页面跳转到验证码 URL
|
||
|
||
**Phase to address:** Phase 2 (AC-02: 请求行为伪装) + Phase 3 (AC-03: 验证码检测)
|
||
|
||
---
|
||
|
||
### Pitfall AC-4: 重试策略不区分错误类型
|
||
|
||
**What goes wrong:**
|
||
对所有错误使用相同的重试策略：
|
||
- 404 Not Found 也在重试
|
||
- 401 Unauthorized 无限重试
|
||
- 429 Rate Limit 重试间隔过短
|
||
|
||
**Why it happens:**
|
||
使用统一的重试装饰器，未针对不同 HTTP 状态码制定策略。现有代码 `retry_on_network_error` 只处理 `NetworkError`，但错误分类不够细致。
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# 错误分类策略
|
||
ERROR_STRATEGIES = {
|
||
# 永不重试
|
||
400: "no_retry", # Bad Request - 修复请求
|
||
401: "no_retry", # Unauthorized - 需要 Cookie 更新
|
||
403: "switch_proxy", # Forbidden - 可能是 IP 封禁
|
||
404: "no_retry", # Not Found - 资源不存在
|
||
|
||
# 有限重试
|
||
429: "slow_retry", # Rate Limit - 长间隔重试
|
||
499: "retry", # Client Closed - 可重试
|
||
|
||
# 正常重试
|
||
500: "retry", # Internal Server Error
|
||
502: "retry", # Bad Gateway
|
||
503: "retry", # Service Unavailable
|
||
504: "retry", # Gateway Timeout
|
||
}
|
||
```
|
||
|
||
**For 429 Rate Limit specifically:**
|
||
```python
|
||
# 尊重 Retry-After 响应头
|
||
def handle_429(response):
|
||
retry_after = response.headers.get("Retry-After")
|
||
if retry_after:
|
||
wait_time = int(retry_after)
|
||
else:
|
||
wait_time = 60 # 默认等待 60 秒
|
||
time.sleep(wait_time)
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 重试日志显示相同的 401/404 错误重复出现
|
||
- 重试次数很快耗尽但问题未解决
|
||
- 代理池快速耗尽
|
||
|
||
**Phase to address:** Phase 4 (AC-04: 错误恢复机制增强)
|
||
|
||
---
|
||
|
||
### Pitfall AC-5: 请求间隔过于规律被识别为机器人
|
||
|
||
**What goes wrong:**
|
||
使用固定间隔（如每 2 秒一个请求）被反爬系统识别：
|
||
- 间隔时间完全相同
|
||
- 请求时间戳呈现明显的周期性
|
||
- 24 小时请求分布不自然（凌晨也高频请求）
|
||
|
||
**Why it happens:**
|
||
开发者使用 `time.sleep(2)` 实现简单的限流，但：
|
||
1. 固定间隔是机器人行为的典型特征
|
||
2. 真实用户行为具有随机性
|
||
3. 现有代码 `random.uniform(min_delay, max_delay)` 是正确方向，但范围可能需要调整
|
||
|
||
**How to avoid:**
|
||
```python
|
||
import random
|
||
import time
|
||
|
||
class BehaviorSimulator:
|
||
def __init__(self):
|
||
self.base_interval = 2.0
|
||
self.jitter_range = 1.0
|
||
|
||
def get_next_delay(self) -> float:
|
||
# 基础间隔 + 随机抖动
|
||
jitter = random.uniform(-self.jitter_range, self.jitter_range)
|
||
delay = self.base_interval + jitter
|
||
|
||
# 偶尔插入较长暂停（模拟阅读时间）
|
||
if random.random() < 0.1: # 10% 概率
|
||
delay += random.uniform(5, 15)
|
||
|
||
return max(0.5, delay) # 保证最小间隔
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 使用随机间隔后成功率显著提升
|
||
- 平台开始频繁要求验证码
|
||
|
||
**Phase to address:** Phase 2 (AC-02: 请求行为伪装)
|
||
|
||
---
|
||
|
||
### Pitfall AC-6: 指数退避无 Jitter 导致重试风暴
|
||
|
||
**What goes wrong:**
|
||
多个爬虫实例同时遇到错误时，因为退避时间相同，会在同一时刻重试，导致"重试风暴"。
|
||
|
||
**Why it happens:**
|
||
标准指数退避算法：`wait = base * 2^attempt`，所有实例计算结果相同。
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# 正确：指数退避 + 随机抖动
|
||
def get_backoff_with_jitter(attempt: int, base: float = 1.0) -> float:
|
||
exponential = base * (2 ** attempt)
|
||
jitter = random.uniform(0, exponential * 0.5) # 0-50% 抖动
|
||
return exponential + jitter
|
||
|
||
# tenacity 配置
|
||
from tenacity import wait_exponential_jitter
|
||
|
||
retry_decorator = retry(
|
||
wait=wait_exponential_jitter(initial=1, max=60, jitter=5),
|
||
stop=stop_after_attempt(3),
|
||
)
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 服务恢复瞬间流量激增
|
||
- 错误在固定时间间隔后重复出现
|
||
|
||
**Phase to address:** Phase 4 (AC-04: 错误恢复机制增强)
|
||
|
||
---
|
||
|
||
## Part 3: LLM Semantic Segmentation Pitfalls (v1.6)
|
||
|
||
> 以下陷阱适用于 v1.6 LLM 语义分段里程碑，重点关注上下文窗口、分段一致性、幻觉检测、成本控制等。
|
||
|
||
### Pitfall LLM-1: Context Window Overflow on Long Videos
|
||
|
||
**What goes wrong:**
|
||
B站视频（尤其是知识类）可能长达1-2小时，字幕内容超过10万token。直接将完整字幕发送给 LLM 会导致 API 错误或截断。
|
||
|
||
**Why it happens:**
|
||
- 2026年的 LLM 虽然号称支持 200k 上下文，但实际在 130k token 左右会变得不可靠
|
||
- Anthropic 研究表明，超过 100k token 的上下文会降低推理质量
|
||
- 简单实现中未考虑 token 计数和分块策略
|
||
|
||
**Consequences:**
|
||
- API 调用失败（4xx 错误）
|
||
- 内容截断导致分段不完整
|
||
- 额外成本：重复处理或手动修复
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 架构设计
|
||
class LLMContentChunker:
|
||
def __init__(self, max_tokens: int = 80_000):
|
||
self.max_tokens = max_tokens
|
||
self.tokenizer = tiktoken.encoding_for_model("gpt-4")
|
||
|
||
def chunk_content(self, content: str) -> List[str]:
|
||
# 1. 首先按现有时间间隔分段（3秒停顿）
|
||
initial_chunks = self.split_by_time_gaps(content)
|
||
|
||
# 2. 合并小段，确保不超过 token 限制
|
||
final_chunks = []
|
||
current_chunk = []
|
||
current_tokens = 0
|
||
|
||
for chunk in initial_chunks:
|
||
chunk_tokens = len(self.tokenizer.encode(chunk))
|
||
if current_tokens + chunk_tokens > self.max_tokens:
|
||
final_chunks.append(" ".join(current_chunk))
|
||
current_chunk = [chunk]
|
||
current_tokens = chunk_tokens
|
||
else:
|
||
current_chunk.append(chunk)
|
||
current_tokens += chunk_tokens
|
||
|
||
if current_chunk:
|
||
final_chunks.append(" ".join(current_chunk))
|
||
|
||
return final_chunks
|
||
```
|
||
|
||
**Two-stage processing strategy:**
|
||
1. **Stage 1**: Fast pre-scan using rules or small model to identify potential topic transition points
|
||
2. **Stage 2**: LLM deep analysis only on suspicious regions
|
||
|
||
**Warning signs:**
|
||
- API 返回 "context_length_exceeded" 错误
|
||
- 某些视频分段结果明显不完整
|
||
- 成本异常高（每次处理 >$5）
|
||
|
||
**Phase to address:** Phase 1 (架构设计：分块策略、两阶段处理)
|
||
|
||
**Sources:**
|
||
- [The LLM context problem in 2026](https://blog.logrocket.com/llm-context-problem/) - LogRocket Blog
|
||
- [Context Window Overflow in 2026](https://redis.io/blog/context-window-overflow/) - Redis Blog
|
||
- [Navigating Frontier LLM Context Window Limitations](https://www.hangryfeed.com/insights/posts/frontier-llm-context-window-limitations-2026-01-15) - HangryFeed
|
||
- [Context Length Comparison: Leading AI Models in 2026](https://www.elvex.com/blog/context-length-comparison-ai-models-2026) - Elvex
|
||
|
||
---
|
||
|
||
### Pitfall LLM-2: Inconsistent Segment Boundaries
|
||
|
||
**What goes wrong:**
|
||
LLM 在不同时间处理相同内容时返回不同的分段边界，导致结果不可重现。
|
||
|
||
**Why it happens:**
|
||
- LLM 的概率性质导致输出不一致
|
||
- Prompt 缺乏明确的边界规则（如"必须在句号处分段"）
|
||
- Temperature 参数设置不当（非零值导致随机性）
|
||
|
||
**Consequences:**
|
||
- 相同视频多次处理结果不同
|
||
- 用户无法信任系统输出
|
||
- 调试困难（无法复现问题）
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - Prompt 工程
|
||
SEMANTIC_SEGMENTATION_PROMPT = """
|
||
你是一个视频字幕语义分析专家。你的任务是将字幕内容按主题分段。
|
||
|
||
**关键规则：**
|
||
1. 必须在句子边界（句号、问号、感叹号）处分段
|
||
2. 绝不在句子中间分段
|
||
3. 每个分段必须包含完整的思想或话题
|
||
4. 如果没有明确主题变化，返回 "CONTINUE"
|
||
|
||
**输出格式（JSON）：**
|
||
{
|
||
"segments": [
|
||
{
|
||
"start_index": 0, # 在原始内容中的字符索引
|
||
"end_index": 150,
|
||
"topic": "黑洞的基本特征", # 如果有明确主题
|
||
"confidence": 0.9 # 0-1 之间的置信度
|
||
}
|
||
],
|
||
"reasoning": "为什么这样分段"
|
||
}
|
||
|
||
**输入内容：**
|
||
{content}
|
||
"""
|
||
|
||
# 使用 temperature=0 获得确定性输出
|
||
client.chat(
|
||
messages=[{"role": "user", "content": prompt}],
|
||
temperature=0, # 关键：确保确定性输出
|
||
response_format={"type": "json_object"}
|
||
)
|
||
```
|
||
|
||
**Phase 2 - 验证机制:**
|
||
```python
|
||
class SegmentValidator:
|
||
def validate_boundaries(self, segments: List[Segment], content: str) -> List[Segment]:
|
||
"""检查分段点是否在句子边界"""
|
||
validated = []
|
||
|
||
for seg in segments:
|
||
# 检查分段点是否在标点符号后
|
||
char_before = content[seg.end_index - 1] if seg.end_index > 0 else ""
|
||
if char_before in "。！？.!?":
|
||
validated.append(seg)
|
||
else:
|
||
# 修正：移动到最近的句子边界
|
||
corrected_end = self.find_next_sentence_boundary(content, seg.end_index)
|
||
validated.append(seg.with_end(corrected_end))
|
||
|
||
return self.merge_short_segments(validated, min_length=50)
|
||
|
||
def find_next_sentence_boundary(self, content: str, position: int) -> int:
|
||
"""找到下一个句子边界"""
|
||
for i in range(position, min(position + 100, len(content))):
|
||
if content[i] in "。！？.!?":
|
||
return i + 1
|
||
return position # 如果找不到，返回原位置
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 相同输入产生不同输出
|
||
- 分段点出现在句子中间（如"我们/来讨论..."）
|
||
- 用户报告"结果每次都不一样"
|
||
|
||
**Phase to address:** Phase 1 (Prompt 工程) + Phase 2 (验证机制)
|
||
|
||
**Sources:**
|
||
- [LLM-Enhanced Semantic Text Segmentation](https://www.mdpi.com/2076-3417/15/19/10849) - MDPI
|
||
- [Best Chunking Strategies for RAG in 2026](https://www.firecrawl.dev/blog/best-chunking-strategies-rag) - Firecrawl
|
||
- [LLM Evaluation Frameworks & Metrics Guide 2026](https://www.mlaidigital.com/blogs/llm-model-evaluation-frameworks-a-complete-guide-for-2026) - MLAI Digital
|
||
|
||
---
|
||
|
||
### Pitfall LLM-3: Hallucinated Topic Titles
|
||
|
||
**What goes wrong:**
|
||
LLM 为分段生成不准确或完全不存在的主题标题。
|
||
|
||
**Why it happens:**
|
||
- LLM 倾向于"合理化"而非"准确化"
|
||
- 视频内容本身缺乏明确主题结构
|
||
- Prompt 未要求标题必须有依据（如引用原文）
|
||
|
||
**Consequences:**
|
||
- 用户被误导（标题与实际内容不符）
|
||
- 信任度下降
|
||
- 知识库内容质量降低
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - Prompt 设计
|
||
TOPIC_GENERATION_PROMPT = """
|
||
为以下字幕分段生成主题标题。
|
||
|
||
**要求：**
|
||
1. **仅在有明确主题时生成标题**
|
||
2. 标题必须基于原文中的关键词或概念
|
||
3. 如果主题不明确，返回 "NO_CLEAR_TOPIC"
|
||
4. 避免编造原文中不存在的内容
|
||
|
||
**输出格式：**
|
||
{
|
||
"topic": "黑洞的形成机制" 或 "NO_CLEAR_TOPIC",
|
||
"evidence": ["原文中的关键词1", "原文中的关键词2"],
|
||
"confidence": 0.8
|
||
}
|
||
|
||
**输入内容：**
|
||
{segment_content}
|
||
"""
|
||
|
||
# Phase 2 - 验证
|
||
class TopicValidator:
|
||
def __init__(self, embedding_model):
|
||
self.embedding_model = embedding_model
|
||
|
||
def validate_topic(self, topic: str, content: str) -> bool:
|
||
"""验证标题与内容的相似度"""
|
||
if topic == "NO_CLEAR_TOPIC":
|
||
return True
|
||
|
||
# 计算标题和内容的 embedding 相似度
|
||
topic_embedding = self.embedding_model.encode(topic)
|
||
content_embedding = self.embedding_model.encode(content)
|
||
|
||
similarity = cosine_similarity([topic_embedding], [content_embedding])[0][0]
|
||
|
||
# 相似度阈值可调
|
||
if similarity < 0.3:
|
||
return False # 标题与内容不符
|
||
|
||
# 检查标题关键词是否出现在内容中
|
||
topic_words = set(topic.split())
|
||
content_words = set(content.split())
|
||
overlap = len(topic_words & content_words) / len(topic_words)
|
||
|
||
return overlap > 0.2 # 至少20%的关键词重叠
|
||
```
|
||
|
||
**User feedback integration:**
|
||
```python
|
||
# Phase 3 - 用户反馈机制
|
||
class FeedbackCollector:
|
||
def record_incorrect_topic(self, video_id: str, segment_id: str, reported_topic: str):
|
||
"""记录用户报告的错误标题"""
|
||
self.feedback_db.insert({
|
||
"video_id": video_id,
|
||
"segment_id": segment_id,
|
||
"reported_topic": reported_topic,
|
||
"timestamp": datetime.now()
|
||
})
|
||
|
||
def get_low_confidence_topics(self, threshold: float = 0.5):
|
||
"""获取低置信度标题用于人工审核"""
|
||
return self.db.query(
|
||
"SELECT * FROM segments WHERE confidence < ?",
|
||
(threshold,)
|
||
)
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 标题与段落内容明显不符
|
||
- 标题包含未在原文中出现的关键概念
|
||
- 用户反馈"标题不对"
|
||
|
||
**Phase to address:** Phase 1 (Prompt 设计) + Phase 2 (验证机制) + Phase 3 (用户反馈)
|
||
|
||
**Sources:**
|
||
- [LLM Hallucinations in 2026: How to Understand and Tackle](https://www.lakera.ai/blog/guide-to-hallucinations-in-large-language-models) - Lakera AI
|
||
- [Hallucination Detection and Mitigation in LLMs](https://arxiv.org/pdf/2601.09929) - arXiv
|
||
- [Detecting hallucinations with LLM-as-a-judge](https://www.datadoghq.com/blog/ai/llm-hallucination-detection/) - Datadog
|
||
- [AI Hallucinations: Definition, Examples & How to Prevent](https://gptzero.me/news/ai-hallucinations-definition-examples/) - GPTZero
|
||
|
||
---
|
||
|
||
### Pitfall LLM-4: Breaking Content Mid-Sentence
|
||
|
||
**What goes wrong:**
|
||
LLM 在句子中间分段，破坏内容完整性。
|
||
|
||
**Why it happens:**
|
||
- Prompt 未明确要求在句子边界处分段
|
||
- LLM 优先考虑"语义完整性"而非"语法完整性"
|
||
- 时间戳信息与文本内容不同步
|
||
|
||
**Consequences:**
|
||
- 段落难以阅读（句子不完整）
|
||
- 后续处理（如摘要生成）质量下降
|
||
- 用户体验差
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - Prompt 工程（已在 Pitfall LLM-2 中覆盖）
|
||
SEMANTIC_SEGMENTATION_PROMPT = """
|
||
**关键规则：**
|
||
1. 必须在句子边界（句号、问号、感叹号）处分段
|
||
2. 绝不在句子中间分段
|
||
3. 每个分段必须以完整句子结束
|
||
|
||
**负面示例（不要这样做）：**
|
||
错误：["我们讨论了黑洞的", "形成机制和观测方法"]
|
||
正确：["我们讨论了黑洞的形成机制。", "接下来是观测方法。"]
|
||
"""
|
||
|
||
# Phase 2 - 后处理
|
||
class SentenceBoundaryProcessor:
|
||
def __init__(self):
|
||
# 使用 NLP 库或正则表达式检测句子边界
|
||
self.sentence_endings = re.compile(r'[。！？.!?]\s*')
|
||
|
||
def enforce_sentence_boundaries(self, segments: List[str]) -> List[str]:
|
||
"""强制在句子边界处分段"""
|
||
processed = []
|
||
|
||
for seg in segments:
|
||
# 检查是否以完整句子结束
|
||
if not self.sentence_endings.search(seg[-1] if seg else ""):
|
||
# 找到下一个句子边界
|
||
next_boundary = self.find_next_boundary(seg)
|
||
if next_boundary < len(seg):
|
||
# 截断到边界
|
||
seg = seg[:next_boundary + 1]
|
||
|
||
# 确保每个分段以标点符号结束
|
||
if seg and not seg[-1] in "。！？.!?":
|
||
seg += "。" # 添加句号
|
||
|
||
processed.append(seg)
|
||
|
||
return processed
|
||
|
||
def find_next_boundary(self, text: str, start: int = 0) -> int:
|
||
"""从指定位置开始查找下一个句子边界"""
|
||
for i in range(start, len(text)):
|
||
if text[i] in "。！？.!?":
|
||
return i
|
||
return len(text) - 1 # 如果找不到，返回末尾
|
||
```
|
||
|
||
**Quality check:**
|
||
```python
|
||
class SegmentQualityChecker:
|
||
def check_segment_completeness(self, segment: str) -> Dict[str, Any]:
|
||
"""检查分段完整性"""
|
||
issues = []
|
||
|
||
# 检查1: 是否以标点符号结束
|
||
if segment and not segment[-1] in "。！？.!?":
|
||
issues.append("missing_punctuation")
|
||
|
||
# 检查2: 是否有小写字母开头（句子中间）
|
||
if segment and segment[0].islower():
|
||
issues.append("starts_mid_sentence")
|
||
|
||
# 检查3: 是否包含截断的句子
|
||
if re.search(r'[，,]\s*$', segment):
|
||
issues.append("ends_with_comma")
|
||
|
||
return {
|
||
"is_complete": len(issues) == 0,
|
||
"issues": issues
|
||
}
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 分段结尾缺少标点符号
|
||
- 分段开头是小写字母（句子中间）
|
||
- 人工审核发现截断的句子
|
||
|
||
**Phase to address:** Phase 1 (Prompt 工程) + Phase 2 (后处理)
|
||
|
||
**Sources:**
|
||
- [WTPSplit - Sentence Boundary Detection](https://github.com/segment-any-text/wtpsplit) - GitHub
|
||
- [Best Chunking Strategies for RAG](https://www.firecrawl.dev/blog/best-chunking-strategies-rag) - Firecrawl
|
||
- [Semantic Boundary Detection for RAG](https://www.inferable.ai/blog/posts/semantic-boundary-detection-rag-agents) - Inferable
|
||
- [How to Build Semantic Chunking](https://oneuptime.com/blog/post/2026-01-30/semantic-chunking/view) - OneUptime
|
||
|
||
---
|
||
|
||
### Pitfall LLM-5: Fallback Strategy Failures
|
||
|
||
**What goes wrong:**
|
||
当 LLM API 失败时，fallback 到规则分段，但两种方法的输出格式不兼容。
|
||
|
||
**Why it happens:**
|
||
- LLM 和规则分段的输出 Schema 设计不同
|
||
- Fallback 逻辑未在架构阶段考虑
|
||
- 错误处理不完善（如超时、速率限制）
|
||
|
||
**Consequences:**
|
||
- Pipeline 在 fallback 时崩溃
|
||
- 用户收到不一致的输出格式
|
||
- 调试困难（无法区分 LLM 结果和 fallback 结果）
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 架构设计：统一接口
|
||
from abc import ABC, abstractmethod
|
||
from typing import Literal
|
||
|
||
class SemanticSegmenterInterface(ABC):
|
||
"""语义分段器统一接口"""
|
||
|
||
@abstractmethod
|
||
def segment(self, content: str, metadata: dict) -> SegmentationResult:
|
||
"""将内容分段
|
||
|
||
Returns:
|
||
SegmentationResult: 统一的结果格式
|
||
"""
|
||
pass
|
||
|
||
class SegmentationResult(BaseModel):
|
||
"""统一的分段结果格式"""
|
||
segments: List[Segment]
|
||
method: Literal["llm", "rule_based", "hybrid"]
|
||
confidence: float
|
||
fallback_used: bool = False
|
||
processing_time_ms: int
|
||
metadata: dict = {}
|
||
|
||
class LLMSegmenter(SemanticSegmenterInterface):
|
||
"""LLM 语义分段器"""
|
||
|
||
def segment(self, content: str, metadata: dict) -> SegmentationResult:
|
||
try:
|
||
start_time = time.time()
|
||
segments = self._call_llm(content)
|
||
return SegmentationResult(
|
||
segments=segments,
|
||
method="llm",
|
||
confidence=0.9,
|
||
processing_time_ms=int((time.time() - start_time) * 1000)
|
||
)
|
||
except Exception as e:
|
||
# Fallback to rule-based
|
||
rule_segmenter = RuleBasedSegmenter()
|
||
result = rule_segmenter.segment(content, metadata)
|
||
result.fallback_used = True
|
||
result.method = "hybrid"
|
||
result.metadata["fallback_reason"] = str(e)
|
||
return result
|
||
|
||
class RuleBasedSegmenter(SemanticSegmenterInterface):
|
||
"""规则分段器（现有实现）"""
|
||
|
||
def segment(self, content: str, metadata: dict) -> SegmentationResult:
|
||
start_time = time.time()
|
||
# 现有的规则分段逻辑
|
||
segments = self._segment_by_rules(content)
|
||
return SegmentationResult(
|
||
segments=segments,
|
||
method="rule_based",
|
||
confidence=0.6,
|
||
processing_time_ms=int((time.time() - start_time) * 1000)
|
||
)
|
||
```
|
||
|
||
**Phase 2 - 错误处理:**
|
||
```python
|
||
class RobustLLMClient:
|
||
def __init__(self, max_retries: int = 3, timeout: int = 60):
|
||
self.max_retries = max_retries
|
||
self.timeout = timeout
|
||
|
||
def call_with_fallback(self, prompt: str) -> str:
|
||
"""带重试和 fallback 的 LLM 调用"""
|
||
last_error = None
|
||
|
||
for attempt in range(self.max_retries):
|
||
try:
|
||
response = self.client.chat(
|
||
messages=[{"role": "user", "content": prompt}],
|
||
timeout=self.timeout
|
||
)
|
||
return response.content
|
||
|
||
except TimeoutError:
|
||
last_error = "timeout"
|
||
wait_time = 2 ** attempt # exponential backoff
|
||
time.sleep(wait_time)
|
||
|
||
except RateLimitError:
|
||
last_error = "rate_limit"
|
||
# 尊重 Retry-After
|
||
retry_after = getattr(e, 'retry_after', 60)
|
||
time.sleep(retry_after)
|
||
|
||
except APIError as e:
|
||
if e.status_code in [401, 403]:
|
||
# 认证错误，不重试
|
||
last_error = f"auth_failed: {e}"
|
||
break
|
||
else:
|
||
last_error = f"api_error: {e}"
|
||
time.sleep(2 ** attempt)
|
||
|
||
# 所有重试失败，触发 fallback
|
||
logger.warning(f"LLM call failed after {self.max_retries} attempts: {last_error}")
|
||
raise FallbackRequiredError(last_error)
|
||
```
|
||
|
||
**Phase 3 - 监控:**
|
||
```python
|
||
class SegmentationMonitor:
|
||
def __init__(self):
|
||
self.metrics = {
|
||
"llm_success": 0,
|
||
"llm_fallback": 0,
|
||
"llm_errors": []
|
||
}
|
||
|
||
def record_success(self):
|
||
self.metrics["llm_success"] += 1
|
||
|
||
def record_fallback(self, reason: str):
|
||
self.metrics["llm_fallback"] += 1
|
||
self.metrics["llm_errors"].append({
|
||
"reason": reason,
|
||
"timestamp": datetime.now().isoformat()
|
||
})
|
||
|
||
def get_health_status(self) -> dict:
|
||
total = self.metrics["llm_success"] + self.metrics["llm_fallback"]
|
||
if total == 0:
|
||
return {"status": "no_data"}
|
||
|
||
fallback_rate = self.metrics["llm_fallback"] / total
|
||
|
||
if fallback_rate > 0.1:
|
||
return {"status": "warning", "fallback_rate": fallback_rate}
|
||
else:
|
||
return {"status": "healthy", "fallback_rate": fallback_rate}
|
||
|
||
def check_alerts(self):
|
||
"""检查是否需要告警"""
|
||
status = self.get_health_status()
|
||
if status.get("fallback_rate", 0) > 0.1:
|
||
logger.error(f"High fallback rate: {status['fallback_rate']:.1%}")
|
||
# 发送告警通知
|
||
```
|
||
|
||
**Warning signs:**
|
||
- Pipeline 在 LLM 失败后崩溃
|
||
- 某些视频输出格式与其他不同
|
||
- 日志中出现大量 API 错误但未处理
|
||
|
||
**Phase to address:** Phase 1 (架构设计) + Phase 2 (错误处理) + Phase 3 (监控)
|
||
|
||
**Sources:**
|
||
- [Agentic Workflows in 2026: LLM Calls with Fallbacks](https://vellum.ai/blog/agentic-workflows-emerging-architectures-and-design-patterns) - Vellum AI
|
||
- [LLM System Design: The Complete Guide](https://www.systemdesignhandbook.com/guides/llm-system-design/) - System Design Handbook
|
||
- [Handling Timeouts and Retries in LLM Systems](https://dasroot.net/posts/2026/02/handling-timeouts-retries-llm-systems/) - Dasroot!
|
||
- [Error Handling & Retries: Making LLM Calls Reliable](https://medium.com/@sonitanishk2003/error-handling-retries-making-llm-calls-reliable-ee7722fc2ea9) - Medium
|
||
|
||
---
|
||
|
||
### Pitfall LLM-6: Video Type Misidentification
|
||
|
||
**What goes wrong:**
|
||
LLM 错误判断视频类型（如将故事类识别为知识类），导致分段策略不匹配。
|
||
|
||
**Why it happens:**
|
||
- 视频内容混合多种类型（如故事中穿插知识讲解）
|
||
- 分类 Prompt 缺乏明确的判断标准
|
||
- LLM 对视频类型的理解与用户期望不一致
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 分类器设计
|
||
VIDEO_TYPE_CLASSIFICATION_PROMPT = """
|
||
分析以下字幕内容，判断视频类型。
|
||
|
||
**视频类型定义：**
|
||
1. **knowledge**: 系统性讲解知识、概念、理论
|
||
- 特征：有明确的知识结构、使用专业术语、循序渐进讲解
|
||
|
||
2. **story**: 叙述故事、事件、经历
|
||
- 特征：有时间线、有人物/事件、强调情节发展
|
||
|
||
3. **dialogue**: 对话、访谈、讨论
|
||
- 特征：多人互动、问答形式、观点交流
|
||
|
||
4. **mixed**: 混合类型
|
||
- 特征：包含以上多种类型的特征
|
||
|
||
**输出格式：**
|
||
{
|
||
"type": "knowledge" | "story" | "dialogue" | "mixed",
|
||
"confidence": 0.0-1.0,
|
||
"reasoning": "判断依据",
|
||
"evidence": ["证据1", "证据2"]
|
||
}
|
||
|
||
**输入内容（前2000字）：**
|
||
{content_preview}
|
||
"""
|
||
|
||
# 使用 few-shot examples
|
||
FEW_SHOT_EXAMPLES = [
|
||
{
|
||
"content": "黑洞是时空展现出极端引力的区域...",
|
||
"type": "knowledge",
|
||
"reasoning": "系统性讲解科学概念"
|
||
},
|
||
{
|
||
"content": "那天我遇到了一个奇怪的人...",
|
||
"type": "story",
|
||
"reasoning": "叙述个人经历和事件"
|
||
}
|
||
]
|
||
```
|
||
|
||
**User override:**
|
||
```python
|
||
class VideoTypeClassifier:
|
||
def __init__(self):
|
||
self.user_overrides = {} # video_id -> user_specified_type
|
||
|
||
def classify(self, content: str, video_id: str = None) -> str:
|
||
# 检查用户是否手动指定了类型
|
||
if video_id and video_id in self.user_overrides:
|
||
return self.user_overrides[video_id]
|
||
|
||
result = self._llm_classify(content)
|
||
|
||
# 低置信度时使用默认策略
|
||
if result["confidence"] < 0.7:
|
||
logger.warning(f"Low confidence classification: {result['confidence']}")
|
||
return "mixed" # 默认使用混合策略
|
||
|
||
return result["type"]
|
||
|
||
def set_user_override(self, video_id: str, video_type: str):
|
||
"""允许用户手动指定视频类型"""
|
||
self.user_overrides[video_id] = video_type
|
||
logger.info(f"User override: {video_id} -> {video_type}")
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 用户报告"分段策略不对"
|
||
- 某类视频的分段质量普遍较差
|
||
- 分类结果与用户直觉不符
|
||
|
||
**Phase to address:** Phase 1 (分类器设计) + Phase 2 (用户反馈)
|
||
|
||
**Sources:**
|
||
- [Multimodal LLMs in 2026: Annotation Challenges](https://www.nextwealth.com/blog/multimodal-llms-in-2026-annotation-challenges-when-ai-needs-to-see-hear-and-read/) - NextWealth
|
||
- [HiVid: LLM-Guided Video Saliency Research](https://arxiv.org/pdf/2602.14214) - arXiv
|
||
|
||
---
|
||
|
||
### Pitfall LLM-7: Cost Explosion
|
||
|
||
**What goes wrong:**
|
||
LLM 调用成本过高，单个视频处理成本 >$1。
|
||
|
||
**Why it happens:**
|
||
- 对每个视频都进行完整的 LLM 分析
|
||
- 未使用缓存（重复处理相同视频）
|
||
- 使用了过于昂贵的模型
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 成本优化：智能缓存
|
||
class CachedLLMSegmenter:
|
||
def __init__(self, cache_dir: str):
|
||
self.cache_dir = Path(cache_dir)
|
||
self.cache_dir.mkdir(exist_ok=True)
|
||
|
||
def get_cache_key(self, content: str) -> str:
|
||
"""生成缓存键"""
|
||
# 使用内容的 hash 作为键
|
||
return hashlib.sha256(content.encode()).hexdigest()
|
||
|
||
def segment(self, content: str) -> SegmentationResult:
|
||
cache_key = self.get_cache_key(content)
|
||
cache_file = self.cache_dir / f"{cache_key}.json"
|
||
|
||
# 检查缓存
|
||
if cache_file.exists():
|
||
logger.info(f"Cache hit: {cache_key}")
|
||
return SegmentationResult.model_validate_json(cache_file.read_text())
|
||
|
||
# 缓存未命中，调用 LLM
|
||
logger.info(f"Cache miss: {cache_key}, calling LLM...")
|
||
result = self._call_llm(content)
|
||
|
||
# 保存到缓存
|
||
cache_file.write_text(result.model_dump_json())
|
||
|
||
return result
|
||
|
||
# 成本优化：使用更便宜的模型
|
||
class TieredLLMStrategy:
|
||
def __init__(self):
|
||
self.cheap_model = "deepseek-chat" # ~$0.28/M tokens
|
||
self.premium_model = "gpt-4" # ~$2.50/M tokens
|
||
|
||
def segment(self, content: str) -> SegmentationResult:
|
||
# 第一步：使用便宜模型快速分析
|
||
quick_result = self._call_cheap_model(content)
|
||
|
||
# 如果 cheap model 置信度高，直接返回
|
||
if quick_result["confidence"] > 0.8:
|
||
return quick_result
|
||
|
||
# 第二步：仅对低置信度情况使用 premium model
|
||
logger.info("Low confidence from cheap model, using premium model")
|
||
return self._call_premium_model(content)
|
||
```
|
||
|
||
**Phase 2 - 监控:**
|
||
```python
|
||
class CostTracker:
|
||
def __init__(self):
|
||
self.costs = []
|
||
|
||
def track_api_call(self, model: str, input_tokens: int, output_tokens: int):
|
||
# 2026年定价（示例）
|
||
PRICING = {
|
||
"deepseek-chat": {"input": 0.14, "output": 0.28}, # per million
|
||
"gpt-4": {"input": 2.50, "output": 10.0},
|
||
}
|
||
|
||
pricing = PRICING.get(model, {"input": 1.0, "output": 1.0})
|
||
cost = (input_tokens * pricing["input"] + output_tokens * pricing["output"]) / 1_000_000
|
||
|
||
self.costs.append({
|
||
"model": model,
|
||
"input_tokens": input_tokens,
|
||
"output_tokens": output_tokens,
|
||
"cost": cost,
|
||
"timestamp": datetime.now()
|
||
})
|
||
|
||
logger.info(f"API call: {model}, {input_tokens}+{output_tokens} tokens, ${cost:.4f}")
|
||
|
||
def get_total_cost(self) -> float:
|
||
return sum(c["cost"] for c in self.costs)
|
||
|
||
def check_cost_alerts(self):
|
||
total = self.get_total_cost()
|
||
if total > 0.5: # 单视频成本阈值
|
||
logger.warning(f"High processing cost: ${total:.2f}")
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 月度账单异常高
|
||
- 单个视频处理成本 >$0.5
|
||
- 频繁调用 LLM 处理相同内容
|
||
|
||
**Phase to address:** Phase 1 (成本优化) + Phase 2 (监控)
|
||
|
||
**Sources:**
|
||
- [LLM Cost Optimization in 2026](https://www.maviklabs.com/blog/llm-cost-optimization-2026) - Mavik Labs
|
||
- [Complete LLM Pricing Comparison 2026](https://www.cloudidr.com/blog/llm-pricing-comparison-2026) - CloudIDR
|
||
- [LLMOps Guide 2026](https://redis.io/blog/large-language-model-operations-guide/) - Redis
|
||
- [LLM API Pricing in 2026: The Complete Cost Comparison](https://www.aimagicx.com/blog/llm-api-pricing-in-2026/) - AI MagicX
|
||
|
||
---
|
||
|
||
### Pitfall LLM-8: Rate Limiting Issues
|
||
|
||
**What goes wrong:**
|
||
批量处理视频时触发 API 速率限制，导致请求失败。
|
||
|
||
**Why it happens:**
|
||
- 未实现请求队列和速率控制
|
||
- 短时间内发送大量请求
|
||
- 未尊重 API 提供商的速率限制
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 速率控制
|
||
import asyncio
|
||
from collections import deque
|
||
|
||
class RateLimitedLLMClient:
|
||
def __init__(self, max_requests_per_second: int = 3):
|
||
self.max_rps = max_requests_per_second
|
||
self.request_times = deque()
|
||
self.semaphore = asyncio.Semaphore(max_requests_per_second)
|
||
|
||
async def call(self, prompt: str) -> str:
|
||
# Token-aware rate limiting
|
||
async with self.semaphore:
|
||
# 等待直到可以发送请求
|
||
await self._wait_for_slot()
|
||
|
||
# 记录请求时间
|
||
now = time.time()
|
||
self.request_times.append(now)
|
||
|
||
# 发送请求
|
||
return await self._api_call(prompt)
|
||
|
||
async def _wait_for_slot(self):
|
||
"""等待直到有可用的请求槽"""
|
||
while len(self.request_times) >= self.max_rps:
|
||
# 检查最早的请求是否已经超过1秒
|
||
if time.time() - self.request_times[0] >= 1.0:
|
||
self.request_times.popleft()
|
||
else:
|
||
# 等待
|
||
await asyncio.sleep(0.1)
|
||
|
||
# Exponential backoff
|
||
class RetryWithBackoff:
|
||
async def call_with_retry(self, func, max_retries: int = 3):
|
||
for attempt in range(max_retries):
|
||
try:
|
||
return await func()
|
||
except RateLimitError:
|
||
if attempt == max_retries - 1:
|
||
raise
|
||
|
||
# Exponential backoff with jitter
|
||
base_wait = 2 ** attempt
|
||
jitter = random.uniform(0, base_wait * 0.5)
|
||
wait_time = base_wait + jitter
|
||
|
||
logger.warning(f"Rate limited, waiting {wait_time:.1f}s")
|
||
await asyncio.sleep(wait_time)
|
||
```
|
||
|
||
**Phase 2 - 批处理策略:**
|
||
```python
|
||
class BatchProcessor:
|
||
def __init__(self, concurrent_limit: int = 3):
|
||
self.concurrent_limit = concurrent_limit
|
||
self.queue = asyncio.Queue()
|
||
|
||
async def process_videos(self, video_urls: List[str]):
|
||
"""批量处理视频"""
|
||
tasks = []
|
||
|
||
for url in video_urls:
|
||
# 创建任务但不立即执行
|
||
task = asyncio.create_task(self._process_single(url))
|
||
tasks.append(task)
|
||
|
||
# 控制并发数
|
||
if len(tasks) >= self.concurrent_limit:
|
||
# 等待一些任务完成
|
||
done, pending = await asyncio.wait(
|
||
tasks,
|
||
return_when=asyncio.FIRST_COMPLETED
|
||
)
|
||
tasks = list(pending)
|
||
|
||
# 等待剩余任务完成
|
||
await asyncio.gather(*tasks)
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 大量 429 错误
|
||
- 请求成功率突然下降
|
||
- API 调用频繁超时
|
||
|
||
**Phase to address:** Phase 1 (速率控制) + Phase 2 (批处理)
|
||
|
||
**Sources:**
|
||
- [Handling Timeouts and Retries in LLM Systems](https://dasroot.net/posts/2026/02/handling-timeouts-retries-llm-systems/) - Dasroot!
|
||
- [Prompt Rate Limits & Batching](https://dev.to/superorange0707/prompt-rate-limits-batching-how-to-stop-your-llm-api-from-melting-down-56e1) - Dev.to
|
||
- [Rate Limiting in AI Gateway](https://www.truefoundry.com/blog/rate-limiting-in-llm-gateway) - TrueFoundry
|
||
- [How to Handle Token Limits and Rate Limits](https://www.typedef.ai/resources/handle-token-limits-rate-limits-large-scale-llm-inference) - Typedef.ai
|
||
|
||
---
|
||
|
||
### Pitfall LLM-9: Integration with Existing Pipeline
|
||
|
||
**What goes wrong:**
|
||
新的 LLM 分段器破坏现有的 SRTProcessor 工作流。
|
||
|
||
**Why it happens:**
|
||
- 接口变更导致向后不兼容
|
||
- 未考虑现有的状态管理和错误处理
|
||
- 测试覆盖不足
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 兼容性设计
|
||
class SemanticSegmenter:
|
||
"""保持现有接口不变"""
|
||
|
||
def __init__(
|
||
self,
|
||
gap_threshold: float = 3.0,
|
||
max_paragraph_chars: int = 500,
|
||
use_llm: bool = False, # 新增：默认不使用 LLM
|
||
llm_client: Optional[SimpleAIClient] = None
|
||
):
|
||
self.gap_threshold = gap_threshold
|
||
self.max_paragraph_chars = max_paragraph_chars
|
||
self.use_llm = use_llm
|
||
self.llm_client = llm_client
|
||
|
||
def segment(self, segments: List[Segment]) -> List[str]:
|
||
"""保持原有方法签名"""
|
||
|
||
if not self.use_llm:
|
||
# 使用现有的规则分段
|
||
return self._segment_by_rules(segments)
|
||
|
||
# 使用 LLM 分段
|
||
return self._segment_by_llm(segments)
|
||
|
||
def _segment_by_rules(self, segments: List[Segment]) -> List[str]:
|
||
"""现有的规则分段逻辑（保持不变）"""
|
||
# ... 现有代码 ...
|
||
pass
|
||
|
||
def _segment_by_llm(self, segments: List[Segment]) -> List[str]:
|
||
"""新的 LLM 分段逻辑"""
|
||
try:
|
||
# LLM 分段
|
||
result = self.llm_client.analyze_segments(segments)
|
||
return result.segments
|
||
except Exception as e:
|
||
# Fallback to rule-based
|
||
logger.warning(f"LLM segmentation failed, falling back to rules: {e}")
|
||
return self._segment_by_rules(segments)
|
||
```
|
||
|
||
**Configuration-based rollout:**
|
||
```python
|
||
# config.yaml
|
||
srt_processing:
|
||
semantic_segmentation:
|
||
method: "rule_based" # or "llm" or "hybrid"
|
||
llm_fallback_enabled: true
|
||
|
||
# 代码中读取配置
|
||
config = load_config("config.yaml")
|
||
segmenter = SemanticSegmenter(
|
||
use_llm=(config.srt_processing.semantic_segmentation.method == "llm")
|
||
)
|
||
```
|
||
|
||
**Testing:**
|
||
```python
|
||
# tests/test_llm_integration.py
|
||
def test_llm_segmentation_compatibility():
|
||
"""确保 LLM 分段与现有流程兼容"""
|
||
# 准备测试数据
|
||
segments = [...]
|
||
|
||
# 测试规则分段（原有行为）
|
||
segmenter = SemanticSegmenter(use_llm=False)
|
||
rule_result = segmenter.segment(segments)
|
||
|
||
# 测试 LLM 分段（新行为）
|
||
llm_segmenter = SemanticSegmenter(use_llm=True, llm_client=mock_client)
|
||
llm_result = llm_segmenter.segment(segments)
|
||
|
||
# 验证输出格式一致
|
||
assert isinstance(rule_result, list)
|
||
assert isinstance(llm_result, list)
|
||
assert all(isinstance(s, str) for s in rule_result)
|
||
assert all(isinstance(s, str) for s in llm_result)
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 现有测试失败
|
||
- Pipeline 在新功能启用后崩溃
|
||
- 输出格式不一致
|
||
|
||
**Phase to address:** Phase 1 (兼容性设计) + Phase 3 (渐进式推出)
|
||
|
||
**Sources:**
|
||
- [LLMOps Guide 2026](https://redis.io/blog/large-language-model-operations-guide/) - Redis
|
||
- [LLM Pipeline Re-Architecture Best Practices](https://www.linkedin.com/top-content/technology/best-practices-in-technology/llm-pipeline-re-architecture-best-practices/) - LinkedIn
|
||
- [How to Build a RAG Pipeline from Scratch in 2026](https://www.kapa.ai/blog/how-to-build-a-rag-pipeline-from-scratch-in-2026) - Kapa.ai
|
||
|
||
---
|
||
|
||
### Pitfall LLM-10: Timeout on Long Videos
|
||
|
||
**What goes wrong:**
|
||
长视频（>1小时）的 LLM 处理超时。
|
||
|
||
**Why it happens:**
|
||
- LLM API 处理时间长（尤其是长上下文）
|
||
- 未设置合理的超时时间
|
||
- 未实现异步处理和进度回调
|
||
|
||
**How to avoid:**
|
||
```python
|
||
# Phase 1 - 超时处理
|
||
class TimeoutAwareLLMClient:
|
||
def __init__(self, timeout: int = 120):
|
||
self.timeout = timeout
|
||
|
||
def call_with_timeout(self, prompt: str) -> str:
|
||
"""设置超时的 LLM 调用"""
|
||
try:
|
||
response = self.client.chat(
|
||
messages=[{"role": "user", "content": prompt}],
|
||
timeout=self.timeout # 设置超时
|
||
)
|
||
return response.content
|
||
except TimeoutError:
|
||
logger.error(f"LLM call timed out after {self.timeout}s")
|
||
raise
|
||
|
||
# Phase 2 - 异步处理
|
||
class AsyncSegmentProcessor:
|
||
async def process_long_video(
|
||
self,
|
||
video_id: str,
|
||
segments: List[Segment],
|
||
progress_callback: Optional[Callable] = None
|
||
) -> SegmentationResult:
|
||
"""异步处理长视频，支持进度回调"""
|
||
|
||
# 分块处理
|
||
chunks = self._split_into_chunks(segments, chunk_size=100)
|
||
|
||
all_segments = []
|
||
for i, chunk in enumerate(chunks):
|
||
if progress_callback:
|
||
progress_callback(i, len(chunks))
|
||
|
||
# 处理单个块
|
||
result = await self._process_chunk_async(chunk)
|
||
all_segments.extend(result.segments)
|
||
|
||
return SegmentationResult(segments=all_segments)
|
||
|
||
def _split_into_chunks(self, segments: List[Segment], chunk_size: int):
|
||
"""将长内容分块"""
|
||
for i in range(0, len(segments), chunk_size):
|
||
yield segments[i:i + chunk_size]
|
||
|
||
# Phase 2 - 进度持久化
|
||
class ResumableProcessor:
|
||
def __init__(self, state_dir: str):
|
||
self.state_dir = Path(state_dir)
|
||
|
||
def process(self, video_id: str, content: str):
|
||
state_file = self.state_dir / f"{video_id}.json"
|
||
|
||
# 检查是否有未完成的状态
|
||
if state_file.exists():
|
||
logger.info(f"Resuming from {state_file}")
|
||
state = json.loads(state_file.read_text())
|
||
completed_chunks = state["completed_chunks"]
|
||
else:
|
||
completed_chunks = []
|
||
|
||
# 继续处理
|
||
chunks = self._split_content(content)
|
||
for i, chunk in enumerate(chunks):
|
||
if i in completed_chunks:
|
||
continue
|
||
|
||
# 处理块
|
||
result = self._process_chunk(chunk)
|
||
completed_chunks.append(i)
|
||
|
||
# 保存进度
|
||
state_file.write_text(json.dumps({
|
||
"video_id": video_id,
|
||
"completed_chunks": completed_chunks,
|
||
"timestamp": datetime.now().isoformat()
|
||
}))
|
||
|
||
# 完成，清理状态文件
|
||
state_file.unlink()
|
||
```
|
||
|
||
**Warning signs:**
|
||
- 某些视频一直处理中
|
||
- 频繁超时错误
|
||
- 用户无法看到处理进度
|
||
|
||
**Phase to address:** Phase 1 (超时处理) + Phase 2 (分块处理)
|
||
|
||
**Sources:**
|
||
- [Handling Timeouts and Retries in LLM Systems](https://dasroot.net/posts/2026/02/handling-timeouts-retries-llm-systems/) - Dasroot!
|
||
- [LLMOps Guide 2026](https://redis.io/blog/large-language-model-operations-guide/) - Redis
|
||
|
||
---
|
||
|
||
## Technical Debt Patterns
|
||
|
||
### Pipeline Technical Debt
|
||
|
||
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|
||
|----------|-------------------|----------------|-----------------|
|
||
| 跳过 Schema 验证 | 快速开发，减少代码 | 后续调试困难，接口不兼容问题频发 | Never |
|
||
| 状态只在内存中 | 简单实现 | 无法恢复，每次失败重头开始 | 仅限一次性任务 |
|
||
| 硬编码路径 | 不用配置 | 迁移困难，环境差异问题 | Never |
|
||
| 自然语言描述步骤输入 | 灵活，文档友好 | LLM 理解不一致，调试困难 | 仅限人类阅读的文档 |
|
||
| 跳过错误处理 | 节省开发时间 | 生产环境频繁崩溃 | 仅限原型/POC |
|
||
| 不记录中间结果 | 节省存储 | 无法断点续传，无法审计 | 仅限幂等操作 |
|
||
|
||
### Anti-Crawler Technical Debt
|
||
|
||
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|
||
|----------|-------------------|----------------|-----------------|
|
||
| 硬编码单个代理 | 快速实现 | 单点故障，被封即全部失败 | 仅开发测试阶段 |
|
||
| 固定请求间隔 | 代码简单 | 被识别为机器人 | **永不可接受** |
|
||
| 忽略 4xx 错误重试 | 减少代码量 | 永久性错误无限重试 | **永不可接受** |
|
||
| 不验证代理可用性 | 启动快 | 运行时大量失败请求 | 仅 MVP 演示 |
|
||
| 禁用 SSL 验证 | 解决证书问题 | 安全风险，可能被中间人攻击 | **永不可接受** |
|
||
|
||
### LLM Segmentation Technical Debt
|
||
|
||
| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
|
||
|----------|-------------------|----------------|-----------------|
|
||
| 直接发送完整内容给 LLM | 实现简单 | 上下文溢出，高成本 | 仅限短视频（<5分钟） |
|
||
| 使用 temperature > 0 | 更多样化的输出 | 结果不一致，无法复现 | **永不可接受** |
|
||
| 跳过 token 计数 | 减少代码 | 不可预测的成本和错误 | 仅限原型 |
|
||
| 硬编码 fallback 逻辑 | 快速实现 | 难以维护，无法监控 | 仅 MVP 阶段 |
|
||
| 不验证 LLM 输出 | 简单实现 | 低质量结果，用户信任下降 | **永不可接受** |
|
||
| 无缓存机制 | 实现简单 | 重复处理，成本爆炸 | **永不可接受** |
|
||
|
||
---
|
||
|
||
## Integration Gotchas
|
||
|
||
### Pipeline Integration
|
||
|
||
| Integration | Common Mistake | Correct Approach |
|
||
|-------------|----------------|------------------|
|
||
| B站 API | 没有处理 403/超时 | 实现重试机制，区分可重试和不可重试错误 |
|
||
| Whisper | 假设输出完美 | 后处理清理（数字格式、标点、分段） |
|
||
| AI API (GLM-5) | 单次调用失败即退出 | 指数退避重试 (1s → 2s → 4s → 8s) |
|
||
| HuggingFace | 没有 HF_TOKEN 仍尝试说话者分离 | 检测 token 存在，无则降级到普通识别 |
|
||
| 知乎 | Cookie 过期导致失败 | 明确错误消息，提示用户更新 Cookie |
|
||
| Git 归档 | 没有配置用户信息 | 检查 KB_GIT_USER/EMAIL，缺失则跳过 Git |
|
||
|
||
### Anti-Crawler Integration (v1.3)
|
||
|
||
| Integration Point | Common Mistake | Correct Approach |
|
||
|-------------------|----------------|------------------|
|
||
| `ZhihuCrawler` 代理配置 | 只设置 `proxy_required=True` 但不处理代理缺失 | 默认 `False`，显式配置时才强制要求 |
|
||
| `CrawlerConfig.from_env()` | 假设环境变量一定存在 | 提供默认值和清晰的错误提示 |
|
||
| `retry_on_network_error` 装饰器 | 对所有函数使用同一重试策略 | 按错误类型区分策略 |
|
||
| Playwright 浏览器管理 | 每次请求创建新浏览器 | 复用浏览器实例，仅在必要时重建 |
|
||
| Cookie 管理 | 硬编码 Cookie 有效期 | 实现过期检测和自动刷新 |
|
||
|
||
### LLM Segmentation Integration (v1.6)
|
||
|
||
| Integration Point | Common Mistake | Correct Approach |
|
||
|-------------------|----------------|------------------|
|
||
| `SemanticSegmenter` 接口 | 破坏现有 `segment()` 方法签名 | 保持兼容，添加 `use_llm` 参数 |
|
||
| SimpleAIClient 调用 | 未设置超时，长视频导致阻塞 | 设置合理超时（60-120秒），异步处理 |
|
||
| 状态管理 | LLM 处理失败未记录到 workflow_state | 记录 fallback 路径，支持断点续传 |
|
||
| 成本追踪 | 未监控 token 使用和成本 | 实现 CostTracker，设置告警阈值 |
|
||
| 现有 SRTProcessor | 直接替换规则分段为 LLM | 保持向后兼容，配置开关控制 |
|
||
|
||
---
|
||
|
||
## Performance Traps
|
||
|
||
### Pipeline Performance
|
||
|
||
| Trap | Symptoms | Prevention | When It Breaks |
|
||
|------|----------|------------|----------------|
|
||
| 内存中累积所有中间结果 | 内存占用持续增长 | 每步完成后写入文件，释放内存 | 处理 1 小时以上视频 |
|
||
| 同步等待 Whisper 完成 | 用户长时间无反馈 | 定时汇报进度（每 2-3 分钟） | 超过 10 分钟的任务 |
|
||
| 单次下载整个视频 | 网络超时，下载失败 | 分块下载，支持断点续传 | 大于 500MB 的视频 |
|
||
| 无限制重试 | 无限循环，资源浪费 | 最大重试次数 + 总超时时间 | 任何外部服务调用 |
|
||
|
||
### Anti-Crawler Performance
|
||
|
||
| Trap | Symptoms | Prevention | When It Breaks |
|
||
|------|----------|------------|----------------|
|
||
| 每次请求验证代理 | 请求延迟显著增加 | 后台异步验证，缓存结果 | >100 代理 |
|
||
| 过于频繁的代理轮换 | 代理池快速耗尽 | 基于会话而非请求轮换 | >10 QPS |
|
||
| 无限制的重试 | 任务永远不结束 | 设置最大重试次数和总超时 | 任何规模 |
|
||
| 同步等待验证码 | 线程阻塞 | 异步检测 + 用户通知 | 任何规模 |
|
||
|
||
### LLM Segmentation Performance
|
||
|
||
| Trap | Symptoms | Prevention | When It Breaks |
|
||
|------|----------|------------|----------------|
|
||
| 一次性发送长内容 | API 超时或上下文溢出 | 分块处理，两阶段策略 | >30分钟视频 |
|
||
| 无缓存机制 | 重复处理相同内容 | 实现 content-based 缓存 | 任何规模 |
|
||
| 同步等待 LLM 响应 | 用户长时间无反馈 | 异步处理 + 进度回调 | >5分钟处理时间 |
|
||
| 单线程批量处理 | 处理队列积压 | 限制并发数 + 请求队列 | >10个视频批量处理 |
|
||
| 未监控成本 | 账单爆炸 | 追踪 token 使用，设置告警 | 任何规模 |
|
||
|
||
---
|
||
|
||
## Security Mistakes
|
||
|
||
### Pipeline Security
|
||
|
||
| Mistake | Risk | Prevention |
|
||
|---------|------|------------|
|
||
| API Key 硬编码 | 泄露到 Git 仓库 | 使用环境变量，.gitignore 排除配置文件 |
|
||
| Cookie 明文存储 | 会话劫持 | 提醒用户 Cookie 敏感性，不提交到 Git |
|
||
|
||
### Anti-Crawler Security
|
||
|
||
| Mistake | Risk | Prevention |
|
||
|---------|------|------------|
|
||
| 代理凭证明文存储 | 凭证泄露 | 使用环境变量，支持 Secret 类型 |
|
||
| 日志输出完整 Cookie | 会话劫持 | 日志脱敏，仅显示前几位 |
|
||
| 忽略 HTTPS 证书验证 | 中间人攻击 | **永不跳过证书验证** |
|
||
| 代理 URL 包含密码 | 密码泄露 | 分离 server/username/password |
|
||
|
||
### LLM Segmentation Security
|
||
|
||
| Mistake | Risk | Prevention |
|
||
|---------|------|------------|
|
||
| Prompt 注入 | LLM 执行恶意指令 | 验证输入，使用结构化 prompt |
|
||
| API Key 泄露到日志 | 凭证泄露 | 日志脱敏，不记录敏感参数 |
|
||
| 未授权访问缓存 | 数据泄露 | 缓存文件权限控制 |
|
||
| 发送敏感内容给 LLM | 隐私泄露 | 内容脱敏，用户明确告知 |
|
||
|
||
---
|
||
|
||
## Platform-Specific Pitfalls
|
||
|
||
### 知乎 (Zhihu) 反爬特性
|
||
|
||
| 特性 | 说明 | 应对策略 |
|
||
|------|------|----------|
|
||
| `x-zse-96` 签名 | 动态签名验证 | 使用 Playwright 浏览器自动处理（现有方案正确） |
|
||
| Cookie 过期检测 | `z_c0` Cookie 有效期约 7 天 | 实现过期检测，触发重新登录 |
|
||
| 登录状态验证 | 未登录时重定向到登录页 | 检测重定向，触发 Cookie 刷新 |
|
||
| 请求频率敏感 | 短时间大量请求触发验证码 | 严格控制请求间隔（2-5秒） |
|
||
| IP 地理位置检测 | 异常地理位置可能触发验证 | 使用国内代理 |
|
||
|
||
### B站 (Bilibili) 反爬特性
|
||
|
||
| 特性 | 说明 | 应对策略 |
|
||
|------|------|----------|
|
||
| 滑动验证码 | 高频访问触发 | 降低频率，使用代理轮换 |
|
||
| 多维度检测 | User-Agent、Cookie、IP 综合判断 | 保持指纹一致性 |
|
||
| 登录必要性 | 部分内容需要登录 | Cookie 持久化管理 |
|
||
| API 限流 | 单 IP 请求频率限制 | 代理轮换 + 请求间隔 |
|
||
|
||
### B站视频内容特性 (v1.6)
|
||
|
||
| 特性 | 说明 | 应对策略 |
|
||
|------|------|----------|
|
||
| 长视频（1-2小时） | 知识类视频常见，token 数量大 | 分块处理，两阶段策略 |
|
||
| 混合类型内容 | 知识+故事+对话混合 | 视频类型识别，自适应分段策略 |
|
||
| 主题切换不明显 | 缺乏明确的过渡词 | 语义分析，识别隐式主题边界 |
|
||
| ASR 错误 | 字幕可能包含语音识别错误 | 与 ASR 纠正步骤配合 |
|
||
|
||
---
|
||
|
||
## "Looks Done But Isn't" Checklist
|
||
|
||
### Pipeline Checklist (v1.0-1.2)
|
||
|
||
- [ ] **状态管理:** 是否每个步骤完成后写入状态？中断后能否恢复？
|
||
- [ ] **错误上下文:** 错误消息是否包含步骤名、输入摘要、可操作建议？
|
||
- [ ] **Schema 验证:** 每个步骤输入/输出是否通过 Pydantic 验证？
|
||
- [ ] **幂等性:** 重复执行同一链接是否产生相同结果？
|
||
- [ ] **环境变量:** 启动时是否检查所有必需变量？
|
||
|
||
### Anti-Crawler Checklist (v1.3)
|
||
|
||
- [ ] **代理池:** 是否验证代理实际可用性？是否测试代理 IP 检测？
|
||
- [ ] **请求间隔:** 是否添加随机抖动？是否模拟人类行为？
|
||
- [ ] **验证码检测:** 是否检测所有常见验证码类型？是否有处理流程？
|
||
- [ ] **错误恢复:** 是否区分可重试/不可重试错误？是否有最大重试限制？
|
||
- [ ] **Cookie 管理:** 是否检测过期？是否支持自动刷新？
|
||
- [ ] **日志脱敏:** 是否避免在日志中输出完整 Cookie/密码？
|
||
- [ ] **集成测试:** 是否测试与现有 Pipeline 的兼容性？
|
||
|
||
### LLM Segmentation Checklist (v1.6)
|
||
|
||
- [ ] **上下文窗口:** 是否实现 token 计数和分块策略？
|
||
- [ ] **输出一致性:** 是否使用 temperature=0？Prompt 是否明确边界规则？
|
||
- [ ] **句子边界:** 是否强制在句子边界处分段？是否有后处理验证？
|
||
- [ ] **标题验证:** 是否验证标题与内容的相似度？低置信度是否标记？
|
||
- [ ] **Fallback:** 是否有统一的输出 Schema？Fallback 是否记录原因？
|
||
- [ ] **成本控制:** 是否实现缓存？是否追踪 token 使用和成本？
|
||
- [ ] **速率限制:** 是否实现请求队列和速率控制？
|
||
- [ ] **超时处理:** 是否设置合理的超时时间？是否支持分块处理？
|
||
- [ ] **向后兼容:** 是否保持现有接口？是否支持配置开关？
|
||
- [ ] **进度可见:** 是否提供进度回调？是否支持断点续传？
|
||
|
||
---
|
||
|
||
## Recovery Strategies
|
||
|
||
### Pipeline Recovery
|
||
|
||
| Pitfall | Recovery Cost | Recovery Steps |
|
||
|---------|---------------|----------------|
|
||
| Statelessness | HIGH | 重新实现状态管理，迁移现有数据 |
|
||
| Error Context Lost | MEDIUM | 增强错误处理，更新所有步骤 |
|
||
| Schema Mismatch | MEDIUM | 更新 Schema，迁移现有数据 |
|
||
|
||
### Anti-Crawler Recovery
|
||
|
||
| Pitfall | Recovery Cost | Recovery Steps |
|
||
|---------|---------------|----------------|
|
||
| IP 泄漏 | HIGH | 1. 停止所有爬虫 2. 更换代理池 3. 等待封禁解除 |
|
||
| 代理轮换被识别 | MEDIUM | 1. 更换代理提供商 2. 调整轮换策略 3. 增加请求间隔 |
|
||
| 验证码检测缺失 | LOW | 1. 添加检测逻辑 2. 清理被污染的代理 3. 重新开始任务 |
|
||
| 重试策略错误 | LOW | 1. 修改错误分类 2. 清理重试队列 3. 重新执行 |
|
||
| Cookie 过期 | LOW | 1. 触发重新登录 2. 更新 Cookie 3. 继续任务 |
|
||
|
||
### LLM Segmentation Recovery
|
||
|
||
| Pitfall | Recovery Cost | Recovery Steps |
|
||
|---------|---------------|----------------|
|
||
| Context Overflow | MEDIUM | 1. 实现分块策略 2. 添加 token 计数器 3. 测试长视频 |
|
||
| Inconsistent Boundaries | HIGH | 1. 修改 Prompt（temperature=0） 2. 实现验证机制 3. 重新处理 |
|
||
| Hallucinated Titles | MEDIUM | 1. 添加标题验证 2. 标记低置信度 3. 用户反馈循环 |
|
||
| Mid-sentence Breaks | MEDIUM | 1. 强化 Prompt 规则 2. 实现后处理 3. 质量检查 |
|
||
| Fallback Failures | HIGH | 1. 重新设计统一 Schema 2. 实现接口抽象 3. 更新所有调用点 |
|
||
| Cost Explosion | HIGH | 1. 实现缓存 2. 添加成本监控 3. 使用更便宜的模型 |
|
||
| Rate Limiting | MEDIUM | 1. 实现请求队列 2. 添加速率控制 3. 配置并发限制 |
|
||
|
||
---
|
||
|
||
## Pitfall-to-Phase Mapping
|
||
|
||
### Pipeline Phases (v1.0-1.2)
|
||
|
||
| Pitfall | Prevention Phase | Verification |
|
||
|---------|------------------|--------------|
|
||
| Statelessness = Restart from Scratch | Phase 1 (PIPE-02, PIPE-04) | 中断执行后可以从断点继续 |
|
||
| Error Propagation Without Context | Phase 1 (PIPE-05, PIPE-07) | 错误消息包含完整上下文和可操作建议 |
|
||
| Assuming Model "Understands Context" | Phase 1 (PIPE-03, PIPE-06) | 每个步骤输入 JSON 自包含所有需要信息 |
|
||
| Parallel Chaos | Phase 1 (设计文档) | 明确文档：串行执行，无并发 |
|
||
| "Automating Bad Processes" | Phase 0 (验证阶段) | 手动执行完整流程，验证每步输出 |
|
||
| Environment Variable Sprawl | Phase 1 (环境检查) | 启动时验证所有必需变量，缺失则报错 |
|
||
| Graceful Degradation That Isn't | Phase 1 (失败处理原则) | 关键步骤失败 = 整体失败，不假装成功 |
|
||
|
||
### Anti-Crawler Phases (v1.3)
|
||
|
||
| Pitfall | Prevention Phase | Verification |
|
||
|---------|------------------|--------------|
|
||
| IP 泄漏 | Phase 1 (AC-01: 代理池管理) | 使用 IP 检测 API 验证代理生效 |
|
||
| 代理轮换规律 | Phase 1 (AC-01: 代理池管理) | 分析日志确认轮换随机性 |
|
||
| 验证码检测缺失 | Phase 3 (AC-03: 验证码检测) | 人工触发验证码测试检测逻辑 |
|
||
| 重试策略错误 | Phase 4 (AC-04: 错误恢复) | 模拟各类错误验证处理正确 |
|
||
| 请求间隔规律 | Phase 2 (AC-02: 行为伪装) | 统计分析请求时间分布 |
|
||
| 退避无 Jitter | Phase 4 (AC-04: 错误恢复) | 多实例并发测试验证无风暴 |
|
||
| 平台特性适配 | Phase 5 (AC-05: 知乎) + Phase 6 (AC-06: B站) | 实际平台测试 |
|
||
|
||
### LLM Segmentation Phases (v1.6)
|
||
|
||
| Pitfall | Prevention Phase | Verification |
|
||
|---------|------------------|--------------|
|
||
| LLM-1: Context Overflow | Phase 1 (架构设计) | 处理1小时视频不溢出 |
|
||
| LLM-2: Inconsistent Boundaries | Phase 1 (Prompt 工程) + Phase 2 (验证) | 多次处理相同内容结果一致 |
|
||
| LLM-3: Hallucinated Titles | Phase 1 (Prompt) + Phase 2 (验证) + Phase 3 (反馈) | 标题与内容相似度检查通过 |
|
||
| LLM-4: Mid-sentence Breaks | Phase 1 (Prompt) + Phase 2 (后处理) | 所有分段在句子边界结束 |
|
||
| LLM-5: Fallback Failures | Phase 1 (架构) + Phase 2 (错误处理) + Phase 3 (监控) | Fallback 率 < 10% |
|
||
| LLM-6: Video Type Misidentification | Phase 1 (分类器) + Phase 2 (用户覆盖) | 分类准确率 > 80% |
|
||
| LLM-7: Cost Explosion | Phase 1 (优化) + Phase 2 (监控) | 单视频成本 < $0.5 |
|
||
| LLM-8: Rate Limiting | Phase 1 (速率控制) + Phase 2 (批处理) | 429 错误率 < 5% |
|
||
| LLM-9: Integration Issues | Phase 1 (兼容性) + Phase 3 (渐进推出) | 现有测试全部通过 |
|
||
| LLM-10: Timeout on Long Videos | Phase 1 (超时) + Phase 2 (分块) | 长视频在合理时间内完成 |
|
||
|
||
---
|
||
|
||
## Sources
|
||
|
||
### Pipeline Sources (v1.0-1.2)
|
||
|
||
- [AI Agent Anti-Patterns: Architectural Pitfalls](https://achan2013.medium.com/ai-agent-anti-patterns-part-1-architectural-pitfalls-that-break-enterprise-agents-before-they-32d211dded43) - MEDIUM confidence
|
||
- [Demystifying Failures in Platform-Orchestrated Workflows](https://arxiv.org/html/2509.23735v2) - HIGH confidence
|
||
- [10 Reasons Multi-Agent Workflows Fail](https://www.infoq.com/presentations/multi-agent-workflow/) - MEDIUM confidence
|
||
- [How Tool Chaining Fails in Production LLM Agents](https://futureagi.substack.com/p/how-tool-chaining-fails-in-production) - MEDIUM confidence
|
||
- [Why Multi-Agent LLM Systems Fail](https://orq.ai/blog/why-do-multi-agent-llm-systems-fail) - MEDIUM confidence
|
||
- [Multi-Step LLM Chains: Best Practices](https://deepchecks.com/orchestrating-multi-step-llm-chains-best-practices/) - HIGH confidence
|
||
- [Data Pipeline State Management: An Underappreciated Challenge](https://www.fivetran.com/blog/data-pipeline-state-management-an-underappreciated-challenge) - HIGH confidence
|
||
- [9 Things I Hate About Environment Variables](https://prefab.cloud/blog/9-problems-with-env-vars/) - HIGH confidence
|
||
- [Mastering Retry Logic Agents: 2025 Best Practices](https://sparkco.ai/blog/mastering-retry-logic-agents-a-deep-dive-into-2025-best-practices) - MEDIUM confidence
|
||
- [Structured Handling of Exceptions in LLM-Driven Agentic Workflows](https://arxiv.org/pdf/2508.07935) - HIGH confidence
|
||
- [Subagent orchestration: The complete 2025 guide](https://www.eesel.ai/blog/subagent-orchestration) - MEDIUM confidence
|
||
- [How Idempotency Keeps Your API Out of Trouble](https://medium.com/@shreyagupta3011/how-idempotency-keeps-your-api-out-of-trouble-7f7848188d39) - HIGH confidence
|
||
- [AI Agent 工作流设计的5 个反模式](https://www.moltbook.com/post/f826b987-641d-4bb5-9938-b825ac5a6b13) - MEDIUM confidence
|
||
- [Workflow Process Automation Gone Wrong](https://www.flowwright.com/workflow-process-automation-gone-wrong-lessons-learned) - MEDIUM confidence
|
||
|
||
### Anti-Crawler Sources (v1.3)
|
||
|
||
**高可信度来源 (官方文档/权威技术文章):**
|
||
- [Firecrawl: 10 Common Web-Scraping Mistakes](https://www.firecrawl.dev/blog/web-scraping-mistakes-and-fixes) - 代理轮换和错误处理最佳实践
|
||
- [ZenRows: Python Requests Retry](https://www.zenrows.com/blog/python-requests-retry) - 指数退避实现
|
||
- [Tenacity Documentation](https://tenacity.readthedocs.io/) - Python 重试库官方文档
|
||
|
||
**中等可信度来源 (技术博客/社区):**
|
||
- [CSDN: 爬虫 Cookie 关键点](https://blog.csdn.net/QuickCode/article/details/153689079) - 请求间隔和 Cookie 处理
|
||
- [知乎专栏: 反反爬虫方法](https://zhuanlan.zhihu.com/p/461669530) - 请求头构造
|
||
- [Stack Overflow: requests proxy bypass](https://stackoverflow.com/questions/28521535/requests-how-to-disable-bypass-proxy) - `trust_env=False` 防止代理泄漏
|
||
|
||
**平台特性 (社区经验):**
|
||
- [CSDN问答: B站反爬虫机制](https://ask.csdn.net/questions/9093164) - B站多维度检测
|
||
- [知乎专栏: 代理IP爬取Bilibili](https://zhuanlan.zhihu.com/p/1888190028963872900) - B站 IP 检测和频率限制
|
||
|
||
**现有代码分析:**
|
||
- `kb_workflow/crawler/zhihu_crawler.py` - 现有知乎爬虫实现
|
||
- `kb_workflow/crawler/config.py` - 现有配置结构
|
||
- `kb_workflow/pipeline/retry.py` - 现有重试机制
|
||
|
||
### LLM Segmentation Sources (v1.6)
|
||
|
||
**Context Window & Token Limits:**
|
||
- [The LLM context problem in 2026](https://blog.logrocket.com/llm-context-problem/) - LogRocket Blog
|
||
- [Context Window Overflow in 2026](https://redis.io/blog/context-window-overflow/) - Redis Blog
|
||
- [Navigating Frontier LLM Context Window Limitations](https://www.hangryfeed.com/insights/posts/frontier-llm-context-window-limitations-2026-01-15) - HangryFeed
|
||
- [Context Length Comparison: Leading AI Models in 2026](https://www.elvex.com/blog/context-length-comparison-ai-models-2026) - Elvex
|
||
|
||
**Segmentation & Consistency:**
|
||
- [LLM-Enhanced Semantic Text Segmentation](https://www.mdpi.com/2076-3417/15/19/10849) - MDPI
|
||
- [Best Chunking Strategies for RAG in 2026](https://www.firecrawl.dev/blog/best-chunking-strategies-rag) - Firecrawl
|
||
- [WTPSplit - Sentence Boundary Detection](https://github.com/segment-any-text/wtpsplit) - GitHub
|
||
- [Semantic Boundary Detection for RAG](https://www.inferable.ai/blog/posts/semantic-boundary-detection-rag-agents) - Inferable
|
||
|
||
**Hallucination Detection:**
|
||
- [LLM Hallucinations in 2026](https://www.lakera.ai/blog/guide-to-hallucinations-in-large-language-models) - Lakera AI
|
||
- [Hallucination Detection and Mitigation in LLMs](https://arxiv.org/pdf/2601.09929) - arXiv
|
||
- [Detecting hallucinations with LLM-as-a-judge](https://www.datadoghq.com/blog/ai/llm-hallucination-detection/) - Datadog
|
||
- [AI Hallucinations: Definition, Examples & How to Prevent](https://gptzero.me/news/ai-hallucinations-definition-examples/) - GPTZero
|
||
|
||
**Fallback & Reliability:**
|
||
- [Agentic Workflows in 2026: LLM Calls with Fallbacks](https://vellum.ai/blog/agentic-workflows-emerging-architectures-and-design-patterns) - Vellum AI
|
||
- [LLM System Design: The Complete Guide](https://www.systemdesignhandbook.com/guides/llm-system-design/) - System Design Handbook
|
||
- [Handling Timeouts and Retries in LLM Systems](https://dasroot.net/posts/2026/02/handling-timeouts-retries-llm-systems/) - Dasroot!
|
||
- [Error Handling & Retries: Making LLM Calls Reliable](https://medium.com/@sonitanishk2003/error-handling-retries-making-llm-calls-reliable-ee7722fc2ea9) - Medium
|
||
|
||
**Cost & Performance:**
|
||
- [LLM Cost Optimization in 2026](https://www.maviklabs.com/blog/llm-cost-optimization-2026) - Mavik Labs
|
||
- [Complete LLM Pricing Comparison 2026](https://www.cloudidr.com/blog/llm-pricing-comparison-2026) - CloudIDR
|
||
- [LLMOps Guide 2026](https://redis.io/blog/large-language-model-operations-guide/) - Redis
|
||
|
||
**Rate Limiting:**
|
||
- [Prompt Rate Limits & Batching](https://dev.to/superorange0707/prompt-rate-limits-batching-how-to-stop-your-llm-api-from-melting-down-56e1) - Dev.to
|
||
- [Rate Limiting in AI Gateway](https://www.truefoundry.com/blog/rate-limiting-in-llm-gateway) - TrueFoundry
|
||
|
||
**Video Content Classification:**
|
||
- [Multimodal LLMs in 2026: Annotation Challenges](https://www.nextwealth.com/blog/multimodal-llms-in-2026-annotation-challenges-when-ai-needs-to-see-hear-and-read/) - NextWealth
|
||
- [HiVid: LLM-Guided Video Saliency Research](https://arxiv.org/pdf/2602.14214) - arXiv
|
||
|
||
**Integration Best Practices:**
|
||
- [LLMOps Guide 2026](https://redis.io/blog/large-language-model-operations-guide/) - Redis
|
||
- [LLM Pipeline Re-Architecture Best Practices](https://www.linkedin.com/top-content/technology/best-practices-in-technology/llm-pipeline-re-architecture-best-practices/) - LinkedIn
|
||
- [How to Build a RAG Pipeline from Scratch in 2026](https://www.kapa.ai/blog/how-to-build-a-rag-pipeline-from-scratch-in-2026) - Kapa.ai
|
||
|
||
---
|
||
*Pitfalls research for: Skill-based Workflow Orchestration + Anti-Crawler Optimization + LLM Semantic Segmentation*
|
||
*Researched: 2026-03-13 (Pipeline), 2026-03-18 (Anti-Crawler), 2026-03-21 (LLM Segmentation)*
|