kb-skills/.planning/research/FEATURES.md

Feature Research

Domain: Skill-based Workflow Orchestration (Content Processing Pipelines) Researched: 2026-03-13 (Updated: 2026-03-18 for v1.3 Anti-Crawler) Confidence: HIGH (v1.0-1.2) / MEDIUM (v1.3 Anti-Crawler - 基于社区实践)

---

v1.3 Milestone: Anti-Crawler Optimization Features

Focus: 增强爬虫反检测能力，降低被封风险，提升爬取稳定性

---

Table Stakes (Users Expect These)

反爬虫优化中用户默认应该有的功能，缺失会让产品感觉不完整。

Feature	Why Expected	Complexity	Notes
代理 IP 轮换	防止单 IP 被封，是反爬虫基础	MEDIUM	当前已有基础代理支持，需扩展为池化管理
请求延迟	避免触发频率限制	LOW	当前已实现随机延迟 (1-3s)，可优化范围
自动重试	网络波动时保证稳定性	LOW	当前已用 tenacity 实现指数退避重试
错误码识别	根据响应状态调整行为	MEDIUM	需区分临时错误 vs 永久错误 vs 封禁

Differentiators (Competitive Advantage)

能让 kb-skills 脱颖而出的功能。不是必须的，但有价值。

Feature	Value Proposition	Complexity	Notes
智能代理轮换	根据代理健康度、响应时间动态选择	HIGH	比简单轮询更有效，降低失败率
请求行为伪装	模拟人类浏览模式	MEDIUM	随机化 User-Agent、请求顺序、停留时间
验证码检测与告警	遇到验证码时暂停并通知，而非静默失败	MEDIUM	个人使用场景：检测后暂停等待手动处理
平台特定适配	针对知乎/B站的不同反爬策略	MEDIUM	知乎 x-zse-96 签名 vs B站 WBI 签名
断点续传增强	记录失败原因，智能恢复	MEDIUM	区分"需要换 IP" vs "需要换策略"

Anti-Features (Commonly Requested, Often Problematic)

看似好但会带来问题的功能。

Feature	Why Requested	Why Problematic	Alternative
自动验证码破解	无人值守运行	1) 需要付费打码服务 2) 违反网站 ToS 3) 增加法律风险	检测后暂停，用户手动处理
高并发爬取	提高效率	1) 更容易触发封禁 2) 增加被检测风险 3) 违背"个人知识库"定位	顺序执行 + 合理延迟
复杂指纹伪装	绕过高级检测	1) 实现复杂 2) 维护成本高 3) 对当前目标网站不必要	基础 User-Agent 轮换足够
SOCKS5 代理池	更高匿名性	1) 住宅代理更有效 2) 当前场景不需要 3) 增加配置复杂度	HTTP/HTTPS 代理足够

---

Platform-Specific Anti-Crawler Measures

知乎 (Zhihu) 反爬策略

检测类型	描述	应对策略
x-zse-96 签名	API 请求需要动态签名	Playwright 自动处理 (当前已实现)
Cookie 验证	需要登录态	二维码扫码登录 (当前已实现)
请求频率限制	单 IP 高频请求触发限制	代理轮换 + 随机延迟
验证码触发	异常行为弹出验证码	检测 + 暂停 + 告警

知乎特征响应:

• 401: Cookie 过期
• 403: 被限制访问
• 验证码页面: HTML 包含 "验证" 或特定元素

B站 (Bilibili) 反爬策略

检测类型	描述	应对策略
WBI 签名	Web API 需要 w_rid + wts 参数	实现签名算法或使用 Playwright
Referer 校验	需要 https://www.bilibili.com/	设置正确 Referer 头
请求频率限制	建议 <60 次/分钟/IP	代理轮换 + 限流
352 风控	异常请求拦截机制	监控错误码，调整行为

B站特征响应:

• -799: 请求过于频繁
• -400: 请求错误
• -352: 风控拦截

---

v1.3 Feature Dependencies

代理 IP 轮换
    └──requires──> 代理池配置 (PROXY_POOL 环境变量或配置文件)
    └──enhances──> 错误恢复机制 (根据错误类型决定是否换 IP)

验证码检测
    └──requires──> 响应内容分析 (HTML/JSON 解析)
    └──triggers──> 用户告警系统 (当前可用日志/控制台输出)

平台特定适配
    └──requires──> 平台识别逻辑 (URL 解析)
    └──requires──> 平台签名算法 (知乎 x-zse-96 / B站 WBI)

错误恢复机制
    └──requires──> 错误分类 (临时/永久/封禁)
    └──requires──> 重试策略配置 (指数退避参数)
    └──conflicts──> 快速失败模式 (某些场景需要立即报错)

请求行为伪装
    └──enhances──> 代理 IP 轮换 (完整伪装链)
    └──enhances──> 验证码规避 (减少触发概率)

Dependency Notes

• 代理 IP 轮换 requires 代理池配置: 需要支持多种配置方式 (环境变量、配置文件、API 获取)
• 验证码检测 requires 响应内容分析: 需要解析 HTML/JSON 识别验证码页面特征
• 平台特定适配 requires 平台签名算法: 知乎 x-zse-96 已通过 Playwright 自动处理；B站 WBI 需要实现签名算法
• 错误恢复机制 conflicts 快速失败模式: 可通过配置选择模式，默认使用重试

---

v1.3 MVP Definition

Launch With (v1.3 MVP)

最低可行产品 - 验证反爬虫优化核心价值。

• 代理池管理 — 支持配置多个代理，轮换使用 — 核心反封能力
• 随机延迟优化 — 扩展延迟范围 (1-10s)，增加随机性 — 低成本高效
• 错误分类与智能重试 — 区分临时错误/永久错误/封禁 — 避免无效重试
• 验证码检测 — 检测验证码页面并暂停 — 避免静默失败

Add After Validation (v1.x)

核心功能验证后添加的增强功能。

• 代理健康检查 — 检测代理可用性，自动剔除失效代理 — 当发现代理失败率高时
• B站 WBI 签名 — 支持 B站 API 签名 — 当需要直接调用 B站 API 时
• 请求行为伪装增强 — User-Agent 轮换、请求头随机化 — 当基础策略不够用时

Future Consideration (v2+)

推迟到产品稳定后的功能。

• 智能代理选择 — 根据历史成功率加权选择 — 需要足够的运行数据
• 自动验证码告警 — 桌面通知/邮件通知 — 当需要无人值守时
• 分布式代理池 — 支持从 API 获取代理列表 — 需要代理服务订阅

---

v1.3 Feature Prioritization Matrix

Feature	User Value	Implementation Cost	Priority
代理池管理	HIGH	MEDIUM	P1
随机延迟优化	HIGH	LOW	P1
错误分类与智能重试	HIGH	MEDIUM	P1
验证码检测	MEDIUM	LOW	P1
代理健康检查	MEDIUM	MEDIUM	P2
User-Agent 轮换	MEDIUM	LOW	P2
B站 WBI 签名	LOW*	MEDIUM	P2
智能代理选择	LOW	HIGH	P3

*注：B站 WBI 签名优先级取决于是否需要直接调用 API（当前 Playwright 方案可能已足够）

Priority key:

• P1: Must have for v1.3
• P2: Should have, add when possible
• P3: Nice to have, future consideration

---

Implementation Complexity Notes

代理池管理 (MEDIUM)

• 配置方式: 环境变量 PROXY_POOL (逗号分隔) 或配置文件
• 轮换策略: Round-robin (简单) vs 加权 (复杂)
• 集成点: 修改 CrawlerConfig 支持代理列表

验证码检测 (LOW-MEDIUM)

• 知乎检测: 检查响应中是否包含 "验证" 或验证码元素
• 通用检测: 检查 HTTP 状态码 + 响应内容特征
• 处理方式: 抛出特定异常，触发暂停和告警

错误分类 (MEDIUM)

• 临时错误: 网络超时、5xx 服务器错误 → 重试
• 永久错误: 404、权限不足 → 失败不重试
• 封禁错误: 403、验证码 → 换 IP/暂停

请求行为伪装 (LOW)

• User-Agent 池: 预定义常见 UA 列表
• 请求头随机化: Accept-Language, Accept-Encoding 顺序随机
• 会话行为: 模拟页面停留、滚动 (Playwright 层面)

---

v1.3 Competitor Feature Analysis

Feature	Scrapy	Requests + tenacity	Our Approach
代理轮换	中间件支持	手动实现	配置驱动的池管理
重试机制	内置	tenacity 库	tenacity (已集成)
限流	AutoThrottle 扩展	手动	配置化延迟范围
验证码	需插件	无	检测 + 暂停告警

---

v1.0-1.2 Baseline Features (Preserved)

Table Stakes (Users Expect These)

Features users assume exist. Missing these = product feels incomplete.

Feature	Why Expected	Complexity	Notes
Pipeline/Sequential Orchestration	Linear processing is the fundamental model - users expect steps to execute in order	LOW	Already in PROJECT.md as core requirement
Idempotent Execution	Re-running same input should produce same output without duplicates	MEDIUM	Critical for reliability; prevents duplicate files on retry
Checkpoint/Resume from Failure	Long-running tasks can fail; users expect to continue from where they left off	MEDIUM	workflow_state.json is planned; checkpointing is industry standard
State Management	Every pipeline needs to track progress - what's done, what's pending	MEDIUM	Step-level state (not task-level) per PROJECT.md
Structured Error Handling	Failures should be captured, reported, and recoverable	LOW	JSON error format with success/error fields already exists
Input/Output Schema Validation	Data contracts ensure reliability; JSON Schema is standard	MEDIUM	PIPE-07 in requirements
Clear Step Boundaries	Users need to understand what each step does, its inputs/outputs	LOW	PIPE-06 requirement
Platform-Specific Content Handlers	Bilibili/Zhihu support is core functionality	MEDIUM	Already implemented for metadata/content extraction
AI-Powered Analysis	Deep analysis is the product's value proposition	HIGH	kb-deep-analyzer exists

Differentiators (Competitive Advantage)

Features that set the product apart. Not required, but valuable.

Feature	Value Proposition	Complexity	Notes
Skill-Based Modular Architecture	Each skill is independent, testable, and reusable; easier maintenance than monolithic pipelines	MEDIUM	SKILL.md specification pattern; already in architecture
Multi-Modal Content Processing	Video (ASR/OCR) + Text unified pipeline; handles diverse content types in one flow	HIGH	Whisper, FunASR, OCR already implemented
Tmux-Based Session Persistence	Long-running operations survive terminal disconnect; GSD-OpenClaw integration	MEDIUM	Unique to this system; TmuxSessionManager exists
Declarative Skill Definition	SKILL.md with YAML frontmatter defines capabilities; AI agents can discover and execute	LOW	Already in place; differentiates from code-only orchestration
Unified JSON Output Format	Consistent success/error structure across all skills; easy programmatic consumption	LOW	Already standardized
Git-Integrated Archival	Processed content versioned and archived automatically	LOW	KB_GIT_ROOT integration
Content Type Detection	Auto-detect video vs article vs QA; route to appropriate processing path	MEDIUM	Link parser handles this

Anti-Features (Commonly Requested, Often Problematic)

Features that seem good but create problems.

Feature	Why Requested	Why Problematic	Alternative
Real-Time/Streaming Processing	Users want instant results	Adds significant complexity; batch processing sufficient for content analysis use case; real-time requires different architecture (WebSocket, event streams)	Keep batch processing; add optional polling/notifications
Concurrent/Parallel Agent Execution	Faster processing	Context sharing between parallel agents is complex; resource contention (API rate limits, model quotas); conflict resolution when results contradict	Sequential orchestration with clear dependencies is more reliable for this use case
Web UI/Dashboard	Visual workflow management	Significant development effort; CLI is sufficient for current use case; UI maintenance overhead	CLI-first; consider status output in structured formats for future visualization
Multi-Tenant/User Management	SaaS-like deployment	Adds auth, isolation, billing complexity; single-user tool currently	Keep single-tenant; future: separate deployment per user
Plugin/Extension System	Third-party integrations	Security concerns; API stability requirements; documentation burden	Keep monolithic but modular skill architecture; internal extensions only
Workflow Visualization	See pipeline graphically	Nice-to-have but not core value; generates maintenance burden	State file (JSON) is machine-readable; can generate viz later
Automatic Retry with Backoff	Handle transient failures	Can mask underlying issues; retry logic is complex; may cause rate limiting	Explicit error reporting; manual retry with checkpoint resume

---

v1.0-1.2 Feature Dependencies

[URL Input]
    └──requires──> [Link Parser Skill]

[Link Parser]
    └──enables──> [Platform-Specific Metadata Extraction]

[Metadata Extraction]
    └──enables──> [Content Download]

[Content Download (Video)]
    └──requires──> [ASR/Transcription]
                          └──requires──> [Whisper OR FunASR]

[Content Download (Video with Text)]
    └──requires──> [OCR Processing]

[ASR/OCR/Text Content]
    └──enables──> [Deep Analysis]

[Deep Analysis]
    └──enables──> [Archive to Git]

[State Management]
    └──enables──> [Checkpoint Resume]
    └──enables──> [Idempotent Execution]

[Schema Validation]
    └──enables──> [Error Detection]
    └──enables──> [Consistent Output Format]

Dependency Notes

• Checkpoint Resume requires State Management: Cannot resume without tracking what completed; workflow_state.json is the mechanism
• Idempotent Execution requires State Management: Need to know if step already completed successfully to skip re-execution
• Schema Validation enables Error Detection: Catch malformed outputs early before they cascade to downstream steps
• Platform-Specific Skills require Link Parser: Cannot extract metadata without knowing platform and content type
• Deep Analysis requires Text Content: ASR/OCR must complete before AI analysis can process

---

v1.0-1.2 MVP Definition

Launch With (v1) - COMPLETED

Minimum viable product — what's needed to validate the concept.

• Sequential Pipeline Orchestration — Core value proposition; linear processing from URL to analysis
• Step-Level State Management (workflow_state.json) — Enables reliability; checkpoint/resume
• Idempotent Execution — Same URL processed twice produces same result; no duplicates
• JSON Schema Validation — Data contracts between steps; error detection
• Structured Error Handling — JSON error format; graceful degradation
• Bilibili/Zhihu Platform Support — Core platforms; metadata + content extraction
• AI Deep Analysis — Product's value output

Add After Validation (v1.x)

Features to add once core is working reliably.

• Enhanced Error Recovery Strategies — 403/timeout specific handling; fallback options
• OCR for Video Text Extraction — Currently implemented but integration may need refinement
• Speaker Diarization — Already implemented; may need pipeline integration
• Multiple ASR Backend Support — Whisper + FunASR selection logic
• Better Progress Visibility — Step completion logging/indicators

Future Consideration (v2+)

Features to defer until product-market fit is established.

• Additional Platforms (YouTube, etc.) — Out of scope per PROJECT.md currently
• Concurrent Processing for Independent Steps — Only if latency becomes critical issue
• Web UI for Monitoring — Consider only if non-technical users need access
• API/Webhook Notifications — If integration with external systems needed

---

v1.0-1.2 Feature Prioritization Matrix

Feature	User Value	Implementation Cost	Priority
Step-Level State Management	HIGH	MEDIUM	P1
Idempotent Execution	HIGH	MEDIUM	P1
JSON Schema Validation	HIGH	LOW	P1
Sequential Pipeline Model	HIGH	LOW	P1
Structured Error Handling	HIGH	LOW	P1
Clear Step Boundaries/Docs	MEDIUM	LOW	P1
Checkpoint/Resume	HIGH	MEDIUM	P2
Platform-Specific Skills	HIGH	MEDIUM	P1 (exists)
AI Deep Analysis	HIGH	HIGH	P1 (exists)
ASR/OCR Integration	MEDIUM	HIGH	P2 (exists)
Error Recovery Strategies	MEDIUM	MEDIUM	P2
Progress Visibility	LOW	LOW	P3
Concurrent Execution	LOW	HIGH	P3
Web UI	LOW	HIGH	P3

Priority key:

• P1: Must have for launch (reliability core)
• P2: Should have, add when possible (robustness)
• P3: Nice to have, future consideration (convenience)

---

Competitor Feature Analysis

Feature	Apache Airflow	Dagster	Prefect	Our Approach
Orchestration Model	DAG-based	Asset-based	Flow-based	Sequential pipeline (simpler)
State Management	Database-backed	Built-in	Built-in	File-based (workflow_state.json)
Checkpoint/Resume	Task-level	Asset-level	Result caching	Step-level
Idempotency	Manual	Built-in checks	Decorator-based	Explicit state check
Schema Validation	Optional	Built-in	Optional	JSON Schema (planned)
Observability	UI Dashboard	UI + Lineage	UI + Logging	JSON output + console
Learning Curve	High	Medium	Low	Low (SKILL.md specs)

Key differentiation: Our skill-based architecture with SKILL.md specifications makes the system more accessible to AI agents and easier to extend than traditional workflow engines. The focus on content processing (not general ETL) allows specialized optimizations.

---

Industry Standard Features (2025-2026)

Based on research, modern data pipeline and workflow orchestration systems have these table stakes:

1. Exactly-Once Processing (E1P) — Data integrity guarantee; each event processed exactly once
2. Fault-Tolerant Architecture — Automatic failover and recovery
3. Data Lineage/Tracking — Know where data came from and how it was transformed
4. Monitoring & Logging — Real-time visibility into pipeline execution
5. Scalable Architecture — Handle varying loads
6. Self-Service Management — Easy pipeline creation and modification
7. Multi-Format Support — Structured, semi-structured, unstructured data

Our coverage:

• E1P: Achieved via idempotent execution
• Fault-Tolerance: Checkpoint/resume covers this
• Lineage: State tracking provides basic lineage
• Monitoring: JSON output + console logging
• Scalability: Not a concern (batch processing, single user)
• Self-Service: SKILL.md specifications enable easy understanding
• Multi-Format: Video + text + images supported

---

Sources

v1.3 Anti-Crawler Research (MEDIUM confidence)

• Bright Data - IP Rotation Best Practices
• ScrapFly - Bypass Anti-Bot Protection
• ScrapeHero - TLS Fingerprint Bypass 2026
• CapMonster - Stop Getting CAPTCHA 2025
• OneUptime - Retry Logic with Exponential Backoff in Python
• AWS Prescriptive Guidance - Retry with Backoff Pattern
• 知乎专栏 - B站 WBI 签名分析
• CSDN - B站视频爬取避坑指南 2024
• Zyte - Rotating Proxies in Python
• Tenacity Library Documentation - 当前项目已使用的重试库

v1.0-1.2 Baseline Research (HIGH confidence)

• Striim: 7 Must-Have Features of Modern Data Pipelines
• Dagster: Data Pipeline Frameworks Key Features 2025
• Microsoft Azure: AI Agent Orchestration Patterns
• Medium: Idempotency Guide for Data Pipelines - MEDIUM confidence
• Fivetran: Data Pipeline State Management - MEDIUM confidence
• PROJECT.md and ARCHITECTURE.md from codebase analysis - HIGH confidence

---

Feature research for: Skill-based Workflow Orchestration (Content Processing) Researched: 2026-03-13 Updated for v1.3 Anti-Crawler: 2026-03-18