docs: update research for v1.3 anti-crawler optimization
PR Test / lint (pull_request) Successful in 2s
PR Test / lint (pull_request) Successful in 2s
- Add architecture research for proxy pool, behavior camouflage, captcha detection, and error recovery - Update features, pitfalls, and stack documentation - Add phase 10 validation checklist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+232
-11
@@ -1,10 +1,206 @@
|
||||
# Feature Research
|
||||
|
||||
**Domain:** Skill-based Workflow Orchestration (Content Processing Pipelines)
|
||||
**Researched:** 2026-03-13
|
||||
**Confidence:** HIGH
|
||||
**Researched:** 2026-03-13 (Updated: 2026-03-18 for v1.3 Anti-Crawler)
|
||||
**Confidence:** HIGH (v1.0-1.2) / MEDIUM (v1.3 Anti-Crawler - 基于社区实践)
|
||||
|
||||
## Feature Landscape
|
||||
---
|
||||
|
||||
## v1.3 Milestone: Anti-Crawler Optimization Features
|
||||
|
||||
**Focus:** 增强爬虫反检测能力,降低被封风险,提升爬取稳定性
|
||||
|
||||
---
|
||||
|
||||
### Table Stakes (Users Expect These)
|
||||
|
||||
反爬虫优化中用户默认应该有的功能,缺失会让产品感觉不完整。
|
||||
|
||||
| Feature | Why Expected | Complexity | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| **代理 IP 轮换** | 防止单 IP 被封,是反爬虫基础 | MEDIUM | 当前已有基础代理支持,需扩展为池化管理 |
|
||||
| **请求延迟** | 避免触发频率限制 | LOW | 当前已实现随机延迟 (1-3s),可优化范围 |
|
||||
| **自动重试** | 网络波动时保证稳定性 | LOW | 当前已用 tenacity 实现指数退避重试 |
|
||||
| **错误码识别** | 根据响应状态调整行为 | MEDIUM | 需区分临时错误 vs 永久错误 vs 封禁 |
|
||||
|
||||
### Differentiators (Competitive Advantage)
|
||||
|
||||
能让 kb-skills 脱颖而出的功能。不是必须的,但有价值。
|
||||
|
||||
| Feature | Value Proposition | Complexity | Notes |
|
||||
|---------|-------------------|------------|-------|
|
||||
| **智能代理轮换** | 根据代理健康度、响应时间动态选择 | HIGH | 比简单轮询更有效,降低失败率 |
|
||||
| **请求行为伪装** | 模拟人类浏览模式 | MEDIUM | 随机化 User-Agent、请求顺序、停留时间 |
|
||||
| **验证码检测与告警** | 遇到验证码时暂停并通知,而非静默失败 | MEDIUM | 个人使用场景:检测后暂停等待手动处理 |
|
||||
| **平台特定适配** | 针对知乎/B站的不同反爬策略 | MEDIUM | 知乎 x-zse-96 签名 vs B站 WBI 签名 |
|
||||
| **断点续传增强** | 记录失败原因,智能恢复 | MEDIUM | 区分"需要换 IP" vs "需要换策略" |
|
||||
|
||||
### Anti-Features (Commonly Requested, Often Problematic)
|
||||
|
||||
看似好但会带来问题的功能。
|
||||
|
||||
| Feature | Why Requested | Why Problematic | Alternative |
|
||||
|---------|---------------|-----------------|-------------|
|
||||
| **自动验证码破解** | 无人值守运行 | 1) 需要付费打码服务 2) 违反网站 ToS 3) 增加法律风险 | 检测后暂停,用户手动处理 |
|
||||
| **高并发爬取** | 提高效率 | 1) 更容易触发封禁 2) 增加被检测风险 3) 违背"个人知识库"定位 | 顺序执行 + 合理延迟 |
|
||||
| **复杂指纹伪装** | 绕过高级检测 | 1) 实现复杂 2) 维护成本高 3) 对当前目标网站不必要 | 基础 User-Agent 轮换足够 |
|
||||
| **SOCKS5 代理池** | 更高匿名性 | 1) 住宅代理更有效 2) 当前场景不需要 3) 增加配置复杂度 | HTTP/HTTPS 代理足够 |
|
||||
|
||||
---
|
||||
|
||||
## Platform-Specific Anti-Crawler Measures
|
||||
|
||||
### 知乎 (Zhihu) 反爬策略
|
||||
|
||||
| 检测类型 | 描述 | 应对策略 |
|
||||
|---------|------|---------|
|
||||
| **x-zse-96 签名** | API 请求需要动态签名 | Playwright 自动处理 (当前已实现) |
|
||||
| **Cookie 验证** | 需要登录态 | 二维码扫码登录 (当前已实现) |
|
||||
| **请求频率限制** | 单 IP 高频请求触发限制 | 代理轮换 + 随机延迟 |
|
||||
| **验证码触发** | 异常行为弹出验证码 | 检测 + 暂停 + 告警 |
|
||||
|
||||
**知乎特征响应:**
|
||||
- 401: Cookie 过期
|
||||
- 403: 被限制访问
|
||||
- 验证码页面: HTML 包含 "验证" 或特定元素
|
||||
|
||||
### B站 (Bilibili) 反爬策略
|
||||
|
||||
| 检测类型 | 描述 | 应对策略 |
|
||||
|---------|------|---------|
|
||||
| **WBI 签名** | Web API 需要 w_rid + wts 参数 | 实现签名算法或使用 Playwright |
|
||||
| **Referer 校验** | 需要 `https://www.bilibili.com/` | 设置正确 Referer 头 |
|
||||
| **请求频率限制** | 建议 <60 次/分钟/IP | 代理轮换 + 限流 |
|
||||
| **352 风控** | 异常请求拦截机制 | 监控错误码,调整行为 |
|
||||
|
||||
**B站特征响应:**
|
||||
- -799: 请求过于频繁
|
||||
- -400: 请求错误
|
||||
- -352: 风控拦截
|
||||
|
||||
---
|
||||
|
||||
## v1.3 Feature Dependencies
|
||||
|
||||
```
|
||||
代理 IP 轮换
|
||||
└──requires──> 代理池配置 (PROXY_POOL 环境变量或配置文件)
|
||||
└──enhances──> 错误恢复机制 (根据错误类型决定是否换 IP)
|
||||
|
||||
验证码检测
|
||||
└──requires──> 响应内容分析 (HTML/JSON 解析)
|
||||
└──triggers──> 用户告警系统 (当前可用日志/控制台输出)
|
||||
|
||||
平台特定适配
|
||||
└──requires──> 平台识别逻辑 (URL 解析)
|
||||
└──requires──> 平台签名算法 (知乎 x-zse-96 / B站 WBI)
|
||||
|
||||
错误恢复机制
|
||||
└──requires──> 错误分类 (临时/永久/封禁)
|
||||
└──requires──> 重试策略配置 (指数退避参数)
|
||||
└──conflicts──> 快速失败模式 (某些场景需要立即报错)
|
||||
|
||||
请求行为伪装
|
||||
└──enhances──> 代理 IP 轮换 (完整伪装链)
|
||||
└──enhances──> 验证码规避 (减少触发概率)
|
||||
```
|
||||
|
||||
### Dependency Notes
|
||||
|
||||
- **代理 IP 轮换 requires 代理池配置:** 需要支持多种配置方式 (环境变量、配置文件、API 获取)
|
||||
- **验证码检测 requires 响应内容分析:** 需要解析 HTML/JSON 识别验证码页面特征
|
||||
- **平台特定适配 requires 平台签名算法:** 知乎 x-zse-96 已通过 Playwright 自动处理;B站 WBI 需要实现签名算法
|
||||
- **错误恢复机制 conflicts 快速失败模式:** 可通过配置选择模式,默认使用重试
|
||||
|
||||
---
|
||||
|
||||
## v1.3 MVP Definition
|
||||
|
||||
### Launch With (v1.3 MVP)
|
||||
|
||||
最低可行产品 - 验证反爬虫优化核心价值。
|
||||
|
||||
- [ ] **代理池管理** — 支持配置多个代理,轮换使用 — 核心反封能力
|
||||
- [ ] **随机延迟优化** — 扩展延迟范围 (1-10s),增加随机性 — 低成本高效
|
||||
- [ ] **错误分类与智能重试** — 区分临时错误/永久错误/封禁 — 避免无效重试
|
||||
- [ ] **验证码检测** — 检测验证码页面并暂停 — 避免静默失败
|
||||
|
||||
### Add After Validation (v1.x)
|
||||
|
||||
核心功能验证后添加的增强功能。
|
||||
|
||||
- [ ] **代理健康检查** — 检测代理可用性,自动剔除失效代理 — 当发现代理失败率高时
|
||||
- [ ] **B站 WBI 签名** — 支持 B站 API 签名 — 当需要直接调用 B站 API 时
|
||||
- [ ] **请求行为伪装增强** — User-Agent 轮换、请求头随机化 — 当基础策略不够用时
|
||||
|
||||
### Future Consideration (v2+)
|
||||
|
||||
推迟到产品稳定后的功能。
|
||||
|
||||
- [ ] **智能代理选择** — 根据历史成功率加权选择 — 需要足够的运行数据
|
||||
- [ ] **自动验证码告警** — 桌面通知/邮件通知 — 当需要无人值守时
|
||||
- [ ] **分布式代理池** — 支持从 API 获取代理列表 — 需要代理服务订阅
|
||||
|
||||
---
|
||||
|
||||
## v1.3 Feature Prioritization Matrix
|
||||
|
||||
| Feature | User Value | Implementation Cost | Priority |
|
||||
|---------|------------|---------------------|----------|
|
||||
| 代理池管理 | HIGH | MEDIUM | P1 |
|
||||
| 随机延迟优化 | HIGH | LOW | P1 |
|
||||
| 错误分类与智能重试 | HIGH | MEDIUM | P1 |
|
||||
| 验证码检测 | MEDIUM | LOW | P1 |
|
||||
| 代理健康检查 | MEDIUM | MEDIUM | P2 |
|
||||
| User-Agent 轮换 | MEDIUM | LOW | P2 |
|
||||
| B站 WBI 签名 | LOW* | MEDIUM | P2 |
|
||||
| 智能代理选择 | LOW | HIGH | P3 |
|
||||
|
||||
*注:B站 WBI 签名优先级取决于是否需要直接调用 API(当前 Playwright 方案可能已足够)
|
||||
|
||||
**Priority key:**
|
||||
- P1: Must have for v1.3
|
||||
- P2: Should have, add when possible
|
||||
- P3: Nice to have, future consideration
|
||||
|
||||
---
|
||||
|
||||
## Implementation Complexity Notes
|
||||
|
||||
### 代理池管理 (MEDIUM)
|
||||
- **配置方式**: 环境变量 `PROXY_POOL` (逗号分隔) 或配置文件
|
||||
- **轮换策略**: Round-robin (简单) vs 加权 (复杂)
|
||||
- **集成点**: 修改 `CrawlerConfig` 支持代理列表
|
||||
|
||||
### 验证码检测 (LOW-MEDIUM)
|
||||
- **知乎检测**: 检查响应中是否包含 "验证" 或验证码元素
|
||||
- **通用检测**: 检查 HTTP 状态码 + 响应内容特征
|
||||
- **处理方式**: 抛出特定异常,触发暂停和告警
|
||||
|
||||
### 错误分类 (MEDIUM)
|
||||
- **临时错误**: 网络超时、5xx 服务器错误 → 重试
|
||||
- **永久错误**: 404、权限不足 → 失败不重试
|
||||
- **封禁错误**: 403、验证码 → 换 IP/暂停
|
||||
|
||||
### 请求行为伪装 (LOW)
|
||||
- **User-Agent 池**: 预定义常见 UA 列表
|
||||
- **请求头随机化**: Accept-Language, Accept-Encoding 顺序随机
|
||||
- **会话行为**: 模拟页面停留、滚动 (Playwright 层面)
|
||||
|
||||
---
|
||||
|
||||
## v1.3 Competitor Feature Analysis
|
||||
|
||||
| Feature | Scrapy | Requests + tenacity | Our Approach |
|
||||
|---------|--------|---------------------|--------------|
|
||||
| 代理轮换 | 中间件支持 | 手动实现 | 配置驱动的池管理 |
|
||||
| 重试机制 | 内置 | tenacity 库 | tenacity (已集成) |
|
||||
| 限流 | AutoThrottle 扩展 | 手动 | 配置化延迟范围 |
|
||||
| 验证码 | 需插件 | 无 | 检测 + 暂停告警 |
|
||||
|
||||
---
|
||||
|
||||
## v1.0-1.2 Baseline Features (Preserved)
|
||||
|
||||
### Table Stakes (Users Expect These)
|
||||
|
||||
@@ -49,9 +245,10 @@ Features that seem good but create problems.
|
||||
| **Plugin/Extension System** | Third-party integrations | Security concerns; API stability requirements; documentation burden | Keep monolithic but modular skill architecture; internal extensions only |
|
||||
| **Workflow Visualization** | See pipeline graphically | Nice-to-have but not core value; generates maintenance burden | State file (JSON) is machine-readable; can generate viz later |
|
||||
| **Automatic Retry with Backoff** | Handle transient failures | Can mask underlying issues; retry logic is complex; may cause rate limiting | Explicit error reporting; manual retry with checkpoint resume |
|
||||
| **Complex Routing/Handoff Patterns** | Dynamic task delegation | Overkill for linear content processing; adds nondeterminism; harder to debug | Sequential orchestration with explicit steps |
|
||||
|
||||
## Feature Dependencies
|
||||
---
|
||||
|
||||
## v1.0-1.2 Feature Dependencies
|
||||
|
||||
```
|
||||
[URL Input]
|
||||
@@ -93,9 +290,11 @@ Features that seem good but create problems.
|
||||
- **Platform-Specific Skills require Link Parser:** Cannot extract metadata without knowing platform and content type
|
||||
- **Deep Analysis requires Text Content:** ASR/OCR must complete before AI analysis can process
|
||||
|
||||
## MVP Definition
|
||||
---
|
||||
|
||||
### Launch With (v1)
|
||||
## v1.0-1.2 MVP Definition
|
||||
|
||||
### Launch With (v1) - COMPLETED
|
||||
|
||||
Minimum viable product — what's needed to validate the concept.
|
||||
|
||||
@@ -126,7 +325,9 @@ Features to defer until product-market fit is established.
|
||||
- [ ] **Web UI for Monitoring** — Consider only if non-technical users need access
|
||||
- [ ] **API/Webhook Notifications** — If integration with external systems needed
|
||||
|
||||
## Feature Prioritization Matrix
|
||||
---
|
||||
|
||||
## v1.0-1.2 Feature Prioritization Matrix
|
||||
|
||||
| Feature | User Value | Implementation Cost | Priority |
|
||||
|---------|------------|---------------------|----------|
|
||||
@@ -150,6 +351,8 @@ Features to defer until product-market fit is established.
|
||||
- P2: Should have, add when possible (robustness)
|
||||
- P3: Nice to have, future consideration (convenience)
|
||||
|
||||
---
|
||||
|
||||
## Competitor Feature Analysis
|
||||
|
||||
| Feature | Apache Airflow | Dagster | Prefect | Our Approach |
|
||||
@@ -164,6 +367,8 @@ Features to defer until product-market fit is established.
|
||||
|
||||
**Key differentiation:** Our skill-based architecture with SKILL.md specifications makes the system more accessible to AI agents and easier to extend than traditional workflow engines. The focus on content processing (not general ETL) allows specialized optimizations.
|
||||
|
||||
---
|
||||
|
||||
## Industry Standard Features (2025-2026)
|
||||
|
||||
Based on research, modern data pipeline and workflow orchestration systems have these table stakes:
|
||||
@@ -185,11 +390,26 @@ Based on research, modern data pipeline and workflow orchestration systems have
|
||||
- Self-Service: SKILL.md specifications enable easy understanding
|
||||
- Multi-Format: Video + text + images supported
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [Striim: 7 Must-Have Features of Modern Data Pipelines](https://www.striim.com/blog/what-is-a-data-pipeline-and-must-have-features-of-modern-data-pipelines/) - HIGH confidence
|
||||
- [Dagster: Data Pipeline Frameworks Key Features 2025](https://dagster.io/guides/data-pipeline-frameworks-key-features-10-tools-to-know-in-2025) - HIGH confidence
|
||||
- [Microsoft Azure: AI Agent Orchestration Patterns](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns) - HIGH confidence
|
||||
### v1.3 Anti-Crawler Research (MEDIUM confidence)
|
||||
- [Bright Data - IP Rotation Best Practices](https://brightdata.com/faqs/ip-rotation/best-practices)
|
||||
- [ScrapFly - Bypass Anti-Bot Protection](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping)
|
||||
- [ScrapeHero - TLS Fingerprint Bypass 2026](https://www.scrapehero.com/tls-fingerprint-bypass-techniques/)
|
||||
- [CapMonster - Stop Getting CAPTCHA 2025](https://capmonster.cloud/en/blog/how-to-stop-getting-captcha-when-scraping-in-2025-with-real-solutions)
|
||||
- [OneUptime - Retry Logic with Exponential Backoff in Python](https://oneuptime.com/blog/post/2025-01-06-python-retry-exponential-backoff/view)
|
||||
- [AWS Prescriptive Guidance - Retry with Backoff Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html)
|
||||
- [知乎专栏 - B站 WBI 签名分析](https://zhuanlan.zhihu.com/p/1961014749807513938)
|
||||
- [CSDN - B站视频爬取避坑指南 2024](https://blog.csdn.net/weixin_29051149/article/details/158076702)
|
||||
- [Zyte - Rotating Proxies in Python](https://www.zyte.com/blog/developers-guide-to-rotating-proxies-in-python/)
|
||||
- [Tenacity Library Documentation](https://github.com/jd/tenacity) - 当前项目已使用的重试库
|
||||
|
||||
### v1.0-1.2 Baseline Research (HIGH confidence)
|
||||
- [Striim: 7 Must-Have Features of Modern Data Pipelines](https://www.striim.com/blog/what-is-a-data-pipeline-and-must-have-features-of-modern-data-pipelines/)
|
||||
- [Dagster: Data Pipeline Frameworks Key Features 2025](https://dagster.io/guides/data-pipeline-frameworks-key-features-10-tools-to-know-in-2025)
|
||||
- [Microsoft Azure: AI Agent Orchestration Patterns](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns)
|
||||
- [Medium: Idempotency Guide for Data Pipelines](https://medium.com/@QuarkAndCode/idempotency-guide-iac-apis-data-pipelines-temporal-workflows-8d85dc280613) - MEDIUM confidence
|
||||
- [Fivetran: Data Pipeline State Management](https://www.fivetran.com/blog/data-pipeline-state-management-an-underappreciated-challenge) - MEDIUM confidence
|
||||
- PROJECT.md and ARCHITECTURE.md from codebase analysis - HIGH confidence
|
||||
@@ -198,3 +418,4 @@ Based on research, modern data pipeline and workflow orchestration systems have
|
||||
|
||||
*Feature research for: Skill-based Workflow Orchestration (Content Processing)*
|
||||
*Researched: 2026-03-13*
|
||||
*Updated for v1.3 Anti-Crawler: 2026-03-18*
|
||||
|
||||
Reference in New Issue
Block a user