docs: complete v1.6 semantic segmentation research

This commit is contained in:
2026-03-21 23:53:55 +08:00
parent cf858027d4
commit 440c885448
5 changed files with 2839 additions and 493 deletions
+172 -360
View File
@@ -1,421 +1,233 @@
# Feature Research
# Feature Landscape: LLM Semantic Segmentation for Video Subtitles
**Domain:** Skill-based Workflow Orchestration (Content Processing Pipelines)
**Researched:** 2026-03-13 (Updated: 2026-03-18 for v1.3 Anti-Crawler)
**Confidence:** HIGH (v1.0-1.2) / MEDIUM (v1.3 Anti-Crawler - 基于社区实践)
**Domain:** LLM-based semantic segmentation for video subtitle processing
**Researched:** 2026-03-21
**Overall confidence:** MEDIUM
---
## Table Stakes
## v1.3 Milestone: Anti-Crawler Optimization Features
**Focus:** 增强爬虫反检测能力,降低被封风险,提升爬取稳定性
---
### Table Stakes (Users Expect These)
反爬虫优化中用户默认应该有的功能,缺失会让产品感觉不完整。
Features users expect in LLM-based subtitle segmentation. Missing = product feels incomplete or "dumb."
| Feature | Why Expected | Complexity | Notes |
|---------|--------------|------------|-------|
| **代理 IP 轮换** | 防止单 IP 被封,是反爬虫基础 | MEDIUM | 当前已有基础代理支持,需扩展为池化管理 |
| **请求延迟** | 避免触发频率限制 | LOW | 当前已实现随机延迟 (1-3s),可优化范围 |
| **自动重试** | 网络波动时保证稳定性 | LOW | 当前已用 tenacity 实现指数退避重试 |
| **错误码识别** | 根据响应状态调整行为 | MEDIUM | 需区分临时错误 vs 永久错误 vs 封禁 |
| **视频类型识别** | 不同类型视频需要不同分段策略 | Medium | 识别知识类、故事类、访谈类、新闻类、vlog等基本类型 |
| **主题边界检测** | 用户期望在话题转换处分段 | High | 基于语义相似度变化检测话题转换点 |
| **内容完整性保护** | 避免在句子或论点中间截断 | High | 确保分段不破坏语义完整性 |
| **自然句边界检测** | 中文字幕需要智能断句 | Medium | 中文没有明确句号,需要语义判断 |
| **主题标题生成** | 每个分段需要描述性标题 | Medium | 为分段生成简洁、准确的主题标题 |
| **基本分段策略** | 区别于简单的时间+长度规则 | Medium | 至少提供2-3种基于语义的分段策略 |
### Differentiators (Competitive Advantage)
## Differentiators
能让 kb-skills 脱颖而出的功能。不是必须的,但有价值。
Features that set product apart. Not expected, but valued.
| Feature | Value Proposition | Complexity | Notes |
|---------|-------------------|------------|-------|
| **智能代理轮换** | 根据代理健康度、响应时间动态选择 | HIGH | 比简单轮询更有效,降低失败率 |
| **请求行为伪装** | 模拟人类浏览模式 | MEDIUM | 随机化 User-Agent、请求顺序、停留时间 |
| **验证码检测与告警** | 遇到验证码时暂停并通知,而非静默失败 | MEDIUM | 个人使用场景:检测后暂停等待手动处理 |
| **平台特定适配** | 针对知乎/B站的不同反爬策略 | MEDIUM | 知乎 x-zse-96 签名 vs B站 WBI 签名 |
| **断点续传增强** | 记录失败原因,智能恢复 | MEDIUM | 区分"需要换 IP" vs "需要换策略" |
| **分层主题结构** | 支持多级主题层次(章-节) | High | 类似书籍目录的层级结构,适用于长视频 |
| **时间戳对齐** | 分段边界与原字幕时间对齐 | Medium | 保持与原始视频的可导航性 |
| **分段质量评分** | 给出分段质量的可信度 | Medium | 帮助用户判断何时需要人工调整 |
| **说话人识别集成** | 结合说话人变化优化分段 | High | 访谈、对话类视频的重要信号 |
| **可配置分段粒度** | 用户可控制分段密度 | Low | 粗粒度(章节)vs 细粒度(话题) |
| **分段合并建议** | 检测过短分段并建议合并 | Medium | 避免碎片化分段 |
| **主题连贯性分析** | 评估分段的内部一致性 | High | 确保单个分段内语义相关 |
| **多模态融合** | 结合视频场景变化辅助分段 | Very High | 需要视频帧分析,超出纯文本范围 |
### Anti-Features (Commonly Requested, Often Problematic)
## Anti-Features
看似好但会带来问题的功能。
Features to explicitly NOT build.
| Feature | Why Requested | Why Problematic | Alternative |
|---------|---------------|-----------------|-------------|
| **自动验证码破解** | 无人值守运行 | 1) 需要付费打码服务 2) 违反网站 ToS 3) 增加法律风险 | 检测后暂停,用户手动处理 |
| **高并发爬取** | 提高效率 | 1) 更容易触发封禁 2) 增加被检测风险 3) 违背"个人知识库"定位 | 顺序执行 + 合理延迟 |
| **复杂指纹伪装** | 绕过高级检测 | 1) 实现复杂 2) 维护成本高 3) 对当前目标网站不必要 | 基础 User-Agent 轮换足够 |
| **SOCKS5 代理池** | 更高匿名性 | 1) 住宅代理更有效 2) 当前场景不需要 3) 增加配置复杂度 | HTTP/HTTPS 代理足够 |
| Anti-Feature | Why Avoid | What to Do Instead |
|--------------|-----------|-------------------|
| **固定大小分段** | 违背语义分段初衷,破坏内容完整性 | 使用语义边界检测 |
| **纯关键词匹配** | 过于简单,无法理解上下文和语义 | 使用嵌入向量和语义相似度 |
| **实时流式分段** | 需要全局上下文才能做出好的分段决策 | 批处理模式,分析完整内容 |
| **多语言混合处理** | 中英文混合分段逻辑复杂,降低质量 | 先分离语言,分别处理 |
| **完全自动化无人工审核** | LLM可能误判边界,需要人工验证 | 提供预览和手动调整界面 |
| **过度细分段** | 产生过多碎片化分段,降低可读性 | 设置最小分段长度和主题阈值 |
---
## Platform-Specific Anti-Crawler Measures
### 知乎 (Zhihu) 反爬策略
| 检测类型 | 描述 | 应对策略 |
|---------|------|---------|
| **x-zse-96 签名** | API 请求需要动态签名 | Playwright 自动处理 (当前已实现) |
| **Cookie 验证** | 需要登录态 | 二维码扫码登录 (当前已实现) |
| **请求频率限制** | 单 IP 高频请求触发限制 | 代理轮换 + 随机延迟 |
| **验证码触发** | 异常行为弹出验证码 | 检测 + 暂停 + 告警 |
**知乎特征响应:**
- 401: Cookie 过期
- 403: 被限制访问
- 验证码页面: HTML 包含 "验证" 或特定元素
### B站 (Bilibili) 反爬策略
| 检测类型 | 描述 | 应对策略 |
|---------|------|---------|
| **WBI 签名** | Web API 需要 w_rid + wts 参数 | 实现签名算法或使用 Playwright |
| **Referer 校验** | 需要 `https://www.bilibili.com/` | 设置正确 Referer 头 |
| **请求频率限制** | 建议 <60 次/分钟/IP | 代理轮换 + 限流 |
| **352 风控** | 异常请求拦截机制 | 监控错误码,调整行为 |
**B站特征响应:**
- -799: 请求过于频繁
- -400: 请求错误
- -352: 风控拦截
---
## v1.3 Feature Dependencies
## Feature Dependencies
```
代理 IP 轮换
└──requires──> 代理池配置 (PROXY_POOL 环境变量或配置文件)
└──enhances──> 错误恢复机制 (根据错误类型决定是否换 IP)
视频类型识别 → 分段策略选择 (类型识别决定使用哪种分段策略)
视频类型识别 → 主题边界检测阈值 (不同类型使用不同的边界敏感度)
验证码检测
└──requires──> 响应内容分析 (HTML/JSON 解析)
└──triggers──> 用户告警系统 (当前可用日志/控制台输出)
自然句边界检测 → 内容完整性保护 (先确定句边界,再保护完整性)
主题边界检测 → 内容完整性保护 (边界不能破坏句完整性和论点完整性)
平台特定适配
└──requires──> 平台识别逻辑 (URL 解析)
└──requires──> 平台签名算法 (知乎 x-zse-96 / B站 WBI)
主题边界检测 → 主题标题生成 (先确定边界,再为每段生成标题)
错误恢复机制
└──requires──> 错误分类 (临时/永久/封禁)
└──requires──> 重试策略配置 (指数退避参数)
└──conflicts──> 快速失败模式 (某些场景需要立即报错)
请求行为伪装
└──enhances──> 代理 IP 轮换 (完整伪装链)
└──enhances──> 验证码规避 (减少触发概率)
分段策略选择 → 主题边界检测 (不同策略使用不同的边界检测算法)
```
### Dependency Notes
## MVP Recommendation for v1.6
- **代理 IP 轮换 requires 代理池配置:** 需要支持多种配置方式 (环境变量、配置文件、API 获取)
- **验证码检测 requires 响应内容分析:** 需要解析 HTML/JSON 识别验证码页面特征
- **平台特定适配 requires 平台签名算法:** 知乎 x-zse-96 已通过 Playwright 自动处理;B站 WBI 需要实现签名算法
- **错误恢复机制 conflicts 快速失败模式:** 可通过配置选择模式,默认使用重试
**Prioritize (MUST-have):**
1. **视频类型识别** - 基础功能,决定后续策略
2. **自然句边界检测** - 中文断句,保护内容完整性
3. **主题边界检测** - 核心功能,实现语义分段
4. **内容完整性保护** - 质量保证,避免破坏语义
5. **主题标题生成** - 用户体验,提供可导航性
6. **基本分段策略** - 至少2种策略(知识类、故事类)
---
**Defer to v2.0:**
- **分层主题结构** - 需要更复杂的算法和更多测试
- **说话人识别集成** - 需要额外的说话人分离模块
- **分段质量评分** - 优化功能,不影响核心价值
- **时间戳对齐** - 优化功能,v1.6可以简化处理
## v1.3 MVP Definition
**Avoid:**
- 固定大小分段、纯关键词匹配、实时流式处理
### Launch With (v1.3 MVP)
## Technical Implementation Insights
最低可行产品 - 验证反爬虫优化核心价值。
### 视频类型识别方法
- [ ] **代理池管理** — 支持配置多个代理,轮换使用 — 核心反封能力
- [ ] **随机延迟优化** — 扩展延迟范围 (1-10s),增加随机性 — 低成本高效
- [ ] **错误分类与智能重试** — 区分临时错误/永久错误/封禁 — 避免无效重试
- [ ] **验证码检测** — 检测验证码页面并暂停 — 避免静默失败
基于2025年研究,视频内容类型分类主要方法:
### Add After Validation (v1.x)
1. **基于文本特征** - 分析标题、描述、字幕文本
- 知识类:教程、课程、讲解性内容
- 故事类:叙事结构、时间线、情节发展
- 访谈类:对话交替、问答模式
- 新闻类:事件报道、时间地点、事实陈述
- Vlog:个人经历、主观感受、日常记录
核心功能验证后添加的增强功能。
2. **LLM分类方法** - 使用few-shot prompting分类
- 提供各类别的示例
- 让LLM分析内容特征
- 输出类别和置信度
- [ ] **代理健康检查** — 检测代理可用性,自动剔除失效代理 — 当发现代理失败率高时
- [ ] **B站 WBI 签名** — 支持 B站 API 签名 — 当需要直接调用 B站 API 时
- [ ] **请求行为伪装增强** — User-Agent 轮换、请求头随机化 — 当基础策略不够用时
### 主题边界检测策略
### Future Consideration (v2+)
基于2025年研究前沿:
推迟到产品稳定后的功能。
1. **语义相似度方法**
- 使用嵌入向量计算段落间相似度
- 相似度骤降点 = 潜在边界
- 适用于:知识类、故事类
- [ ] **智能代理选择** — 根据历史成功率加权选择 — 需要足够的运行数据
- [ ] **自动验证码告警** — 桌面通知/邮件通知 — 当需要无人值守时
- [ ] **分布式代理池** — 支持从 API 获取代理列表 — 需要代理服务订阅
2. **词频变化检测**
- 检测sharp word-frequency dips
- 词汇分布变化 = 话题转换
- 适用于:新闻类、访谈类
---
3. **LLM零样本边界检测**
- 直接让LLM识别话题转换点
- 提供上下文窗口
- 输出边界位置和理由
## v1.3 Feature Prioritization Matrix
4. **强化学习方法** (BoundRL, 2025)
- 训练模型学习边界模式
- 需要标注数据
- 可能超出v1.6范围
| Feature | User Value | Implementation Cost | Priority |
|---------|------------|---------------------|----------|
| 代理池管理 | HIGH | MEDIUM | P1 |
| 随机延迟优化 | HIGH | LOW | P1 |
| 错误分类与智能重试 | HIGH | MEDIUM | P1 |
| 验证码检测 | MEDIUM | LOW | P1 |
| 代理健康检查 | MEDIUM | MEDIUM | P2 |
| User-Agent 轮换 | MEDIUM | LOW | P2 |
| B站 WBI 签名 | LOW* | MEDIUM | P2 |
| 智能代理选择 | LOW | HIGH | P3 |
### 中文句边界检测
*注:B站 WBI 签名优先级取决于是否需要直接调用 API(当前 Playwright 方案可能已足够)
2025年研究方向:
**Priority key:**
- P1: Must have for v1.3
- P2: Should have, add when possible
- P3: Nice to have, future consideration
1. **边界增强解码器** (Boundary-Enhanced Decoder)
- 专门针对中文分词和断句
- 神经注意力机制
---
2. **多任务学习方法**
- 联合学习分词和断句
- 提高边界识别准确性
## Implementation Complexity Notes
3. **实践方法** (适合v1.6)
- 使用标点符号(句号、问号、感叹号)
- 语义停顿检测(基于短语结构)
- LLM辅助断句(对歧义情况)
### 代理池管理 (MEDIUM)
- **配置方式**: 环境变量 `PROXY_POOL` (逗号分隔) 或配置文件
- **轮换策略**: Round-robin (简单) vs 加权 (复杂)
- **集成点**: 修改 `CrawlerConfig` 支持代理列表
### 内容完整性保护
### 验证码检测 (LOW-MEDIUM)
- **知乎检测**: 检查响应中是否包含 "验证" 或验证码元素
- **通用检测**: 检查 HTTP 状态码 + 响应内容特征
- **处理方式**: 抛出特定异常,触发暂停和告警
基于2025年研究:
### 错误分类 (MEDIUM)
- **临时错误**: 网络超时、5xx 服务器错误 → 重试
- **永久错误**: 404、权限不足 → 失败不重试
- **封禁错误**: 403、验证码 → 换 IP/暂停
1. **语义连贯性评估**
- 确保分段内语义一致
- 避免在论点中间截断
### 请求行为伪装 (LOW)
- **User-Agent 池**: 预定义常见 UA 列表
- **请求头随机化**: Accept-Language, Accept-Encoding 顺序随机
- **会话行为**: 模拟页面停留、滚动 (Playwright 层面)
2. **修辞结构理论** (RST)
- 识别文本的修辞关系
- 保持修辞结构的完整性
---
3. **实践策略** (v1.6可实现)
- 不在句子中间分段
- 识别并保护论证结构(前提-结论)
- 检测并列结构(首先、其次、最后)
## v1.3 Competitor Feature Analysis
### 主题标题生成
| Feature | Scrapy | Requests + tenacity | Our Approach |
|---------|--------|---------------------|--------------|
| 代理轮换 | 中间件支持 | 手动实现 | 配置驱动的池管理 |
| 重试机制 | 内置 | tenacity 库 | tenacity (已集成) |
| 限流 | AutoThrottle 扩展 | 手动 | 配置化延迟范围 |
| 验证码 | 需插件 | 无 | 检测 + 暂停告警 |
2025年方法:
---
1. **LLM摘要式标题**
- 让LLM生成分段的简短摘要
- 限制长度(如10-15字)
2. **关键词提取**
- 提取分段的关键词
- 组合成描述性标题
## v1.0-1.2 Baseline Features (Preserved)
3. **两阶段方法**
- 第一阶段:检测边界
- 第二阶段:为每段生成标题
### Table Stakes (Users Expect These)
## Complexity Breakdown
Features users assume exist. Missing these = product feels incomplete.
| Complexity Level | Features | Rationale |
|------------------|----------|-----------|
| **Low** | 可配置分段粒度 | UI/配置项,不涉及核心算法 |
| **Medium** | 视频类型识别、自然句边界检测、主题标题生成、时间戳对齐、分段合并建议 | 有成熟方法,可直接应用LLM |
| **High** | 主题边界检测、内容完整性保护、分层主题结构、分段质量评分、说话人识别集成 | 需要复杂算法或多阶段处理 |
| **Very High** | 多模态融合、主题连贯性分析 | 需要额外模块或大量计算资源 |
| Feature | Why Expected | Complexity | Notes |
|---------|--------------|------------|-------|
| **Pipeline/Sequential Orchestration** | Linear processing is the fundamental model - users expect steps to execute in order | LOW | Already in PROJECT.md as core requirement |
| **Idempotent Execution** | Re-running same input should produce same output without duplicates | MEDIUM | Critical for reliability; prevents duplicate files on retry |
| **Checkpoint/Resume from Failure** | Long-running tasks can fail; users expect to continue from where they left off | MEDIUM | workflow_state.json is planned; checkpointing is industry standard |
| **State Management** | Every pipeline needs to track progress - what's done, what's pending | MEDIUM | Step-level state (not task-level) per PROJECT.md |
| **Structured Error Handling** | Failures should be captured, reported, and recoverable | LOW | JSON error format with success/error fields already exists |
| **Input/Output Schema Validation** | Data contracts ensure reliability; JSON Schema is standard | MEDIUM | PIPE-07 in requirements |
| **Clear Step Boundaries** | Users need to understand what each step does, its inputs/outputs | LOW | PIPE-06 requirement |
| **Platform-Specific Content Handlers** | Bilibili/Zhihu support is core functionality | MEDIUM | Already implemented for metadata/content extraction |
| **AI-Powered Analysis** | Deep analysis is the product's value proposition | HIGH | kb-deep-analyzer exists |
## Video Type-Specific Strategies
### Differentiators (Competitive Advantage)
### 知识类视频
- **分段依据**: 主题/知识点变化
- **边界信号**: "接下来"、"其次"、"另外"等过渡词
- **标题风格**: 描述性,反映知识点
- **分段长度**: 中等(2-5分钟)
### 故事类视频
- **分段依据**: 情节发展、场景变化
- **边界信号**: 时间转换、地点变化、人物变化
- **标题风格**: 叙事性,反映情节发展
- **分段长度**: 较长(5-10分钟)
Features that set the product apart. Not required, but valuable.
### 访谈类视频
- **分段依据**: 问题/话题转换
- **边界信号**: 主持人提问、话题切换
- **标题风格**: 问答式,反映讨论主题
- **分段长度**: 中等(3-6分钟)
### 新闻类视频
- **分段依据**: 新闻条目
- **边界信号**: "接下来是"、"据报道"
- **标题风格**: 信息性,反映新闻事件
- **分段长度**: 较短(1-3分钟)
| Feature | Value Proposition | Complexity | Notes |
|---------|-------------------|------------|-------|
| **Skill-Based Modular Architecture** | Each skill is independent, testable, and reusable; easier maintenance than monolithic pipelines | MEDIUM | SKILL.md specification pattern; already in architecture |
| **Multi-Modal Content Processing** | Video (ASR/OCR) + Text unified pipeline; handles diverse content types in one flow | HIGH | Whisper, FunASR, OCR already implemented |
| **Tmux-Based Session Persistence** | Long-running operations survive terminal disconnect; GSD-OpenClaw integration | MEDIUM | Unique to this system; TmuxSessionManager exists |
| **Declarative Skill Definition** | SKILL.md with YAML frontmatter defines capabilities; AI agents can discover and execute | LOW | Already in place; differentiates from code-only orchestration |
| **Unified JSON Output Format** | Consistent success/error structure across all skills; easy programmatic consumption | LOW | Already standardized |
| **Git-Integrated Archival** | Processed content versioned and archived automatically | LOW | KB_GIT_ROOT integration |
| **Content Type Detection** | Auto-detect video vs article vs QA; route to appropriate processing path | MEDIUM | Link parser handles this |
### Anti-Features (Commonly Requested, Often Problematic)
Features that seem good but create problems.
| Feature | Why Requested | Why Problematic | Alternative |
|---------|---------------|-----------------|-------------|
| **Real-Time/Streaming Processing** | Users want instant results | Adds significant complexity; batch processing sufficient for content analysis use case; real-time requires different architecture (WebSocket, event streams) | Keep batch processing; add optional polling/notifications |
| **Concurrent/Parallel Agent Execution** | Faster processing | Context sharing between parallel agents is complex; resource contention (API rate limits, model quotas); conflict resolution when results contradict | Sequential orchestration with clear dependencies is more reliable for this use case |
| **Web UI/Dashboard** | Visual workflow management | Significant development effort; CLI is sufficient for current use case; UI maintenance overhead | CLI-first; consider status output in structured formats for future visualization |
| **Multi-Tenant/User Management** | SaaS-like deployment | Adds auth, isolation, billing complexity; single-user tool currently | Keep single-tenant; future: separate deployment per user |
| **Plugin/Extension System** | Third-party integrations | Security concerns; API stability requirements; documentation burden | Keep monolithic but modular skill architecture; internal extensions only |
| **Workflow Visualization** | See pipeline graphically | Nice-to-have but not core value; generates maintenance burden | State file (JSON) is machine-readable; can generate viz later |
| **Automatic Retry with Backoff** | Handle transient failures | Can mask underlying issues; retry logic is complex; may cause rate limiting | Explicit error reporting; manual retry with checkpoint resume |
---
## v1.0-1.2 Feature Dependencies
```
[URL Input]
└──requires──> [Link Parser Skill]
[Link Parser]
└──enables──> [Platform-Specific Metadata Extraction]
[Metadata Extraction]
└──enables──> [Content Download]
[Content Download (Video)]
└──requires──> [ASR/Transcription]
└──requires──> [Whisper OR FunASR]
[Content Download (Video with Text)]
└──requires──> [OCR Processing]
[ASR/OCR/Text Content]
└──enables──> [Deep Analysis]
[Deep Analysis]
└──enables──> [Archive to Git]
[State Management]
└──enables──> [Checkpoint Resume]
└──enables──> [Idempotent Execution]
[Schema Validation]
└──enables──> [Error Detection]
└──enables──> [Consistent Output Format]
```
### Dependency Notes
- **Checkpoint Resume requires State Management:** Cannot resume without tracking what completed; workflow_state.json is the mechanism
- **Idempotent Execution requires State Management:** Need to know if step already completed successfully to skip re-execution
- **Schema Validation enables Error Detection:** Catch malformed outputs early before they cascade to downstream steps
- **Platform-Specific Skills require Link Parser:** Cannot extract metadata without knowing platform and content type
- **Deep Analysis requires Text Content:** ASR/OCR must complete before AI analysis can process
---
## v1.0-1.2 MVP Definition
### Launch With (v1) - COMPLETED
Minimum viable product — what's needed to validate the concept.
- [x] **Sequential Pipeline Orchestration** — Core value proposition; linear processing from URL to analysis
- [x] **Step-Level State Management (workflow_state.json)** — Enables reliability; checkpoint/resume
- [x] **Idempotent Execution** — Same URL processed twice produces same result; no duplicates
- [x] **JSON Schema Validation** — Data contracts between steps; error detection
- [x] **Structured Error Handling** — JSON error format; graceful degradation
- [x] **Bilibili/Zhihu Platform Support** — Core platforms; metadata + content extraction
- [x] **AI Deep Analysis** — Product's value output
### Add After Validation (v1.x)
Features to add once core is working reliably.
- [ ] **Enhanced Error Recovery Strategies** — 403/timeout specific handling; fallback options
- [ ] **OCR for Video Text Extraction** — Currently implemented but integration may need refinement
- [ ] **Speaker Diarization** — Already implemented; may need pipeline integration
- [ ] **Multiple ASR Backend Support** — Whisper + FunASR selection logic
- [ ] **Better Progress Visibility** — Step completion logging/indicators
### Future Consideration (v2+)
Features to defer until product-market fit is established.
- [ ] **Additional Platforms (YouTube, etc.)** — Out of scope per PROJECT.md currently
- [ ] **Concurrent Processing for Independent Steps** — Only if latency becomes critical issue
- [ ] **Web UI for Monitoring** — Consider only if non-technical users need access
- [ ] **API/Webhook Notifications** — If integration with external systems needed
---
## v1.0-1.2 Feature Prioritization Matrix
| Feature | User Value | Implementation Cost | Priority |
|---------|------------|---------------------|----------|
| Step-Level State Management | HIGH | MEDIUM | P1 |
| Idempotent Execution | HIGH | MEDIUM | P1 |
| JSON Schema Validation | HIGH | LOW | P1 |
| Sequential Pipeline Model | HIGH | LOW | P1 |
| Structured Error Handling | HIGH | LOW | P1 |
| Clear Step Boundaries/Docs | MEDIUM | LOW | P1 |
| Checkpoint/Resume | HIGH | MEDIUM | P2 |
| Platform-Specific Skills | HIGH | MEDIUM | P1 (exists) |
| AI Deep Analysis | HIGH | HIGH | P1 (exists) |
| ASR/OCR Integration | MEDIUM | HIGH | P2 (exists) |
| Error Recovery Strategies | MEDIUM | MEDIUM | P2 |
| Progress Visibility | LOW | LOW | P3 |
| Concurrent Execution | LOW | HIGH | P3 |
| Web UI | LOW | HIGH | P3 |
**Priority key:**
- P1: Must have for launch (reliability core)
- P2: Should have, add when possible (robustness)
- P3: Nice to have, future consideration (convenience)
---
## Competitor Feature Analysis
| Feature | Apache Airflow | Dagster | Prefect | Our Approach |
|---------|----------------|---------|---------|--------------|
| **Orchestration Model** | DAG-based | Asset-based | Flow-based | Sequential pipeline (simpler) |
| **State Management** | Database-backed | Built-in | Built-in | File-based (workflow_state.json) |
| **Checkpoint/Resume** | Task-level | Asset-level | Result caching | Step-level |
| **Idempotency** | Manual | Built-in checks | Decorator-based | Explicit state check |
| **Schema Validation** | Optional | Built-in | Optional | JSON Schema (planned) |
| **Observability** | UI Dashboard | UI + Lineage | UI + Logging | JSON output + console |
| **Learning Curve** | High | Medium | Low | Low (SKILL.md specs) |
**Key differentiation:** Our skill-based architecture with SKILL.md specifications makes the system more accessible to AI agents and easier to extend than traditional workflow engines. The focus on content processing (not general ETL) allows specialized optimizations.
---
## Industry Standard Features (2025-2026)
Based on research, modern data pipeline and workflow orchestration systems have these table stakes:
1. **Exactly-Once Processing (E1P)** — Data integrity guarantee; each event processed exactly once
2. **Fault-Tolerant Architecture** — Automatic failover and recovery
3. **Data Lineage/Tracking** — Know where data came from and how it was transformed
4. **Monitoring & Logging** — Real-time visibility into pipeline execution
5. **Scalable Architecture** — Handle varying loads
6. **Self-Service Management** — Easy pipeline creation and modification
7. **Multi-Format Support** — Structured, semi-structured, unstructured data
**Our coverage:**
- E1P: Achieved via idempotent execution
- Fault-Tolerance: Checkpoint/resume covers this
- Lineage: State tracking provides basic lineage
- Monitoring: JSON output + console logging
- Scalability: Not a concern (batch processing, single user)
- Self-Service: SKILL.md specifications enable easy understanding
- Multi-Format: Video + text + images supported
---
### Vlog类视频
- **分段依据**: 活动/场景变化
- **边界信号**: 时间跳跃、地点变化、活动切换
- **标题风格**: 描述性,反映活动内容
- **分段长度**: 变化大(1-8分钟)
## Sources
### v1.3 Anti-Crawler Research (MEDIUM confidence)
- [Bright Data - IP Rotation Best Practices](https://brightdata.com/faqs/ip-rotation/best-practices)
- [ScrapFly - Bypass Anti-Bot Protection](https://scrapfly.io/blog/posts/how-to-bypass-anti-bot-protection-when-web-scraping)
- [ScrapeHero - TLS Fingerprint Bypass 2026](https://www.scrapehero.com/tls-fingerprint-bypass-techniques/)
- [CapMonster - Stop Getting CAPTCHA 2025](https://capmonster.cloud/en/blog/how-to-stop-getting-captcha-when-scraping-in-2025-with-real-solutions)
- [OneUptime - Retry Logic with Exponential Backoff in Python](https://oneuptime.com/blog/post/2025-01-06-python-retry-exponential-backoff/view)
- [AWS Prescriptive Guidance - Retry with Backoff Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html)
- [知乎专栏 - B站 WBI 签名分析](https://zhuanlan.zhihu.com/p/1961014749807513938)
- [CSDN - B站视频爬取避坑指南 2024](https://blog.csdn.net/weixin_29051149/article/details/158076702)
- [Zyte - Rotating Proxies in Python](https://www.zyte.com/blog/developers-guide-to-rotating-proxies-in-python/)
- [Tenacity Library Documentation](https://github.com/jd/tenacity) - 当前项目已使用的重试库
**HIGH Confidence:**
- [LLM-Enhanced Semantic Text Segmentation - MDPI Applied Sciences 2025](https://www.mdpi.com/2076-3417/15/19/10849)
- [Beyond Transcripts: A Renewed Perspective on Audio Chaptering - arXiv 2025](https://arxiv.org/html/2602.08979v1)
- [Multimodal Fusion and Coherence Modeling for Video Topic Segmentation - ACL Findings 2025](https://aclanthology.org/2025.findings-acl.904.pdf)
- [LoRA Fine-Tuning for Table-of-Contents Generation - Interspeech 2025](https://www.isca-archive.org/interspeech_2025/freisinger25_interspeech.pdf)
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs - CVPR 2025](https://cvpr.thecvf.com/virtual/2025/poster/33157)
### v1.0-1.2 Baseline Research (HIGH confidence)
- [Striim: 7 Must-Have Features of Modern Data Pipelines](https://www.striim.com/blog/what-is-a-data-pipeline-and-must-have-features-of-modern-data-pipelines/)
- [Dagster: Data Pipeline Frameworks Key Features 2025](https://dagster.io/guides/data-pipeline-frameworks-key-features-10-tools-to-know-in-2025)
- [Microsoft Azure: AI Agent Orchestration Patterns](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns)
- [Medium: Idempotency Guide for Data Pipelines](https://medium.com/@QuarkAndCode/idempotency-guide-iac-apis-data-pipelines-temporal-workflows-8d85dc280613) - MEDIUM confidence
- [Fivetran: Data Pipeline State Management](https://www.fivetran.com/blog/data-pipeline-state-management-an-underappreciated-challenge) - MEDIUM confidence
- PROJECT.md and ARCHITECTURE.md from codebase analysis - HIGH confidence
**MEDIUM Confidence:**
- [SEGLLM: Topic-Oriented Call Segmentation Via LLM-Based - IEEE](https://ieeexplore.ieee.org/document/10446156/)
- [BoundRL: Efficient Structured Text Segmentation - arXiv 2025](https://arxiv.org/abs/2510.20151)
- [A boundary enhanced multi-task neural attention approach - Nature 2025](https://www.nature.com/articles/s41598-025-25317-5)
- [Parsing Through Boundaries in Chinese Word Segmentation - arXiv 2025](https://arxiv.org/html/2503.23091v1)
- [RAG 2.0: Advanced Chunking Strategies with Examples - Medium](https://medium.com/@visrow/rag-2-0-advanced-chunking-strategies-with-examples-d87d03adf6d1)
- [Best Chunking Strategies for RAG (and LLMs) in 2026 - Firecrawl](https://www.firecrawl.dev/blog/best-chunking-strategies-rag)
---
*Feature research for: Skill-based Workflow Orchestration (Content Processing)*
*Researched: 2026-03-13*
*Updated for v1.3 Anti-Crawler: 2026-03-18*
**LOW Confidence (WebSearch only):**
- Video content type classification methods (需要验证具体实现)
- 说话人识别集成方法 (需要额外调研具体技术方案)