12 KiB
12 KiB
Feature Research
Domain: Skill-based Workflow Orchestration (Content Processing Pipelines) Researched: 2026-03-13 Confidence: HIGH
Feature Landscape
Table Stakes (Users Expect These)
Features users assume exist. Missing these = product feels incomplete.
| Feature | Why Expected | Complexity | Notes |
|---|---|---|---|
| Pipeline/Sequential Orchestration | Linear processing is the fundamental model - users expect steps to execute in order | LOW | Already in PROJECT.md as core requirement |
| Idempotent Execution | Re-running same input should produce same output without duplicates | MEDIUM | Critical for reliability; prevents duplicate files on retry |
| Checkpoint/Resume from Failure | Long-running tasks can fail; users expect to continue from where they left off | MEDIUM | workflow_state.json is planned; checkpointing is industry standard |
| State Management | Every pipeline needs to track progress - what's done, what's pending | MEDIUM | Step-level state (not task-level) per PROJECT.md |
| Structured Error Handling | Failures should be captured, reported, and recoverable | LOW | JSON error format with success/error fields already exists |
| Input/Output Schema Validation | Data contracts ensure reliability; JSON Schema is standard | MEDIUM | PIPE-07 in requirements |
| Clear Step Boundaries | Users need to understand what each step does, its inputs/outputs | LOW | PIPE-06 requirement |
| Platform-Specific Content Handlers | Bilibili/Zhihu support is core functionality | MEDIUM | Already implemented for metadata/content extraction |
| AI-Powered Analysis | Deep analysis is the product's value proposition | HIGH | kb-deep-analyzer exists |
Differentiators (Competitive Advantage)
Features that set the product apart. Not required, but valuable.
| Feature | Value Proposition | Complexity | Notes |
|---|---|---|---|
| Skill-Based Modular Architecture | Each skill is independent, testable, and reusable; easier maintenance than monolithic pipelines | MEDIUM | SKILL.md specification pattern; already in architecture |
| Multi-Modal Content Processing | Video (ASR/OCR) + Text unified pipeline; handles diverse content types in one flow | HIGH | Whisper, FunASR, OCR already implemented |
| Tmux-Based Session Persistence | Long-running operations survive terminal disconnect; GSD-OpenClaw integration | MEDIUM | Unique to this system; TmuxSessionManager exists |
| Declarative Skill Definition | SKILL.md with YAML frontmatter defines capabilities; AI agents can discover and execute | LOW | Already in place; differentiates from code-only orchestration |
| Unified JSON Output Format | Consistent success/error structure across all skills; easy programmatic consumption | LOW | Already standardized |
| Git-Integrated Archival | Processed content versioned and archived automatically | LOW | KB_GIT_ROOT integration |
| Content Type Detection | Auto-detect video vs article vs QA; route to appropriate processing path | MEDIUM | Link parser handles this |
Anti-Features (Commonly Requested, Often Problematic)
Features that seem good but create problems.
| Feature | Why Requested | Why Problematic | Alternative |
|---|---|---|---|
| Real-Time/Streaming Processing | Users want instant results | Adds significant complexity; batch processing sufficient for content analysis use case; real-time requires different architecture (WebSocket, event streams) | Keep batch processing; add optional polling/notifications |
| Concurrent/Parallel Agent Execution | Faster processing | Context sharing between parallel agents is complex; resource contention (API rate limits, model quotas); conflict resolution when results contradict | Sequential orchestration with clear dependencies is more reliable for this use case |
| Web UI/Dashboard | Visual workflow management | Significant development effort; CLI is sufficient for current use case; UI maintenance overhead | CLI-first; consider status output in structured formats for future visualization |
| Multi-Tenant/User Management | SaaS-like deployment | Adds auth, isolation, billing complexity; single-user tool currently | Keep single-tenant; future: separate deployment per user |
| Plugin/Extension System | Third-party integrations | Security concerns; API stability requirements; documentation burden | Keep monolithic but modular skill architecture; internal extensions only |
| Workflow Visualization | See pipeline graphically | Nice-to-have but not core value; generates maintenance burden | State file (JSON) is machine-readable; can generate viz later |
| Automatic Retry with Backoff | Handle transient failures | Can mask underlying issues; retry logic is complex; may cause rate limiting | Explicit error reporting; manual retry with checkpoint resume |
| Complex Routing/Handoff Patterns | Dynamic task delegation | Overkill for linear content processing; adds nondeterminism; harder to debug | Sequential orchestration with explicit steps |
Feature Dependencies
[URL Input]
└──requires──> [Link Parser Skill]
[Link Parser]
└──enables──> [Platform-Specific Metadata Extraction]
[Metadata Extraction]
└──enables──> [Content Download]
[Content Download (Video)]
└──requires──> [ASR/Transcription]
└──requires──> [Whisper OR FunASR]
[Content Download (Video with Text)]
└──requires──> [OCR Processing]
[ASR/OCR/Text Content]
└──enables──> [Deep Analysis]
[Deep Analysis]
└──enables──> [Archive to Git]
[State Management]
└──enables──> [Checkpoint Resume]
└──enables──> [Idempotent Execution]
[Schema Validation]
└──enables──> [Error Detection]
└──enables──> [Consistent Output Format]
Dependency Notes
- Checkpoint Resume requires State Management: Cannot resume without tracking what completed; workflow_state.json is the mechanism
- Idempotent Execution requires State Management: Need to know if step already completed successfully to skip re-execution
- Schema Validation enables Error Detection: Catch malformed outputs early before they cascade to downstream steps
- Platform-Specific Skills require Link Parser: Cannot extract metadata without knowing platform and content type
- Deep Analysis requires Text Content: ASR/OCR must complete before AI analysis can process
MVP Definition
Launch With (v1)
Minimum viable product — what's needed to validate the concept.
- Sequential Pipeline Orchestration — Core value proposition; linear processing from URL to analysis
- Step-Level State Management (workflow_state.json) — Enables reliability; checkpoint/resume
- Idempotent Execution — Same URL processed twice produces same result; no duplicates
- JSON Schema Validation — Data contracts between steps; error detection
- Structured Error Handling — JSON error format; graceful degradation
- Bilibili/Zhihu Platform Support — Core platforms; metadata + content extraction
- AI Deep Analysis — Product's value output
Add After Validation (v1.x)
Features to add once core is working reliably.
- Enhanced Error Recovery Strategies — 403/timeout specific handling; fallback options
- OCR for Video Text Extraction — Currently implemented but integration may need refinement
- Speaker Diarization — Already implemented; may need pipeline integration
- Multiple ASR Backend Support — Whisper + FunASR selection logic
- Better Progress Visibility — Step completion logging/indicators
Future Consideration (v2+)
Features to defer until product-market fit is established.
- Additional Platforms (YouTube, etc.) — Out of scope per PROJECT.md currently
- Concurrent Processing for Independent Steps — Only if latency becomes critical issue
- Web UI for Monitoring — Consider only if non-technical users need access
- API/Webhook Notifications — If integration with external systems needed
Feature Prioritization Matrix
| Feature | User Value | Implementation Cost | Priority |
|---|---|---|---|
| Step-Level State Management | HIGH | MEDIUM | P1 |
| Idempotent Execution | HIGH | MEDIUM | P1 |
| JSON Schema Validation | HIGH | LOW | P1 |
| Sequential Pipeline Model | HIGH | LOW | P1 |
| Structured Error Handling | HIGH | LOW | P1 |
| Clear Step Boundaries/Docs | MEDIUM | LOW | P1 |
| Checkpoint/Resume | HIGH | MEDIUM | P2 |
| Platform-Specific Skills | HIGH | MEDIUM | P1 (exists) |
| AI Deep Analysis | HIGH | HIGH | P1 (exists) |
| ASR/OCR Integration | MEDIUM | HIGH | P2 (exists) |
| Error Recovery Strategies | MEDIUM | MEDIUM | P2 |
| Progress Visibility | LOW | LOW | P3 |
| Concurrent Execution | LOW | HIGH | P3 |
| Web UI | LOW | HIGH | P3 |
Priority key:
- P1: Must have for launch (reliability core)
- P2: Should have, add when possible (robustness)
- P3: Nice to have, future consideration (convenience)
Competitor Feature Analysis
| Feature | Apache Airflow | Dagster | Prefect | Our Approach |
|---|---|---|---|---|
| Orchestration Model | DAG-based | Asset-based | Flow-based | Sequential pipeline (simpler) |
| State Management | Database-backed | Built-in | Built-in | File-based (workflow_state.json) |
| Checkpoint/Resume | Task-level | Asset-level | Result caching | Step-level |
| Idempotency | Manual | Built-in checks | Decorator-based | Explicit state check |
| Schema Validation | Optional | Built-in | Optional | JSON Schema (planned) |
| Observability | UI Dashboard | UI + Lineage | UI + Logging | JSON output + console |
| Learning Curve | High | Medium | Low | Low (SKILL.md specs) |
Key differentiation: Our skill-based architecture with SKILL.md specifications makes the system more accessible to AI agents and easier to extend than traditional workflow engines. The focus on content processing (not general ETL) allows specialized optimizations.
Industry Standard Features (2025-2026)
Based on research, modern data pipeline and workflow orchestration systems have these table stakes:
- Exactly-Once Processing (E1P) — Data integrity guarantee; each event processed exactly once
- Fault-Tolerant Architecture — Automatic failover and recovery
- Data Lineage/Tracking — Know where data came from and how it was transformed
- Monitoring & Logging — Real-time visibility into pipeline execution
- Scalable Architecture — Handle varying loads
- Self-Service Management — Easy pipeline creation and modification
- Multi-Format Support — Structured, semi-structured, unstructured data
Our coverage:
- E1P: Achieved via idempotent execution
- Fault-Tolerance: Checkpoint/resume covers this
- Lineage: State tracking provides basic lineage
- Monitoring: JSON output + console logging
- Scalability: Not a concern (batch processing, single user)
- Self-Service: SKILL.md specifications enable easy understanding
- Multi-Format: Video + text + images supported
Sources
- Striim: 7 Must-Have Features of Modern Data Pipelines - HIGH confidence
- Dagster: Data Pipeline Frameworks Key Features 2025 - HIGH confidence
- Microsoft Azure: AI Agent Orchestration Patterns - HIGH confidence
- Medium: Idempotency Guide for Data Pipelines - MEDIUM confidence
- Fivetran: Data Pipeline State Management - MEDIUM confidence
- PROJECT.md and ARCHITECTURE.md from codebase analysis - HIGH confidence
Feature research for: Skill-based Workflow Orchestration (Content Processing) Researched: 2026-03-13