docs: complete v1.6 semantic segmentation research
This commit is contained in:
@@ -0,0 +1,494 @@
|
||||
# Technology Stack - LLM Semantic Segmentation for SRT Subtitles
|
||||
|
||||
**Project:** kb-skills v1.6 - Semantic Segmentation Improvement
|
||||
**Researched:** 2026-03-21
|
||||
**Focus:** Stack additions for LLM-based semantic segmentation of B站 SRT subtitles
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**无需新的外部依赖** - 基于现有SimpleAIClient扩展,使用Pydantic v2进行结构化输出验证。
|
||||
|
||||
核心架构:
|
||||
- **Prompt Engineering**: 两阶段分析(视频类型分类 → 主题分段)
|
||||
- **Response Parsing**: Pydantic v2 models with fallback
|
||||
- **Integration**: 扩展现有SemanticSegmenter,保持向后兼容
|
||||
|
||||
---
|
||||
|
||||
## Required Libraries (None New)
|
||||
|
||||
### Existing Stack (Already Available)
|
||||
| Library | Version | Purpose | Why |
|
||||
|---------|---------|---------|-----|
|
||||
| Pydantic | v2 | Schema validation | Already in use, perfect for LLM output validation |
|
||||
| SimpleAIClient | Custom | LLM calls | Existing integration, supports zhipu/openai/minimax |
|
||||
| Python | 3.12+ | Type hints & modern features | Match existing requirement |
|
||||
|
||||
### NO New Dependencies Required
|
||||
All functionality can be built using existing stack. The Pydantic v2 already available provides robust JSON parsing and validation capabilities needed for LLM response handling.
|
||||
|
||||
---
|
||||
|
||||
## LLM Prompt Engineering Approach
|
||||
|
||||
### Two-Stage Analysis Strategy
|
||||
|
||||
Based on research from [Chapter-Llama (CVPR 2025)](https://arxiv.org/html/2504.00072v1) and [Automate Video Chaptering with LLMs](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/), a two-stage approach provides better accuracy than single-shot analysis:
|
||||
|
||||
#### Stage 1: Video Type Detection
|
||||
**Purpose:** Classify video type to apply appropriate segmentation strategy
|
||||
|
||||
**Prompt Template:**
|
||||
```python
|
||||
VIDEO_TYPE_PROMPT = """你是一个视频内容分析专家。请根据以下字幕内容,判断视频类型。
|
||||
|
||||
【字幕内容】(前5000字符)
|
||||
{text_preview}
|
||||
|
||||
【视频信息】
|
||||
标题:{title}
|
||||
时长:{duration_str}
|
||||
|
||||
【任务】判断视频类型,从以下类别中选择最匹配的一个:
|
||||
- tutorial: 教程类(步骤清晰、教学导向)
|
||||
- knowledge: 知识类(讲解概念、理论分析)
|
||||
- story: 故事类(叙事、情节发展)
|
||||
- interview: 访谈类(对话、问答)
|
||||
- commentary: 评论类(观点表达、分析讨论)
|
||||
- mixed: 混合类型(包含多种元素)
|
||||
|
||||
【输出格式】只输出类型代码(如 tutorial),不要解释。"""
|
||||
```
|
||||
|
||||
**Response Format:** Plain text type code (single word)
|
||||
|
||||
**Why Simple Output:**
|
||||
- Reduces parsing complexity
|
||||
- Lower error rate
|
||||
- Faster processing
|
||||
- Can be cached for similar content
|
||||
|
||||
#### Stage 2: Semantic Segmentation
|
||||
**Purpose:** Identify topic boundaries and generate chapter titles
|
||||
|
||||
**Prompt Template:**
|
||||
```python
|
||||
SEMANTIC_SEGMENTATION_PROMPT = """你是一个视频内容结构化专家。请将以下字幕按语义主题分段。
|
||||
|
||||
【字幕内容】
|
||||
{full_text}
|
||||
|
||||
【视频类型】:{video_type}
|
||||
【视频信息】
|
||||
标题:{title}
|
||||
总时长:{duration_str}
|
||||
|
||||
【任务】
|
||||
1. 分析字幕内容,识别主题变化边界
|
||||
2. 将内容分为{target_segments}个段落
|
||||
3. 为每个段落生成简洁标题(不超过15字)
|
||||
|
||||
【分段原则】
|
||||
- 保持论点完整性,不要在句子中间截断
|
||||
- 在话题转换处分段
|
||||
- 考虑{video_type}的特点:
|
||||
* tutorial: 按步骤/章节分段
|
||||
* knowledge: 按概念/论点分段
|
||||
* story: 按情节发展分段
|
||||
* interview: 按话题/问答分段
|
||||
* commentary: 按观点段落分段
|
||||
* mixed: 综合以上策略
|
||||
|
||||
【输出格式】必须严格遵循以下JSON格式:
|
||||
{{
|
||||
"segments": [
|
||||
{{
|
||||
"title": "段落标题",
|
||||
"start_index": 0,
|
||||
"end_index": 150,
|
||||
"summary": "段落主题摘要(30字以内)"
|
||||
}}
|
||||
],
|
||||
"overall_summary": "整体内容摘要(100字以内)"
|
||||
}}
|
||||
|
||||
注意:
|
||||
- start_index 和 end_index 是字符在原文中的位置(从0开始)
|
||||
- 确保segments覆盖全部内容,不遗漏不重复
|
||||
- 只输出JSON,不要包含其他内容"""
|
||||
```
|
||||
|
||||
**Response Format:** Structured JSON with Pydantic validation
|
||||
|
||||
**Why JSON Output:**
|
||||
- Machine-readable for precise mapping
|
||||
- Supports Pydantic validation
|
||||
- Enables error recovery
|
||||
- Matches existing workflow patterns
|
||||
|
||||
---
|
||||
|
||||
## Response Format Specifications
|
||||
|
||||
### Pydantic Schemas for LLM Output
|
||||
|
||||
Based on best practices from [Pydantic validation guides](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/):
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from typing import List
|
||||
|
||||
class VideoSegment(BaseModel):
|
||||
"""Single semantic segment"""
|
||||
|
||||
title: str = Field(..., min_length=1, max_length=50, description="Segment title")
|
||||
start_index: int = Field(..., ge=0, description="Start character position in text")
|
||||
end_index: int = Field(..., ge=0, description="End character position in text")
|
||||
summary: str = Field(..., min_length=1, max_length=100, description="Segment summary")
|
||||
|
||||
@field_validator('end_index')
|
||||
@classmethod
|
||||
def validate_bounds(cls, v, info):
|
||||
"""Ensure end_index > start_index"""
|
||||
if 'start_index' in info.data and v <= info.data['start_index']:
|
||||
raise ValueError('end_index must be greater than start_index')
|
||||
return v
|
||||
|
||||
class SegmentationResult(BaseModel):
|
||||
"""Complete segmentation result from LLM"""
|
||||
|
||||
segments: List[VideoSegment] = Field(..., min_length=1, description="List of segments")
|
||||
overall_summary: str = Field(..., min_length=1, max_length=200, description="Overall summary")
|
||||
|
||||
@field_validator('segments')
|
||||
@classmethod
|
||||
def validate_segments(cls, v):
|
||||
"""Ensure segments are sorted and non-overlapping"""
|
||||
if not v:
|
||||
raise ValueError('segments cannot be empty')
|
||||
|
||||
# Sort by start_index
|
||||
sorted_segments = sorted(v, key=lambda x: x.start_index)
|
||||
|
||||
# Check for overlaps
|
||||
for i in range(len(sorted_segments) - 1):
|
||||
if sorted_segments[i].end_index > sorted_segments[i + 1].start_index:
|
||||
raise ValueError(f'Segments {i} and {i+1} overlap')
|
||||
|
||||
return sorted_segments
|
||||
```
|
||||
|
||||
### Response Parsing Strategy
|
||||
|
||||
**Three-tier fallback strategy** (inspired by [reliable LLM outputs](https://www.freecodecamp.org/news/how-to-keep-llm-outputs-predictable-using-pydantic-validation/)):
|
||||
|
||||
```python
|
||||
def parse_llm_response(response_text: str, total_length: int) -> SegmentationResult:
|
||||
"""Parse LLM response with multiple fallback strategies"""
|
||||
|
||||
# Strategy 1: Direct JSON parsing
|
||||
try:
|
||||
return SegmentationResult.model_validate_json(response_text)
|
||||
except Exception as e:
|
||||
logger.warning(f"Direct JSON parsing failed: {e}")
|
||||
|
||||
# Strategy 2: Extract JSON from markdown code blocks
|
||||
try:
|
||||
import re
|
||||
json_match = re.search(r'```json\s*(.*?)\s*```', response_text, re.DOTALL)
|
||||
if json_match:
|
||||
return SegmentationResult.model_validate_json(json_match.group(1))
|
||||
except Exception as e:
|
||||
logger.warning(f"Markdown JSON extraction failed: {e}")
|
||||
|
||||
# Strategy 3: Heuristic fallback - simple equal-length segments
|
||||
logger.warning("All parsing strategies failed, using heuristic fallback")
|
||||
return _generate_fallback_segments(total_length)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with Existing SimpleAIClient
|
||||
|
||||
### Architecture Pattern
|
||||
|
||||
Based on existing `ASRCorrector` pattern in `kb_workflow/srt/corrector.py`:
|
||||
|
||||
```python
|
||||
from typing import Optional, Protocol
|
||||
from kb_workflow.deep_analyzer import SimpleAIClient
|
||||
|
||||
class AIClientProtocol(Protocol):
|
||||
"""AI Client Protocol (already exists)"""
|
||||
def generate(self, prompt: str, max_tokens: int = 16000, disable_reasoning: bool = None) -> str:
|
||||
...
|
||||
|
||||
class SemanticSegmentationEngine:
|
||||
"""LLM-powered semantic segmentation engine"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ai_client: Optional[AIClientProtocol] = None,
|
||||
enable_fallback: bool = True
|
||||
):
|
||||
self.ai_client = ai_client
|
||||
self.enable_fallback = enable_fallback
|
||||
|
||||
def detect_video_type(self, text_preview: str, metadata: dict) -> str:
|
||||
"""Stage 1: Detect video type"""
|
||||
if self.ai_client is None:
|
||||
return "mixed" # Default fallback
|
||||
|
||||
prompt = VIDEO_TYPE_PROMPT.format(
|
||||
text_preview=text_preview[:5000],
|
||||
title=metadata.get('title', ''),
|
||||
duration_str=self._format_duration(metadata.get('duration', 0))
|
||||
)
|
||||
|
||||
try:
|
||||
response = self.ai_client.generate(prompt, max_tokens=100, disable_reasoning=True)
|
||||
return response.strip().lower()
|
||||
except Exception as e:
|
||||
logger.warning(f"Video type detection failed: {e}")
|
||||
return "mixed"
|
||||
|
||||
def segment_semantically(
|
||||
self,
|
||||
full_text: str,
|
||||
video_type: str,
|
||||
metadata: dict,
|
||||
target_segments: int = 5
|
||||
) -> SegmentationResult:
|
||||
"""Stage 2: Perform semantic segmentation"""
|
||||
if self.ai_client is None:
|
||||
return self._generate_fallback_segments(len(full_text))
|
||||
|
||||
prompt = SEMANTIC_SEGMENTATION_PROMPT.format(
|
||||
full_text=full_text,
|
||||
video_type=video_type,
|
||||
title=metadata.get('title', ''),
|
||||
duration_str=self._format_duration(metadata.get('duration', 0)),
|
||||
target_segments=target_segments
|
||||
)
|
||||
|
||||
try:
|
||||
response = self.ai_client.generate(prompt, max_tokens=4000, disable_reasoning=False)
|
||||
return parse_llm_response(response, len(full_text))
|
||||
except Exception as e:
|
||||
logger.warning(f"Semantic segmentation failed: {e}")
|
||||
if self.enable_fallback:
|
||||
return self._generate_fallback_segments(len(full_text))
|
||||
raise
|
||||
```
|
||||
|
||||
### Integration Point in SRTProcessor
|
||||
|
||||
Minimal changes to existing `kb_workflow/pipeline/processors/srt.py`:
|
||||
|
||||
```python
|
||||
class SRTProcessor(BaseProcessor):
|
||||
def __init__(
|
||||
self,
|
||||
state_manager: Optional[WorkflowStateManager] = None,
|
||||
enable_correction: bool = True,
|
||||
enable_semantic_segmentation: bool = True # NEW parameter
|
||||
):
|
||||
super().__init__(state_manager)
|
||||
self.enable_correction = enable_correction
|
||||
self.enable_semantic_segmentation = enable_semantic_segmentation
|
||||
self.segmenter = SemanticSegmenter()
|
||||
self.corrector = None
|
||||
self.semantic_engine = None # NEW
|
||||
|
||||
# Lazy initialization
|
||||
if enable_semantic_segmentation:
|
||||
try:
|
||||
from kb_workflow.srt.semantic_engine import SemanticSegmentationEngine
|
||||
from kb_workflow.deep_analyzer import SimpleAIClient
|
||||
|
||||
ai_client = SimpleAIClient()
|
||||
if ai_client.api_key:
|
||||
self.semantic_engine = SemanticSegmentationEngine(ai_client=ai_client)
|
||||
logger.info("Semantic segmentation engine initialized")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to initialize semantic engine: {e}")
|
||||
|
||||
async def process(self, input_data: SRTInput) -> SRTOutput:
|
||||
# ... existing parsing code ...
|
||||
|
||||
# Step 2: Enhanced semantic segmentation
|
||||
if self.semantic_engine and self.enable_semantic_segmentation:
|
||||
logger.info("Using LLM-based semantic segmentation")
|
||||
paragraphs = self._semantic_segment_with_llm(segments, metadata)
|
||||
else:
|
||||
logger.info("Using rule-based segmentation")
|
||||
paragraphs = self.segmenter.segment(segments)
|
||||
|
||||
# ... rest of existing pipeline ...
|
||||
|
||||
def _semantic_segment_with_llm(
|
||||
self,
|
||||
segments: List[Segment],
|
||||
metadata: dict
|
||||
) -> List[str]:
|
||||
"""Use LLM for intelligent segmentation"""
|
||||
# Combine segments into full text
|
||||
full_text = ' '.join(seg.text for seg in segments)
|
||||
|
||||
# Stage 1: Detect video type
|
||||
video_type = self.semantic_engine.detect_video_type(full_text, metadata)
|
||||
|
||||
# Stage 2: Semantic segmentation
|
||||
result = self.semantic_engine.segment_semantically(
|
||||
full_text=full_text,
|
||||
video_type=video_type,
|
||||
metadata=metadata,
|
||||
target_segments=self._calculate_target_segments(len(full_text))
|
||||
)
|
||||
|
||||
# Extract paragraph texts based on indices
|
||||
paragraphs = [
|
||||
full_text[seg.start_index:seg.end_index].strip()
|
||||
for seg in result.segments
|
||||
]
|
||||
|
||||
return paragraphs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What NOT to Add (Scope Boundaries)
|
||||
|
||||
### Explicitly Out of Scope
|
||||
|
||||
| Feature | Why NOT to Add | Alternative |
|
||||
|---------|----------------|-------------|
|
||||
| **LangChain** | Over-engineering for single LLM call | Direct SimpleAIClient usage |
|
||||
| **New LLM providers** | Existing 3 providers sufficient | Extend zhipu/openai/minimax |
|
||||
| **Vector embeddings** | Not needed for segmentation | Plain text analysis |
|
||||
| **Chunking strategies** | LLM handles full context | Single-pass analysis |
|
||||
| **Streaming responses** | Batch processing is fine | Await full response |
|
||||
| **Multi-modal analysis** | Only text (SRT) available | Text-only prompts |
|
||||
| **Dedicated segmentation models** | LLM is more flexible | Prompt engineering |
|
||||
| **Database storage** | Not needed for pipeline | In-memory processing |
|
||||
| **Real-time processing** | Batch is acceptable | Async processing |
|
||||
| **Custom tokenizer** | Not needed for this use case | Character-based indexing |
|
||||
| **Caching layer** | Adds complexity | Simple in-memory memoization |
|
||||
|
||||
### Architecture Anti-Patterns to Avoid
|
||||
|
||||
**From [video chaptering research](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687):**
|
||||
|
||||
1. **❌ Single-shot segmentation**
|
||||
- Problem: Too complex for one prompt
|
||||
- Solution: Two-stage approach (type → segment)
|
||||
|
||||
2. **❌ Fixed segment count**
|
||||
- Problem: Ignores content structure
|
||||
- Solution: Adaptive target based on length
|
||||
|
||||
3. **❌ Pure time-based segmentation**
|
||||
- Problem: Breaks semantic coherence
|
||||
- Solution: Content-aware boundaries
|
||||
|
||||
4. **❌ JSON parsing without validation**
|
||||
- Problem: Malformed LLM output
|
||||
- Solution: Pydantic + fallback strategies
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
### Phase 1: Core Segmentation (MVP)
|
||||
1. Video type detection prompt
|
||||
2. Semantic segmentation prompt
|
||||
3. Pydantic response models
|
||||
4. Basic integration with SRTProcessor
|
||||
|
||||
### Phase 2: Robustness (Post-MVP)
|
||||
1. Fallback strategies
|
||||
2. Error recovery
|
||||
3. Logging and debugging
|
||||
4. Performance optimization
|
||||
|
||||
### Phase 3: Enhancement (Future)
|
||||
1. Caching for similar content
|
||||
2. Adaptive segment count
|
||||
3. User feedback integration
|
||||
4. A/B testing framework
|
||||
|
||||
---
|
||||
|
||||
## Configuration Strategy
|
||||
|
||||
### Environment Variables (No New Variables Needed)
|
||||
|
||||
Reuse existing AI configuration:
|
||||
```bash
|
||||
# Already in use
|
||||
AI_API_KEY=your_key
|
||||
AI_PROVIDER=zhipu # or openai, minimax
|
||||
AI_MODEL=glm-5
|
||||
AI_BASE_URL=https://open.bigmodel.cn/api/coding/paas/v4
|
||||
```
|
||||
|
||||
### Optional Parameters (YAML Config)
|
||||
|
||||
Add to existing `.kb_env.yaml`:
|
||||
```yaml
|
||||
semantic_segmentation:
|
||||
enabled: true
|
||||
enable_fallback: true
|
||||
default_video_type: mixed
|
||||
min_segment_length: 200 # characters
|
||||
max_segment_length: 2000 # characters
|
||||
target_segment_count: 5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- Pydantic model validation
|
||||
- Response parsing strategies
|
||||
- Fallback mechanisms
|
||||
|
||||
### Integration Tests
|
||||
- End-to-end SRT processing
|
||||
- Error recovery paths
|
||||
- Performance benchmarks
|
||||
|
||||
### Manual Testing
|
||||
- Real B站 videos
|
||||
- Different video types
|
||||
- Edge cases (very short/long content)
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Prompt Engineering & Segmentation
|
||||
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (CVPR 2025)](https://arxiv.org/html/2504.00072v1) - HIGH confidence, academic research on LLM-based video chaptering
|
||||
- [Automate Video Chaptering with LLMs and TF-IDF](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/) - MEDIUM confidence, practical implementation guide
|
||||
- [AI-Based Video Analyzer for Automated Chaptering and Summarization](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687) - MEDIUM confidence, real-world application
|
||||
- [Azure AI Video Indexer with LLM Prompts](https://learn.microsoft.com/en-us/azure/azure-video-indexer/prompt-overview) - HIGH confidence, official documentation
|
||||
|
||||
### Pydantic Validation
|
||||
- [The Complete Guide to Using Pydantic for Validating LLM Outputs](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/) - HIGH confidence, comprehensive guide
|
||||
- [A Practical Guide on Structuring LLM Outputs with Pydantic](https://nuno-bispo.medium.com/a-practical-guide-on-structuring-llm-outputs-with-pydantic-68f9f363e26e) - HIGH confidence, practical examples
|
||||
- [Building Reliable LLM Applications with Pydantic Validation](https://pub.towardsai.net/building-reliable-llm-applications-with-pydantic-validation-7241a93cebb3) - MEDIUM confidence, reliability patterns
|
||||
|
||||
### LLM Best Practices
|
||||
- [The Ultimate Guide to Prompt Engineering in 2026](https://www.lakera.ai/blog/prompt-engineering-guide) - HIGH confidence, current best practices
|
||||
- [Context Engineering for Video Understanding](https://www.twelvelabs.io/blog/context-engineering-for-video-understanding) - MEDIUM confidence, video-specific context design
|
||||
- [Harnessing LLMs for Automated Video Content Analysis](https://arxiv.org/html/2406.19528v3) - HIGH confidence, academic research
|
||||
|
||||
### Video Analysis
|
||||
- [Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025)](https://cvpr.thecvf.com/virtual/2025/poster/33157) - HIGH confidence, 2025 research
|
||||
- [Long Video Summarization via Key Moment Extraction (2026)](https://aclanthology.org/2026.iwsds-1.22.pdf) - HIGH confidence, 2026 research
|
||||
|
||||
### Confidence Notes
|
||||
- HIGH: Academic papers, official documentation, peer-reviewed sources
|
||||
- MEDIUM: Industry blogs, practical tutorials, real-world implementations
|
||||
- LOW: Single-source claims, unverified practices (avoided in this document)
|
||||
Reference in New Issue
Block a user