495 lines
18 KiB
Markdown
495 lines
18 KiB
Markdown
# Technology Stack - LLM Semantic Segmentation for SRT Subtitles
|
|
|
|
**Project:** kb-skills v1.6 - Semantic Segmentation Improvement
|
|
**Researched:** 2026-03-21
|
|
**Focus:** Stack additions for LLM-based semantic segmentation of B站 SRT subtitles
|
|
|
|
## Executive Summary
|
|
|
|
**无需新的外部依赖** - 基于现有SimpleAIClient扩展，使用Pydantic v2进行结构化输出验证。
|
|
|
|
核心架构：
|
|
- **Prompt Engineering**: 两阶段分析（视频类型分类 → 主题分段）
|
|
- **Response Parsing**: Pydantic v2 models with fallback
|
|
- **Integration**: 扩展现有SemanticSegmenter，保持向后兼容
|
|
|
|
---
|
|
|
|
## Required Libraries (None New)
|
|
|
|
### Existing Stack (Already Available)
|
|
| Library | Version | Purpose | Why |
|
|
|---------|---------|---------|-----|
|
|
| Pydantic | v2 | Schema validation | Already in use, perfect for LLM output validation |
|
|
| SimpleAIClient | Custom | LLM calls | Existing integration, supports zhipu/openai/minimax |
|
|
| Python | 3.12+ | Type hints & modern features | Match existing requirement |
|
|
|
|
### NO New Dependencies Required
|
|
All functionality can be built using existing stack. The Pydantic v2 already available provides robust JSON parsing and validation capabilities needed for LLM response handling.
|
|
|
|
---
|
|
|
|
## LLM Prompt Engineering Approach
|
|
|
|
### Two-Stage Analysis Strategy
|
|
|
|
Based on research from [Chapter-Llama (CVPR 2025)](https://arxiv.org/html/2504.00072v1) and [Automate Video Chaptering with LLMs](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/), a two-stage approach provides better accuracy than single-shot analysis:
|
|
|
|
#### Stage 1: Video Type Detection
|
|
**Purpose:** Classify video type to apply appropriate segmentation strategy
|
|
|
|
**Prompt Template:**
|
|
```python
|
|
VIDEO_TYPE_PROMPT = """你是一个视频内容分析专家。请根据以下字幕内容，判断视频类型。
|
|
|
|
【字幕内容】（前5000字符）
|
|
{text_preview}
|
|
|
|
【视频信息】
|
|
标题：{title}
|
|
时长：{duration_str}
|
|
|
|
【任务】判断视频类型，从以下类别中选择最匹配的一个：
|
|
- tutorial: 教程类（步骤清晰、教学导向）
|
|
- knowledge: 知识类（讲解概念、理论分析）
|
|
- story: 故事类（叙事、情节发展）
|
|
- interview: 访谈类（对话、问答）
|
|
- commentary: 评论类（观点表达、分析讨论）
|
|
- mixed: 混合类型（包含多种元素）
|
|
|
|
【输出格式】只输出类型代码（如 tutorial），不要解释。"""
|
|
```
|
|
|
|
**Response Format:** Plain text type code (single word)
|
|
|
|
**Why Simple Output:**
|
|
- Reduces parsing complexity
|
|
- Lower error rate
|
|
- Faster processing
|
|
- Can be cached for similar content
|
|
|
|
#### Stage 2: Semantic Segmentation
|
|
**Purpose:** Identify topic boundaries and generate chapter titles
|
|
|
|
**Prompt Template:**
|
|
```python
|
|
SEMANTIC_SEGMENTATION_PROMPT = """你是一个视频内容结构化专家。请将以下字幕按语义主题分段。
|
|
|
|
【字幕内容】
|
|
{full_text}
|
|
|
|
【视频类型】：{video_type}
|
|
【视频信息】
|
|
标题：{title}
|
|
总时长：{duration_str}
|
|
|
|
【任务】
|
|
1. 分析字幕内容，识别主题变化边界
|
|
2. 将内容分为{target_segments}个段落
|
|
3. 为每个段落生成简洁标题（不超过15字）
|
|
|
|
【分段原则】
|
|
- 保持论点完整性，不要在句子中间截断
|
|
- 在话题转换处分段
|
|
- 考虑{video_type}的特点：
|
|
* tutorial: 按步骤/章节分段
|
|
* knowledge: 按概念/论点分段
|
|
* story: 按情节发展分段
|
|
* interview: 按话题/问答分段
|
|
* commentary: 按观点段落分段
|
|
* mixed: 综合以上策略
|
|
|
|
【输出格式】必须严格遵循以下JSON格式：
|
|
{{
|
|
"segments": [
|
|
{{
|
|
"title": "段落标题",
|
|
"start_index": 0,
|
|
"end_index": 150,
|
|
"summary": "段落主题摘要（30字以内）"
|
|
}}
|
|
],
|
|
"overall_summary": "整体内容摘要（100字以内）"
|
|
}}
|
|
|
|
注意：
|
|
- start_index 和 end_index 是字符在原文中的位置（从0开始）
|
|
- 确保segments覆盖全部内容，不遗漏不重复
|
|
- 只输出JSON，不要包含其他内容"""
|
|
```
|
|
|
|
**Response Format:** Structured JSON with Pydantic validation
|
|
|
|
**Why JSON Output:**
|
|
- Machine-readable for precise mapping
|
|
- Supports Pydantic validation
|
|
- Enables error recovery
|
|
- Matches existing workflow patterns
|
|
|
|
---
|
|
|
|
## Response Format Specifications
|
|
|
|
### Pydantic Schemas for LLM Output
|
|
|
|
Based on best practices from [Pydantic validation guides](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/):
|
|
|
|
```python
|
|
from pydantic import BaseModel, Field, field_validator
|
|
from typing import List
|
|
|
|
class VideoSegment(BaseModel):
|
|
"""Single semantic segment"""
|
|
|
|
title: str = Field(..., min_length=1, max_length=50, description="Segment title")
|
|
start_index: int = Field(..., ge=0, description="Start character position in text")
|
|
end_index: int = Field(..., ge=0, description="End character position in text")
|
|
summary: str = Field(..., min_length=1, max_length=100, description="Segment summary")
|
|
|
|
@field_validator('end_index')
|
|
@classmethod
|
|
def validate_bounds(cls, v, info):
|
|
"""Ensure end_index > start_index"""
|
|
if 'start_index' in info.data and v <= info.data['start_index']:
|
|
raise ValueError('end_index must be greater than start_index')
|
|
return v
|
|
|
|
class SegmentationResult(BaseModel):
|
|
"""Complete segmentation result from LLM"""
|
|
|
|
segments: List[VideoSegment] = Field(..., min_length=1, description="List of segments")
|
|
overall_summary: str = Field(..., min_length=1, max_length=200, description="Overall summary")
|
|
|
|
@field_validator('segments')
|
|
@classmethod
|
|
def validate_segments(cls, v):
|
|
"""Ensure segments are sorted and non-overlapping"""
|
|
if not v:
|
|
raise ValueError('segments cannot be empty')
|
|
|
|
# Sort by start_index
|
|
sorted_segments = sorted(v, key=lambda x: x.start_index)
|
|
|
|
# Check for overlaps
|
|
for i in range(len(sorted_segments) - 1):
|
|
if sorted_segments[i].end_index > sorted_segments[i + 1].start_index:
|
|
raise ValueError(f'Segments {i} and {i+1} overlap')
|
|
|
|
return sorted_segments
|
|
```
|
|
|
|
### Response Parsing Strategy
|
|
|
|
**Three-tier fallback strategy** (inspired by [reliable LLM outputs](https://www.freecodecamp.org/news/how-to-keep-llm-outputs-predictable-using-pydantic-validation/)):
|
|
|
|
```python
|
|
def parse_llm_response(response_text: str, total_length: int) -> SegmentationResult:
|
|
"""Parse LLM response with multiple fallback strategies"""
|
|
|
|
# Strategy 1: Direct JSON parsing
|
|
try:
|
|
return SegmentationResult.model_validate_json(response_text)
|
|
except Exception as e:
|
|
logger.warning(f"Direct JSON parsing failed: {e}")
|
|
|
|
# Strategy 2: Extract JSON from markdown code blocks
|
|
try:
|
|
import re
|
|
json_match = re.search(r'```json\s*(.*?)\s*```', response_text, re.DOTALL)
|
|
if json_match:
|
|
return SegmentationResult.model_validate_json(json_match.group(1))
|
|
except Exception as e:
|
|
logger.warning(f"Markdown JSON extraction failed: {e}")
|
|
|
|
# Strategy 3: Heuristic fallback - simple equal-length segments
|
|
logger.warning("All parsing strategies failed, using heuristic fallback")
|
|
return _generate_fallback_segments(total_length)
|
|
```
|
|
|
|
---
|
|
|
|
## Integration with Existing SimpleAIClient
|
|
|
|
### Architecture Pattern
|
|
|
|
Based on existing `ASRCorrector` pattern in `kb_workflow/srt/corrector.py`:
|
|
|
|
```python
|
|
from typing import Optional, Protocol
|
|
from kb_workflow.deep_analyzer import SimpleAIClient
|
|
|
|
class AIClientProtocol(Protocol):
|
|
"""AI Client Protocol (already exists)"""
|
|
def generate(self, prompt: str, max_tokens: int = 16000, disable_reasoning: bool = None) -> str:
|
|
...
|
|
|
|
class SemanticSegmentationEngine:
|
|
"""LLM-powered semantic segmentation engine"""
|
|
|
|
def __init__(
|
|
self,
|
|
ai_client: Optional[AIClientProtocol] = None,
|
|
enable_fallback: bool = True
|
|
):
|
|
self.ai_client = ai_client
|
|
self.enable_fallback = enable_fallback
|
|
|
|
def detect_video_type(self, text_preview: str, metadata: dict) -> str:
|
|
"""Stage 1: Detect video type"""
|
|
if self.ai_client is None:
|
|
return "mixed" # Default fallback
|
|
|
|
prompt = VIDEO_TYPE_PROMPT.format(
|
|
text_preview=text_preview[:5000],
|
|
title=metadata.get('title', ''),
|
|
duration_str=self._format_duration(metadata.get('duration', 0))
|
|
)
|
|
|
|
try:
|
|
response = self.ai_client.generate(prompt, max_tokens=100, disable_reasoning=True)
|
|
return response.strip().lower()
|
|
except Exception as e:
|
|
logger.warning(f"Video type detection failed: {e}")
|
|
return "mixed"
|
|
|
|
def segment_semantically(
|
|
self,
|
|
full_text: str,
|
|
video_type: str,
|
|
metadata: dict,
|
|
target_segments: int = 5
|
|
) -> SegmentationResult:
|
|
"""Stage 2: Perform semantic segmentation"""
|
|
if self.ai_client is None:
|
|
return self._generate_fallback_segments(len(full_text))
|
|
|
|
prompt = SEMANTIC_SEGMENTATION_PROMPT.format(
|
|
full_text=full_text,
|
|
video_type=video_type,
|
|
title=metadata.get('title', ''),
|
|
duration_str=self._format_duration(metadata.get('duration', 0)),
|
|
target_segments=target_segments
|
|
)
|
|
|
|
try:
|
|
response = self.ai_client.generate(prompt, max_tokens=4000, disable_reasoning=False)
|
|
return parse_llm_response(response, len(full_text))
|
|
except Exception as e:
|
|
logger.warning(f"Semantic segmentation failed: {e}")
|
|
if self.enable_fallback:
|
|
return self._generate_fallback_segments(len(full_text))
|
|
raise
|
|
```
|
|
|
|
### Integration Point in SRTProcessor
|
|
|
|
Minimal changes to existing `kb_workflow/pipeline/processors/srt.py`:
|
|
|
|
```python
|
|
class SRTProcessor(BaseProcessor):
|
|
def __init__(
|
|
self,
|
|
state_manager: Optional[WorkflowStateManager] = None,
|
|
enable_correction: bool = True,
|
|
enable_semantic_segmentation: bool = True # NEW parameter
|
|
):
|
|
super().__init__(state_manager)
|
|
self.enable_correction = enable_correction
|
|
self.enable_semantic_segmentation = enable_semantic_segmentation
|
|
self.segmenter = SemanticSegmenter()
|
|
self.corrector = None
|
|
self.semantic_engine = None # NEW
|
|
|
|
# Lazy initialization
|
|
if enable_semantic_segmentation:
|
|
try:
|
|
from kb_workflow.srt.semantic_engine import SemanticSegmentationEngine
|
|
from kb_workflow.deep_analyzer import SimpleAIClient
|
|
|
|
ai_client = SimpleAIClient()
|
|
if ai_client.api_key:
|
|
self.semantic_engine = SemanticSegmentationEngine(ai_client=ai_client)
|
|
logger.info("Semantic segmentation engine initialized")
|
|
except Exception as e:
|
|
logger.warning(f"Failed to initialize semantic engine: {e}")
|
|
|
|
async def process(self, input_data: SRTInput) -> SRTOutput:
|
|
# ... existing parsing code ...
|
|
|
|
# Step 2: Enhanced semantic segmentation
|
|
if self.semantic_engine and self.enable_semantic_segmentation:
|
|
logger.info("Using LLM-based semantic segmentation")
|
|
paragraphs = self._semantic_segment_with_llm(segments, metadata)
|
|
else:
|
|
logger.info("Using rule-based segmentation")
|
|
paragraphs = self.segmenter.segment(segments)
|
|
|
|
# ... rest of existing pipeline ...
|
|
|
|
def _semantic_segment_with_llm(
|
|
self,
|
|
segments: List[Segment],
|
|
metadata: dict
|
|
) -> List[str]:
|
|
"""Use LLM for intelligent segmentation"""
|
|
# Combine segments into full text
|
|
full_text = ' '.join(seg.text for seg in segments)
|
|
|
|
# Stage 1: Detect video type
|
|
video_type = self.semantic_engine.detect_video_type(full_text, metadata)
|
|
|
|
# Stage 2: Semantic segmentation
|
|
result = self.semantic_engine.segment_semantically(
|
|
full_text=full_text,
|
|
video_type=video_type,
|
|
metadata=metadata,
|
|
target_segments=self._calculate_target_segments(len(full_text))
|
|
)
|
|
|
|
# Extract paragraph texts based on indices
|
|
paragraphs = [
|
|
full_text[seg.start_index:seg.end_index].strip()
|
|
for seg in result.segments
|
|
]
|
|
|
|
return paragraphs
|
|
```
|
|
|
|
---
|
|
|
|
## What NOT to Add (Scope Boundaries)
|
|
|
|
### Explicitly Out of Scope
|
|
|
|
| Feature | Why NOT to Add | Alternative |
|
|
|---------|----------------|-------------|
|
|
| **LangChain** | Over-engineering for single LLM call | Direct SimpleAIClient usage |
|
|
| **New LLM providers** | Existing 3 providers sufficient | Extend zhipu/openai/minimax |
|
|
| **Vector embeddings** | Not needed for segmentation | Plain text analysis |
|
|
| **Chunking strategies** | LLM handles full context | Single-pass analysis |
|
|
| **Streaming responses** | Batch processing is fine | Await full response |
|
|
| **Multi-modal analysis** | Only text (SRT) available | Text-only prompts |
|
|
| **Dedicated segmentation models** | LLM is more flexible | Prompt engineering |
|
|
| **Database storage** | Not needed for pipeline | In-memory processing |
|
|
| **Real-time processing** | Batch is acceptable | Async processing |
|
|
| **Custom tokenizer** | Not needed for this use case | Character-based indexing |
|
|
| **Caching layer** | Adds complexity | Simple in-memory memoization |
|
|
|
|
### Architecture Anti-Patterns to Avoid
|
|
|
|
**From [video chaptering research](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687):**
|
|
|
|
1. **❌ Single-shot segmentation**
|
|
- Problem: Too complex for one prompt
|
|
- Solution: Two-stage approach (type → segment)
|
|
|
|
2. **❌ Fixed segment count**
|
|
- Problem: Ignores content structure
|
|
- Solution: Adaptive target based on length
|
|
|
|
3. **❌ Pure time-based segmentation**
|
|
- Problem: Breaks semantic coherence
|
|
- Solution: Content-aware boundaries
|
|
|
|
4. **❌ JSON parsing without validation**
|
|
- Problem: Malformed LLM output
|
|
- Solution: Pydantic + fallback strategies
|
|
|
|
---
|
|
|
|
## Implementation Priority
|
|
|
|
### Phase 1: Core Segmentation (MVP)
|
|
1. Video type detection prompt
|
|
2. Semantic segmentation prompt
|
|
3. Pydantic response models
|
|
4. Basic integration with SRTProcessor
|
|
|
|
### Phase 2: Robustness (Post-MVP)
|
|
1. Fallback strategies
|
|
2. Error recovery
|
|
3. Logging and debugging
|
|
4. Performance optimization
|
|
|
|
### Phase 3: Enhancement (Future)
|
|
1. Caching for similar content
|
|
2. Adaptive segment count
|
|
3. User feedback integration
|
|
4. A/B testing framework
|
|
|
|
---
|
|
|
|
## Configuration Strategy
|
|
|
|
### Environment Variables (No New Variables Needed)
|
|
|
|
Reuse existing AI configuration:
|
|
```bash
|
|
# Already in use
|
|
AI_API_KEY=your_key
|
|
AI_PROVIDER=zhipu # or openai, minimax
|
|
AI_MODEL=glm-5
|
|
AI_BASE_URL=https://open.bigmodel.cn/api/coding/paas/v4
|
|
```
|
|
|
|
### Optional Parameters (YAML Config)
|
|
|
|
Add to existing `.kb_env.yaml`:
|
|
```yaml
|
|
semantic_segmentation:
|
|
enabled: true
|
|
enable_fallback: true
|
|
default_video_type: mixed
|
|
min_segment_length: 200 # characters
|
|
max_segment_length: 2000 # characters
|
|
target_segment_count: 5
|
|
```
|
|
|
|
---
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Tests
|
|
- Pydantic model validation
|
|
- Response parsing strategies
|
|
- Fallback mechanisms
|
|
|
|
### Integration Tests
|
|
- End-to-end SRT processing
|
|
- Error recovery paths
|
|
- Performance benchmarks
|
|
|
|
### Manual Testing
|
|
- Real B站 videos
|
|
- Different video types
|
|
- Edge cases (very short/long content)
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
### Prompt Engineering & Segmentation
|
|
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (CVPR 2025)](https://arxiv.org/html/2504.00072v1) - HIGH confidence, academic research on LLM-based video chaptering
|
|
- [Automate Video Chaptering with LLMs and TF-IDF](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/) - MEDIUM confidence, practical implementation guide
|
|
- [AI-Based Video Analyzer for Automated Chaptering and Summarization](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687) - MEDIUM confidence, real-world application
|
|
- [Azure AI Video Indexer with LLM Prompts](https://learn.microsoft.com/en-us/azure/azure-video-indexer/prompt-overview) - HIGH confidence, official documentation
|
|
|
|
### Pydantic Validation
|
|
- [The Complete Guide to Using Pydantic for Validating LLM Outputs](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/) - HIGH confidence, comprehensive guide
|
|
- [A Practical Guide on Structuring LLM Outputs with Pydantic](https://nuno-bispo.medium.com/a-practical-guide-on-structuring-llm-outputs-with-pydantic-68f9f363e26e) - HIGH confidence, practical examples
|
|
- [Building Reliable LLM Applications with Pydantic Validation](https://pub.towardsai.net/building-reliable-llm-applications-with-pydantic-validation-7241a93cebb3) - MEDIUM confidence, reliability patterns
|
|
|
|
### LLM Best Practices
|
|
- [The Ultimate Guide to Prompt Engineering in 2026](https://www.lakera.ai/blog/prompt-engineering-guide) - HIGH confidence, current best practices
|
|
- [Context Engineering for Video Understanding](https://www.twelvelabs.io/blog/context-engineering-for-video-understanding) - MEDIUM confidence, video-specific context design
|
|
- [Harnessing LLMs for Automated Video Content Analysis](https://arxiv.org/html/2406.19528v3) - HIGH confidence, academic research
|
|
|
|
### Video Analysis
|
|
- [Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025)](https://cvpr.thecvf.com/virtual/2025/poster/33157) - HIGH confidence, 2025 research
|
|
- [Long Video Summarization via Key Moment Extraction (2026)](https://aclanthology.org/2026.iwsds-1.22.pdf) - HIGH confidence, 2026 research
|
|
|
|
### Confidence Notes
|
|
- HIGH: Academic papers, official documentation, peer-reviewed sources
|
|
- MEDIUM: Industry blogs, practical tutorials, real-world implementations
|
|
- LOW: Single-source claims, unverified practices (avoided in this document)
|