# Technology Stack - LLM Semantic Segmentation for SRT Subtitles

**Project:** kb-skills v1.6 - Semantic Segmentation Improvement
**Researched:** 2026-03-21
**Focus:** Stack additions for LLM-based semantic segmentation of B站 SRT subtitles

## Executive Summary

**无需新的外部依赖** - 基于现有SimpleAIClient扩展，使用Pydantic v2进行结构化输出验证。

核心架构：
- **Prompt Engineering**: 两阶段分析（视频类型分类 → 主题分段）
- **Response Parsing**: Pydantic v2 models with fallback
- **Integration**: 扩展现有SemanticSegmenter，保持向后兼容

---

## Required Libraries (None New)

### Existing Stack (Already Available)
| Library | Version | Purpose | Why |
|---------|---------|---------|-----|
| Pydantic | v2 | Schema validation | Already in use, perfect for LLM output validation |
| SimpleAIClient | Custom | LLM calls | Existing integration, supports zhipu/openai/minimax |
| Python | 3.12+ | Type hints & modern features | Match existing requirement |

### NO New Dependencies Required
All functionality can be built using existing stack. The Pydantic v2 already available provides robust JSON parsing and validation capabilities needed for LLM response handling.

---

## LLM Prompt Engineering Approach

### Two-Stage Analysis Strategy

Based on research from [Chapter-Llama (CVPR 2025)](https://arxiv.org/html/2504.00072v1) and [Automate Video Chaptering with LLMs](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/), a two-stage approach provides better accuracy than single-shot analysis:

#### Stage 1: Video Type Detection
**Purpose:** Classify video type to apply appropriate segmentation strategy

**Prompt Template:**
```python
VIDEO_TYPE_PROMPT = """你是一个视频内容分析专家。请根据以下字幕内容，判断视频类型。

【字幕内容】（前5000字符）
{text_preview}

【视频信息】
标题：{title}
时长：{duration_str}

【任务】判断视频类型，从以下类别中选择最匹配的一个：
- tutorial: 教程类（步骤清晰、教学导向）
- knowledge: 知识类（讲解概念、理论分析）
- story: 故事类（叙事、情节发展）
- interview: 访谈类（对话、问答）
- commentary: 评论类（观点表达、分析讨论）
- mixed: 混合类型（包含多种元素）

【输出格式】只输出类型代码（如 tutorial），不要解释。"""
```

**Response Format:** Plain text type code (single word)

**Why Simple Output:**
- Reduces parsing complexity
- Lower error rate
- Faster processing
- Can be cached for similar content

#### Stage 2: Semantic Segmentation
**Purpose:** Identify topic boundaries and generate chapter titles

**Prompt Template:**
```python
SEMANTIC_SEGMENTATION_PROMPT = """你是一个视频内容结构化专家。请将以下字幕按语义主题分段。

【字幕内容】
{full_text}

【视频类型】：{video_type}
【视频信息】
标题：{title}
总时长：{duration_str}

【任务】
1. 分析字幕内容，识别主题变化边界
2. 将内容分为{target_segments}个段落
3. 为每个段落生成简洁标题（不超过15字）

【分段原则】
- 保持论点完整性，不要在句子中间截断
- 在话题转换处分段
- 考虑{video_type}的特点：
  * tutorial: 按步骤/章节分段
  * knowledge: 按概念/论点分段
  * story: 按情节发展分段
  * interview: 按话题/问答分段
  * commentary: 按观点段落分段
  * mixed: 综合以上策略

【输出格式】必须严格遵循以下JSON格式：
{{
  "segments": [
    {{
      "title": "段落标题",
      "start_index": 0,
      "end_index": 150,
      "summary": "段落主题摘要（30字以内）"
    }}
  ],
  "overall_summary": "整体内容摘要（100字以内）"
}}

注意：
- start_index 和 end_index 是字符在原文中的位置（从0开始）
- 确保segments覆盖全部内容，不遗漏不重复
- 只输出JSON，不要包含其他内容"""
```

**Response Format:** Structured JSON with Pydantic validation

**Why JSON Output:**
- Machine-readable for precise mapping
- Supports Pydantic validation
- Enables error recovery
- Matches existing workflow patterns

---

## Response Format Specifications

### Pydantic Schemas for LLM Output

Based on best practices from [Pydantic validation guides](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/):

```python
from pydantic import BaseModel, Field, field_validator
from typing import List

class VideoSegment(BaseModel):
    """Single semantic segment"""

    title: str = Field(..., min_length=1, max_length=50, description="Segment title")
    start_index: int = Field(..., ge=0, description="Start character position in text")
    end_index: int = Field(..., ge=0, description="End character position in text")
    summary: str = Field(..., min_length=1, max_length=100, description="Segment summary")

    @field_validator('end_index')
    @classmethod
    def validate_bounds(cls, v, info):
        """Ensure end_index > start_index"""
        if 'start_index' in info.data and v <= info.data['start_index']:
            raise ValueError('end_index must be greater than start_index')
        return v

class SegmentationResult(BaseModel):
    """Complete segmentation result from LLM"""

    segments: List[VideoSegment] = Field(..., min_length=1, description="List of segments")
    overall_summary: str = Field(..., min_length=1, max_length=200, description="Overall summary")

    @field_validator('segments')
    @classmethod
    def validate_segments(cls, v):
        """Ensure segments are sorted and non-overlapping"""
        if not v:
            raise ValueError('segments cannot be empty')

        # Sort by start_index
        sorted_segments = sorted(v, key=lambda x: x.start_index)

        # Check for overlaps
        for i in range(len(sorted_segments) - 1):
            if sorted_segments[i].end_index > sorted_segments[i + 1].start_index:
                raise ValueError(f'Segments {i} and {i+1} overlap')

        return sorted_segments
```

### Response Parsing Strategy

**Three-tier fallback strategy** (inspired by [reliable LLM outputs](https://www.freecodecamp.org/news/how-to-keep-llm-outputs-predictable-using-pydantic-validation/)):

```python
def parse_llm_response(response_text: str, total_length: int) -> SegmentationResult:
    """Parse LLM response with multiple fallback strategies"""

    # Strategy 1: Direct JSON parsing
    try:
        return SegmentationResult.model_validate_json(response_text)
    except Exception as e:
        logger.warning(f"Direct JSON parsing failed: {e}")

    # Strategy 2: Extract JSON from markdown code blocks
    try:
        import re
        json_match = re.search(r'```json\s*(.*?)\s*```', response_text, re.DOTALL)
        if json_match:
            return SegmentationResult.model_validate_json(json_match.group(1))
    except Exception as e:
        logger.warning(f"Markdown JSON extraction failed: {e}")

    # Strategy 3: Heuristic fallback - simple equal-length segments
    logger.warning("All parsing strategies failed, using heuristic fallback")
    return _generate_fallback_segments(total_length)
```

---

## Integration with Existing SimpleAIClient

### Architecture Pattern

Based on existing `ASRCorrector` pattern in `kb_workflow/srt/corrector.py`:

```python
from typing import Optional, Protocol
from kb_workflow.deep_analyzer import SimpleAIClient

class AIClientProtocol(Protocol):
    """AI Client Protocol (already exists)"""
    def generate(self, prompt: str, max_tokens: int = 16000, disable_reasoning: bool = None) -> str:
        ...

class SemanticSegmentationEngine:
    """LLM-powered semantic segmentation engine"""

    def __init__(
        self,
        ai_client: Optional[AIClientProtocol] = None,
        enable_fallback: bool = True
    ):
        self.ai_client = ai_client
        self.enable_fallback = enable_fallback

    def detect_video_type(self, text_preview: str, metadata: dict) -> str:
        """Stage 1: Detect video type"""
        if self.ai_client is None:
            return "mixed"  # Default fallback

        prompt = VIDEO_TYPE_PROMPT.format(
            text_preview=text_preview[:5000],
            title=metadata.get('title', ''),
            duration_str=self._format_duration(metadata.get('duration', 0))
        )

        try:
            response = self.ai_client.generate(prompt, max_tokens=100, disable_reasoning=True)
            return response.strip().lower()
        except Exception as e:
            logger.warning(f"Video type detection failed: {e}")
            return "mixed"

    def segment_semantically(
        self,
        full_text: str,
        video_type: str,
        metadata: dict,
        target_segments: int = 5
    ) -> SegmentationResult:
        """Stage 2: Perform semantic segmentation"""
        if self.ai_client is None:
            return self._generate_fallback_segments(len(full_text))

        prompt = SEMANTIC_SEGMENTATION_PROMPT.format(
            full_text=full_text,
            video_type=video_type,
            title=metadata.get('title', ''),
            duration_str=self._format_duration(metadata.get('duration', 0)),
            target_segments=target_segments
        )

        try:
            response = self.ai_client.generate(prompt, max_tokens=4000, disable_reasoning=False)
            return parse_llm_response(response, len(full_text))
        except Exception as e:
            logger.warning(f"Semantic segmentation failed: {e}")
            if self.enable_fallback:
                return self._generate_fallback_segments(len(full_text))
            raise
```

### Integration Point in SRTProcessor

Minimal changes to existing `kb_workflow/pipeline/processors/srt.py`:

```python
class SRTProcessor(BaseProcessor):
    def __init__(
        self,
        state_manager: Optional[WorkflowStateManager] = None,
        enable_correction: bool = True,
        enable_semantic_segmentation: bool = True  # NEW parameter
    ):
        super().__init__(state_manager)
        self.enable_correction = enable_correction
        self.enable_semantic_segmentation = enable_semantic_segmentation
        self.segmenter = SemanticSegmenter()
        self.corrector = None
        self.semantic_engine = None  # NEW

        # Lazy initialization
        if enable_semantic_segmentation:
            try:
                from kb_workflow.srt.semantic_engine import SemanticSegmentationEngine
                from kb_workflow.deep_analyzer import SimpleAIClient

                ai_client = SimpleAIClient()
                if ai_client.api_key:
                    self.semantic_engine = SemanticSegmentationEngine(ai_client=ai_client)
                    logger.info("Semantic segmentation engine initialized")
            except Exception as e:
                logger.warning(f"Failed to initialize semantic engine: {e}")

    async def process(self, input_data: SRTInput) -> SRTOutput:
        # ... existing parsing code ...

        # Step 2: Enhanced semantic segmentation
        if self.semantic_engine and self.enable_semantic_segmentation:
            logger.info("Using LLM-based semantic segmentation")
            paragraphs = self._semantic_segment_with_llm(segments, metadata)
        else:
            logger.info("Using rule-based segmentation")
            paragraphs = self.segmenter.segment(segments)

        # ... rest of existing pipeline ...

    def _semantic_segment_with_llm(
        self,
        segments: List[Segment],
        metadata: dict
    ) -> List[str]:
        """Use LLM for intelligent segmentation"""
        # Combine segments into full text
        full_text = ' '.join(seg.text for seg in segments)

        # Stage 1: Detect video type
        video_type = self.semantic_engine.detect_video_type(full_text, metadata)

        # Stage 2: Semantic segmentation
        result = self.semantic_engine.segment_semantically(
            full_text=full_text,
            video_type=video_type,
            metadata=metadata,
            target_segments=self._calculate_target_segments(len(full_text))
        )

        # Extract paragraph texts based on indices
        paragraphs = [
            full_text[seg.start_index:seg.end_index].strip()
            for seg in result.segments
        ]

        return paragraphs
```

---

## What NOT to Add (Scope Boundaries)

### Explicitly Out of Scope

| Feature | Why NOT to Add | Alternative |
|---------|----------------|-------------|
| **LangChain** | Over-engineering for single LLM call | Direct SimpleAIClient usage |
| **New LLM providers** | Existing 3 providers sufficient | Extend zhipu/openai/minimax |
| **Vector embeddings** | Not needed for segmentation | Plain text analysis |
| **Chunking strategies** | LLM handles full context | Single-pass analysis |
| **Streaming responses** | Batch processing is fine | Await full response |
| **Multi-modal analysis** | Only text (SRT) available | Text-only prompts |
| **Dedicated segmentation models** | LLM is more flexible | Prompt engineering |
| **Database storage** | Not needed for pipeline | In-memory processing |
| **Real-time processing** | Batch is acceptable | Async processing |
| **Custom tokenizer** | Not needed for this use case | Character-based indexing |
| **Caching layer** | Adds complexity | Simple in-memory memoization |

### Architecture Anti-Patterns to Avoid

**From [video chaptering research](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687):**

1. **❌ Single-shot segmentation**
   - Problem: Too complex for one prompt
   - Solution: Two-stage approach (type → segment)

2. **❌ Fixed segment count**
   - Problem: Ignores content structure
   - Solution: Adaptive target based on length

3. **❌ Pure time-based segmentation**
   - Problem: Breaks semantic coherence
   - Solution: Content-aware boundaries

4. **❌ JSON parsing without validation**
   - Problem: Malformed LLM output
   - Solution: Pydantic + fallback strategies

---

## Implementation Priority

### Phase 1: Core Segmentation (MVP)
1. Video type detection prompt
2. Semantic segmentation prompt
3. Pydantic response models
4. Basic integration with SRTProcessor

### Phase 2: Robustness (Post-MVP)
1. Fallback strategies
2. Error recovery
3. Logging and debugging
4. Performance optimization

### Phase 3: Enhancement (Future)
1. Caching for similar content
2. Adaptive segment count
3. User feedback integration
4. A/B testing framework

---

## Configuration Strategy

### Environment Variables (No New Variables Needed)

Reuse existing AI configuration:
```bash
# Already in use
AI_API_KEY=your_key
AI_PROVIDER=zhipu  # or openai, minimax
AI_MODEL=glm-5
AI_BASE_URL=https://open.bigmodel.cn/api/coding/paas/v4
```

### Optional Parameters (YAML Config)

Add to existing `.kb_env.yaml`:
```yaml
semantic_segmentation:
  enabled: true
  enable_fallback: true
  default_video_type: mixed
  min_segment_length: 200  # characters
  max_segment_length: 2000  # characters
  target_segment_count: 5
```

---

## Testing Strategy

### Unit Tests
- Pydantic model validation
- Response parsing strategies
- Fallback mechanisms

### Integration Tests
- End-to-end SRT processing
- Error recovery paths
- Performance benchmarks

### Manual Testing
- Real B站 videos
- Different video types
- Edge cases (very short/long content)

---

## Sources

### Prompt Engineering & Segmentation
- [Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (CVPR 2025)](https://arxiv.org/html/2504.00072v1) - HIGH confidence, academic research on LLM-based video chaptering
- [Automate Video Chaptering with LLMs and TF-IDF](https://towardsdatascience.com/automate-video-chaptering-with-llms-and-tf-idf-f6569fd4d32b/) - MEDIUM confidence, practical implementation guide
- [AI-Based Video Analyzer for Automated Chaptering and Summarization](https://medium.com/codex/ai-based-video-analyzer-for-automated-chaptering-and-summarization-2ca65f35b687) - MEDIUM confidence, real-world application
- [Azure AI Video Indexer with LLM Prompts](https://learn.microsoft.com/en-us/azure/azure-video-indexer/prompt-overview) - HIGH confidence, official documentation

### Pydantic Validation
- [The Complete Guide to Using Pydantic for Validating LLM Outputs](https://machinelearningmastery.com/the-complete-guide-to-using-pydantic-for-validating-llm-outputs/) - HIGH confidence, comprehensive guide
- [A Practical Guide on Structuring LLM Outputs with Pydantic](https://nuno-bispo.medium.com/a-practical-guide-on-structuring-llm-outputs-with-pydantic-68f9f363e26e) - HIGH confidence, practical examples
- [Building Reliable LLM Applications with Pydantic Validation](https://pub.towardsai.net/building-reliable-llm-applications-with-pydantic-validation-7241a93cebb3) - MEDIUM confidence, reliability patterns

### LLM Best Practices
- [The Ultimate Guide to Prompt Engineering in 2026](https://www.lakera.ai/blog/prompt-engineering-guide) - HIGH confidence, current best practices
- [Context Engineering for Video Understanding](https://www.twelvelabs.io/blog/context-engineering-for-video-understanding) - MEDIUM confidence, video-specific context design
- [Harnessing LLMs for Automated Video Content Analysis](https://arxiv.org/html/2406.19528v3) - HIGH confidence, academic research

### Video Analysis
- [Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025)](https://cvpr.thecvf.com/virtual/2025/poster/33157) - HIGH confidence, 2025 research
- [Long Video Summarization via Key Moment Extraction (2026)](https://aclanthology.org/2026.iwsds-1.22.pdf) - HIGH confidence, 2026 research

### Confidence Notes
- HIGH: Academic papers, official documentation, peer-reviewed sources
- MEDIUM: Industry blogs, practical tutorials, real-world implementations
- LOW: Single-source claims, unverified practices (avoided in this document)