kb-skills/.planning/research/STACK_SEMANTIC_SEGMENTATION.md

Technology Stack - LLM Semantic Segmentation for SRT Subtitles

Project: kb-skills v1.6 - Semantic Segmentation Improvement Researched: 2026-03-21 Focus: Stack additions for LLM-based semantic segmentation of B站 SRT subtitles

Executive Summary

无需新的外部依赖 - 基于现有SimpleAIClient扩展，使用Pydantic v2进行结构化输出验证。

核心架构：

• Prompt Engineering: 两阶段分析（视频类型分类 → 主题分段）
• Response Parsing: Pydantic v2 models with fallback
• Integration: 扩展现有SemanticSegmenter，保持向后兼容

---

Required Libraries (None New)

Existing Stack (Already Available)

Library	Version	Purpose	Why
Pydantic	v2	Schema validation	Already in use, perfect for LLM output validation
SimpleAIClient	Custom	LLM calls	Existing integration, supports zhipu/openai/minimax
Python	3.12+	Type hints & modern features	Match existing requirement

NO New Dependencies Required

All functionality can be built using existing stack. The Pydantic v2 already available provides robust JSON parsing and validation capabilities needed for LLM response handling.

---

LLM Prompt Engineering Approach

Two-Stage Analysis Strategy

Based on research from Chapter-Llama (CVPR 2025) and Automate Video Chaptering with LLMs, a two-stage approach provides better accuracy than single-shot analysis:

Stage 1: Video Type Detection

Purpose: Classify video type to apply appropriate segmentation strategy

Prompt Template:

VIDEO_TYPE_PROMPT = """你是一个视频内容分析专家。请根据以下字幕内容，判断视频类型。

【字幕内容】（前5000字符）
{text_preview}

【视频信息】
标题：{title}
时长：{duration_str}

【任务】判断视频类型，从以下类别中选择最匹配的一个：
- tutorial: 教程类（步骤清晰、教学导向）
- knowledge: 知识类（讲解概念、理论分析）
- story: 故事类（叙事、情节发展）
- interview: 访谈类（对话、问答）
- commentary: 评论类（观点表达、分析讨论）
- mixed: 混合类型（包含多种元素）

【输出格式】只输出类型代码（如 tutorial），不要解释。"""

Response Format: Plain text type code (single word)

Why Simple Output:

• Reduces parsing complexity
• Lower error rate
• Faster processing
• Can be cached for similar content

Stage 2: Semantic Segmentation

Purpose: Identify topic boundaries and generate chapter titles

Prompt Template:

SEMANTIC_SEGMENTATION_PROMPT = """你是一个视频内容结构化专家。请将以下字幕按语义主题分段。

【字幕内容】
{full_text}

【视频类型】：{video_type}
【视频信息】
标题：{title}
总时长：{duration_str}

【任务】
1. 分析字幕内容，识别主题变化边界
2. 将内容分为{target_segments}个段落
3. 为每个段落生成简洁标题（不超过15字）

【分段原则】
- 保持论点完整性，不要在句子中间截断
- 在话题转换处分段
- 考虑{video_type}的特点：
  * tutorial: 按步骤/章节分段
  * knowledge: 按概念/论点分段
  * story: 按情节发展分段
  * interview: 按话题/问答分段
  * commentary: 按观点段落分段
  * mixed: 综合以上策略

【输出格式】必须严格遵循以下JSON格式：
{{
  "segments": [
    {{
      "title": "段落标题",
      "start_index": 0,
      "end_index": 150,
      "summary": "段落主题摘要（30字以内）"
    }}
  ],
  "overall_summary": "整体内容摘要（100字以内）"
}}

注意：
- start_index 和 end_index 是字符在原文中的位置（从0开始）
- 确保segments覆盖全部内容，不遗漏不重复
- 只输出JSON，不要包含其他内容"""

Response Format: Structured JSON with Pydantic validation

Why JSON Output:

• Machine-readable for precise mapping
• Supports Pydantic validation
• Enables error recovery
• Matches existing workflow patterns

---

Response Format Specifications

Pydantic Schemas for LLM Output

Based on best practices from Pydantic validation guides:

from pydantic import BaseModel, Field, field_validator
from typing import List

class VideoSegment(BaseModel):
    """Single semantic segment"""

    title: str = Field(..., min_length=1, max_length=50, description="Segment title")
    start_index: int = Field(..., ge=0, description="Start character position in text")
    end_index: int = Field(..., ge=0, description="End character position in text")
    summary: str = Field(..., min_length=1, max_length=100, description="Segment summary")

    @field_validator('end_index')
    @classmethod
    def validate_bounds(cls, v, info):
        """Ensure end_index > start_index"""
        if 'start_index' in info.data and v <= info.data['start_index']:
            raise ValueError('end_index must be greater than start_index')
        return v

class SegmentationResult(BaseModel):
    """Complete segmentation result from LLM"""

    segments: List[VideoSegment] = Field(..., min_length=1, description="List of segments")
    overall_summary: str = Field(..., min_length=1, max_length=200, description="Overall summary")

    @field_validator('segments')
    @classmethod
    def validate_segments(cls, v):
        """Ensure segments are sorted and non-overlapping"""
        if not v:
            raise ValueError('segments cannot be empty')

        # Sort by start_index
        sorted_segments = sorted(v, key=lambda x: x.start_index)

        # Check for overlaps
        for i in range(len(sorted_segments) - 1):
            if sorted_segments[i].end_index > sorted_segments[i + 1].start_index:
                raise ValueError(f'Segments {i} and {i+1} overlap')

        return sorted_segments

Response Parsing Strategy

Three-tier fallback strategy (inspired by reliable LLM outputs):

def parse_llm_response(response_text: str, total_length: int) -> SegmentationResult:
    """Parse LLM response with multiple fallback strategies"""

    # Strategy 1: Direct JSON parsing
    try:
        return SegmentationResult.model_validate_json(response_text)
    except Exception as e:
        logger.warning(f"Direct JSON parsing failed: {e}")

    # Strategy 2: Extract JSON from markdown code blocks
    try:
        import re
        json_match = re.search(r'```json\s*(.*?)\s*```', response_text, re.DOTALL)
        if json_match:
            return SegmentationResult.model_validate_json(json_match.group(1))
    except Exception as e:
        logger.warning(f"Markdown JSON extraction failed: {e}")

    # Strategy 3: Heuristic fallback - simple equal-length segments
    logger.warning("All parsing strategies failed, using heuristic fallback")
    return _generate_fallback_segments(total_length)

---

Integration with Existing SimpleAIClient

Architecture Pattern

Based on existing ASRCorrector pattern in kb_workflow/srt/corrector.py:

from typing import Optional, Protocol
from kb_workflow.deep_analyzer import SimpleAIClient

class AIClientProtocol(Protocol):
    """AI Client Protocol (already exists)"""
    def generate(self, prompt: str, max_tokens: int = 16000, disable_reasoning: bool = None) -> str:
        ...

class SemanticSegmentationEngine:
    """LLM-powered semantic segmentation engine"""

    def __init__(
        self,
        ai_client: Optional[AIClientProtocol] = None,
        enable_fallback: bool = True
    ):
        self.ai_client = ai_client
        self.enable_fallback = enable_fallback

    def detect_video_type(self, text_preview: str, metadata: dict) -> str:
        """Stage 1: Detect video type"""
        if self.ai_client is None:
            return "mixed"  # Default fallback

        prompt = VIDEO_TYPE_PROMPT.format(
            text_preview=text_preview[:5000],
            title=metadata.get('title', ''),
            duration_str=self._format_duration(metadata.get('duration', 0))
        )

        try:
            response = self.ai_client.generate(prompt, max_tokens=100, disable_reasoning=True)
            return response.strip().lower()
        except Exception as e:
            logger.warning(f"Video type detection failed: {e}")
            return "mixed"

    def segment_semantically(
        self,
        full_text: str,
        video_type: str,
        metadata: dict,
        target_segments: int = 5
    ) -> SegmentationResult:
        """Stage 2: Perform semantic segmentation"""
        if self.ai_client is None:
            return self._generate_fallback_segments(len(full_text))

        prompt = SEMANTIC_SEGMENTATION_PROMPT.format(
            full_text=full_text,
            video_type=video_type,
            title=metadata.get('title', ''),
            duration_str=self._format_duration(metadata.get('duration', 0)),
            target_segments=target_segments
        )

        try:
            response = self.ai_client.generate(prompt, max_tokens=4000, disable_reasoning=False)
            return parse_llm_response(response, len(full_text))
        except Exception as e:
            logger.warning(f"Semantic segmentation failed: {e}")
            if self.enable_fallback:
                return self._generate_fallback_segments(len(full_text))
            raise

Integration Point in SRTProcessor

Minimal changes to existing kb_workflow/pipeline/processors/srt.py:

class SRTProcessor(BaseProcessor):
    def __init__(
        self,
        state_manager: Optional[WorkflowStateManager] = None,
        enable_correction: bool = True,
        enable_semantic_segmentation: bool = True  # NEW parameter
    ):
        super().__init__(state_manager)
        self.enable_correction = enable_correction
        self.enable_semantic_segmentation = enable_semantic_segmentation
        self.segmenter = SemanticSegmenter()
        self.corrector = None
        self.semantic_engine = None  # NEW

        # Lazy initialization
        if enable_semantic_segmentation:
            try:
                from kb_workflow.srt.semantic_engine import SemanticSegmentationEngine
                from kb_workflow.deep_analyzer import SimpleAIClient

                ai_client = SimpleAIClient()
                if ai_client.api_key:
                    self.semantic_engine = SemanticSegmentationEngine(ai_client=ai_client)
                    logger.info("Semantic segmentation engine initialized")
            except Exception as e:
                logger.warning(f"Failed to initialize semantic engine: {e}")

    async def process(self, input_data: SRTInput) -> SRTOutput:
        # ... existing parsing code ...

        # Step 2: Enhanced semantic segmentation
        if self.semantic_engine and self.enable_semantic_segmentation:
            logger.info("Using LLM-based semantic segmentation")
            paragraphs = self._semantic_segment_with_llm(segments, metadata)
        else:
            logger.info("Using rule-based segmentation")
            paragraphs = self.segmenter.segment(segments)

        # ... rest of existing pipeline ...

    def _semantic_segment_with_llm(
        self,
        segments: List[Segment],
        metadata: dict
    ) -> List[str]:
        """Use LLM for intelligent segmentation"""
        # Combine segments into full text
        full_text = ' '.join(seg.text for seg in segments)

        # Stage 1: Detect video type
        video_type = self.semantic_engine.detect_video_type(full_text, metadata)

        # Stage 2: Semantic segmentation
        result = self.semantic_engine.segment_semantically(
            full_text=full_text,
            video_type=video_type,
            metadata=metadata,
            target_segments=self._calculate_target_segments(len(full_text))
        )

        # Extract paragraph texts based on indices
        paragraphs = [
            full_text[seg.start_index:seg.end_index].strip()
            for seg in result.segments
        ]

        return paragraphs

---

What NOT to Add (Scope Boundaries)

Explicitly Out of Scope

Feature	Why NOT to Add	Alternative
LangChain	Over-engineering for single LLM call	Direct SimpleAIClient usage
New LLM providers	Existing 3 providers sufficient	Extend zhipu/openai/minimax
Vector embeddings	Not needed for segmentation	Plain text analysis
Chunking strategies	LLM handles full context	Single-pass analysis
Streaming responses	Batch processing is fine	Await full response
Multi-modal analysis	Only text (SRT) available	Text-only prompts
Dedicated segmentation models	LLM is more flexible	Prompt engineering
Database storage	Not needed for pipeline	In-memory processing
Real-time processing	Batch is acceptable	Async processing
Custom tokenizer	Not needed for this use case	Character-based indexing
Caching layer	Adds complexity	Simple in-memory memoization

Architecture Anti-Patterns to Avoid

From video chaptering research:

1. ❌ Single-shot segmentation

   • Problem: Too complex for one prompt
   • Solution: Two-stage approach (type → segment)

2. ❌ Fixed segment count

   • Problem: Ignores content structure
   • Solution: Adaptive target based on length

3. ❌ Pure time-based segmentation

   • Problem: Breaks semantic coherence
   • Solution: Content-aware boundaries

4. ❌ JSON parsing without validation

   • Problem: Malformed LLM output
   • Solution: Pydantic + fallback strategies

---

Implementation Priority

Phase 1: Core Segmentation (MVP)

1. Video type detection prompt
2. Semantic segmentation prompt
3. Pydantic response models
4. Basic integration with SRTProcessor

Phase 2: Robustness (Post-MVP)

1. Fallback strategies
2. Error recovery
3. Logging and debugging
4. Performance optimization

Phase 3: Enhancement (Future)

1. Caching for similar content
2. Adaptive segment count
3. User feedback integration
4. A/B testing framework

---

Configuration Strategy

Environment Variables (No New Variables Needed)

Reuse existing AI configuration:

# Already in use
AI_API_KEY=your_key
AI_PROVIDER=zhipu  # or openai, minimax
AI_MODEL=glm-5
AI_BASE_URL=https://open.bigmodel.cn/api/coding/paas/v4

Optional Parameters (YAML Config)

Add to existing .kb_env.yaml:

semantic_segmentation:
  enabled: true
  enable_fallback: true
  default_video_type: mixed
  min_segment_length: 200  # characters
  max_segment_length: 2000  # characters
  target_segment_count: 5

---

Testing Strategy

Unit Tests

• Pydantic model validation
• Response parsing strategies
• Fallback mechanisms

Integration Tests

• End-to-end SRT processing
• Error recovery paths
• Performance benchmarks

Manual Testing

• Real B站 videos
• Different video types
• Edge cases (very short/long content)

---

Sources

Prompt Engineering & Segmentation

• Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMs (CVPR 2025) - HIGH confidence, academic research on LLM-based video chaptering
• Automate Video Chaptering with LLMs and TF-IDF - MEDIUM confidence, practical implementation guide
• AI-Based Video Analyzer for Automated Chaptering and Summarization - MEDIUM confidence, real-world application
• Azure AI Video Indexer with LLM Prompts - HIGH confidence, official documentation

Pydantic Validation

• The Complete Guide to Using Pydantic for Validating LLM Outputs - HIGH confidence, comprehensive guide
• A Practical Guide on Structuring LLM Outputs with Pydantic - HIGH confidence, practical examples
• Building Reliable LLM Applications with Pydantic Validation - MEDIUM confidence, reliability patterns

LLM Best Practices

• The Ultimate Guide to Prompt Engineering in 2026 - HIGH confidence, current best practices
• Context Engineering for Video Understanding - MEDIUM confidence, video-specific context design
• Harnessing LLMs for Automated Video Content Analysis - HIGH confidence, academic research

Video Analysis

• Learning Video LLM with Streaming Speech Transcription at Scale (CVPR 2025) - HIGH confidence, 2025 research
• Long Video Summarization via Key Moment Extraction (2026) - HIGH confidence, 2026 research

Confidence Notes

• HIGH: Academic papers, official documentation, peer-reviewed sources
• MEDIUM: Industry blogs, practical tutorials, real-world implementations
• LOW: Single-source claims, unverified practices (avoided in this document)