fix/crawler-proxy-config #41

Merged
starmen1265 merged 3 commits from fix/crawler-proxy-config into main 2026-03-21 23:34:07 +08:00
Showing only changes of commit d039537f1d - Show all commits
@@ -0,0 +1,60 @@
# Semantic Segmentation for SRT Processing
**Status:** Backlog
**Priority:** Medium
**Source:** User feedback during v1.5 testing
## Problem
Current `SemanticSegmenter` uses simple heuristic rules:
- Time gap > 3 seconds
- Paragraph length > 500 characters
This approach cannot identify **semantic topic transitions**.
### Example Case
Video: BV1MiPSz2ErY (科学透镜 - 5 topics in one video)
1. 银河系中心的超大黑洞人马座A*其实不是黑洞
2. 银河系和仙女座星系等构成的本星系群陷入巨大物质薄片
3. 科学家为世界上97%的建筑生成3D模型
4. 雷暴的形成
5. 闪电的形成
Current segmentation mixes these topics together.
## Potential Solutions
### Option 1: Transition Keyword Detection (Simple)
- Detect common transition phrases: "接下来", "另外", "第二个", etc.
- **Pros:** Fast, no API calls
- **Cons:** Low accuracy, many videos lack explicit transitions
### Option 2: LLM-Based Segmentation (Accurate)
- Ask LLM to analyze entire subtitle and identify topic boundaries
- **Pros:** Highest accuracy, understands semantics
- **Cons:** Extra API cost, token limits for long videos
### Option 3: Two-Stage Approach (Balanced)
- Stage 1: Simple rule-based segmentation (current)
- Stage 2: LLM checks paragraph beginnings for new topics
- **Pros:** Most cases use simple rules, LLM only for key decisions
- **Cons:** More complex implementation
### Option 4: Embedding-Based Similarity
- Calculate semantic similarity between adjacent paragraphs
- Low similarity = topic boundary
- **Pros:** No LLM API calls, runs locally
- **Cons:** Requires embedding model, implementation complexity
## Recommendation
**Phase 1 (Quick Win):** Transition keyword detection + improve prompt
**Phase 2 (Full Solution):** Two-stage approach with optional LLM semantic segmentation
## Success Criteria
1. Topics are separated correctly
2. Each paragraph covers a single coherent topic
3. No false splits within a topic