--- phase: 14-robustness-cost plan: 04 type: execute wave: 2 depends_on: - 14-03 files_modified: - kb_workflow/srt/semantic_engine.py autonomous: true requirements: - COST-01 must_haves: truths: - "长视频字幕(>5000字符)自动分块处理" - "分块按句子边界分割,保持语义完整性" - "每个分块独立调用 LLM 分段,结果直接拼接" artifacts: - path: "kb_workflow/srt/semantic_engine.py" provides: "UnifiedSegmentationStrategy 分块处理逻辑" exports: ["_split_into_chunks", "_merge_segment_results"] key_links: - from: "_split_into_chunks()" to: "ASRCorrector._correct_long_text()" via: "复用分批模式" pattern: "按句子边界分割" - from: "segment()" to: "_split_into_chunks()" via: "长文本分块" pattern: "len(content) > 5000" --- 实现长视频字幕的分块处理,避免 LLM 上下文窗口溢出。 Purpose: 处理超过 5000 字符的长字幕,按句子边界分割后独立分段。 Output: 带分块处理的 UnifiedSegmentationStrategy.segment() 方法。 @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md @.planning/phases/14-robustness-cost/14-CONTEXT.md @.planning/phases/14-robustness-cost/14-RESEARCH.md @.planning/phases/14-robustness-cost/14-VALIDATION.md @.planning/ROADMAP.md @.planning/REQUIREMENTS.md @.planning/STATE.md @kb_workflow/srt/semantic_engine.py @kb_workflow/srt/corrector.py @.planning/phases/14-robustness-cost/14-03-PLAN.md From kb_workflow/srt/corrector.py (分批处理模式): ```python def _correct_long_text(self, text: str) -> str: """分批处理长文本 按句子分割,每批不超过 batch_size 字符。 """ # 按句子分割(简单分割,基于句号、问号、感叹号) sentences = [] current = "" for char in text: current += char if char in '。!?\n': sentences.append(current) current = "" # 添加最后一个不完整的句子 if current: sentences.append(current) # 分批处理 batches = [] current_batch = [] current_length = 0 for sentence in sentences: if current_length + len(sentence) > self.batch_size: if current_batch: batches.append(''.join(current_batch)) current_batch = [sentence] current_length = len(sentence) else: current_batch.append(sentence) current_length += len(sentence) if current_batch: batches.append(''.join(current_batch)) # 处理每个批次 corrected_batches = [] for batch in batches: corrected = self._correct_batch(batch) corrected_batches.append(corrected) return ''.join(corrected_batches) ``` 关键观察: - 按标点符号(。!?\n)分割句子 - 每批不超过 batch_size (5000) - 处理完每批后直接拼接结果 From 14-CONTEXT.md (User Decisions): ```python # 分块处理 - 分块阈值: 5000 字符(约 10-15 分钟字幕) - 分块方式: 按句子边界分割(句号、问号、感叹号等) - 结果合并: 直接拼接各分块结果 - 分块策略: 长字幕按句子边界分成多个 5000 字符以内的块 ``` From 14-RESEARCH.md (Open Questions): > **分块后的分段索引如何计算?** > - What we know: 每个分块独立分段,结果直接拼接 > - What's unclear: VideoSegment.start_index 和 end_index 是相对于原文还是分块? > - Recommendation: 相对于原文,需要在分块时记录每个分块的起始偏移量,分段后累加偏移量 决策: start_index/end_index 相对于原文,需要记录分块偏移量。 Task 1: 实现长文本分块处理 kb_workflow/srt/semantic_engine.py - 添加 _split_into_chunks() 方法,按句子边界分割长文本 - 分块阈值: 5000 字符 - 添加 _merge_segment_results() 方法,合并分块结果 - 修正 VideoSegment.start_index 和 end_index 为相对于原文的位置 - 短文本(<= 5000 字符)直接处理,不分块 在 UnifiedSegmentationStrategy 类中添加分块处理逻辑: 1. 添加 _split_into_chunks() 方法(复用 ASRCorrector 模式): ```python def _split_into_chunks(self, text: str, max_chunk_size: int = 5000) -> List[tuple[str, int]]: """按句子边界分割文本为多个块 Args: text: 待分割的文本 max_chunk_size: 每块的最大字符数 Returns: List of (chunk_text, offset) 元组,offset 是该块在原文中的起始位置 """ # 按句子分割 sentences = [] current = "" for char in text: current += char if char in '。!?\n': sentences.append(current) current = "" # 添加最后一个不完整的句子 if current: sentences.append(current) # 分块 chunks = [] current_chunk = [] current_length = 0 current_offset = 0 for sentence in sentences: sentence_len = len(sentence) if current_length + sentence_len > max_chunk_size: if current_chunk: chunk_text = ''.join(current_chunk) chunks.append((chunk_text, current_offset)) current_offset += len(chunk_text) current_chunk = [sentence] current_length = sentence_len else: current_chunk.append(sentence) current_length += sentence_len # 添加最后一个块 if current_chunk: chunk_text = ''.join(current_chunk) chunks.append((chunk_text, current_offset)) return chunks ``` 2. 添加 _merge_segment_results() 方法: ```python def _merge_segment_results( self, chunk_results: List[tuple[List[VideoSegment], int]] ) -> List[VideoSegment]: """合并分块的分段结果 Args: chunk_results: List of (segments, offset) 元组 Returns: 合并后的分段列表,修正 start_index 和 end_index 为相对于原文的位置 """ all_segments = [] for segments, offset in chunk_results: for seg in segments: # 修正索引为相对于原文的位置 corrected_seg = VideoSegment( title=seg.title, content=seg.content, start_index=seg.start_index + offset, end_index=seg.end_index + offset ) all_segments.append(corrected_seg) return all_segments ``` 3. 修改 segment() 方法,添加分块逻辑: ```python def segment(self, segments: List, metadata: dict) -> List[VideoSegment]: # AI 不可用,直接使用规则分段 if self.ai_client is None: logger.info("AI client unavailable, using fallback segmenter") paragraphs = self.fallback_segmenter.segment(segments) return [ VideoSegment(title=None, content=para, start_index=0, end_index=0) for para in paragraphs ] # 合并字幕文本 content = " ".join(seg.text for seg in segments) # 检查是否需要分块 if len(content) <= 5000: # 短文本直接处理 return self._segment_single_chunk(content, metadata, segments) else: # 长文本分块处理 return self._segment_multiple_chunks(content, metadata, segments) ``` 4. 添加 _segment_single_chunk() 和 _segment_multiple_chunks() 辅助方法: ```python def _segment_single_chunk( self, content: str, metadata: dict, fallback_segments: List ) -> List[VideoSegment]: """处理单个文本块(不分块)""" prompt = self._build_prompt(content=content, metadata=metadata) # 调用 LLM(带重试和降级) # ... (现有的 LLM 调用逻辑) def _segment_multiple_chunks( self, content: str, metadata: dict, fallback_segments: List ) -> List[VideoSegment]: """处理多个文本块(分块)""" logger.info(f"Content too long ({len(content)} chars), splitting into chunks") # 分块 chunks = self._split_into_chunks(content, max_chunk_size=5000) logger.info(f"Split into {len(chunks)} chunks") # 处理每个分块 chunk_results = [] for i, (chunk_text, offset) in enumerate(chunks, 1): logger.info(f"Processing chunk {i}/{len(chunks)} (offset={offset}, length={len(chunk_text)})") # 构建该分块的 prompt chunk_metadata = metadata.copy() chunk_metadata['chunk_info'] = f"第 {i}/{len(chunks)} 块" prompt = self._build_prompt(content=chunk_text, metadata=chunk_metadata) try: # 调用 LLM llm_output = self.ai_client.generate(prompt, max_tokens=8000, disable_reasoning=True) segments = self._parse_result(llm_output, fallback_segments=fallback_segments) chunk_results.append((segments, offset)) logger.info(f"Chunk {i} segmented successfully: {len(segments)} segments") except Exception as e: logger.warning(f"Chunk {i} failed: {e}, using fallback for this chunk") # 该分块降级到规则分段 chunk_segments = self._fallback_chunk(chunk_text, offset) chunk_results.append((chunk_segments, offset)) # 合并结果 return self._merge_segment_results(chunk_results) ``` 遵守用户决策: - 分块阈值: 5000 字符 - 按句子边界分割(。!?\n) - 直接拼接各分块结果 - start_index/end_index 相对于原文 pytest tests/test_semantic_robustness.py::test_long_text_chunking -x - 长文本(>5000字符)自动分块处理 - 按句子边界分割,保持语义完整性 - 每个分块独立调用 LLM 分段 - 分块结果直接拼接,start_index/end_index 相对于原文 - 分块失败时该分块降级到规则分段 - 短文本(<=5000字符)不分块,直接处理 ## 整体验证标准 1. **分块触发**: 超过 5000 字符的文本自动分块 2. **句子边界**: 分块在标点符号(。!?\n)后分割 3. **索引修正**: VideoSegment.start_index 和 end_index 相对于原文 4. **分块降级**: 单个分块失败时降级到规则分段,不影响其他分块 5. **日志记录**: 记录分块信息(块数、偏移量、长度) 运行完整测试套件: ```bash python3 -m pytest tests/test_semantic_robustness.py::test_long_text_chunking -v -x ``` 1. 长文本(>5000字符)自动分块处理 2. 每个分块不超过 5000 字符 3. 分块边界在句子末尾(标点符号后) 4. VideoSegment.start_index/end_index 相对于原文位置 5. 分块失败时该分块降级到规则分段 6. 所有测试通过 执行完成后创建 `.planning/phases/14-robustness-cost/14-04-SUMMARY.md`