--- phase: 14-robustness-cost plan: 00 type: execute wave: 0 depends_on: [] files_modified: - tests/test_semantic_robustness.py - tests/conftest.py - tests/test_srt_processor.py autonomous: true requirements: - ROBUST-01 - ROBUST-02 - ROBUST-03 - ROBUST-04 - COST-01 - COST-02 - COST-03 must_haves: truths: - "所有计划任务都有对应的自动化测试" - "测试框架配置正确,可以运行测试" - "Mock fixtures 可用,模拟 LLM 客户端行为" artifacts: - path: "tests/test_semantic_robustness.py" provides: "鲁棒性和成本控制测试套件" exports: ["test_*"] - path: "tests/conftest.py" provides: "共享测试 fixtures" exports: ["mock_ai_client", "sample_segments", "sample_metadata"] - path: "tests/test_srt_processor.py" provides: "SRTProcessor 集成测试" exports: ["test_semantic_segmentation_toggle"] key_links: - from: "test_semantic_robustness.py" to: "conftest.py" via: "pytest fixtures" pattern: "@pytest.fixture" --- 创建阶段 14 的测试基础设施,覆盖所有鲁棒性和成本控制功能。 Purpose: 确保所有计划任务都有对应的自动化测试,遵循 TDD 原则。 Output: 完整的测试套件,包括鲁棒性测试、成本控制测试、集成测试。 @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md @.planning/phases/14-robustness-cost/14-CONTEXT.md @.planning/phases/14-robustness-cost/14-RESEARCH.md @.planning/phases/14-robustness-cost/14-VALIDATION.md @.planning/ROADMAP.md @.planning/REQUIREMENTS.md @.planning/STATE.md @tests/conftest.py @tests/test_semantic_engine.py @kb_workflow/srt/semantic_engine.py From tests/conftest.py (existing fixtures): ```python import pytest from unittest.mock import Mock, MagicMock from kb_workflow.srt.parser import Segment @pytest.fixture def mock_ai_client(): """Mock AI client""" client = Mock() client.api_key = "test_key" client.generate = Mock(return_value="Test response") return client @pytest.fixture def sample_segments(): """示例字幕分段""" return [ Segment(index=1, start_time=0.0, end_time=1.0, text="第一句"), Segment(index=2, start_time=1.0, end_time=2.0, text="第二句"), ] @pytest.fixture def sample_metadata(): """示例视频元数据""" return { "title": "测试视频", "duration": 120, "category": "知识" } ``` From 14-VALIDATION.md (Wave 0 Requirements): ```python # Wave 0 Requirements - [ ] tests/test_semantic_robustness.py — 覆盖 ROBUST-01, ROBUST-02, ROBUST-03, ROBUST-04, COST-01, COST-02 - [ ] tests/test_srt_processor.py — 覆盖 COST-03(配置开关) - [ ] tests/conftest.py — 共享 fixtures(mock LLM client, 样本字幕数据) - [ ] 框架安装: 无需安装,pytest 已配置 # Mock Strategy 使用 unittest.mock.patch 模拟 SimpleAIClient.generate(),测试不同失败场景(网络错误、解析失败、超时等) ``` From 14-RESEARCH.md (Test Requirements): | Req ID | Behavior | Test Type | Automated Command | |--------|----------|-----------|-------------------| | ROBUST-01 | LLM 调用失败时自动降级 | unit | test_llm_failure_fallback | | ROBUST-02 | 解析失败时重试或降级 | unit | test_parse_failure_retry | | ROBUST-03 | AI 不可用时使用规则分段 | unit | test_ai_unavailable_fallback | | ROBUST-04 | 降级日志记录 | unit | test_fallback_logging | | COST-01 | 长文本分块处理 | unit | test_long_text_chunking | | COST-02 | 缓存命中避免重复调用 | unit | test_cache_hit | | COST-03 | 配置开关控制 | integration | test_semantic_segmentation_toggle | Task 1: 创建 test_semantic_robustness.py 测试文件 tests/test_semantic_robustness.py 创建 tests/test_semantic_robustness.py,包含所有鲁棒性和成本控制测试: 1. 导入依赖: ```python import pytest import json from unittest.mock import Mock, patch, MagicMock from pathlib import Path from requests.exceptions import ConnectionError, Timeout, HTTPError from kb_workflow.srt.semantic_engine import ( UnifiedSegmentationStrategy, SemanticSegmentationEngine, VideoSegment, SegmentationResult ) from kb_workflow.srt.cache import SemanticCache, compute_content_hash ``` 2. 创建测试类和方法(遵循 14-VALIDATION.md 的测试映射): **TestLLMFailureFallback (ROBUST-01)**: - test_llm_failure_fallback: 模拟 LLM 调用失败(网络错误),验证降级到规则分段 - test_recoverable_error_retry: 模拟可恢复错误(ConnectionError),验证重试 1 次 - test_non_recoverable_error_no_retry: 模拟不可恢复错误(HTTPError 400),验证直接降级 **TestParseFailureRetry (ROBUST-02, ROBUST-04)**: - test_parse_failure_retry: 模拟 JSON 解析失败(JSONDecodeError),验证重试后降级 - test_validation_error_no_retry: 模拟 Pydantic 验证失败(ValidationError),验证直接降级 - test_fallback_logging: 验证降级时记录 warning 级别日志,包含错误类型和原因 **TestAIAvailabilityFallback (ROBUST-03)**: - test_ai_unavailable_fallback: 模拟 AI 客户端未配置(api_key 为空),验证使用规则分段 - test_ai_client_none: 模拟 AI 客户端初始化失败,验证使用规则分段 **TestLongTextChunking (COST-01)**: - test_long_text_chunking: 使用超过 5000 字符的文本,验证分块处理 - test_chunk_sentence_boundary: 验证分块在句子边界(标点符号后)分割 - test_chunk_index_correction: 验证 VideoSegment.start_index/end_index 相对于原文 **TestCacheHit (COST-02)**: - test_cache_hit: 第一次调用 LLM,第二次调用从缓存读取 - test_cache_key_hash: 验证缓存键使用 SHA256 哈希 - test_cache_miss_content_change: 内容变化时缓存未命中 - test_cache_file_format: 验证缓存文件格式(JSON + version 字段) 3. 使用 Mock 模拟 LLM 客户端: ```python @pytest.fixture def mock_ai_client_with_failure(): """模拟失败的 AI 客户端""" client = Mock() client.api_key = "test_key" client.generate = Mock(side_effect=ConnectionError("Network error")) return client @pytest.fixture def mock_ai_client_with_parse_error(): """模拟解析失败的 AI 客户端""" client = Mock() client.api_key = "test_key" # 第一次返回无效 JSON,第二次返回有效 JSON client.generate = Mock(side_effect=[ "Invalid JSON", '{"video_type": "知识类", "segments": [], "actual_segment_count": 0}' ]) return client ``` 4. 示例测试实现: ```python def test_llm_failure_fallback(sample_segments, sample_metadata): """测试 LLM 调用失败时降级到规则分段""" strategy = UnifiedSegmentationStrategy( ai_client=mock_ai_client_with_failure() ) result = strategy.segment(sample_segments, sample_metadata) # 验证降级到规则分段(返回 VideoSegment 列表) assert isinstance(result, list) assert all(isinstance(seg, VideoSegment) for seg in result) # 规则分段没有标题 assert all(seg.title is None for seg in result) def test_recoverable_error_retry(sample_segments, sample_metadata): """测试可恢复错误重试 1 次""" client = Mock() client.api_key = "test_key" # 第一次失败,第二次成功 client.generate = Mock(side_effect=[ ConnectionError("Network error"), '{"video_type": "知识类", "segments": [], "actual_segment_count": 0}' ]) strategy = UnifiedSegmentationStrategy(ai_client=client) result = strategy.segment(sample_segments, sample_metadata) # 验证重试了 2 次 assert client.generate.call_count == 2 ``` 确保所有测试都是 RED 状态(功能尚未实现)。 pytest tests/test_semantic_robustness.py -v --tb=no - test_semantic_robustness.py 文件创建,包含所有测试 - 所有测试都是 RED 状态(预期失败) - 测试覆盖所有需求(ROBUST-01 到 COST-03) - Mock fixtures 正确模拟 LLM 客户端行为 Task 2: 更新 conftest.py 添加新 fixtures tests/conftest.py 更新 tests/conftest.py,添加阶段 14 需要的新 fixtures: 1. 添加长文本样本(用于分块测试): ```python @pytest.fixture def long_text_sample(): """长文本样本(>5000 字符,用于分块测试)""" # 生成一个超过 5000 字符的文本,包含多个句子 sentences = ["这是一个测试句子。" * 10] * 15 # 约 6000 字符 return " ".join(sentences) ``` 2. 添加模拟失败的 AI 客户端 fixtures: ```python @pytest.fixture def mock_ai_client_network_error(): """模拟网络错误的 AI 客户端""" client = Mock() client.api_key = "test_key" client.generate = Mock(side_effect=ConnectionError("Network error")) return client @pytest.fixture def mock_ai_client_timeout(): """模拟超时的 AI 客户端""" client = Mock() client.api_key = "test_key" client.generate = Mock(side_effect=Timeout("Request timeout")) return client @pytest.fixture def mock_ai_client_rate_limit(): """模拟限流的 AI 客户端""" response = Mock() response.status_code = 429 client = Mock() client.api_key = "test_key" client.generate = Mock(side_effect=HTTPError(response=response)) return client @pytest.fixture def mock_ai_client_api_error(): """模拟 API 错误的 AI 客户端(不可恢复)""" response = Mock() response.status_code = 400 client = Mock() client.api_key = "test_key" client.generate = Mock(side_effect=HTTPError(response=response)) return client @pytest.fixture def mock_ai_client_parse_error(): """模拟解析失败的 AI 客户端""" client = Mock() client.api_key = "test_key" # 第一次返回无效 JSON client.generate = Mock(return_value="Invalid JSON response") return client @pytest.fixture def mock_ai_client_success(): """模拟成功的 AI 客户端""" client = Mock() client.api_key = "test_key" client.generate = Mock(return_value='''{ "video_type": "知识类", "segments": [ { "title": "测试标题", "content": "测试内容", "start_index": 0, "end_index": 10 } ], "actual_segment_count": 1 }''') return client ``` 3. 添加缓存相关的 fixtures: ```python @pytest.fixture def temp_cache_dir(tmp_path): """临时缓存目录""" cache_dir = tmp_path / "cache" / "semantic" return cache_dir @pytest.fixture def semantic_cache(temp_cache_dir): """语义分段缓存实例""" from kb_workflow.srt.cache import SemanticCache return SemanticCache(cache_dir=temp_cache_dir) ``` 4. 添加长字幕分段样本: ```python @pytest.fixture def long_subtitle_segments(): """长字幕分段样本(>5000 字符)""" segments = [] for i in range(100): segments.append(Segment( index=i + 1, start_time=i * 1.0, end_time=(i + 1) * 1.0, text=f"这是第 {i + 1} 句字幕内容。" * 5 )) return segments ``` pytest tests/conftest.py --collect-only - conftest.py 添加所有需要的 fixtures - Fixtures 包括: 模拟失败的 AI 客户端、长文本样本、缓存目录 - 所有 fixtures 可以正确导入和使用 Task 3: 在 test_srt_processor.py 中添加配置开关测试 tests/test_srt_processor.py 在 tests/test_srt_processor.py 中添加配置开关测试(COST-03): 1. 如果文件不存在,创建基础文件: ```python import pytest from unittest.mock import Mock, patch from pathlib import Path from kb_workflow.srt.processor import SRTProcessor from kb_workflow.srt.parser import SRTParser ``` 2. 添加 TestSemanticSegmentationToggle 测试类: ```python class TestSemanticSegmentationToggle: """测试语义分段配置开关 (COST-03)""" def test_semantic_segmentation_enabled(self, sample_srt_file): """测试启用语义分段""" with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client: mock_client.return_value = Mock(api_key="test_key") processor = SRTProcessor( config={}, enable_semantic_segmentation=True ) # 验证创建了 SemanticSegmentationEngine assert processor.semantic_engine is not None assert processor.segmenter is None def test_semantic_segmentation_disabled(self, sample_srt_file): """测试禁用语义分段(默认)""" processor = SRTProcessor( config={}, enable_semantic_segmentation=False ) # 验证使用规则分段器 assert processor.semantic_engine is None assert processor.segmenter is not None def test_semantic_segmentation_with_cache(self, sample_srt_file): """测试启用语义分段和缓存""" with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client: mock_client.return_value = Mock(api_key="test_key") processor = SRTProcessor( config={}, enable_semantic_segmentation=True, enable_cache=True ) # 验证创建了缓存 assert processor.semantic_engine is not None # 验证策略使用了缓存 assert processor.semantic_engine.strategy.cache is not None def test_semantic_segmentation_without_cache(self, sample_srt_file): """测试启用语义分段但禁用缓存""" with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client: mock_client.return_value = Mock(api_key="test_key") processor = SRTProcessor( config={}, enable_semantic_segmentation=True, enable_cache=False ) # 验证没有创建缓存 assert processor.semantic_engine is not None assert processor.semantic_engine.strategy.cache is None @pytest.fixture def sample_srt_file(self, tmp_path): """创建示例 SRT 文件""" srt_content = """1 00:00:00,000 --> 00:00:01,000 第一句字幕 2 00:00:01,000 --> 00:00:02,000 第二句字幕 """ srt_file = tmp_path / "test.srt" srt_file.write_text(srt_content, encoding='utf-8') return srt_file ``` 3. 确保测试是 RED 状态(功能尚未实现)。 pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle -v --tb=no - test_srt_processor.py 添加配置开关测试 - 测试覆盖启用/禁用语义分段、启用/禁用缓存 - 所有测试都是 RED 状态(预期失败) ## 整体验证标准 1. **测试文件创建**: test_semantic_robustness.py 包含所有鲁棒性和成本控制测试 2. **Fixtures 更新**: conftest.py 添加所有需要的 fixtures 3. **集成测试**: test_srt_processor.py 添加配置开关测试 4. **测试状态**: 所有测试都是 RED 状态(功能尚未实现) 运行测试验证文件结构: ```bash # 验证测试文件存在 ls tests/test_semantic_robustness.py tests/conftest.py tests/test_srt_processor.py # 验证测试可以收集 python3 -m pytest tests/test_semantic_robustness.py --collect-only python3 -m pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle --collect-only # 验证测试是 RED 状态 python3 -m pytest tests/test_semantic_robustness.py -v --tb=no ``` 1. test_semantic_robustness.py 文件创建,包含 6 个测试类 2. conftest.py 添加 10+ 个新 fixtures 3. test_srt_processor.py 添加 TestSemanticSegmentationToggle 测试类 4. 所有测试都是 RED 状态(预期失败) 5. 测试覆盖所有需求(ROBUST-01 到 COST-03) 执行完成后创建 `.planning/phases/14-robustness-cost/14-00-SUMMARY.md`