511 lines
18 KiB
Markdown
511 lines
18 KiB
Markdown
---
|
||
phase: 14-robustness-cost
|
||
plan: 00
|
||
type: execute
|
||
wave: 0
|
||
depends_on: []
|
||
files_modified:
|
||
- tests/test_semantic_robustness.py
|
||
- tests/conftest.py
|
||
- tests/test_srt_processor.py
|
||
autonomous: true
|
||
requirements:
|
||
- ROBUST-01
|
||
- ROBUST-02
|
||
- ROBUST-03
|
||
- ROBUST-04
|
||
- COST-01
|
||
- COST-02
|
||
- COST-03
|
||
must_haves:
|
||
truths:
|
||
- "所有计划任务都有对应的自动化测试"
|
||
- "测试框架配置正确，可以运行测试"
|
||
- "Mock fixtures 可用，模拟 LLM 客户端行为"
|
||
artifacts:
|
||
- path: "tests/test_semantic_robustness.py"
|
||
provides: "鲁棒性和成本控制测试套件"
|
||
exports: ["test_*"]
|
||
- path: "tests/conftest.py"
|
||
provides: "共享测试 fixtures"
|
||
exports: ["mock_ai_client", "sample_segments", "sample_metadata"]
|
||
- path: "tests/test_srt_processor.py"
|
||
provides: "SRTProcessor 集成测试"
|
||
exports: ["test_semantic_segmentation_toggle"]
|
||
key_links:
|
||
- from: "test_semantic_robustness.py"
|
||
to: "conftest.py"
|
||
via: "pytest fixtures"
|
||
pattern: "@pytest.fixture"
|
||
---
|
||
|
||
<objective>
|
||
创建阶段 14 的测试基础设施，覆盖所有鲁棒性和成本控制功能。
|
||
|
||
Purpose: 确保所有计划任务都有对应的自动化测试，遵循 TDD 原则。
|
||
Output: 完整的测试套件，包括鲁棒性测试、成本控制测试、集成测试。
|
||
</objective>
|
||
|
||
<execution_context>
|
||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||
</execution_context>
|
||
|
||
<context>
|
||
@.planning/phases/14-robustness-cost/14-CONTEXT.md
|
||
@.planning/phases/14-robustness-cost/14-RESEARCH.md
|
||
@.planning/phases/14-robustness-cost/14-VALIDATION.md
|
||
@.planning/ROADMAP.md
|
||
@.planning/REQUIREMENTS.md
|
||
@.planning/STATE.md
|
||
|
||
@tests/conftest.py
|
||
@tests/test_semantic_engine.py
|
||
@kb_workflow/srt/semantic_engine.py
|
||
</context>
|
||
|
||
<interfaces>
|
||
<!-- Key fixtures and test patterns from existing test files -->
|
||
|
||
From tests/conftest.py (existing fixtures):
|
||
```python
|
||
import pytest
|
||
from unittest.mock import Mock, MagicMock
|
||
from kb_workflow.srt.parser import Segment
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client():
|
||
"""Mock AI client"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(return_value="Test response")
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def sample_segments():
|
||
"""示例字幕分段"""
|
||
return [
|
||
Segment(index=1, start_time=0.0, end_time=1.0, text="第一句"),
|
||
Segment(index=2, start_time=1.0, end_time=2.0, text="第二句"),
|
||
]
|
||
|
||
@pytest.fixture
|
||
def sample_metadata():
|
||
"""示例视频元数据"""
|
||
return {
|
||
"title": "测试视频",
|
||
"duration": 120,
|
||
"category": "知识"
|
||
}
|
||
```
|
||
|
||
From 14-VALIDATION.md (Wave 0 Requirements):
|
||
```python
|
||
# Wave 0 Requirements
|
||
- [ ] tests/test_semantic_robustness.py — 覆盖 ROBUST-01, ROBUST-02, ROBUST-03, ROBUST-04, COST-01, COST-02
|
||
- [ ] tests/test_srt_processor.py — 覆盖 COST-03（配置开关）
|
||
- [ ] tests/conftest.py — 共享 fixtures（mock LLM client, 样本字幕数据）
|
||
- [ ] 框架安装: 无需安装，pytest 已配置
|
||
|
||
# Mock Strategy
|
||
使用 unittest.mock.patch 模拟 SimpleAIClient.generate()，测试不同失败场景（网络错误、解析失败、超时等）
|
||
```
|
||
|
||
From 14-RESEARCH.md (Test Requirements):
|
||
| Req ID | Behavior | Test Type | Automated Command |
|
||
|--------|----------|-----------|-------------------|
|
||
| ROBUST-01 | LLM 调用失败时自动降级 | unit | test_llm_failure_fallback |
|
||
| ROBUST-02 | 解析失败时重试或降级 | unit | test_parse_failure_retry |
|
||
| ROBUST-03 | AI 不可用时使用规则分段 | unit | test_ai_unavailable_fallback |
|
||
| ROBUST-04 | 降级日志记录 | unit | test_fallback_logging |
|
||
| COST-01 | 长文本分块处理 | unit | test_long_text_chunking |
|
||
| COST-02 | 缓存命中避免重复调用 | unit | test_cache_hit |
|
||
| COST-03 | 配置开关控制 | integration | test_semantic_segmentation_toggle |
|
||
</interfaces>
|
||
|
||
<tasks>
|
||
|
||
<task type="auto">
|
||
<name>Task 1: 创建 test_semantic_robustness.py 测试文件</name>
|
||
<files>tests/test_semantic_robustness.py</files>
|
||
<action>
|
||
创建 tests/test_semantic_robustness.py，包含所有鲁棒性和成本控制测试:
|
||
|
||
1. 导入依赖:
|
||
```python
|
||
import pytest
|
||
import json
|
||
from unittest.mock import Mock, patch, MagicMock
|
||
from pathlib import Path
|
||
from requests.exceptions import ConnectionError, Timeout, HTTPError
|
||
|
||
from kb_workflow.srt.semantic_engine import (
|
||
UnifiedSegmentationStrategy,
|
||
SemanticSegmentationEngine,
|
||
VideoSegment,
|
||
SegmentationResult
|
||
)
|
||
from kb_workflow.srt.cache import SemanticCache, compute_content_hash
|
||
```
|
||
|
||
2. 创建测试类和方法（遵循 14-VALIDATION.md 的测试映射）:
|
||
|
||
**TestLLMFailureFallback (ROBUST-01)**:
|
||
- test_llm_failure_fallback: 模拟 LLM 调用失败（网络错误），验证降级到规则分段
|
||
- test_recoverable_error_retry: 模拟可恢复错误（ConnectionError），验证重试 1 次
|
||
- test_non_recoverable_error_no_retry: 模拟不可恢复错误（HTTPError 400），验证直接降级
|
||
|
||
**TestParseFailureRetry (ROBUST-02, ROBUST-04)**:
|
||
- test_parse_failure_retry: 模拟 JSON 解析失败（JSONDecodeError），验证重试后降级
|
||
- test_validation_error_no_retry: 模拟 Pydantic 验证失败（ValidationError），验证直接降级
|
||
- test_fallback_logging: 验证降级时记录 warning 级别日志，包含错误类型和原因
|
||
|
||
**TestAIAvailabilityFallback (ROBUST-03)**:
|
||
- test_ai_unavailable_fallback: 模拟 AI 客户端未配置（api_key 为空），验证使用规则分段
|
||
- test_ai_client_none: 模拟 AI 客户端初始化失败，验证使用规则分段
|
||
|
||
**TestLongTextChunking (COST-01)**:
|
||
- test_long_text_chunking: 使用超过 5000 字符的文本，验证分块处理
|
||
- test_chunk_sentence_boundary: 验证分块在句子边界（标点符号后）分割
|
||
- test_chunk_index_correction: 验证 VideoSegment.start_index/end_index 相对于原文
|
||
|
||
**TestCacheHit (COST-02)**:
|
||
- test_cache_hit: 第一次调用 LLM，第二次调用从缓存读取
|
||
- test_cache_key_hash: 验证缓存键使用 SHA256 哈希
|
||
- test_cache_miss_content_change: 内容变化时缓存未命中
|
||
- test_cache_file_format: 验证缓存文件格式（JSON + version 字段）
|
||
|
||
3. 使用 Mock 模拟 LLM 客户端:
|
||
```python
|
||
@pytest.fixture
|
||
def mock_ai_client_with_failure():
|
||
"""模拟失败的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(side_effect=ConnectionError("Network error"))
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_with_parse_error():
|
||
"""模拟解析失败的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
# 第一次返回无效 JSON，第二次返回有效 JSON
|
||
client.generate = Mock(side_effect=[
|
||
"Invalid JSON",
|
||
'{"video_type": "知识类", "segments": [], "actual_segment_count": 0}'
|
||
])
|
||
return client
|
||
```
|
||
|
||
4. 示例测试实现:
|
||
```python
|
||
def test_llm_failure_fallback(sample_segments, sample_metadata):
|
||
"""测试 LLM 调用失败时降级到规则分段"""
|
||
strategy = UnifiedSegmentationStrategy(
|
||
ai_client=mock_ai_client_with_failure()
|
||
)
|
||
|
||
result = strategy.segment(sample_segments, sample_metadata)
|
||
|
||
# 验证降级到规则分段（返回 VideoSegment 列表）
|
||
assert isinstance(result, list)
|
||
assert all(isinstance(seg, VideoSegment) for seg in result)
|
||
# 规则分段没有标题
|
||
assert all(seg.title is None for seg in result)
|
||
|
||
def test_recoverable_error_retry(sample_segments, sample_metadata):
|
||
"""测试可恢复错误重试 1 次"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
# 第一次失败，第二次成功
|
||
client.generate = Mock(side_effect=[
|
||
ConnectionError("Network error"),
|
||
'{"video_type": "知识类", "segments": [], "actual_segment_count": 0}'
|
||
])
|
||
|
||
strategy = UnifiedSegmentationStrategy(ai_client=client)
|
||
result = strategy.segment(sample_segments, sample_metadata)
|
||
|
||
# 验证重试了 2 次
|
||
assert client.generate.call_count == 2
|
||
```
|
||
|
||
确保所有测试都是 RED 状态（功能尚未实现）。
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_semantic_robustness.py -v --tb=no</automated>
|
||
</verify>
|
||
<done>
|
||
- test_semantic_robustness.py 文件创建，包含所有测试
|
||
- 所有测试都是 RED 状态（预期失败）
|
||
- 测试覆盖所有需求（ROBUST-01 到 COST-03）
|
||
- Mock fixtures 正确模拟 LLM 客户端行为
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto">
|
||
<name>Task 2: 更新 conftest.py 添加新 fixtures</name>
|
||
<files>tests/conftest.py</files>
|
||
<action>
|
||
更新 tests/conftest.py，添加阶段 14 需要的新 fixtures:
|
||
|
||
1. 添加长文本样本（用于分块测试）:
|
||
```python
|
||
@pytest.fixture
|
||
def long_text_sample():
|
||
"""长文本样本（>5000 字符，用于分块测试）"""
|
||
# 生成一个超过 5000 字符的文本，包含多个句子
|
||
sentences = ["这是一个测试句子。" * 10] * 15 # 约 6000 字符
|
||
return " ".join(sentences)
|
||
```
|
||
|
||
2. 添加模拟失败的 AI 客户端 fixtures:
|
||
```python
|
||
@pytest.fixture
|
||
def mock_ai_client_network_error():
|
||
"""模拟网络错误的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(side_effect=ConnectionError("Network error"))
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_timeout():
|
||
"""模拟超时的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(side_effect=Timeout("Request timeout"))
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_rate_limit():
|
||
"""模拟限流的 AI 客户端"""
|
||
response = Mock()
|
||
response.status_code = 429
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(side_effect=HTTPError(response=response))
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_api_error():
|
||
"""模拟 API 错误的 AI 客户端（不可恢复）"""
|
||
response = Mock()
|
||
response.status_code = 400
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(side_effect=HTTPError(response=response))
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_parse_error():
|
||
"""模拟解析失败的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
# 第一次返回无效 JSON
|
||
client.generate = Mock(return_value="Invalid JSON response")
|
||
return client
|
||
|
||
@pytest.fixture
|
||
def mock_ai_client_success():
|
||
"""模拟成功的 AI 客户端"""
|
||
client = Mock()
|
||
client.api_key = "test_key"
|
||
client.generate = Mock(return_value='''{
|
||
"video_type": "知识类",
|
||
"segments": [
|
||
{
|
||
"title": "测试标题",
|
||
"content": "测试内容",
|
||
"start_index": 0,
|
||
"end_index": 10
|
||
}
|
||
],
|
||
"actual_segment_count": 1
|
||
}''')
|
||
return client
|
||
```
|
||
|
||
3. 添加缓存相关的 fixtures:
|
||
```python
|
||
@pytest.fixture
|
||
def temp_cache_dir(tmp_path):
|
||
"""临时缓存目录"""
|
||
cache_dir = tmp_path / "cache" / "semantic"
|
||
return cache_dir
|
||
|
||
@pytest.fixture
|
||
def semantic_cache(temp_cache_dir):
|
||
"""语义分段缓存实例"""
|
||
from kb_workflow.srt.cache import SemanticCache
|
||
return SemanticCache(cache_dir=temp_cache_dir)
|
||
```
|
||
|
||
4. 添加长字幕分段样本:
|
||
```python
|
||
@pytest.fixture
|
||
def long_subtitle_segments():
|
||
"""长字幕分段样本（>5000 字符）"""
|
||
segments = []
|
||
for i in range(100):
|
||
segments.append(Segment(
|
||
index=i + 1,
|
||
start_time=i * 1.0,
|
||
end_time=(i + 1) * 1.0,
|
||
text=f"这是第 {i + 1} 句字幕内容。" * 5
|
||
))
|
||
return segments
|
||
```
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/conftest.py --collect-only</automated>
|
||
</verify>
|
||
<done>
|
||
- conftest.py 添加所有需要的 fixtures
|
||
- Fixtures 包括: 模拟失败的 AI 客户端、长文本样本、缓存目录
|
||
- 所有 fixtures 可以正确导入和使用
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto">
|
||
<name>Task 3: 在 test_srt_processor.py 中添加配置开关测试</name>
|
||
<files>tests/test_srt_processor.py</files>
|
||
<action>
|
||
在 tests/test_srt_processor.py 中添加配置开关测试（COST-03）:
|
||
|
||
1. 如果文件不存在，创建基础文件:
|
||
```python
|
||
import pytest
|
||
from unittest.mock import Mock, patch
|
||
from pathlib import Path
|
||
|
||
from kb_workflow.srt.processor import SRTProcessor
|
||
from kb_workflow.srt.parser import SRTParser
|
||
```
|
||
|
||
2. 添加 TestSemanticSegmentationToggle 测试类:
|
||
```python
|
||
class TestSemanticSegmentationToggle:
|
||
"""测试语义分段配置开关 (COST-03)"""
|
||
|
||
def test_semantic_segmentation_enabled(self, sample_srt_file):
|
||
"""测试启用语义分段"""
|
||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||
mock_client.return_value = Mock(api_key="test_key")
|
||
|
||
processor = SRTProcessor(
|
||
config={},
|
||
enable_semantic_segmentation=True
|
||
)
|
||
|
||
# 验证创建了 SemanticSegmentationEngine
|
||
assert processor.semantic_engine is not None
|
||
assert processor.segmenter is None
|
||
|
||
def test_semantic_segmentation_disabled(self, sample_srt_file):
|
||
"""测试禁用语义分段（默认）"""
|
||
processor = SRTProcessor(
|
||
config={},
|
||
enable_semantic_segmentation=False
|
||
)
|
||
|
||
# 验证使用规则分段器
|
||
assert processor.semantic_engine is None
|
||
assert processor.segmenter is not None
|
||
|
||
def test_semantic_segmentation_with_cache(self, sample_srt_file):
|
||
"""测试启用语义分段和缓存"""
|
||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||
mock_client.return_value = Mock(api_key="test_key")
|
||
|
||
processor = SRTProcessor(
|
||
config={},
|
||
enable_semantic_segmentation=True,
|
||
enable_cache=True
|
||
)
|
||
|
||
# 验证创建了缓存
|
||
assert processor.semantic_engine is not None
|
||
# 验证策略使用了缓存
|
||
assert processor.semantic_engine.strategy.cache is not None
|
||
|
||
def test_semantic_segmentation_without_cache(self, sample_srt_file):
|
||
"""测试启用语义分段但禁用缓存"""
|
||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||
mock_client.return_value = Mock(api_key="test_key")
|
||
|
||
processor = SRTProcessor(
|
||
config={},
|
||
enable_semantic_segmentation=True,
|
||
enable_cache=False
|
||
)
|
||
|
||
# 验证没有创建缓存
|
||
assert processor.semantic_engine is not None
|
||
assert processor.semantic_engine.strategy.cache is None
|
||
|
||
@pytest.fixture
|
||
def sample_srt_file(self, tmp_path):
|
||
"""创建示例 SRT 文件"""
|
||
srt_content = """1
|
||
00:00:00,000 --> 00:00:01,000
|
||
第一句字幕
|
||
|
||
2
|
||
00:00:01,000 --> 00:00:02,000
|
||
第二句字幕
|
||
"""
|
||
srt_file = tmp_path / "test.srt"
|
||
srt_file.write_text(srt_content, encoding='utf-8')
|
||
return srt_file
|
||
```
|
||
|
||
3. 确保测试是 RED 状态（功能尚未实现）。
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle -v --tb=no</automated>
|
||
</verify>
|
||
<done>
|
||
- test_srt_processor.py 添加配置开关测试
|
||
- 测试覆盖启用/禁用语义分段、启用/禁用缓存
|
||
- 所有测试都是 RED 状态（预期失败）
|
||
</done>
|
||
</task>
|
||
|
||
</tasks>
|
||
|
||
<verification>
|
||
## 整体验证标准
|
||
|
||
1. **测试文件创建**: test_semantic_robustness.py 包含所有鲁棒性和成本控制测试
|
||
2. **Fixtures 更新**: conftest.py 添加所有需要的 fixtures
|
||
3. **集成测试**: test_srt_processor.py 添加配置开关测试
|
||
4. **测试状态**: 所有测试都是 RED 状态（功能尚未实现）
|
||
|
||
运行测试验证文件结构:
|
||
```bash
|
||
# 验证测试文件存在
|
||
ls tests/test_semantic_robustness.py tests/conftest.py tests/test_srt_processor.py
|
||
|
||
# 验证测试可以收集
|
||
python3 -m pytest tests/test_semantic_robustness.py --collect-only
|
||
python3 -m pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle --collect-only
|
||
|
||
# 验证测试是 RED 状态
|
||
python3 -m pytest tests/test_semantic_robustness.py -v --tb=no
|
||
```
|
||
</verification>
|
||
|
||
<success_criteria>
|
||
1. test_semantic_robustness.py 文件创建，包含 6 个测试类
|
||
2. conftest.py 添加 10+ 个新 fixtures
|
||
3. test_srt_processor.py 添加 TestSemanticSegmentationToggle 测试类
|
||
4. 所有测试都是 RED 状态（预期失败）
|
||
5. 测试覆盖所有需求（ROBUST-01 到 COST-03）
|
||
</success_criteria>
|
||
|
||
<output>
|
||
执行完成后创建 `.planning/phases/14-robustness-cost/14-00-SUMMARY.md`
|
||
</output>
|