docs(14): create phase 14 plans (5 plans + wave 0 tests)
This commit is contained in:
@@ -0,0 +1,510 @@
|
||||
---
|
||||
phase: 14-robustness-cost
|
||||
plan: 00
|
||||
type: execute
|
||||
wave: 0
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- tests/test_semantic_robustness.py
|
||||
- tests/conftest.py
|
||||
- tests/test_srt_processor.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
- ROBUST-01
|
||||
- ROBUST-02
|
||||
- ROBUST-03
|
||||
- ROBUST-04
|
||||
- COST-01
|
||||
- COST-02
|
||||
- COST-03
|
||||
must_haves:
|
||||
truths:
|
||||
- "所有计划任务都有对应的自动化测试"
|
||||
- "测试框架配置正确,可以运行测试"
|
||||
- "Mock fixtures 可用,模拟 LLM 客户端行为"
|
||||
artifacts:
|
||||
- path: "tests/test_semantic_robustness.py"
|
||||
provides: "鲁棒性和成本控制测试套件"
|
||||
exports: ["test_*"]
|
||||
- path: "tests/conftest.py"
|
||||
provides: "共享测试 fixtures"
|
||||
exports: ["mock_ai_client", "sample_segments", "sample_metadata"]
|
||||
- path: "tests/test_srt_processor.py"
|
||||
provides: "SRTProcessor 集成测试"
|
||||
exports: ["test_semantic_segmentation_toggle"]
|
||||
key_links:
|
||||
- from: "test_semantic_robustness.py"
|
||||
to: "conftest.py"
|
||||
via: "pytest fixtures"
|
||||
pattern: "@pytest.fixture"
|
||||
---
|
||||
|
||||
<objective>
|
||||
创建阶段 14 的测试基础设施,覆盖所有鲁棒性和成本控制功能。
|
||||
|
||||
Purpose: 确保所有计划任务都有对应的自动化测试,遵循 TDD 原则。
|
||||
Output: 完整的测试套件,包括鲁棒性测试、成本控制测试、集成测试。
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/14-robustness-cost/14-CONTEXT.md
|
||||
@.planning/phases/14-robustness-cost/14-RESEARCH.md
|
||||
@.planning/phases/14-robustness-cost/14-VALIDATION.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/REQUIREMENTS.md
|
||||
@.planning/STATE.md
|
||||
|
||||
@tests/conftest.py
|
||||
@tests/test_semantic_engine.py
|
||||
@kb_workflow/srt/semantic_engine.py
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- Key fixtures and test patterns from existing test files -->
|
||||
|
||||
From tests/conftest.py (existing fixtures):
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock
|
||||
from kb_workflow.srt.parser import Segment
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client():
|
||||
"""Mock AI client"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(return_value="Test response")
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def sample_segments():
|
||||
"""示例字幕分段"""
|
||||
return [
|
||||
Segment(index=1, start_time=0.0, end_time=1.0, text="第一句"),
|
||||
Segment(index=2, start_time=1.0, end_time=2.0, text="第二句"),
|
||||
]
|
||||
|
||||
@pytest.fixture
|
||||
def sample_metadata():
|
||||
"""示例视频元数据"""
|
||||
return {
|
||||
"title": "测试视频",
|
||||
"duration": 120,
|
||||
"category": "知识"
|
||||
}
|
||||
```
|
||||
|
||||
From 14-VALIDATION.md (Wave 0 Requirements):
|
||||
```python
|
||||
# Wave 0 Requirements
|
||||
- [ ] tests/test_semantic_robustness.py — 覆盖 ROBUST-01, ROBUST-02, ROBUST-03, ROBUST-04, COST-01, COST-02
|
||||
- [ ] tests/test_srt_processor.py — 覆盖 COST-03(配置开关)
|
||||
- [ ] tests/conftest.py — 共享 fixtures(mock LLM client, 样本字幕数据)
|
||||
- [ ] 框架安装: 无需安装,pytest 已配置
|
||||
|
||||
# Mock Strategy
|
||||
使用 unittest.mock.patch 模拟 SimpleAIClient.generate(),测试不同失败场景(网络错误、解析失败、超时等)
|
||||
```
|
||||
|
||||
From 14-RESEARCH.md (Test Requirements):
|
||||
| Req ID | Behavior | Test Type | Automated Command |
|
||||
|--------|----------|-----------|-------------------|
|
||||
| ROBUST-01 | LLM 调用失败时自动降级 | unit | test_llm_failure_fallback |
|
||||
| ROBUST-02 | 解析失败时重试或降级 | unit | test_parse_failure_retry |
|
||||
| ROBUST-03 | AI 不可用时使用规则分段 | unit | test_ai_unavailable_fallback |
|
||||
| ROBUST-04 | 降级日志记录 | unit | test_fallback_logging |
|
||||
| COST-01 | 长文本分块处理 | unit | test_long_text_chunking |
|
||||
| COST-02 | 缓存命中避免重复调用 | unit | test_cache_hit |
|
||||
| COST-03 | 配置开关控制 | integration | test_semantic_segmentation_toggle |
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: 创建 test_semantic_robustness.py 测试文件</name>
|
||||
<files>tests/test_semantic_robustness.py</files>
|
||||
<action>
|
||||
创建 tests/test_semantic_robustness.py,包含所有鲁棒性和成本控制测试:
|
||||
|
||||
1. 导入依赖:
|
||||
```python
|
||||
import pytest
|
||||
import json
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
from pathlib import Path
|
||||
from requests.exceptions import ConnectionError, Timeout, HTTPError
|
||||
|
||||
from kb_workflow.srt.semantic_engine import (
|
||||
UnifiedSegmentationStrategy,
|
||||
SemanticSegmentationEngine,
|
||||
VideoSegment,
|
||||
SegmentationResult
|
||||
)
|
||||
from kb_workflow.srt.cache import SemanticCache, compute_content_hash
|
||||
```
|
||||
|
||||
2. 创建测试类和方法(遵循 14-VALIDATION.md 的测试映射):
|
||||
|
||||
**TestLLMFailureFallback (ROBUST-01)**:
|
||||
- test_llm_failure_fallback: 模拟 LLM 调用失败(网络错误),验证降级到规则分段
|
||||
- test_recoverable_error_retry: 模拟可恢复错误(ConnectionError),验证重试 1 次
|
||||
- test_non_recoverable_error_no_retry: 模拟不可恢复错误(HTTPError 400),验证直接降级
|
||||
|
||||
**TestParseFailureRetry (ROBUST-02, ROBUST-04)**:
|
||||
- test_parse_failure_retry: 模拟 JSON 解析失败(JSONDecodeError),验证重试后降级
|
||||
- test_validation_error_no_retry: 模拟 Pydantic 验证失败(ValidationError),验证直接降级
|
||||
- test_fallback_logging: 验证降级时记录 warning 级别日志,包含错误类型和原因
|
||||
|
||||
**TestAIAvailabilityFallback (ROBUST-03)**:
|
||||
- test_ai_unavailable_fallback: 模拟 AI 客户端未配置(api_key 为空),验证使用规则分段
|
||||
- test_ai_client_none: 模拟 AI 客户端初始化失败,验证使用规则分段
|
||||
|
||||
**TestLongTextChunking (COST-01)**:
|
||||
- test_long_text_chunking: 使用超过 5000 字符的文本,验证分块处理
|
||||
- test_chunk_sentence_boundary: 验证分块在句子边界(标点符号后)分割
|
||||
- test_chunk_index_correction: 验证 VideoSegment.start_index/end_index 相对于原文
|
||||
|
||||
**TestCacheHit (COST-02)**:
|
||||
- test_cache_hit: 第一次调用 LLM,第二次调用从缓存读取
|
||||
- test_cache_key_hash: 验证缓存键使用 SHA256 哈希
|
||||
- test_cache_miss_content_change: 内容变化时缓存未命中
|
||||
- test_cache_file_format: 验证缓存文件格式(JSON + version 字段)
|
||||
|
||||
3. 使用 Mock 模拟 LLM 客户端:
|
||||
```python
|
||||
@pytest.fixture
|
||||
def mock_ai_client_with_failure():
|
||||
"""模拟失败的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(side_effect=ConnectionError("Network error"))
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_with_parse_error():
|
||||
"""模拟解析失败的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
# 第一次返回无效 JSON,第二次返回有效 JSON
|
||||
client.generate = Mock(side_effect=[
|
||||
"Invalid JSON",
|
||||
'{"video_type": "知识类", "segments": [], "actual_segment_count": 0}'
|
||||
])
|
||||
return client
|
||||
```
|
||||
|
||||
4. 示例测试实现:
|
||||
```python
|
||||
def test_llm_failure_fallback(sample_segments, sample_metadata):
|
||||
"""测试 LLM 调用失败时降级到规则分段"""
|
||||
strategy = UnifiedSegmentationStrategy(
|
||||
ai_client=mock_ai_client_with_failure()
|
||||
)
|
||||
|
||||
result = strategy.segment(sample_segments, sample_metadata)
|
||||
|
||||
# 验证降级到规则分段(返回 VideoSegment 列表)
|
||||
assert isinstance(result, list)
|
||||
assert all(isinstance(seg, VideoSegment) for seg in result)
|
||||
# 规则分段没有标题
|
||||
assert all(seg.title is None for seg in result)
|
||||
|
||||
def test_recoverable_error_retry(sample_segments, sample_metadata):
|
||||
"""测试可恢复错误重试 1 次"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
# 第一次失败,第二次成功
|
||||
client.generate = Mock(side_effect=[
|
||||
ConnectionError("Network error"),
|
||||
'{"video_type": "知识类", "segments": [], "actual_segment_count": 0}'
|
||||
])
|
||||
|
||||
strategy = UnifiedSegmentationStrategy(ai_client=client)
|
||||
result = strategy.segment(sample_segments, sample_metadata)
|
||||
|
||||
# 验证重试了 2 次
|
||||
assert client.generate.call_count == 2
|
||||
```
|
||||
|
||||
确保所有测试都是 RED 状态(功能尚未实现)。
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_semantic_robustness.py -v --tb=no</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- test_semantic_robustness.py 文件创建,包含所有测试
|
||||
- 所有测试都是 RED 状态(预期失败)
|
||||
- 测试覆盖所有需求(ROBUST-01 到 COST-03)
|
||||
- Mock fixtures 正确模拟 LLM 客户端行为
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: 更新 conftest.py 添加新 fixtures</name>
|
||||
<files>tests/conftest.py</files>
|
||||
<action>
|
||||
更新 tests/conftest.py,添加阶段 14 需要的新 fixtures:
|
||||
|
||||
1. 添加长文本样本(用于分块测试):
|
||||
```python
|
||||
@pytest.fixture
|
||||
def long_text_sample():
|
||||
"""长文本样本(>5000 字符,用于分块测试)"""
|
||||
# 生成一个超过 5000 字符的文本,包含多个句子
|
||||
sentences = ["这是一个测试句子。" * 10] * 15 # 约 6000 字符
|
||||
return " ".join(sentences)
|
||||
```
|
||||
|
||||
2. 添加模拟失败的 AI 客户端 fixtures:
|
||||
```python
|
||||
@pytest.fixture
|
||||
def mock_ai_client_network_error():
|
||||
"""模拟网络错误的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(side_effect=ConnectionError("Network error"))
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_timeout():
|
||||
"""模拟超时的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(side_effect=Timeout("Request timeout"))
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_rate_limit():
|
||||
"""模拟限流的 AI 客户端"""
|
||||
response = Mock()
|
||||
response.status_code = 429
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(side_effect=HTTPError(response=response))
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_api_error():
|
||||
"""模拟 API 错误的 AI 客户端(不可恢复)"""
|
||||
response = Mock()
|
||||
response.status_code = 400
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(side_effect=HTTPError(response=response))
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_parse_error():
|
||||
"""模拟解析失败的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
# 第一次返回无效 JSON
|
||||
client.generate = Mock(return_value="Invalid JSON response")
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ai_client_success():
|
||||
"""模拟成功的 AI 客户端"""
|
||||
client = Mock()
|
||||
client.api_key = "test_key"
|
||||
client.generate = Mock(return_value='''{
|
||||
"video_type": "知识类",
|
||||
"segments": [
|
||||
{
|
||||
"title": "测试标题",
|
||||
"content": "测试内容",
|
||||
"start_index": 0,
|
||||
"end_index": 10
|
||||
}
|
||||
],
|
||||
"actual_segment_count": 1
|
||||
}''')
|
||||
return client
|
||||
```
|
||||
|
||||
3. 添加缓存相关的 fixtures:
|
||||
```python
|
||||
@pytest.fixture
|
||||
def temp_cache_dir(tmp_path):
|
||||
"""临时缓存目录"""
|
||||
cache_dir = tmp_path / "cache" / "semantic"
|
||||
return cache_dir
|
||||
|
||||
@pytest.fixture
|
||||
def semantic_cache(temp_cache_dir):
|
||||
"""语义分段缓存实例"""
|
||||
from kb_workflow.srt.cache import SemanticCache
|
||||
return SemanticCache(cache_dir=temp_cache_dir)
|
||||
```
|
||||
|
||||
4. 添加长字幕分段样本:
|
||||
```python
|
||||
@pytest.fixture
|
||||
def long_subtitle_segments():
|
||||
"""长字幕分段样本(>5000 字符)"""
|
||||
segments = []
|
||||
for i in range(100):
|
||||
segments.append(Segment(
|
||||
index=i + 1,
|
||||
start_time=i * 1.0,
|
||||
end_time=(i + 1) * 1.0,
|
||||
text=f"这是第 {i + 1} 句字幕内容。" * 5
|
||||
))
|
||||
return segments
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/conftest.py --collect-only</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- conftest.py 添加所有需要的 fixtures
|
||||
- Fixtures 包括: 模拟失败的 AI 客户端、长文本样本、缓存目录
|
||||
- 所有 fixtures 可以正确导入和使用
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: 在 test_srt_processor.py 中添加配置开关测试</name>
|
||||
<files>tests/test_srt_processor.py</files>
|
||||
<action>
|
||||
在 tests/test_srt_processor.py 中添加配置开关测试(COST-03):
|
||||
|
||||
1. 如果文件不存在,创建基础文件:
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from pathlib import Path
|
||||
|
||||
from kb_workflow.srt.processor import SRTProcessor
|
||||
from kb_workflow.srt.parser import SRTParser
|
||||
```
|
||||
|
||||
2. 添加 TestSemanticSegmentationToggle 测试类:
|
||||
```python
|
||||
class TestSemanticSegmentationToggle:
|
||||
"""测试语义分段配置开关 (COST-03)"""
|
||||
|
||||
def test_semantic_segmentation_enabled(self, sample_srt_file):
|
||||
"""测试启用语义分段"""
|
||||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||||
mock_client.return_value = Mock(api_key="test_key")
|
||||
|
||||
processor = SRTProcessor(
|
||||
config={},
|
||||
enable_semantic_segmentation=True
|
||||
)
|
||||
|
||||
# 验证创建了 SemanticSegmentationEngine
|
||||
assert processor.semantic_engine is not None
|
||||
assert processor.segmenter is None
|
||||
|
||||
def test_semantic_segmentation_disabled(self, sample_srt_file):
|
||||
"""测试禁用语义分段(默认)"""
|
||||
processor = SRTProcessor(
|
||||
config={},
|
||||
enable_semantic_segmentation=False
|
||||
)
|
||||
|
||||
# 验证使用规则分段器
|
||||
assert processor.semantic_engine is None
|
||||
assert processor.segmenter is not None
|
||||
|
||||
def test_semantic_segmentation_with_cache(self, sample_srt_file):
|
||||
"""测试启用语义分段和缓存"""
|
||||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||||
mock_client.return_value = Mock(api_key="test_key")
|
||||
|
||||
processor = SRTProcessor(
|
||||
config={},
|
||||
enable_semantic_segmentation=True,
|
||||
enable_cache=True
|
||||
)
|
||||
|
||||
# 验证创建了缓存
|
||||
assert processor.semantic_engine is not None
|
||||
# 验证策略使用了缓存
|
||||
assert processor.semantic_engine.strategy.cache is not None
|
||||
|
||||
def test_semantic_segmentation_without_cache(self, sample_srt_file):
|
||||
"""测试启用语义分段但禁用缓存"""
|
||||
with patch('kb_workflow.srt.semantic_engine.SimpleAIClient') as mock_client:
|
||||
mock_client.return_value = Mock(api_key="test_key")
|
||||
|
||||
processor = SRTProcessor(
|
||||
config={},
|
||||
enable_semantic_segmentation=True,
|
||||
enable_cache=False
|
||||
)
|
||||
|
||||
# 验证没有创建缓存
|
||||
assert processor.semantic_engine is not None
|
||||
assert processor.semantic_engine.strategy.cache is None
|
||||
|
||||
@pytest.fixture
|
||||
def sample_srt_file(self, tmp_path):
|
||||
"""创建示例 SRT 文件"""
|
||||
srt_content = """1
|
||||
00:00:00,000 --> 00:00:01,000
|
||||
第一句字幕
|
||||
|
||||
2
|
||||
00:00:01,000 --> 00:00:02,000
|
||||
第二句字幕
|
||||
"""
|
||||
srt_file = tmp_path / "test.srt"
|
||||
srt_file.write_text(srt_content, encoding='utf-8')
|
||||
return srt_file
|
||||
```
|
||||
|
||||
3. 确保测试是 RED 状态(功能尚未实现)。
|
||||
</action>
|
||||
<verify>
|
||||
<automated>pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle -v --tb=no</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- test_srt_processor.py 添加配置开关测试
|
||||
- 测试覆盖启用/禁用语义分段、启用/禁用缓存
|
||||
- 所有测试都是 RED 状态(预期失败)
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
## 整体验证标准
|
||||
|
||||
1. **测试文件创建**: test_semantic_robustness.py 包含所有鲁棒性和成本控制测试
|
||||
2. **Fixtures 更新**: conftest.py 添加所有需要的 fixtures
|
||||
3. **集成测试**: test_srt_processor.py 添加配置开关测试
|
||||
4. **测试状态**: 所有测试都是 RED 状态(功能尚未实现)
|
||||
|
||||
运行测试验证文件结构:
|
||||
```bash
|
||||
# 验证测试文件存在
|
||||
ls tests/test_semantic_robustness.py tests/conftest.py tests/test_srt_processor.py
|
||||
|
||||
# 验证测试可以收集
|
||||
python3 -m pytest tests/test_semantic_robustness.py --collect-only
|
||||
python3 -m pytest tests/test_srt_processor.py::test_semantic_segmentation_toggle --collect-only
|
||||
|
||||
# 验证测试是 RED 状态
|
||||
python3 -m pytest tests/test_semantic_robustness.py -v --tb=no
|
||||
```
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
1. test_semantic_robustness.py 文件创建,包含 6 个测试类
|
||||
2. conftest.py 添加 10+ 个新 fixtures
|
||||
3. test_srt_processor.py 添加 TestSemanticSegmentationToggle 测试类
|
||||
4. 所有测试都是 RED 状态(预期失败)
|
||||
5. 测试覆盖所有需求(ROBUST-01 到 COST-03)
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
执行完成后创建 `.planning/phases/14-robustness-cost/14-00-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user