starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
488 Commits 3 Branches 4 Tags
fix/crawler-proxy-config
Commit Graph

488 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
starmen1265 a1076dfcb4 feat(semantic): 添加广告植入内容识别功能
PR Test / lint (pull_request) Successful in 1s
Details 
在LLM语义分段Prompt中添加广告识别要求：

【分段要求】新增：
- 第7条：要求识别UP主广告植入内容
- 明确常见的电商平台推广关键词（转转、京东、得物、淘宝、拼多多等）
- 要求将广告内容单独分段，title设为"广告"或"推广"

【注意事项】新增：
- 添加广告识别提示，列出常见广告植入特征：
  * 突然话题转换到商品推荐
  * 电商相关词汇（回收、卖、优惠、下单等）
  * 具体品牌名称提及
  * 推销语气的风格变化
  * 与视频主题无关的内容

测试结果：
- 转转广告被正确识别为独立段落，标题为"广告"
- 广告内容不再混入正常内容段落中

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-24 00:36:20 +08:00
starmen1265 bbc8816d16 refactor(semantic): 移除语义分段降级功能，LLM失败直接报错
PR Test / lint (pull_request) Successful in 1s
Details 
主要变更：
- 移除 _segment_single_chunk() 中的降级逻辑，失败直接抛出 RuntimeError
- 移除 _segment_multiple_chunks() 中的降级逻辑，分块失败直接抛出异常
- 移除 segment() 中 AI 客户端不可用时的降级逻辑，直接报错
- _fallback_chunk() 方法已弃用，调用时直接抛出异常
- VideoSegment 标题长度限制从 15 字增加到 30 字
- 将 ValidationError 从不可恢复错误改为可恢复错误，允许重试
- 更新相关 docstring，将 fallback_segmenter 标记为已弃用

现在 LLM 语义分段失败时会抛出明确的 RuntimeError 异常，
包含失败原因（HTTP 错误、JSON 解析错误等）。

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-24 00:20:28 +08:00
starmen1265 cb181c9d9f chore: complete v1.6 milestone - 语义分段改进
PR Test / lint (pull_request) Successful in 2s
Details
v1.6 		2026-03-23 23:20:28 +08:00
starmen1265 5dc8e32924 docs(phase-15): complete phase execution - output compatibility and configuration integration 2026-03-23 23:06:19 +08:00
starmen1265 7a1da86851 docs(15-02): complete configuration integration plan 
- Created SUMMARY.md documenting config integration
- Updated STATE.md with decisions and metrics
- Updated ROADMAP.md progress (3/4 summaries complete)
- Marked requirements OUTPUT-01, OUTPUT-02, OUTPUT-03 complete

Plan 15-02 complete:
- SRTProcessor reads enable_semantic_segmentation from KBSettings
- Config priority: parameter > KBSettings > default (True)
- 34/34 tests passing (9 new tests added)
- Duration: ~15 minutes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:04:23 +08:00
starmen1265 79c678b48a docs(15-03): complete backward compatibility verification plan 
- Created SUMMARY.md: 3 tasks, 9 tests, all passed
- Updated STATE.md: recorded decisions and session info
- Updated ROADMAP.md: phase 15 progress (3/4 summaries)
- Marked CONFIG-02 and CONFIG-03 complete

Backward compatibility verified:
- Rule-based segmentation unchanged (CONFIG-02)
- AnalyzeProcessor independent (CONFIG-03)
- API calls remain compatible
- All 24 tests pass (9 new + 15 existing)

Duration: 3min

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:04:17 +08:00
starmen1265 5983e95352 feat(15-02): integrate KBSettings config for semantic segmentation 
- Modified SRTProcessor to read enable_semantic_segmentation from KBSettings
- Changed constructor parameter default from False to None (None = read from config)
- Added config priority: constructor parameter > KBSettings > default (True)
- Added logging for config status (INFO level)
- Added comprehensive tests for config integration (TestConfigIntegration)
- Added tests for output format compatibility (TestOutputFormatBenefits)
- Fixed MockAIClient to accept disable_reasoning parameter
- All 34 tests passing

Requirements satisfied:
- CONFIG-01: SRTProcessor reads from KBSettings
- CONFIG-02: Disabled config uses rule-based segmentation
- OUTPUT-01: LLM segmentation outputs H2 titles
- OUTPUT-02: title=null skips title output
- OUTPUT-03: Rule-based segmentation outputs "## 内容"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:03:54 +08:00
starmen1265 5debdd6f36 test(15-03): add API compatibility tests 
- test_existing_api_compatibility: 验证不传参数时使用配置默认值
- test_explicit_false_overrides_config: 验证显式传 False 覆盖配置
- test_migration_path: 验证两种启用语义分段的方式

Task 3 完成：API 调用兼容性验证
所有向后兼容性测试通过（9/9）

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:03:40 +08:00
starmen1265 7371c6e8c0 test(15-03): add AnalyzeProcessor independence tests 
- test_analyze_processor_unaffected_by_config: 验证 AnalyzeProcessor 不受 enable_semantic_segmentation 配置影响
- test_analyze_processor_normal_initialization: 验证 AnalyzeProcessor 正常初始化
- test_analyze_processor_output_unchanged: 验证 AnalyzeProcessor 输出格式未改变

Task 2 完成：AnalyzeProcessor 独立性验证（CONFIG-03）
现有 test_analyze.py 测试全部通过，未被配置变更破坏

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:03:12 +08:00
starmen1265 9e33df7154 docs(15-00): record Phase 15 Plan 00 completion 
更新项目状态和进度：

- 完成 Phase 15 Plan 00（测试基础设施）
- 进度: 92% (44/48 计划完成)
- 执行时长: 94秒
- 任务数: 4
- 文件数: 3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:02:25 +08:00
starmen1265 8577571eda test(15-03): add rule-based segmentation backward compatibility tests 
- test_rule_based_segmentation_unchanged: 验证 enable_semantic_segmentation=False 时使用规则分段器
- test_rule_based_no_llm_calls: 验证规则分段不调用 LLM
- test_rule_based_thresholds_unchanged: 验证规则分段阈值（时间/长度）未改变

Task 1 完成：规则分段向后兼容性验证（CONFIG-02）

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:02:20 +08:00
starmen1265 2df27e277f docs(15-01): complete enable_semantic_segmentation config integration plan 2026-03-23 23:02:15 +08:00
starmen1265 1c1c88e2cf test(15-00): add test infrastructure placeholders for Phase 15 
创建了测试基础设施，为后续计划提供测试框架。

## 创建的文件

1. **tests/test_processors/test_srt.py** - SRTProcessor 配置集成测试占位符
   - TestConfigIntegration 测试类
   - 5 个测试占位符（pytest.skip）

2. **tests/test_backward_compat.py** - 向后兼容性测试
   - TestBackwardCompatibility 测试类
   - 部分测试已实现（来自计划 15-01）
   - 部分测试占位符（pytest.skip）

3. **tests/conftest.py** - 添加共享 fixtures
   - temp_env_vars fixture：临时环境变量管理
   - mock_kbsettings fixture：Mock KBSettings 实例
   - 添加 os 模块导入

## 偏离说明

计划要求所有测试都是占位符（pytest.skip），但实际上：
- test_config.py 已由计划 15-01 实现完整测试
- test_backward_compat.py 有部分实现

原因：计划 15-01 在 15-00 之前执行，导致测试不是占位符。

## 验证

- pytest 发现所有新增测试类和测试函数
- 测试文件结构符合项目规范

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:02:12 +08:00
starmen1265 8303cea018 feat(15-01): add enable_semantic_segmentation field to KBSettings 
- Add enable_semantic_segmentation: bool field with default=True
- Allow override via KB_ENABLE_SEMANTIC_SEGMENTATION env var
- Pydantic automatically parses boolean strings (true/false, yes/no, 1/0)
- Update docstring to document new field

CONFIG-01: LLM segmentation can be enabled/disabled via configuration

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:01:29 +08:00
starmen1265 f2a1847557 test(15-01): add failing test for enable_semantic_segmentation field 
- Test KBSettings has enable_semantic_segmentation field with bool type
- Test default value is True
- Test environment variable override (KB_ENABLE_SEMANTIC_SEGMENTATION)
- Test boolean string parsing (true/false, yes/no, 1/0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 23:01:20 +08:00
starmen1265 bc26765d17 fix(15-00): add <automated> verification commands for Nyquist compliance 2026-03-23 22:54:31 +08:00
starmen1265 405761c6ea docs(phase-15): create phase 15 plans 2026-03-23 22:53:07 +08:00
starmen1265 6ea21ad565 docs(15): add validation strategy 2026-03-23 22:50:47 +08:00
starmen1265 f2038c4676 docs(phase-15): research configuration integration and output compatibility 2026-03-23 22:50:22 +08:00
starmen1265 39ee6e1fb8 docs(state): record phase 15 context session 2026-03-23 22:48:54 +08:00
starmen1265 f484efb3b8 docs(15): capture phase context 2026-03-23 22:48:51 +08:00
starmen1265 db2c1602e5 docs(phase-14): complete phase execution 2026-03-23 22:45:06 +08:00
starmen1265 1eacc23c99 fix(14-04): correct Segment constructor parameter 
Change Segment(index=1, ...) to Segment(id=1, ...) to match
the actual Segment class definition in parser.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:43:10 +08:00
starmen1265 ffc6b85081 docs(14-04): complete long text chunking plan 2026-03-23 22:43:05 +08:00
starmen1265 865d06ba10 docs(14-05): update ROADMAP.md for Plan 14-05 completion 
- Mark Plan 14-05 as complete in plans list
- Update progress table: Phase 14 now 5/6 plans complete
- Mark Phase 13 as 7/7 complete (corrected)
- Update Phase 13 completion date to 2026-03-22
- Note Plan 14-04 partially implemented in 14-05

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:42:57 +08:00
starmen1265 50141a5545 docs(14-05): update STATE.md for Plan 14-05 completion 
- Update current position to Phase 14 Plan 05 (completed)
- Update progress to 34% (42/44 plans)
- Add Phase 14 Plan 05 decisions to Accumulated Context
- Update performance metrics table with Plan 14-05 data
- Update session continuity information

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:42:29 +08:00
starmen1265 ee5062ca9c test(14-04): 添加长文本分块处理测试 
- 添加 TestSplitIntoChunks 测试类：测试文本分块功能
- 添加 TestLongTextChunking 测试类：测试长文本自动分块分段
- 添加 TestChunkIndexCorrection 测试类：测试分块索引修正
- 添加 TestChunkFallback 测试类：测试单个分块失败时降级到规则分段

测试覆盖：
- 分块边界条件（空文本、短文本、长文本）
- 句子边界分割（。！？\n）
- 索引修正（start_index/end_index 相对于原文）
- 分块失败降级

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:41:55 +08:00
starmen1265 88302adb9d docs(14-05): create SUMMARY.md for Plan 14-05 
Summarize the execution of Plan 14-05:
- Completed 3 tasks (integrate cache, integrate SRTProcessor, fix tests)
- 3 atomic commits
- All tests passing (28/28 robustness, 23/23 SRTProcessor)
- Cache integration and configuration toggle working as expected
- Bonus: Implemented chunking methods from Plan 14-04

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:41:55 +08:00
starmen1265 537f009934 test(14-05): fix semantic segmentation toggle tests 
- Fix patch path for SimpleAIClient (deep_analyzer not semantic_engine)
- Remove config parameter from SRTProcessor initialization
- Add proper assertions for cache enabled/disabled states
- Import Mock from unittest.mock
- Remove duplicate test functions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:41:34 +08:00
starmen1265 bb321d9cd4 feat(14-05): integrate cache into SRTProcessor 
- Add enable_cache parameter to SRTProcessor.__init__()
- Create SemanticCache instance when semantic segmentation is enabled
- Pass cache to UnifiedSegmentationStrategy through SemanticSegmentationEngine
- Add logging for cache status (enabled/disabled)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:40:47 +08:00
starmen1265 faff7903a4 feat(14-04): 实现长文本分块处理方法 
- 添加 _split_into_chunks() 方法，按句子边界分割长文本（5000字符阈值）
- 添加 _merge_segment_results() 方法，合并分块结果并修正索引
- 添加 _segment_single_chunk() 方法，处理单个文本块
- 添加 _segment_multiple_chunks() 方法，处理多个文本块（分块）
- 添加 _fallback_chunk() 方法，用于单个分块失败时的降级处理
- 修改 segment() 方法，添加分块判断逻辑（len(content) > 5000）

遵守用户决策：
- 分块阈值: 5000 字符
- 按句子边界分割（。！？\n）
- 直接拼接各分块结果
- start_index/end_index 相对于原文

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:40:31 +08:00
starmen1265 87a1740cd3 feat(14-05): integrate cache into UnifiedSegmentationStrategy 
- Add cache parameter to UnifiedSegmentationStrategy.__init__()
- Add cache logic to segment() method (check before LLM, save after)
- Pass cache parameter through SemanticSegmentationEngine.__init__()
- Implement missing chunking methods (_segment_single_chunk, _segment_multiple_chunks, etc.)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:40:24 +08:00
starmen1265 83b89d4725 docs(14-02): complete parse error retry and fallback plan 2026-03-23 22:39:10 +08:00
starmen1265 dfbbabceea docs(14-01): complete LLM call graceful degradation strategy 
- 添加可恢复错误检测和重试逻辑
- 实现自动降级到规则分段
- 18 个测试全部通过
- 执行时间: 219s

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:39:02 +08:00
starmen1265 5c32cfdf69 test(14-01): 实现 LLM 调用的可恢复错误检测和重试逻辑 
- 添加 _is_recoverable_error() 静态方法判断错误类型
- 可恢复错误: ConnectionError, Timeout, HTTPError (429/500/502/503/504)
- 不可恢复错误: 其他 HTTPError (4xx), ValidationError
- 重试逻辑: 仅对可恢复错误重试 1 次，立即重试（无延迟）
- 改进日志记录: 重试用 info 级别，降级用 warning 级别
- 降级时记录具体错误类型和消息

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:38:28 +08:00
starmen1265 e4c704a582 docs(14-00): complete test infrastructure plan 
- Create 14-00-SUMMARY.md documenting test infrastructure
- Update STATE.md with position and progress
- Update ROADMAP.md for Phase 14

Summary details:
- 26 tests created/modified covering ROBUST-01 to COST-03
- Test infrastructure ready for Phase 14 implementation plans
- TDD RED state established for unimplemented features
- 18 tests passing (existing functionality)
- 8 tests failing (expected - features not yet implemented)
 2026-03-23 22:38:18 +08:00
starmen1265 92c300bb21 test(14-00): add test infrastructure for robustness and cost control 
- Add comprehensive test suite in test_semantic_robustness.py
  - TestLLMFailureFallback: LLM调用失败自动降级 (ROBUST-01)
  - TestNonRecoverableErrorDetection: 不可恢复错误检测
  - TestRecoverableErrorDetection: 可恢复错误检测
  - TestRetryMechanism: 重试机制测试
  - TestFallbackToRuleBased: 降级到规则分段 (ROBUST-03)
  - TestLoggingBehavior: 日志记录行为 (ROBUST-04)
  - TestParseFailureRetry: 解析失败重试 (ROBUST-02)
  - TestExtractJSON: JSON提取功能

- Update conftest.py with Phase 14 fixtures:
  - Mock AI clients for different failure scenarios
  - Long text samples for chunking tests (COST-01)
  - Cache-related fixtures (COST-02)
  - Long subtitle segments for testing

- Add TestSemanticSegmentationToggle to test_srt_processor.py (COST-03):
  - Test enabling/disabling semantic segmentation
  - Test cache enable/disable functionality
  - Test configuration switches

All tests follow TDD RED state - failing tests for future implementation
 2026-03-23 22:37:49 +08:00
starmen1265 bf359628fc docs(14-03): complete semantic cache implementation plan 
- Created SUMMARY.md with implementation details
- Updated STATE.md with new decisions and session info
- Updated ROADMAP.md progress (1/6 plans complete)
- Marked COST-02 requirement complete

Duration: 1 min (72s)
Tasks: 1 (TDD: RED → GREEN)
Tests: 11 passed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:36:50 +08:00
starmen1265 970c13a93b test(14-02): add failing test for parse error retry and fallback 
- Test JSONDecodeError triggers retry 1 time before fallback
- Test ValidationError directly fallbacks without retry
- Test parse error logs LLM output fragment (first 500 chars)
- Test fallback logs error type and message
- Test retry uses INFO level logging

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:36:05 +08:00
starmen1265 baf7c0f521 feat(14-03): implement SemanticCache for content-based caching 
- compute_content_hash(): 计算 SHA256 内容哈希
- SemanticCache.get(): 从缓存读取 SegmentationResult
- SemanticCache.set(): 保存 SegmentationResult 到缓存
- 支持版本控制（CACHE_VERSION = '1.0'）
- 缓存存储在 .planning/cache/semantic/ 目录
- 错误容错：缓存失败不影响主流程
- 自动创建缓存目录

遵守用户决策：
- 缓存键: SHA256 内容哈希
- 缓存存储: 本地文件系统
- 缓存 TTL: 永久有效
- 缓存内容: 完整的 SegmentationResult

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:35:57 +08:00
starmen1265 b6e923c4a4 test(14-03): add failing tests for SemanticCache 
- TestComputeContentHash: 测试内容哈希计算（SHA256）
- TestSemanticCache: 测试缓存读写功能
- 测试覆盖：缓存命中/未命中、版本控制、错误处理、多缓存项

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-23 22:35:38 +08:00
starmen1265 c378a0eb32 docs(14): set nyquist compliant 2026-03-23 22:34:19 +08:00
starmen1265 f8275a9846 docs(14): move plan 04 to wave 2 2026-03-23 22:33:53 +08:00
starmen1265 1f51ea1408 docs(14): update ROADMAP with plan list and wave structure 2026-03-23 20:57:00 +08:00
starmen1265 cefb79d311 docs(14): create phase 14 plans (5 plans + wave 0 tests) 2026-03-23 20:56:54 +08:00
starmen1265 3c9859c30d docs(14): add validation strategy 2026-03-23 20:54:44 +08:00
starmen1265 6d53ce47d4 docs(14): research phase 14 robustness and cost optimization 2026-03-23 20:53:52 +08:00
starmen1265 897ef95a87 docs(state): record phase 14 context session 2026-03-23 20:51:32 +08:00
starmen1265 2dde3a3dc8 docs(14): capture phase context 2026-03-23 20:51:29 +08:00
starmen1265 e445986d6d docs(phase-13): complete phase execution 2026-03-22 00:34:15 +08:00