starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
389 Commits 3 Branches 4 Tags
7023a57644520c96f858080dd08ff6d224607628
Commit Graph

389 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
starmen1265 7023a57644 test(12-01): add failing tests for SRT parser module 
- Test standard SRT format parsing
- Test special character handling
- Test error handling for invalid/empty content
- Test file parsing functionality

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-19 23:31:36 +08:00
starmen1265 40337940d8 docs(12-02): fix autonomous flag for checkpoint task 2026-03-19 23:28:06 +08:00
starmen1265 56039c5956 docs(12-srt-simplifier): create phase 12 plans 2026-03-19 23:25:43 +08:00
starmen1265 978613461b docs: create milestone v1.5 requirements and roadmap 2026-03-19 23:18:50 +08:00
starmen1265 d69b85cb1e docs: start milestone v1.5 B站字幕简化处理 2026-03-19 23:13:43 +08:00
starmen1265 7915808413 fix(cli): use json mode for model_dump serialization
PR Test / lint (pull_request) Successful in 2s
Details 
Ensure Pydantic model serializes to JSON-compatible format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-19 22:59:36 +08:00
starmen1265 fa46062698 chore: archive v1.4 milestone - clean up moved files
PR Test / lint (pull_request) Successful in 2s
Details 
- Delete .planning/REQUIREMENTS.md (archived to milestones/)
- Delete .planning/phases/11-stealth-fingerprinting/* (moved to milestones/v1.4-phases/)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-19 00:05:27 +08:00
starmen1265 0e558d2e3d chore: archive v1.4 phase directories 2026-03-19 00:05:03 +08:00
starmen1265 a58fbfb4a7 chore: complete v1.4 Zhihu Anti-Crawler milestone v1.4 2026-03-19 00:04:04 +08:00
starmen1265 679b7d1440 docs(phase-11): complete phase execution - stealth fingerprinting 2026-03-18 23:59:31 +08:00
starmen1265 b81eec4284 docs(11-02): complete stealth integration plan 
- Created 11-02-SUMMARY.md with execution results
- Updated STATE.md: v1.4 milestone complete, 100% progress
- Updated ROADMAP.md: Phase 11 complete, all plans done
- Updated REQUIREMENTS.md: VERIFY-01, VERIFY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:56:09 +08:00
starmen1265 08cbe24d08 feat(11-02): integrate stealth fingerprint masking into ZhihuCrawler 
- Add stealth_config parameter to ZhihuCrawler.__init__
- Add --disable-blink-features=AutomationControlled to launch args
- Call apply_stealth_sync after browser context creation
- Maintain backward compatibility (stealth enabled by default)
- Add 8 integration tests covering stealth behavior and compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:53:22 +08:00
starmen1265 56c21f3f36 test(11-02): add failing tests for stealth integration in ZhihuCrawler 
- Test crawler accepts stealth_config parameter
- Test default stealth enabled behavior
- Test stealth can be disabled
- Test _start_browser_with_proxy calls apply_stealth_sync
- Test launch args include --disable-blink-features
- Test backward compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:52:23 +08:00
starmen1265 b9f74e5a7e docs(11-01): complete stealth fingerprint masking plan 
- Created 11-01-SUMMARY.md with execution results
- Updated STATE.md: Plan 1/2 complete, 50% progress
- Updated ROADMAP.md: 11-01 marked complete
- Updated REQUIREMENTS.md: STEALTH-01 to STEALTH-04 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:50:24 +08:00
starmen1265 e152b8ea05 feat(11-01): create stealth fingerprint masking module 
- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:47:50 +08:00
starmen1265 d6026660a9 docs(11): create phase 11 stealth fingerprinting plans 2026-03-18 23:41:43 +08:00
starmen1265 bfb6ab9c0a docs(11): add validation strategy 2026-03-18 23:35:58 +08:00
starmen1265 76da0f04d4 docs(11): research stealth fingerprinting domain 2026-03-18 23:34:59 +08:00
starmen1265 9e30e042a8 docs: create v1.4 Zhihu Anti-Crawler roadmap (1 phase, 2 plans) 2026-03-18 23:19:25 +08:00
starmen1265 eb90773422 docs: define v1.4 Zhihu Anti-Crawler requirements 2026-03-18 23:16:35 +08:00
starmen1265 a443cbde73 docs: start milestone v1.4 Zhihu Anti-Crawler 2026-03-18 23:09:06 +08:00
starmen1265 a9ead75da7 chore: complete v1.3 Proxy Pool milestone v1.3 2026-03-18 22:45:17 +08:00
starmen1265 d92b872b6c chore: remove phase 11 (Platform Adaptation) 2026-03-18 22:39:29 +08:00
starmen1265 d7f32b6b93 chore: remove phases 11-13 (Behavior Camouflage, CAPTCHA Detection, Error Recovery) 2026-03-18 22:35:48 +08:00
starmen1265 2561de6ad7 docs: update research for v1.3 anti-crawler optimization
PR Test / lint (pull_request) Successful in 2s
Details 
- Add architecture research for proxy pool, behavior camouflage,
  captcha detection, and error recovery
- Update features, pitfalls, and stack documentation
- Add phase 10 validation checklist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 20:45:14 +08:00
starmen1265 0c63503309 docs: resolve debug kb_workflow-module-conflict 2026-03-18 20:38:30 +08:00
starmen1265 c30b0bce88 docs(phase-10): complete proxy pool phase execution 2026-03-18 16:55:18 +08:00
starmen1265 bcfef48676 docs(10-02): complete health check and crawler integration plan 2026-03-18 16:51:16 +08:00
starmen1265 b66bcc08b1 feat(10-02): integrate ProxyPool into ZhihuCrawler 
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:48:48 +08:00
starmen1265 36a45095c1 test(10-02): add health check tests for proxy pool 
- test_fail_threshold: verify 3 consecutive failures mark proxy unavailable
- test_success_resets_count: verify success resets failure count
- test_cooldown_recovery: verify proxy recovers after cooldown
- test_cooldown_not_elapsed: verify proxy stays unavailable during cooldown
- test_all_unavailable_fallback: verify fallback to first proxy when all fail
- test_get_available_count: verify available proxy count tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:46:37 +08:00
starmen1265 96dfdce59c docs(10-01): complete proxy pool core plan 
- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:43:34 +08:00
starmen1265 c58a7d08b6 feat(10-01): implement proxy pool core functionality 
- ProxyEntry: tracks proxy state, failures, and recovery
- ProxyPoolConfig: YAML configuration model
- ProxyPool: manages multiple proxies with round-robin rotation
- from_yaml(): load from ~/.kb_proxies.yaml
- from_env_fallback(): backward compatible env var loading
- get_next_proxy(): round-robin with lazy recovery check

Implements PROXY-01 (multi-proxy config) and PROXY-02 (rotation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:39:39 +08:00
starmen1265 bcf1a88613 test(10-01): add failing tests for proxy pool module 
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults

RED phase: tests fail because proxy_pool module does not exist yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:38:07 +08:00
starmen1265 e896bd6765 docs(10): create phase plan for proxy pool 2026-03-18 16:27:33 +08:00
starmen1265 fe22f94358 docs(10): research proxy pool domain 2026-03-18 16:18:38 +08:00
starmen1265 1b58bf9364 docs(10): capture phase context for proxy pool 2026-03-18 16:06:31 +08:00
starmen1265 9d9de810b7 docs: create milestone v1.3 roadmap (5 phases) 2026-03-18 14:17:14 +08:00
starmen1265 887ced6075 docs: define v1.3 anti-crawler requirements 2026-03-18 13:07:46 +08:00
starmen1265 e4f96be0c4 docs: start milestone v1.3 Anti-Crawler Optimization 2026-03-18 11:59:39 +08:00
starmen1265 9c8ac34138 fix(kb_workflow): resolve module/package naming conflict
PR Test / lint (pull_request) Successful in 1s
Details 
Rename kb_workflow.py to cli.py to avoid conflict with kb_workflow/ package.
Add __main__.py to support `python -m kb_workflow` execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 10:35:33 +08:00
starmen1265 9d2433322a fix(crawler): set proxy_required=False by default
PR Test / lint (pull_request) Successful in 2s
Details 
旁路由 (192.168.1.200) provides automatic system-level proxy.
No explicit HTTP_PROXY/HTTPS_PROXY configuration needed.
 2026-03-18 00:04:48 +08:00
starmen1265 e1d3218a44 Merge branch 'fix/ci-deploy-reset' 2026-03-18 00:03:43 +08:00
starmen1265 c09c8588f0 fix(crawler): resolve circular import in zhihu_crawler 
Circular import chain:
  zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler

Fix: Lazy import get_settings() inside _get_cookie() method instead of
module-level import.
 2026-03-18 00:03:02 +08:00
starmen1265 d88f1a3b8d Merge pull request 'docs(skill): update kb_workflow SKILL.md to reflect current implementation' (#28) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/28
 2026-03-17 23:52:57 +08:00
starmen1265 6a819b6c80 docs(skill): update kb_workflow SKILL.md to reflect current implementation
PR Test / lint (pull_request) Successful in 2s
Details 
- Fix CLI path: kb-workflow -> kb_workflow
- Add login command documentation for Zhihu
- Update directory structure to match actual paths
- Add environment variable checks section
- Document strict failure handling principle
- Update technical architecture diagram
 2026-03-17 23:50:42 +08:00
starmen1265 5ccabec3e4 Merge pull request 'fix(ci): correct skill directory names in deploy.yml' (#27) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/27
 2026-03-17 23:41:51 +08:00
starmen1265 228cd1cf8c fix(ci): correct skill directory names in deploy.yml
PR Test / lint (pull_request) Successful in 2s
Details 
- Fix kb-workflow -> kb_workflow (underscore not hyphen)
- Remove non-existent skill directories from the list
- Only include skills that actually exist in the repo
 2026-03-17 23:40:12 +08:00
starmen1265 289be86fc6 Merge pull request 'fix/ci-deploy-reset' (#26) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/26
 2026-03-17 23:28:17 +08:00
starmen1265 7744f7ac1d docs(phase-09): complete phase execution - pipeline integration verified
PR Test / lint (pull_request) Successful in 2s
Details
2026-03-17 23:25:08 +08:00
starmen1265 248a1f2b8c docs(09-01): complete ZhihuAdapter integration plan 
- Integrated ZhihuCrawler into ZhihuAdapter
- Added comment crawling and formatting
- Implemented cookie auto-login flow
- Added idempotent execution via IdempotencyChecker

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 23:17:39 +08:00