starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix/crawler-proxy-config #31

Merged
starmen1265 merged 15 commits from fix/crawler-proxy-config into main 2026-03-18 20:48:33 +08:00
Conversation 0 Commits 15 Files Changed 20
+4344 -163

15 Commits

Author SHA1 Message Date
starmen1265 2561de6ad7 docs: update research for v1.3 anti-crawler optimization
PR Test / lint (pull_request) Successful in 2s
Details 
- Add architecture research for proxy pool, behavior camouflage,
  captcha detection, and error recovery
- Update features, pitfalls, and stack documentation
- Add phase 10 validation checklist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 20:45:14 +08:00
starmen1265 0c63503309 docs: resolve debug kb_workflow-module-conflict 2026-03-18 20:38:30 +08:00
starmen1265 c30b0bce88 docs(phase-10): complete proxy pool phase execution 2026-03-18 16:55:18 +08:00
starmen1265 bcfef48676 docs(10-02): complete health check and crawler integration plan 2026-03-18 16:51:16 +08:00
starmen1265 b66bcc08b1 feat(10-02): integrate ProxyPool into ZhihuCrawler 
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:48:48 +08:00
starmen1265 36a45095c1 test(10-02): add health check tests for proxy pool 
- test_fail_threshold: verify 3 consecutive failures mark proxy unavailable
- test_success_resets_count: verify success resets failure count
- test_cooldown_recovery: verify proxy recovers after cooldown
- test_cooldown_not_elapsed: verify proxy stays unavailable during cooldown
- test_all_unavailable_fallback: verify fallback to first proxy when all fail
- test_get_available_count: verify available proxy count tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:46:37 +08:00
starmen1265 96dfdce59c docs(10-01): complete proxy pool core plan 
- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:43:34 +08:00
starmen1265 c58a7d08b6 feat(10-01): implement proxy pool core functionality 
- ProxyEntry: tracks proxy state, failures, and recovery
- ProxyPoolConfig: YAML configuration model
- ProxyPool: manages multiple proxies with round-robin rotation
- from_yaml(): load from ~/.kb_proxies.yaml
- from_env_fallback(): backward compatible env var loading
- get_next_proxy(): round-robin with lazy recovery check

Implements PROXY-01 (multi-proxy config) and PROXY-02 (rotation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:39:39 +08:00
starmen1265 bcf1a88613 test(10-01): add failing tests for proxy pool module 
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults

RED phase: tests fail because proxy_pool module does not exist yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:38:07 +08:00
starmen1265 e896bd6765 docs(10): create phase plan for proxy pool 2026-03-18 16:27:33 +08:00
starmen1265 fe22f94358 docs(10): research proxy pool domain 2026-03-18 16:18:38 +08:00
starmen1265 1b58bf9364 docs(10): capture phase context for proxy pool 2026-03-18 16:06:31 +08:00
starmen1265 9d9de810b7 docs: create milestone v1.3 roadmap (5 phases) 2026-03-18 14:17:14 +08:00
starmen1265 887ced6075 docs: define v1.3 anti-crawler requirements 2026-03-18 13:07:46 +08:00
starmen1265 e4f96be0c4 docs: start milestone v1.3 Anti-Crawler Optimization 2026-03-18 11:59:39 +08:00