fix/crawler-proxy-config #31

2026-03-18T20:45:53+08:00

starmen1265 commented

2026-03-18 20:45:53 +08:00

No description provided.

starmen1265 added 15 commits 2026-03-18 20:45:55 +08:00

docs: start milestone v1.3 Anti-Crawler Optimization e4f96be0c4

docs: define v1.3 anti-crawler requirements 887ced6075

docs: create milestone v1.3 roadmap (5 phases) 9d9de810b7

docs(10): capture phase context for proxy pool 1b58bf9364

docs(10): research proxy pool domain fe22f94358

docs(10): create phase plan for proxy pool e896bd6765

test(10-01): add failing tests for proxy pool module bcf1a88613

- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults

RED phase: tests fail because proxy_pool module does not exist yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat(10-01): implement proxy pool core functionality c58a7d08b6

- ProxyEntry: tracks proxy state, failures, and recovery
- ProxyPoolConfig: YAML configuration model
- ProxyPool: manages multiple proxies with round-robin rotation
- from_yaml(): load from ~/.kb_proxies.yaml
- from_env_fallback(): backward compatible env var loading
- get_next_proxy(): round-robin with lazy recovery check

Implements PROXY-01 (multi-proxy config) and PROXY-02 (rotation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs(10-01): complete proxy pool core plan 96dfdce59c

- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

test(10-02): add health check tests for proxy pool 36a45095c1

- test_fail_threshold: verify 3 consecutive failures mark proxy unavailable
- test_success_resets_count: verify success resets failure count
- test_cooldown_recovery: verify proxy recovers after cooldown
- test_cooldown_not_elapsed: verify proxy stays unavailable during cooldown
- test_all_unavailable_fallback: verify fallback to first proxy when all fail
- test_get_available_count: verify available proxy count tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat(10-02): integrate ProxyPool into ZhihuCrawler b66bcc08b1

- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs(10-02): complete health check and crawler integration plan bcfef48676

docs(phase-10): complete proxy pool phase execution c30b0bce88

docs: resolve debug kb_workflow-module-conflict 0c63503309

docs: update research for v1.3 anti-crawler optimization

PR Test / lint (pull_request) Successful in 2s

Details

2561de6ad7

- Add architecture research for proxy pool, behavior camouflage,
  captcha detection, and error recovery
- Update features, pitfalls, and stack documentation
- Add phase 10 validation checklist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

starmen1265 merged commit 22fd634498 into main

2026-03-18 20:48:33 +08:00

starmen1265 referenced this issue from a commit

2026-03-18 20:48:34 +08:00

Merge pull request 'fix/crawler-proxy-config' (#31) from fix/crawler-proxy-config into main

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starmen1265/kb-skills#31