- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add architecture research for proxy pool, behavior camouflage,
captcha detection, and error recovery
- Update features, pitfalls, and stack documentation
- Add phase 10 validation checklist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config
Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults
RED phase: tests fail because proxy_pool module does not exist yet
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename kb_workflow.py to cli.py to avoid conflict with kb_workflow/ package.
Add __main__.py to support `python -m kb_workflow` execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix kb-workflow -> kb_workflow (underscore not hyphen)
- Remove non-existent skill directories from the list
- Only include skills that actually exist in the repo
- Replace ZhihuPlaywrightExtractor with ZhihuCrawler
- Add comment crawling with include_comments=True by default
- Implement cookie expiration detection and QR code login trigger
- Add idempotency check via IdempotencyChecker
- Add comment formatting with "## 热门评论" section
- Convert HTML content to Markdown using html2text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Added 5 mock fixtures for ZhihuCrawler integration testing
- Added 6 test skeletons (marked skip) for Wave 1 implementation
- Updated STATE.md with Phase 9 position
- Updated ROADMAP.md with plan progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
- Add 08-02-SUMMARY.md with execution details
- Update STATE.md with decisions and metrics
- Mark CONT-03 requirement complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
starmen1265