starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
503 Commits 3 Branches 4 Tags
main
Commit Graph

12 Commits

Author SHA1 Message Date
starmen1265 56c21f3f36 test(11-02): add failing tests for stealth integration in ZhihuCrawler 
- Test crawler accepts stealth_config parameter
- Test default stealth enabled behavior
- Test stealth can be disabled
- Test _start_browser_with_proxy calls apply_stealth_sync
- Test launch args include --disable-blink-features
- Test backward compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:52:23 +08:00
starmen1265 e152b8ea05 feat(11-01): create stealth fingerprint masking module 
- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:47:50 +08:00
starmen1265 b66bcc08b1 feat(10-02): integrate ProxyPool into ZhihuCrawler 
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:48:48 +08:00
starmen1265 36a45095c1 test(10-02): add health check tests for proxy pool 
- test_fail_threshold: verify 3 consecutive failures mark proxy unavailable
- test_success_resets_count: verify success resets failure count
- test_cooldown_recovery: verify proxy recovers after cooldown
- test_cooldown_not_elapsed: verify proxy stays unavailable during cooldown
- test_all_unavailable_fallback: verify fallback to first proxy when all fail
- test_get_available_count: verify available proxy count tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:46:37 +08:00
starmen1265 bcf1a88613 test(10-01): add failing tests for proxy pool module 
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults

RED phase: tests fail because proxy_pool module does not exist yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:38:07 +08:00
starmen1265 123e307522 docs(08-03): complete anti-crawl measures plan 
- Proxy configuration from HTTP_PROXY/HTTPS_PROXY
- Random delay (1-3s) between requests
- Exponential backoff retry (3 attempts)
- x-zse-96 signature handled by browser

All 44 crawler tests passing

Requirements: NET-01, NET-02, NET-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 22:20:38 +08:00
starmen1265 9e1452d2a9 feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
 2026-03-17 22:14:47 +08:00
starmen1265 626fb73a58 feat(08-03): create CrawlerConfig for proxy and rate limiting 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:57:09 +08:00
starmen1265 f41e75a1b2 feat(08-02): implement crawl_comments method with hot sorting 
- Add COMMENT_API_TEMPLATE for comment API URL
- Implement crawl_comments() returning CommentsResult
- Use order=hot for popular comments sorting
- Only fetch first-level comments (no nested replies)
- Add include_comments parameter to crawl_answer()
- Export CommentAuthor, CommentData, CommentsResult in __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:56:47 +08:00
starmen1265 8cc46965c1 test(08-02): add failing tests for comment crawling 
- Test crawl_comments returns CommentsResult
- Test comments use hot sorting (order=hot)
- Test comment has required fields (content, author, like_count)
- Test no nested replies (first-level only)
- Test include_comments parameter control

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:54:42 +08:00
starmen1265 912773fc65 test(08-01): add failing tests for ZhihuCrawler 
- Add test_answer_crawler.py with tests for crawl_answer()
- Add test_article_crawler.py with tests for crawl_article()
- Add test_metadata_extraction.py with tests for author/metadata parsing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:08:13 +08:00
starmen1265 a102a75b28 feat(08-01): create crawler data models and test scaffolding 
- Add Pydantic models: AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
- Create test fixtures in conftest.py with mock browser and sample API responses
- Set up crawler module with __init__.py exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:06:33 +08:00