starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
378 Commits 3 Branches 4 Tags
08cbe24d087c20d29311b436e969809dab9bebf2
Commit Graph

12 Commits

Author SHA1 Message Date
starmen1265 08cbe24d08 feat(11-02): integrate stealth fingerprint masking into ZhihuCrawler 
- Add stealth_config parameter to ZhihuCrawler.__init__
- Add --disable-blink-features=AutomationControlled to launch args
- Call apply_stealth_sync after browser context creation
- Maintain backward compatibility (stealth enabled by default)
- Add 8 integration tests covering stealth behavior and compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:53:22 +08:00
starmen1265 e152b8ea05 feat(11-01): create stealth fingerprint masking module 
- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:47:50 +08:00
starmen1265 b66bcc08b1 feat(10-02): integrate ProxyPool into ZhihuCrawler 
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:48:48 +08:00
starmen1265 c58a7d08b6 feat(10-01): implement proxy pool core functionality 
- ProxyEntry: tracks proxy state, failures, and recovery
- ProxyPoolConfig: YAML configuration model
- ProxyPool: manages multiple proxies with round-robin rotation
- from_yaml(): load from ~/.kb_proxies.yaml
- from_env_fallback(): backward compatible env var loading
- get_next_proxy(): round-robin with lazy recovery check

Implements PROXY-01 (multi-proxy config) and PROXY-02 (rotation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:39:39 +08:00
starmen1265 9d2433322a fix(crawler): set proxy_required=False by default
PR Test / lint (pull_request) Successful in 2s
Details 
旁路由 (192.168.1.200) provides automatic system-level proxy.
No explicit HTTP_PROXY/HTTPS_PROXY configuration needed.
 2026-03-18 00:04:48 +08:00
starmen1265 c09c8588f0 fix(crawler): resolve circular import in zhihu_crawler 
Circular import chain:
  zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler

Fix: Lazy import get_settings() inside _get_cookie() method instead of
module-level import.
 2026-03-18 00:03:02 +08:00
starmen1265 9e1452d2a9 feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
 2026-03-17 22:14:47 +08:00
starmen1265 626fb73a58 feat(08-03): create CrawlerConfig for proxy and rate limiting 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:57:09 +08:00
starmen1265 f41e75a1b2 feat(08-02): implement crawl_comments method with hot sorting 
- Add COMMENT_API_TEMPLATE for comment API URL
- Implement crawl_comments() returning CommentsResult
- Use order=hot for popular comments sorting
- Only fetch first-level comments (no nested replies)
- Add include_comments parameter to crawl_answer()
- Export CommentAuthor, CommentData, CommentsResult in __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:56:47 +08:00
starmen1265 b1f57922da feat(08-02): add CommentData models for comment crawling 
- Add CommentAuthor model for comment author info
- Add CommentData model with content, author, likes, child_count
- Add CommentsResult model for crawl operation result
- Update CrawlResult to include optional comments field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:53:17 +08:00
starmen1265 aa96bc963c feat(08-01): implement ZhihuCrawler with API interception 
- Add ZhihuCrawler class with crawl_answer() and crawl_article() methods
- Implement ZhihuApiInterceptor for capturing /api/v4/ responses
- Add _parse_author(), _parse_metadata(), _parse_answer_data(), _parse_article_data() helpers
- Support DOM parsing fallback for zhuanlan articles
- Export ZhihuCrawler and ZhihuApiInterceptor from __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:10:10 +08:00
starmen1265 a102a75b28 feat(08-01): create crawler data models and test scaffolding 
- Add Pydantic models: AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
- Create test fixtures in conftest.py with mock browser and sample API responses
- Set up crawler module with __init__.py exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:06:33 +08:00