starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
328 Commits 3 Branches 4 Tags
c09c8588f0a3dd207071e41b2fffc0f7a144975b
Commit Graph

7 Commits

Author SHA1 Message Date
starmen1265 c09c8588f0 fix(crawler): resolve circular import in zhihu_crawler 
Circular import chain:
  zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler

Fix: Lazy import get_settings() inside _get_cookie() method instead of
module-level import.
 2026-03-18 00:03:02 +08:00
starmen1265 9e1452d2a9 feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
 2026-03-17 22:14:47 +08:00
starmen1265 626fb73a58 feat(08-03): create CrawlerConfig for proxy and rate limiting 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:57:09 +08:00
starmen1265 f41e75a1b2 feat(08-02): implement crawl_comments method with hot sorting 
- Add COMMENT_API_TEMPLATE for comment API URL
- Implement crawl_comments() returning CommentsResult
- Use order=hot for popular comments sorting
- Only fetch first-level comments (no nested replies)
- Add include_comments parameter to crawl_answer()
- Export CommentAuthor, CommentData, CommentsResult in __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:56:47 +08:00
starmen1265 b1f57922da feat(08-02): add CommentData models for comment crawling 
- Add CommentAuthor model for comment author info
- Add CommentData model with content, author, likes, child_count
- Add CommentsResult model for crawl operation result
- Update CrawlResult to include optional comments field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:53:17 +08:00
starmen1265 aa96bc963c feat(08-01): implement ZhihuCrawler with API interception 
- Add ZhihuCrawler class with crawl_answer() and crawl_article() methods
- Implement ZhihuApiInterceptor for capturing /api/v4/ responses
- Add _parse_author(), _parse_metadata(), _parse_answer_data(), _parse_article_data() helpers
- Support DOM parsing fallback for zhuanlan articles
- Export ZhihuCrawler and ZhihuApiInterceptor from __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:10:10 +08:00
starmen1265 a102a75b28 feat(08-01): create crawler data models and test scaffolding 
- Add Pydantic models: AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
- Create test fixtures in conftest.py with mock browser and sample API responses
- Set up crawler module with __init__.py exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 21:06:33 +08:00