starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
378 Commits 3 Branches 4 Tags
08cbe24d087c20d29311b436e969809dab9bebf2
Commit Graph

378 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
starmen1265 08cbe24d08 feat(11-02): integrate stealth fingerprint masking into ZhihuCrawler 
- Add stealth_config parameter to ZhihuCrawler.__init__
- Add --disable-blink-features=AutomationControlled to launch args
- Call apply_stealth_sync after browser context creation
- Maintain backward compatibility (stealth enabled by default)
- Add 8 integration tests covering stealth behavior and compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:53:22 +08:00
starmen1265 56c21f3f36 test(11-02): add failing tests for stealth integration in ZhihuCrawler 
- Test crawler accepts stealth_config parameter
- Test default stealth enabled behavior
- Test stealth can be disabled
- Test _start_browser_with_proxy calls apply_stealth_sync
- Test launch args include --disable-blink-features
- Test backward compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:52:23 +08:00
starmen1265 b9f74e5a7e docs(11-01): complete stealth fingerprint masking plan 
- Created 11-01-SUMMARY.md with execution results
- Updated STATE.md: Plan 1/2 complete, 50% progress
- Updated ROADMAP.md: 11-01 marked complete
- Updated REQUIREMENTS.md: STEALTH-01 to STEALTH-04 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:50:24 +08:00
starmen1265 e152b8ea05 feat(11-01): create stealth fingerprint masking module 
- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 23:47:50 +08:00
starmen1265 d6026660a9 docs(11): create phase 11 stealth fingerprinting plans 2026-03-18 23:41:43 +08:00
starmen1265 bfb6ab9c0a docs(11): add validation strategy 2026-03-18 23:35:58 +08:00
starmen1265 76da0f04d4 docs(11): research stealth fingerprinting domain 2026-03-18 23:34:59 +08:00
starmen1265 9e30e042a8 docs: create v1.4 Zhihu Anti-Crawler roadmap (1 phase, 2 plans) 2026-03-18 23:19:25 +08:00
starmen1265 eb90773422 docs: define v1.4 Zhihu Anti-Crawler requirements 2026-03-18 23:16:35 +08:00
starmen1265 a443cbde73 docs: start milestone v1.4 Zhihu Anti-Crawler 2026-03-18 23:09:06 +08:00
starmen1265 a9ead75da7 chore: complete v1.3 Proxy Pool milestone v1.3 2026-03-18 22:45:17 +08:00
starmen1265 d92b872b6c chore: remove phase 11 (Platform Adaptation) 2026-03-18 22:39:29 +08:00
starmen1265 d7f32b6b93 chore: remove phases 11-13 (Behavior Camouflage, CAPTCHA Detection, Error Recovery) 2026-03-18 22:35:48 +08:00
starmen1265 2561de6ad7 docs: update research for v1.3 anti-crawler optimization
PR Test / lint (pull_request) Successful in 2s
Details 
- Add architecture research for proxy pool, behavior camouflage,
  captcha detection, and error recovery
- Update features, pitfalls, and stack documentation
- Add phase 10 validation checklist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 20:45:14 +08:00
starmen1265 0c63503309 docs: resolve debug kb_workflow-module-conflict 2026-03-18 20:38:30 +08:00
starmen1265 c30b0bce88 docs(phase-10): complete proxy pool phase execution 2026-03-18 16:55:18 +08:00
starmen1265 bcfef48676 docs(10-02): complete health check and crawler integration plan 2026-03-18 16:51:16 +08:00
starmen1265 b66bcc08b1 feat(10-02): integrate ProxyPool into ZhihuCrawler 
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config

Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:48:48 +08:00
starmen1265 36a45095c1 test(10-02): add health check tests for proxy pool 
- test_fail_threshold: verify 3 consecutive failures mark proxy unavailable
- test_success_resets_count: verify success resets failure count
- test_cooldown_recovery: verify proxy recovers after cooldown
- test_cooldown_not_elapsed: verify proxy stays unavailable during cooldown
- test_all_unavailable_fallback: verify fallback to first proxy when all fail
- test_get_available_count: verify available proxy count tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:46:37 +08:00
starmen1265 96dfdce59c docs(10-01): complete proxy pool core plan 
- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:43:34 +08:00
starmen1265 c58a7d08b6 feat(10-01): implement proxy pool core functionality 
- ProxyEntry: tracks proxy state, failures, and recovery
- ProxyPoolConfig: YAML configuration model
- ProxyPool: manages multiple proxies with round-robin rotation
- from_yaml(): load from ~/.kb_proxies.yaml
- from_env_fallback(): backward compatible env var loading
- get_next_proxy(): round-robin with lazy recovery check

Implements PROXY-01 (multi-proxy config) and PROXY-02 (rotation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:39:39 +08:00
starmen1265 bcf1a88613 test(10-01): add failing tests for proxy pool module 
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults

RED phase: tests fail because proxy_pool module does not exist yet

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 16:38:07 +08:00
starmen1265 e896bd6765 docs(10): create phase plan for proxy pool 2026-03-18 16:27:33 +08:00
starmen1265 fe22f94358 docs(10): research proxy pool domain 2026-03-18 16:18:38 +08:00
starmen1265 1b58bf9364 docs(10): capture phase context for proxy pool 2026-03-18 16:06:31 +08:00
starmen1265 9d9de810b7 docs: create milestone v1.3 roadmap (5 phases) 2026-03-18 14:17:14 +08:00
starmen1265 887ced6075 docs: define v1.3 anti-crawler requirements 2026-03-18 13:07:46 +08:00
starmen1265 e4f96be0c4 docs: start milestone v1.3 Anti-Crawler Optimization 2026-03-18 11:59:39 +08:00
starmen1265 9c8ac34138 fix(kb_workflow): resolve module/package naming conflict
PR Test / lint (pull_request) Successful in 1s
Details 
Rename kb_workflow.py to cli.py to avoid conflict with kb_workflow/ package.
Add __main__.py to support `python -m kb_workflow` execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-18 10:35:33 +08:00
starmen1265 9d2433322a fix(crawler): set proxy_required=False by default
PR Test / lint (pull_request) Successful in 2s
Details 
旁路由 (192.168.1.200) provides automatic system-level proxy.
No explicit HTTP_PROXY/HTTPS_PROXY configuration needed.
 2026-03-18 00:04:48 +08:00
starmen1265 e1d3218a44 Merge branch 'fix/ci-deploy-reset' 2026-03-18 00:03:43 +08:00
starmen1265 c09c8588f0 fix(crawler): resolve circular import in zhihu_crawler 
Circular import chain:
  zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler

Fix: Lazy import get_settings() inside _get_cookie() method instead of
module-level import.
 2026-03-18 00:03:02 +08:00
starmen1265 d88f1a3b8d Merge pull request 'docs(skill): update kb_workflow SKILL.md to reflect current implementation' (#28) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/28
 2026-03-17 23:52:57 +08:00
starmen1265 6a819b6c80 docs(skill): update kb_workflow SKILL.md to reflect current implementation
PR Test / lint (pull_request) Successful in 2s
Details 
- Fix CLI path: kb-workflow -> kb_workflow
- Add login command documentation for Zhihu
- Update directory structure to match actual paths
- Add environment variable checks section
- Document strict failure handling principle
- Update technical architecture diagram
 2026-03-17 23:50:42 +08:00
starmen1265 5ccabec3e4 Merge pull request 'fix(ci): correct skill directory names in deploy.yml' (#27) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/27
 2026-03-17 23:41:51 +08:00
starmen1265 228cd1cf8c fix(ci): correct skill directory names in deploy.yml
PR Test / lint (pull_request) Successful in 2s
Details 
- Fix kb-workflow -> kb_workflow (underscore not hyphen)
- Remove non-existent skill directories from the list
- Only include skills that actually exist in the repo
 2026-03-17 23:40:12 +08:00
starmen1265 289be86fc6 Merge pull request 'fix/ci-deploy-reset' (#26) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Details 
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/26
 2026-03-17 23:28:17 +08:00
starmen1265 7744f7ac1d docs(phase-09): complete phase execution - pipeline integration verified
PR Test / lint (pull_request) Successful in 2s
Details
2026-03-17 23:25:08 +08:00
starmen1265 248a1f2b8c docs(09-01): complete ZhihuAdapter integration plan 
- Integrated ZhihuCrawler into ZhihuAdapter
- Added comment crawling and formatting
- Implemented cookie auto-login flow
- Added idempotent execution via IdempotencyChecker

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 23:17:39 +08:00
starmen1265 76f06c330b feat(09-01): integrate ZhihuCrawler into ZhihuAdapter 
- Replace ZhihuPlaywrightExtractor with ZhihuCrawler
- Add comment crawling with include_comments=True by default
- Implement cookie expiration detection and QR code login trigger
- Add idempotency check via IdempotencyChecker
- Add comment formatting with "## 热门评论" section
- Convert HTML content to Markdown using html2text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 23:14:22 +08:00
starmen1265 22ed60a5ab docs(09-00): complete Wave 0 test scaffolding plan 
- Added 5 mock fixtures for ZhihuCrawler integration testing
- Added 6 test skeletons (marked skip) for Wave 1 implementation
- Updated STATE.md with Phase 9 position
- Updated ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 23:02:40 +08:00
starmen1265 95218c18cd test(09-00): add Wave 0 test scaffolding for Pipeline integration 
- Add mock fixtures for ZhihuCrawler CrawlResult models:
  - mock_crawl_answer_result: simulates answer crawl with comments
  - mock_crawl_article_result: simulates zhuanlan article crawl
  - mock_comments_list: test data for comment formatting
  - mock_zhihu_login_result: simulates QR code login success
  - mock_crawl_failed_result: for error handling tests

- Add test skeletons (marked skip) for Wave 1 implementation:
  - test_fetch_metadata_answer
  - test_fetch_metadata_article
  - test_fetch_content_with_comments
  - test_cookie_expired_triggers_login
  - test_idempotent_execution
  - test_comment_formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 22:59:07 +08:00
starmen1265 bee696b96d docs(09): complete phase 9 planning with 2 plans in 2 waves 2026-03-17 22:54:48 +08:00
starmen1265 1c42443dce fix(09): revise plans based on checker feedback 2026-03-17 22:53:00 +08:00
starmen1265 ec18ea9197 docs(09): create phase 9 plans 2026-03-17 22:46:21 +08:00
starmen1265 45ba9c973f docs(09): research phase 9 domain 2026-03-17 22:37:38 +08:00
starmen1265 b3c7386098 docs(state): record phase 9 context session 2026-03-17 22:32:56 +08:00
starmen1265 e6b62f42e7 docs(09): capture phase context 2026-03-17 22:32:44 +08:00
starmen1265 007b0d70ef docs(08): complete phase 08 - Zhihu Crawler 
Phase 08: Crawler - All 3 plans complete

Verification: 7/7 must-haves passed
Tests: 44 passed

Requirements completed:
- CONT-01: Crawl Zhihu answers
- CONT-02: Crawl Zhihu articles
- CONT-03: Crawl comments (optional)
- CONT-04: Metadata extraction
- NET-01: Proxy configuration
- NET-02: Mandatory proxy
- NET-03: Rate limiting & retry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 22:23:56 +08:00
starmen1265 123e307522 docs(08-03): complete anti-crawl measures plan 
- Proxy configuration from HTTP_PROXY/HTTPS_PROXY
- Random delay (1-3s) between requests
- Exponential backoff retry (3 attempts)
- x-zse-96 signature handled by browser

All 44 crawler tests passing

Requirements: NET-01, NET-02, NET-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 22:20:38 +08:00