starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix/ci-deploy-reset #26

Merged
starmen1265 merged 52 commits from fix/ci-deploy-reset into main 2026-03-17 23:28:19 +08:00
Conversation 0 Commits 52 Files Changed 63
+10618 -297
starmen1265 commented 2026-03-17 23:28:06 +08:00
Owner
Copy Link
Copy Source
No description provided.
starmen1265 added 52 commits 2026-03-17 23:28:07 +08:00
docs: start milestone v1.2 Zhihu Crawler 9077a783e8
docs: define milestone v1.2 requirements dcba34b643
docs: create milestone v1.2 roadmap (3 phases) 367b2816c4
docs(07): capture phase context 87c7e50c4f
docs(state): record phase 7 context session 58487b0fa3
docs(07): research phase domain - login auth system 1d1f3309b2
docs(07): create phase 7 login auth plans 14434df4eb
docs(07): add phase plans (3 plans, 2 waves) d6a8690bc7
test(07-02): add failing tests for Cookie JSON parsing af3536aa92 
- Test parse_json_cookies for Chrome Cookie Editor format
- Test domain filtering for zhihu.com
- Test empty array handling
- Test malformed JSON error handling
- Test file not found error handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(07-02): implement parse_json_cookies function 2360fcc99e 
- Parse Chrome Cookie Editor JSON format
- Filter cookies by zhihu.com domain
- Handle malformed JSON with ValueError
- Handle file not found with ValueError
- Return cookie string in "name=value; name2=value2" format

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(07-01): add auth test fixtures f90de6b8a6 
- Create tests/test_auth/conftest.py with pytest fixtures
- mock_browser: Mock PlaywrightBrowser for unit testing
- mock_canvas: Mock canvas element for QR code screenshot
- sample_cookies: Test cookie data with z_c0 login cookie
- sample_cookies_no_login: Test cookies without login state
- mock_env_file: Temporary .kb_env file fixture
- intermediate_path: Temporary intermediate directory fixture

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(07-02): add failing tests for Cookie persistence 19c377ef65 
- Test save_cookie creates env file
- Test save_cookie updates existing line
- Test save_cookie appends new line
- Test save_cookie preserves other lines
- Test save_cookie sets file permissions to 600
- Test load_cookie returns saved value
- Test load_cookie returns empty if file not exists

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(07-01): add failing tests for ZhihuLogin QR code login fb9bc2d1ed 
TDD RED phase - tests expect ZhihuLogin class that doesn't exist yet:

- test_login_page_access: verify correct login URL
- test_qrcode_screenshot: verify QR code capture and save
- test_cookie_detection_with_z_c0: verify login detection
- test_cookie_detection_without_z_c0: verify no login when missing z_c0
- test_cookie_extraction: verify cookie string format
- test_login_timeout: verify timeout after 60s polling
- test_result_model_success: verify success result model
- test_result_model_failure: verify failure result model

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(07-02): implement CookieManager class 00245c58cc 
- CookieManager saves cookies to ~/.kb_env file
- Updates existing ZHIHU_COOKIE line or appends new
- Preserves other configuration lines
- Sets file permissions to 600 for security
- Loads cookies from env file
- Clears get_settings cache after saving

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(07-02): complete cookie import & persistence plan 771ca554b3
feat(07-01): implement ZhihuLogin QR code login class 072d83da35 
TDD GREEN phase - implementation passes all 8 tests:

- ZhihuLoginResult: Pydantic model for login results
- ZhihuLogin: Main class with QR code login flow
  - LOGIN_URL: https://www.zhihu.com/signin
  - CANVAS_SELECTORS: Multiple fallback selectors for QR canvas
  - _find_qrcode_canvas(): Find canvas with fallback selectors
  - _save_qrcode_image(): Save QR code to intermediate directory
  - check_login_cookie(): Detect z_c0 cookie for login state
  - get_cookies_str(): Extract cookie string from context
  - _wait_for_login(): Poll with configurable timeout (tenacity)
  - login_by_qrcode(): Main entry point for login flow

Key design decisions:
- Uses tenacity Retrying for configurable timeout
- QR code saved to intermediate_path with timestamp
- Multiple canvas selectors for robustness
- Proper browser lifecycle management

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(07-02): add self-check verification to summary 799bf09acc 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(07-01): complete QR Code Login Core plan 4228b1e3a1
test(07-03): add failing tests for cookie validation 32bb22008b 
- Test validate_cookie_valid - valid cookie returns user info
- Test validate_cookie_expired - expired cookie returns False
- Test validate_cookie_empty - empty cookie returns False
- Test validate_cookie_malformed - network error handling
- Test validate_cookie_other_status_code - non-200/401 handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(07-03): implement cookie validation functionality 919a4673e6 
- Add CookieValidationResult Pydantic model with valid, username, user_id, reason fields
- Add validate_cookie async method to ZhihuLogin class
- Validate cookie via Zhihu /api/v4/me endpoint
- Handle 200, 401, and other status codes appropriately
- Handle network errors gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(07-03): add failing tests for login CLI subcommand 94c85baac9 
- Test login_default_mode - default QR code login flow
- Test login_import_cookie - --import-cookie JSON import
- Test login_status - --status cookie validation
- Test login_integration - successful login saves cookie
- Test login_error_handling - exception handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(07-03): implement login CLI subcommand d9861aa6c3 
- Add login_command function with QR code login, cookie import, and status check
- Add login subcommand with --import-cookie and --status options
- Update CLI help text with login examples
- Fix mock paths in tests to match actual import locations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(07-03): complete Login CLI Subcommand plan e8eaca519f 
- Cookie validation via /api/v4/me endpoint
- login CLI subcommand with 3 modes: scan, import, status
- AUTH-04 requirement complete: Cookie expiry detection
- Phase 7 Login Auth complete, ready for Phase 8 Crawler

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(08): capture phase context for crawler implementation 87adf8aca6
docs(state): record phase 08 context session ba33e0e518
docs(08): research crawler implementation domain 1464c9b707
docs(08): add validation strategy c93237abde
docs(08): create phase 8 plans for crawler implementation 6d2d6b26e9
fix(08): resolve Wave 1 file conflict - move 08-02 to Wave 2 334aac87c7
feat(08-01): create crawler data models and test scaffolding a102a75b28 
- Add Pydantic models: AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
- Create test fixtures in conftest.py with mock browser and sample API responses
- Set up crawler module with __init__.py exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(08-01): add failing tests for ZhihuCrawler 912773fc65 
- Add test_answer_crawler.py with tests for crawl_answer()
- Add test_article_crawler.py with tests for crawl_article()
- Add test_metadata_extraction.py with tests for author/metadata parsing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(08-01): implement ZhihuCrawler with API interception aa96bc963c 
- Add ZhihuCrawler class with crawl_answer() and crawl_article() methods
- Implement ZhihuApiInterceptor for capturing /api/v4/ responses
- Add _parse_author(), _parse_metadata(), _parse_answer_data(), _parse_article_data() helpers
- Support DOM parsing fallback for zhuanlan articles
- Export ZhihuCrawler and ZhihuApiInterceptor from __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(08-01): complete zhihu content crawler plan f0ee4ca5e2 
- ZhihuCrawler class with API interception
- AnswerData, ArticleData models
- 15 tests passing

Requirements: CONT-01, CONT-02, CONT-04 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(08-02): add CommentData models for comment crawling b1f57922da 
- Add CommentAuthor model for comment author info
- Add CommentData model with content, author, likes, child_count
- Add CommentsResult model for crawl operation result
- Update CrawlResult to include optional comments field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test(08-02): add failing tests for comment crawling 8cc46965c1 
- Test crawl_comments returns CommentsResult
- Test comments use hot sorting (order=hot)
- Test comment has required fields (content, author, like_count)
- Test no nested replies (first-level only)
- Test include_comments parameter control

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(08-02): implement crawl_comments method with hot sorting f41e75a1b2 
- Add COMMENT_API_TEMPLATE for comment API URL
- Implement crawl_comments() returning CommentsResult
- Use order=hot for popular comments sorting
- Only fetch first-level comments (no nested replies)
- Add include_comments parameter to crawl_answer()
- Export CommentAuthor, CommentData, CommentsResult in __init__.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(08-03): create CrawlerConfig for proxy and rate limiting 626fb73a58 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(08-02): complete comment crawler plan 6a4f13b2f6 
- Add 08-02-SUMMARY.md with execution details
- Update STATE.md with decisions and metrics
- Mark CONT-03 requirement complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler 9e1452d2a9 
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
docs(08-03): complete anti-crawl measures plan 123e307522 
- Proxy configuration from HTTP_PROXY/HTTPS_PROXY
- Random delay (1-3s) between requests
- Exponential backoff retry (3 attempts)
- x-zse-96 signature handled by browser

All 44 crawler tests passing

Requirements: NET-01, NET-02, NET-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(08): complete phase 08 - Zhihu Crawler 007b0d70ef 
Phase 08: Crawler - All 3 plans complete

Verification: 7/7 must-haves passed
Tests: 44 passed

Requirements completed:
- CONT-01: Crawl Zhihu answers
- CONT-02: Crawl Zhihu articles
- CONT-03: Crawl comments (optional)
- CONT-04: Metadata extraction
- NET-01: Proxy configuration
- NET-02: Mandatory proxy
- NET-03: Rate limiting & retry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(09): capture phase context e6b62f42e7
docs(state): record phase 9 context session b3c7386098
docs(09): research phase 9 domain 45ba9c973f
docs(09): create phase 9 plans ec18ea9197
fix(09): revise plans based on checker feedback 1c42443dce
docs(09): complete phase 9 planning with 2 plans in 2 waves bee696b96d
test(09-00): add Wave 0 test scaffolding for Pipeline integration 95218c18cd 
- Add mock fixtures for ZhihuCrawler CrawlResult models:
  - mock_crawl_answer_result: simulates answer crawl with comments
  - mock_crawl_article_result: simulates zhuanlan article crawl
  - mock_comments_list: test data for comment formatting
  - mock_zhihu_login_result: simulates QR code login success
  - mock_crawl_failed_result: for error handling tests

- Add test skeletons (marked skip) for Wave 1 implementation:
  - test_fetch_metadata_answer
  - test_fetch_metadata_article
  - test_fetch_content_with_comments
  - test_cookie_expired_triggers_login
  - test_idempotent_execution
  - test_comment_formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(09-00): complete Wave 0 test scaffolding plan 22ed60a5ab 
- Added 5 mock fixtures for ZhihuCrawler integration testing
- Added 6 test skeletons (marked skip) for Wave 1 implementation
- Updated STATE.md with Phase 9 position
- Updated ROADMAP.md with plan progress

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(09-01): integrate ZhihuCrawler into ZhihuAdapter 76f06c330b 
- Replace ZhihuPlaywrightExtractor with ZhihuCrawler
- Add comment crawling with include_comments=True by default
- Implement cookie expiration detection and QR code login trigger
- Add idempotency check via IdempotencyChecker
- Add comment formatting with "## 热门评论" section
- Convert HTML content to Markdown using html2text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(09-01): complete ZhihuAdapter integration plan 248a1f2b8c 
- Integrated ZhihuCrawler into ZhihuAdapter
- Added comment crawling and formatting
- Implemented cookie auto-login flow
- Added idempotent execution via IdempotencyChecker

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs(phase-09): complete phase execution - pipeline integration verified
PR Test / lint (pull_request) Successful in 2s
Details
7744f7ac1d
starmen1265 merged commit 289be86fc6 into main 2026-03-17 23:28:19 +08:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
starmen1265
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starmen1265/kb-skills#26
Delete Branch "fix/ci-deploy-reset"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?