- Test crawler accepts stealth_config parameter
- Test default stealth enabled behavior
- Test stealth can be disabled
- Test _start_browser_with_proxy calls apply_stealth_sync
- Test launch args include --disable-blink-features
- Test backward compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add StealthConfig Pydantic model with all playwright-stealth options
- Add apply_stealth_sync/apply_stealth_async functions for browser context
- Add get_default_stealth_config convenience function
- Add 9 unit tests covering config defaults, stealth application, and disabled mode
- Update conftest.py with stealth_config and mock_browser_context fixtures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add proxy_pool parameter to ZhihuCrawler constructor
- Add _current_proxy tracking for health check feedback
- Implement _parse_proxy_url static method for URL parsing
- Implement _on_request_success/failure for proxy health feedback
- Prioritize proxy_pool over config.proxy for multi-proxy support
- Maintain backward compatibility with single proxy config
Tests:
- test_crawler_accepts_proxy_pool
- test_crawler_without_proxy_pool (backward compatibility)
- test_parse_proxy_url (HTTP, auth, SOCKS5)
- test_proxy_pool_takes_priority_over_config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test load from YAML file with multiple proxies
- Test load from environment variable (backward compatibility)
- Test round-robin rotation
- Test empty pool handling
- Test single proxy handling
- Test ProxyEntry defaults
RED phase: tests fail because proxy_pool module does not exist yet
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace ZhihuPlaywrightExtractor with ZhihuCrawler
- Add comment crawling with include_comments=True by default
- Implement cookie expiration detection and QR code login trigger
- Add idempotency check via IdempotencyChecker
- Add comment formatting with "## 热门评论" section
- Convert HTML content to Markdown using html2text
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Integrate proxy configuration via _start_browser_with_proxy()
- Add random delay via _apply_random_delay()
- Add retry logic via _crawl_with_retry using tenacity
- Add ZhihuApiInterceptor tests for x-zse-96 handling
- All tests pass
- Add CrawlerConfig with ProxyConfig, delay, and retry settings
- Support HTTP_PROXY/HTTPS_PROXY environment variable parsing
- Add proxy_required validation with clear error message
- Add get_crawler_config singleton for configuration access
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add COMMENT_API_TEMPLATE for comment API URL
- Implement crawl_comments() returning CommentsResult
- Use order=hot for popular comments sorting
- Only fetch first-level comments (no nested replies)
- Add include_comments parameter to crawl_answer()
- Export CommentAuthor, CommentData, CommentsResult in __init__.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test crawl_comments returns CommentsResult
- Test comments use hot sorting (order=hot)
- Test comment has required fields (content, author, like_count)
- Test no nested replies (first-level only)
- Test include_comments parameter control
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test_answer_crawler.py with tests for crawl_answer()
- Add test_article_crawler.py with tests for crawl_article()
- Add test_metadata_extraction.py with tests for author/metadata parsing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Pydantic models: AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
- Create test fixtures in conftest.py with mock browser and sample API responses
- Set up crawler module with __init__.py exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add login_command function with QR code login, cookie import, and status check
- Add login subcommand with --import-cookie and --status options
- Update CLI help text with login examples
- Fix mock paths in tests to match actual import locations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test validate_cookie_valid - valid cookie returns user info
- Test validate_cookie_expired - expired cookie returns False
- Test validate_cookie_empty - empty cookie returns False
- Test validate_cookie_malformed - network error handling
- Test validate_cookie_other_status_code - non-200/401 handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase - implementation passes all 8 tests:
- ZhihuLoginResult: Pydantic model for login results
- ZhihuLogin: Main class with QR code login flow
- LOGIN_URL: https://www.zhihu.com/signin
- CANVAS_SELECTORS: Multiple fallback selectors for QR canvas
- _find_qrcode_canvas(): Find canvas with fallback selectors
- _save_qrcode_image(): Save QR code to intermediate directory
- check_login_cookie(): Detect z_c0 cookie for login state
- get_cookies_str(): Extract cookie string from context
- _wait_for_login(): Poll with configurable timeout (tenacity)
- login_by_qrcode(): Main entry point for login flow
Key design decisions:
- Uses tenacity Retrying for configurable timeout
- QR code saved to intermediate_path with timestamp
- Multiple canvas selectors for robustness
- Proper browser lifecycle management
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD RED phase - tests expect ZhihuLogin class that doesn't exist yet:
- test_login_page_access: verify correct login URL
- test_qrcode_screenshot: verify QR code capture and save
- test_cookie_detection_with_z_c0: verify login detection
- test_cookie_detection_without_z_c0: verify no login when missing z_c0
- test_cookie_extraction: verify cookie string format
- test_login_timeout: verify timeout after 60s polling
- test_result_model_success: verify success result model
- test_result_model_failure: verify failure result model
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test save_cookie creates env file
- Test save_cookie updates existing line
- Test save_cookie appends new line
- Test save_cookie preserves other lines
- Test save_cookie sets file permissions to 600
- Test load_cookie returns saved value
- Test load_cookie returns empty if file not exists
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create tests/test_auth/conftest.py with pytest fixtures
- mock_browser: Mock PlaywrightBrowser for unit testing
- mock_canvas: Mock canvas element for QR code screenshot
- sample_cookies: Test cookie data with z_c0 login cookie
- sample_cookies_no_login: Test cookies without login state
- mock_env_file: Temporary .kb_env file fixture
- intermediate_path: Temporary intermediate directory fixture
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test parse_json_cookies for Chrome Cookie Editor format
- Test domain filtering for zhihu.com
- Test empty array handling
- Test malformed JSON error handling
- Test file not found error handling
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix download processor to return actual author name (e.g. "HOLA小测佬")
instead of B站用户ID (mid, e.g. "406636263")
- Add 5 test cases for author extraction logic
- Resolves report display issue where UP主名称 showed numeric ID
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove sys.path manipulation from test file
- Use standard package import from kb_workflow.metadata
- Test file now follows the same import pattern as production code
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove sys.path manipulation from kb_workflow.py
- Update CLI imports to use kb_workflow.pipeline prefix
- Update all test file imports and patch paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove kb-bilibili-metadata/ and kb-zhihu-metadata/ directories
- Create kb_workflow/__init__.py to export main and process_command
- Update test_cli/test_process_command.py imports for new package structure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix test_workflow_skips_completed_steps to mock _ask_rerun_step
- Fix test_fetch_video_metadata_success to use AsyncMock
- Document other pre-existing test issues in deferred-items.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test pipeline includes archive step
- Test archive uses platform directory structure
- Test no date-based subdirectories in archive
- Update default steps assertion to include archive
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test archive copies to correct platform directory
- Test filename format {YYYYMMDD}_{content_id}_analysis.md
- Test skip when step already completed
- Test detect existing archive by content_id
- Test handle missing analysis file with error
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Zhihu HTML structure includes noscript fallback images:
<figure><img data-actualsrc=...><noscript><img src=...></noscript></figure>
After fixing lazy-load (replacing src with data-actualsrc), both imgs
have real URLs, causing duplicates. The fix removes noscript tags before
processing img elements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add content_dir field to ASRInput schema
- Pass content_dir from pipeline when creating ASRInput
- Modify _try_official_subtitle to use content_dir for output
- Modify _run_whisper_basic to use content_dir for output
- Add validation for empty video_path in Whisper fallback
Root cause: DownloadProcessor sets content_file="" for video types,
causing Path("").parent to resolve to current directory instead of
content directory. Subtitle files were written to wrong location.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test full workflow execution from URL to analysis
- Test workflow error handling
- Test state persistence
- Test workflow resumability (skip completed steps)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test process --help shows usage
- Test process <url> calls Pipeline.run()
- Test invalid URL returns error
- Test --output-dir flag
- Test --no-skip flag
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test PipelineInput validates url field (required)
- Test PipelineOutput contains results dict
- Test PipelineConfig has default steps list
- Test PipelineConfig allows custom output_dir
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AnalyzeProcessor wraps existing DeepAnalyzer for pipeline integration
- Loads metadata and content from JSON files
- Generates analysis reports as Markdown with frontmatter
- Records state on success and failure
- Supports both video and article content types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test schema validation for AnalyzeInput (metadata_file, content_file, content_type, output_file)
- Test schema validation for AnalyzeOutput (analysis_file, summary, main_topics, key_insights, tags, stats)
- Test AnalyzeProcessor integration with DeepAnalyzer
- Test state recording on success and failure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor downloads article/answer/zhuanlan content
- Video content is skipped (handled by ASR in Phase 3)
- Content files saved with correct naming convention
- DownloadProcessor exported from kb_pipeline.processors
- All 17 tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD GREEN phase:
- DownloadProcessor inherits from BaseProcessor
- Fetches metadata from Bilibili and Zhihu adapters
- Saves metadata to content_dir as JSON
- Records state on success/failure
- Skips step if already completed (idempotency)
- All metadata tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD RED phase:
- Test DownloadProcessor fetches Bilibili video metadata correctly
- Test DownloadProcessor fetches Zhihu answer metadata correctly
- Test DownloadProcessor records state on success
- Test DownloadProcessor records state on failure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TDD RED phase:
- Test DownloadInput validates required fields (content_dir, content_type)
- Test DownloadOutput validates required fields (metadata_file, content_file, title, author)
- Test DownloadOutput handles both video and article content types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseProcessor wraps LinkParser for URL parsing
- Returns ParseOutput with platform, content_type, content_id
- Records state when state_manager is provided
- Handles unknown URLs with success=False
- Export BaseProcessor, ParseProcessor, ASRProcessor from __init__.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ParseInput extends StepInput with url field
- ParseOutput extends StepOutput with platform, content_type, content_id, resolved_url, original_url
- Tests for schema validation and inheritance
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test BaseProcessor cannot be instantiated directly
- Test subclass must implement process() method
- Test state_manager attribute is provided
- Test STEP_NAME class attribute requirement
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix test_state_recording_failure to use correct mock pattern
- _try_official_subtitle returning None triggers _Whisper` fallback
- Ensure proper exception propagation for accurate failure recording
- All 22 tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ASRInput schema with video_path and content_type fields
- Add ASROutput schema with tool/has_subtitle/output_files/segments/text/duration_seconds/speakers
- Create test_processors directory with conftest fixtures
- Add comprehensive schema tests (8 test cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test answer content extraction
- Test zhuanlan (article) content extraction
- Test cookie-based authentication
- Test login page detection
- Test AuthError handling for missing/expired cookies
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test for article content returning Markdown format
- Add test for YAML frontmatter with required fields
- Add test for video content returning empty (ASR handling)
- Add test for unsupported content type error
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add test for abstract class instantiation prevention
- Add test for incomplete implementation detection
- Add test for complete implementation instantiation
- Add test for async method signatures
- Add shared fixtures for platform tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
starmen1265