--- phase: 11-stealth-fingerprinting plan: 01 type: execute wave: 1 depends_on: [] files_modified: - kb_workflow/crawler/stealth/__init__.py - kb_workflow/crawler/stealth/stealth_config.py - tests/test_crawler/conftest.py - tests/test_crawler/test_stealth.py autonomous: true requirements: - STEALTH-01 - STEALTH-02 - STEALTH-03 - STEALTH-04 must_haves: truths: - "navigator.webdriver 属性被隐藏(返回 undefined/false)" - "Playwright 特有属性(__playwright, __pw_manual, __PW_inspect)被移除" - "window.chrome 对象被模拟为正常 Chrome 浏览器结构" - "User-Agent 不包含 HeadlessChrome 字符串" artifacts: - path: "kb_workflow/crawler/stealth/__init__.py" provides: "Stealth 模块入口,导出 StealthConfig 和 apply_stealth" min_lines: 10 - path: "kb_workflow/crawler/stealth/stealth_config.py" provides: "StealthConfig Pydantic 模型和 apply_stealth_async 函数" min_lines: 50 - path: "tests/test_crawler/test_stealth.py" provides: "Stealth 功能单元测试" min_lines: 100 key_links: - from: "kb_workflow/crawler/stealth/stealth_config.py" to: "playwright_stealth.Stealth" via: "from playwright_stealth import Stealth" pattern: "from playwright_stealth" --- 创建浏览器指纹伪装模块,使用 playwright-stealth 库隐藏 Playwright 自动化特征。 Purpose: 为 ZhihuCrawler 提供可复用的指纹伪装能力,绕过知乎的浏览器指纹检测 Output: kb_workflow/crawler/stealth/ 模块 + 单元测试 @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md @.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md @.planning/phases/11-stealth-fingerprinting/11-VALIDATION.md From kb_workflow/crawler/config.py: ```python class CrawlerConfig(BaseModel): proxy: Optional[ProxyConfig] = Field(None, description="Proxy configuration") proxy_required: bool = Field(default=False) min_delay: float = Field(default=1.0) max_delay: float = Field(default=3.0) ``` From kb_workflow/crawler/zhihu_crawler.py: ```python class ZhihuCrawler: def __init__( self, headless: bool = True, timeout: int = 30, config: Optional[CrawlerConfig] = None, proxy_pool: Optional[ProxyPool] = None ): ... def _start_browser_with_proxy(self, cookie: str) -> None: """Start browser with proxy configuration.""" self._playwright = sync_playwright().start() self._browser = self._playwright.chromium.launch(headless=self.headless) context_options = { 'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' } ... self._context = self._browser.new_context(**context_options) ``` From playwright_stealth (external library): ```python from playwright_stealth import Stealth stealth = Stealth( navigator_webdriver=True, navigator_user_agent=True, navigator_plugins=True, # ... other options ) stealth.apply_stealth_async(context) # for async stealth.apply_stealth_sync(context) # for sync ``` Task 1: 创建 StealthConfig 模块 kb_workflow/crawler/stealth/__init__.py, kb_workflow/crawler/stealth/stealth_config.py - Test 1: StealthConfig 默认值正确(stealth_enabled=True, navigator_webdriver=True 等) - Test 2: apply_stealth_sync 在 mock context 上正确调用 playwright_stealth - Test 3: get_default_stealth_config() 返回正确的默认配置 1. 创建 `kb_workflow/crawler/stealth/` 目录 2. 创建 `kb_workflow/crawler/stealth/__init__.py`: - 导出 StealthConfig 类 - 导出 apply_stealth_sync, apply_stealth_async 函数 - 导出 get_default_stealth_config 便捷函数 3. 创建 `kb_workflow/crawler/stealth/stealth_config.py`: ```python """Stealth configuration for browser fingerprint masking. Uses playwright-stealth library to hide Playwright automation signatures. """ from typing import Optional from playwright.sync_api import BrowserContext from playwright.async_api import BrowserContext as AsyncBrowserContext from pydantic import BaseModel, Field from playwright_stealth import Stealth class StealthConfig(BaseModel): """Configuration for browser stealth settings. All options map to playwright-stealth evasion scripts. Default values optimized for Zhihu anti-detection. """ stealth_enabled: bool = Field( default=True, description="Enable stealth mode (set False to disable)" ) # Navigator evasion options navigator_webdriver: bool = Field( default=True, description="Hide navigator.webdriver property (STEALTH-01)" ) navigator_user_agent: bool = Field( default=True, description="Replace HeadlessChrome in User-Agent (STEALTH-04)" ) navigator_plugins: bool = Field( default=True, description="Mock navigator.plugins array" ) navigator_languages: bool = Field( default=True, description="Set navigator.languages" ) navigator_platform: bool = Field( default=True, description="Set navigator.platform" ) navigator_vendor: bool = Field( default=True, description="Set navigator.vendor" ) navigator_permissions: bool = Field( default=True, description="Override navigator.permissions" ) # Chrome object evasion (STEALTH-03) chrome_app: bool = Field( default=True, description="Mock chrome.app object" ) chrome_csi: bool = Field( default=True, description="Mock chrome.csi function" ) chrome_load_times: bool = Field( default=True, description="Mock chrome.loadTimes function" ) # WebGL and rendering webgl_vendor: bool = Field( default=True, description="Override WebGL vendor/renderer strings" ) # Other evasion options hairline: bool = Field( default=True, description="Fix hairline detection" ) iframe_content_window: bool = Field( default=True, description="Fix iframe contentWindow" ) media_codecs: bool = Field( default=True, description="Mock media codecs" ) error_prototype: bool = Field( default=True, description="Fix error prototype chain" ) sec_ch_ua: bool = Field( default=True, description="Set sec-ch-ua headers" ) def create_stealth(self) -> Stealth: """Create playwright-stealth Stealth instance from config. Returns: Configured Stealth instance """ return Stealth( navigator_webdriver=self.navigator_webdriver, navigator_user_agent=self.navigator_user_agent, navigator_plugins=self.navigator_plugins, navigator_languages=self.navigator_languages, navigator_platform=self.navigator_platform, navigator_vendor=self.navigator_vendor, navigator_permissions=self.navigator_permissions, chrome_app=self.chrome_app, chrome_csi=self.chrome_csi, chrome_load_times=self.chrome_load_times, webgl_vendor=self.webgl_vendor, hairline=self.hairline, iframe_content_window=self.iframe_content_window, media_codecs=self.media_codecs, error_prototype=self.error_prototype, sec_ch_ua=self.sec_ch_ua, ) def apply_stealth_sync( context: BrowserContext, config: Optional[StealthConfig] = None ) -> None: """Apply stealth settings to browser context (synchronous). Args: context: Playwright BrowserContext to apply stealth to config: StealthConfig instance (uses defaults if not provided) """ if config is None: config = StealthConfig() if not config.stealth_enabled: return stealth = config.create_stealth() stealth.apply_stealth_sync(context) async def apply_stealth_async( context: AsyncBrowserContext, config: Optional[StealthConfig] = None ) -> None: """Apply stealth settings to browser context (asynchronous). Args: context: Playwright async BrowserContext to apply stealth to config: StealthConfig instance (uses defaults if not provided) """ if config is None: config = StealthConfig() if not config.stealth_enabled: return stealth = config.create_stealth() await stealth.apply_stealth_async(context) def get_default_stealth_config() -> StealthConfig: """Get default StealthConfig for Zhihu crawling. Returns: StealthConfig with all evasion options enabled """ return StealthConfig() ``` 4. 创建 `kb_workflow/crawler/stealth/__init__.py`: ```python """Stealth module for browser fingerprint masking.""" from .stealth_config import ( StealthConfig, apply_stealth_sync, apply_stealth_async, get_default_stealth_config, ) __all__ = [ "StealthConfig", "apply_stealth_sync", "apply_stealth_async", "get_default_stealth_config", ] ``` pytest tests/test_crawler/test_stealth.py::test_stealth_config_defaults -x - StealthConfig 类创建完成,包含所有 playwright-stealth 选项 - apply_stealth_sync/apply_stealth_async 函数可用 - 单元测试通过 Task 2: 创建 Stealth 功能单元测试 tests/test_crawler/test_stealth.py, tests/test_crawler/conftest.py - Test 1: test_stealth_config_defaults - 默认配置正确 - Test 2: test_stealth_config_disabled - stealth_enabled=False 时不应用伪装 - Test 3: test_apply_stealth_sync_calls_library - apply_stealth_sync 正确调用 playwright_stealth - Test 4: test_get_default_stealth_config - 返回默认配置实例 - Test 5: test_stealth_options_mapping - 配置选项正确映射到 Stealth 类 1. 在 `tests/test_crawler/conftest.py` 中添加 stealth_config fixture: ```python @pytest.fixture def stealth_config(): """Default StealthConfig for testing.""" from kb_workflow.crawler.stealth import StealthConfig return StealthConfig() @pytest.fixture def mock_browser_context(): """Mock Playwright BrowserContext for stealth testing.""" from unittest.mock import MagicMock context = MagicMock() context.add_init_script = MagicMock() return context ``` 2. 创建 `tests/test_crawler/test_stealth.py`: ```python """Tests for stealth fingerprint masking module.""" import pytest from unittest.mock import MagicMock, patch, call class TestStealthConfig: """Tests for StealthConfig class.""" def test_stealth_config_defaults(self): """Test that default StealthConfig has all options enabled.""" from kb_workflow.crawler.stealth import StealthConfig config = StealthConfig() assert config.stealth_enabled is True # STEALTH-01: navigator.webdriver hiding assert config.navigator_webdriver is True # STEALTH-04: User-Agent modification assert config.navigator_user_agent is True # STEALTH-03: Chrome properties assert config.chrome_app is True assert config.chrome_csi is True assert config.chrome_load_times is True # Other important options assert config.navigator_plugins is True assert config.webgl_vendor is True def test_stealth_config_disabled(self): """Test that stealth can be disabled.""" from kb_workflow.crawler.stealth import StealthConfig config = StealthConfig(stealth_enabled=False) assert config.stealth_enabled is False def test_create_stealth_returns_stealth_instance(self): """Test that create_stealth() returns configured Stealth.""" from kb_workflow.crawler.stealth import StealthConfig from playwright_stealth import Stealth config = StealthConfig() stealth = config.create_stealth() assert isinstance(stealth, Stealth) def test_stealth_options_mapping(self): """Test that StealthConfig options map correctly to Stealth.""" from kb_workflow.crawler.stealth import StealthConfig config = StealthConfig( navigator_webdriver=True, navigator_user_agent=False, # Disable one option chrome_app=True, ) stealth = config.create_stealth() # Verify the Stealth object was created with correct options # The Stealth class stores these as attributes assert stealth.navigator_webdriver is True assert stealth.navigator_user_agent is False class TestApplyStealth: """Tests for apply_stealth functions.""" def test_apply_stealth_sync_calls_library(self, mock_browser_context): """Test that apply_stealth_sync calls playwright-stealth correctly.""" from kb_workflow.crawler.stealth import apply_stealth_sync, StealthConfig config = StealthConfig() with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class: mock_stealth_instance = MagicMock() mock_stealth_class.return_value = mock_stealth_instance apply_stealth_sync(mock_browser_context, config) # Verify Stealth was created with config options mock_stealth_class.assert_called_once() # Verify apply_stealth_sync was called on the Stealth instance mock_stealth_instance.apply_stealth_sync.assert_called_once_with( mock_browser_context ) def test_apply_stealth_sync_disabled_skips_stealth(self, mock_browser_context): """Test that stealth is skipped when disabled.""" from kb_workflow.crawler.stealth import apply_stealth_sync, StealthConfig config = StealthConfig(stealth_enabled=False) with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class: apply_stealth_sync(mock_browser_context, config) # Stealth should not be created when disabled mock_stealth_class.assert_not_called() def test_apply_stealth_sync_default_config(self, mock_browser_context): """Test that apply_stealth_sync works with default config.""" from kb_workflow.crawler.stealth import apply_stealth_sync with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class: mock_stealth_instance = MagicMock() mock_stealth_class.return_value = mock_stealth_instance # Should work without explicit config apply_stealth_sync(mock_browser_context) mock_stealth_class.assert_called_once() class TestGetDefaultStealthConfig: """Tests for get_default_stealth_config function.""" def test_get_default_stealth_config(self): """Test that get_default_stealth_config returns StealthConfig.""" from kb_workflow.crawler.stealth import get_default_stealth_config config = get_default_stealth_config() assert config.stealth_enabled is True assert isinstance(config, type(config)) # Is a StealthConfig def test_get_default_stealth_config_is_fresh_instance(self): """Test that each call returns a new instance.""" from kb_workflow.crawler.stealth import get_default_stealth_config config1 = get_default_stealth_config() config2 = get_default_stealth_config() # Should be equal but not the same object assert config1 == config2 assert config1 is not config2 # Wave 0 completion marker WAVE_0_COMPLETE = True ``` pytest tests/test_crawler/test_stealth.py -x -v - test_stealth.py 创建完成,包含 5 个测试用例 - conftest.py 添加了 stealth_config 和 mock_browser_context fixtures - 所有测试通过 **Per-task verification:** - Task 1: `pytest tests/test_crawler/test_stealth.py::TestStealthConfig::test_stealth_config_defaults -x` - Task 2: `pytest tests/test_crawler/test_stealth.py -x -v` **Wave verification:** - All 5 tests in test_stealth.py pass - No regression in existing tests: `pytest tests/test_crawler/ -x` - [ ] kb_workflow/crawler/stealth/ 模块创建完成 - [ ] StealthConfig 类包含所有 playwright-stealth 选项 - [ ] apply_stealth_sync/apply_stealth_async 函数可用 - [ ] 5 个单元测试全部通过 - [ ] 无现有测试回归 After completion, create `.planning/phases/11-stealth-fingerprinting/11-01-SUMMARY.md`