This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
kb-skills/.planning/phases/11-stealth-fingerprinting/11-01-PLAN.md
T

537 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
phase: 11-stealth-fingerprinting
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- kb_workflow/crawler/stealth/__init__.py
- kb_workflow/crawler/stealth/stealth_config.py
- tests/test_crawler/conftest.py
- tests/test_crawler/test_stealth.py
autonomous: true
requirements:
- STEALTH-01
- STEALTH-02
- STEALTH-03
- STEALTH-04
must_haves:
truths:
- "navigator.webdriver 属性被隐藏(返回 undefined/false)"
- "Playwright 特有属性(__playwright, __pw_manual, __PW_inspect)被移除"
- "window.chrome 对象被模拟为正常 Chrome 浏览器结构"
- "User-Agent 不包含 HeadlessChrome 字符串"
artifacts:
- path: "kb_workflow/crawler/stealth/__init__.py"
provides: "Stealth 模块入口,导出 StealthConfig 和 apply_stealth"
min_lines: 10
- path: "kb_workflow/crawler/stealth/stealth_config.py"
provides: "StealthConfig Pydantic 模型和 apply_stealth_async 函数"
min_lines: 50
- path: "tests/test_crawler/test_stealth.py"
provides: "Stealth 功能单元测试"
min_lines: 100
key_links:
- from: "kb_workflow/crawler/stealth/stealth_config.py"
to: "playwright_stealth.Stealth"
via: "from playwright_stealth import Stealth"
pattern: "from playwright_stealth"
---
<objective>
创建浏览器指纹伪装模块,使用 playwright-stealth 库隐藏 Playwright 自动化特征。
Purpose: 为 ZhihuCrawler 提供可复用的指纹伪装能力,绕过知乎的浏览器指纹检测
Output: kb_workflow/crawler/stealth/ 模块 + 单元测试
</objective>
<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md
@.planning/phases/11-stealth-fingerprinting/11-VALIDATION.md
<interfaces>
<!-- Key interfaces from existing codebase -->
From kb_workflow/crawler/config.py:
```python
class CrawlerConfig(BaseModel):
proxy: Optional[ProxyConfig] = Field(None, description="Proxy configuration")
proxy_required: bool = Field(default=False)
min_delay: float = Field(default=1.0)
max_delay: float = Field(default=3.0)
```
From kb_workflow/crawler/zhihu_crawler.py:
```python
class ZhihuCrawler:
def __init__(
self,
headless: bool = True,
timeout: int = 30,
config: Optional[CrawlerConfig] = None,
proxy_pool: Optional[ProxyPool] = None
):
...
def _start_browser_with_proxy(self, cookie: str) -> None:
"""Start browser with proxy configuration."""
self._playwright = sync_playwright().start()
self._browser = self._playwright.chromium.launch(headless=self.headless)
context_options = {
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
...
self._context = self._browser.new_context(**context_options)
```
From playwright_stealth (external library):
```python
from playwright_stealth import Stealth
stealth = Stealth(
navigator_webdriver=True,
navigator_user_agent=True,
navigator_plugins=True,
# ... other options
)
stealth.apply_stealth_async(context) # for async
stealth.apply_stealth_sync(context) # for sync
```
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: 创建 StealthConfig 模块</name>
<files>kb_workflow/crawler/stealth/__init__.py, kb_workflow/crawler/stealth/stealth_config.py</files>
<behavior>
- Test 1: StealthConfig 默认值正确(stealth_enabled=True, navigator_webdriver=True 等)
- Test 2: apply_stealth_sync 在 mock context 上正确调用 playwright_stealth
- Test 3: get_default_stealth_config() 返回正确的默认配置
</behavior>
<action>
1. 创建 `kb_workflow/crawler/stealth/` 目录
2. 创建 `kb_workflow/crawler/stealth/__init__.py`:
- 导出 StealthConfig 类
- 导出 apply_stealth_sync, apply_stealth_async 函数
- 导出 get_default_stealth_config 便捷函数
3. 创建 `kb_workflow/crawler/stealth/stealth_config.py`:
```python
"""Stealth configuration for browser fingerprint masking.
Uses playwright-stealth library to hide Playwright automation signatures.
"""
from typing import Optional
from playwright.sync_api import BrowserContext
from playwright.async_api import BrowserContext as AsyncBrowserContext
from pydantic import BaseModel, Field
from playwright_stealth import Stealth
class StealthConfig(BaseModel):
"""Configuration for browser stealth settings.
All options map to playwright-stealth evasion scripts.
Default values optimized for Zhihu anti-detection.
"""
stealth_enabled: bool = Field(
default=True,
description="Enable stealth mode (set False to disable)"
)
# Navigator evasion options
navigator_webdriver: bool = Field(
default=True,
description="Hide navigator.webdriver property (STEALTH-01)"
)
navigator_user_agent: bool = Field(
default=True,
description="Replace HeadlessChrome in User-Agent (STEALTH-04)"
)
navigator_plugins: bool = Field(
default=True,
description="Mock navigator.plugins array"
)
navigator_languages: bool = Field(
default=True,
description="Set navigator.languages"
)
navigator_platform: bool = Field(
default=True,
description="Set navigator.platform"
)
navigator_vendor: bool = Field(
default=True,
description="Set navigator.vendor"
)
navigator_permissions: bool = Field(
default=True,
description="Override navigator.permissions"
)
# Chrome object evasion (STEALTH-03)
chrome_app: bool = Field(
default=True,
description="Mock chrome.app object"
)
chrome_csi: bool = Field(
default=True,
description="Mock chrome.csi function"
)
chrome_load_times: bool = Field(
default=True,
description="Mock chrome.loadTimes function"
)
# WebGL and rendering
webgl_vendor: bool = Field(
default=True,
description="Override WebGL vendor/renderer strings"
)
# Other evasion options
hairline: bool = Field(
default=True,
description="Fix hairline detection"
)
iframe_content_window: bool = Field(
default=True,
description="Fix iframe contentWindow"
)
media_codecs: bool = Field(
default=True,
description="Mock media codecs"
)
error_prototype: bool = Field(
default=True,
description="Fix error prototype chain"
)
sec_ch_ua: bool = Field(
default=True,
description="Set sec-ch-ua headers"
)
def create_stealth(self) -> Stealth:
"""Create playwright-stealth Stealth instance from config.
Returns:
Configured Stealth instance
"""
return Stealth(
navigator_webdriver=self.navigator_webdriver,
navigator_user_agent=self.navigator_user_agent,
navigator_plugins=self.navigator_plugins,
navigator_languages=self.navigator_languages,
navigator_platform=self.navigator_platform,
navigator_vendor=self.navigator_vendor,
navigator_permissions=self.navigator_permissions,
chrome_app=self.chrome_app,
chrome_csi=self.chrome_csi,
chrome_load_times=self.chrome_load_times,
webgl_vendor=self.webgl_vendor,
hairline=self.hairline,
iframe_content_window=self.iframe_content_window,
media_codecs=self.media_codecs,
error_prototype=self.error_prototype,
sec_ch_ua=self.sec_ch_ua,
)
def apply_stealth_sync(
context: BrowserContext,
config: Optional[StealthConfig] = None
) -> None:
"""Apply stealth settings to browser context (synchronous).
Args:
context: Playwright BrowserContext to apply stealth to
config: StealthConfig instance (uses defaults if not provided)
"""
if config is None:
config = StealthConfig()
if not config.stealth_enabled:
return
stealth = config.create_stealth()
stealth.apply_stealth_sync(context)
async def apply_stealth_async(
context: AsyncBrowserContext,
config: Optional[StealthConfig] = None
) -> None:
"""Apply stealth settings to browser context (asynchronous).
Args:
context: Playwright async BrowserContext to apply stealth to
config: StealthConfig instance (uses defaults if not provided)
"""
if config is None:
config = StealthConfig()
if not config.stealth_enabled:
return
stealth = config.create_stealth()
await stealth.apply_stealth_async(context)
def get_default_stealth_config() -> StealthConfig:
"""Get default StealthConfig for Zhihu crawling.
Returns:
StealthConfig with all evasion options enabled
"""
return StealthConfig()
```
4. 创建 `kb_workflow/crawler/stealth/__init__.py`:
```python
"""Stealth module for browser fingerprint masking."""
from .stealth_config import (
StealthConfig,
apply_stealth_sync,
apply_stealth_async,
get_default_stealth_config,
)
__all__ = [
"StealthConfig",
"apply_stealth_sync",
"apply_stealth_async",
"get_default_stealth_config",
]
```
</action>
<verify>
<automated>pytest tests/test_crawler/test_stealth.py::test_stealth_config_defaults -x</automated>
</verify>
<done>
- StealthConfig 类创建完成,包含所有 playwright-stealth 选项
- apply_stealth_sync/apply_stealth_async 函数可用
- 单元测试通过
</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: 创建 Stealth 功能单元测试</name>
<files>tests/test_crawler/test_stealth.py, tests/test_crawler/conftest.py</files>
<behavior>
- Test 1: test_stealth_config_defaults - 默认配置正确
- Test 2: test_stealth_config_disabled - stealth_enabled=False 时不应用伪装
- Test 3: test_apply_stealth_sync_calls_library - apply_stealth_sync 正确调用 playwright_stealth
- Test 4: test_get_default_stealth_config - 返回默认配置实例
- Test 5: test_stealth_options_mapping - 配置选项正确映射到 Stealth 类
</behavior>
<action>
1. 在 `tests/test_crawler/conftest.py` 中添加 stealth_config fixture:
```python
@pytest.fixture
def stealth_config():
"""Default StealthConfig for testing."""
from kb_workflow.crawler.stealth import StealthConfig
return StealthConfig()
@pytest.fixture
def mock_browser_context():
"""Mock Playwright BrowserContext for stealth testing."""
from unittest.mock import MagicMock
context = MagicMock()
context.add_init_script = MagicMock()
return context
```
2. 创建 `tests/test_crawler/test_stealth.py`:
```python
"""Tests for stealth fingerprint masking module."""
import pytest
from unittest.mock import MagicMock, patch, call
class TestStealthConfig:
"""Tests for StealthConfig class."""
def test_stealth_config_defaults(self):
"""Test that default StealthConfig has all options enabled."""
from kb_workflow.crawler.stealth import StealthConfig
config = StealthConfig()
assert config.stealth_enabled is True
# STEALTH-01: navigator.webdriver hiding
assert config.navigator_webdriver is True
# STEALTH-04: User-Agent modification
assert config.navigator_user_agent is True
# STEALTH-03: Chrome properties
assert config.chrome_app is True
assert config.chrome_csi is True
assert config.chrome_load_times is True
# Other important options
assert config.navigator_plugins is True
assert config.webgl_vendor is True
def test_stealth_config_disabled(self):
"""Test that stealth can be disabled."""
from kb_workflow.crawler.stealth import StealthConfig
config = StealthConfig(stealth_enabled=False)
assert config.stealth_enabled is False
def test_create_stealth_returns_stealth_instance(self):
"""Test that create_stealth() returns configured Stealth."""
from kb_workflow.crawler.stealth import StealthConfig
from playwright_stealth import Stealth
config = StealthConfig()
stealth = config.create_stealth()
assert isinstance(stealth, Stealth)
def test_stealth_options_mapping(self):
"""Test that StealthConfig options map correctly to Stealth."""
from kb_workflow.crawler.stealth import StealthConfig
config = StealthConfig(
navigator_webdriver=True,
navigator_user_agent=False, # Disable one option
chrome_app=True,
)
stealth = config.create_stealth()
# Verify the Stealth object was created with correct options
# The Stealth class stores these as attributes
assert stealth.navigator_webdriver is True
assert stealth.navigator_user_agent is False
class TestApplyStealth:
"""Tests for apply_stealth functions."""
def test_apply_stealth_sync_calls_library(self, mock_browser_context):
"""Test that apply_stealth_sync calls playwright-stealth correctly."""
from kb_workflow.crawler.stealth import apply_stealth_sync, StealthConfig
config = StealthConfig()
with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class:
mock_stealth_instance = MagicMock()
mock_stealth_class.return_value = mock_stealth_instance
apply_stealth_sync(mock_browser_context, config)
# Verify Stealth was created with config options
mock_stealth_class.assert_called_once()
# Verify apply_stealth_sync was called on the Stealth instance
mock_stealth_instance.apply_stealth_sync.assert_called_once_with(
mock_browser_context
)
def test_apply_stealth_sync_disabled_skips_stealth(self, mock_browser_context):
"""Test that stealth is skipped when disabled."""
from kb_workflow.crawler.stealth import apply_stealth_sync, StealthConfig
config = StealthConfig(stealth_enabled=False)
with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class:
apply_stealth_sync(mock_browser_context, config)
# Stealth should not be created when disabled
mock_stealth_class.assert_not_called()
def test_apply_stealth_sync_default_config(self, mock_browser_context):
"""Test that apply_stealth_sync works with default config."""
from kb_workflow.crawler.stealth import apply_stealth_sync
with patch('kb_workflow.crawler.stealth.stealth_config.Stealth') as mock_stealth_class:
mock_stealth_instance = MagicMock()
mock_stealth_class.return_value = mock_stealth_instance
# Should work without explicit config
apply_stealth_sync(mock_browser_context)
mock_stealth_class.assert_called_once()
class TestGetDefaultStealthConfig:
"""Tests for get_default_stealth_config function."""
def test_get_default_stealth_config(self):
"""Test that get_default_stealth_config returns StealthConfig."""
from kb_workflow.crawler.stealth import get_default_stealth_config
config = get_default_stealth_config()
assert config.stealth_enabled is True
assert isinstance(config, type(config)) # Is a StealthConfig
def test_get_default_stealth_config_is_fresh_instance(self):
"""Test that each call returns a new instance."""
from kb_workflow.crawler.stealth import get_default_stealth_config
config1 = get_default_stealth_config()
config2 = get_default_stealth_config()
# Should be equal but not the same object
assert config1 == config2
assert config1 is not config2
# Wave 0 completion marker
WAVE_0_COMPLETE = True
```
</action>
<verify>
<automated>pytest tests/test_crawler/test_stealth.py -x -v</automated>
</verify>
<done>
- test_stealth.py 创建完成,包含 5 个测试用例
- conftest.py 添加了 stealth_config 和 mock_browser_context fixtures
- 所有测试通过
</done>
</task>
</tasks>
<verification>
**Per-task verification:**
- Task 1: `pytest tests/test_crawler/test_stealth.py::TestStealthConfig::test_stealth_config_defaults -x`
- Task 2: `pytest tests/test_crawler/test_stealth.py -x -v`
**Wave verification:**
- All 5 tests in test_stealth.py pass
- No regression in existing tests: `pytest tests/test_crawler/ -x`
</verification>
<success_criteria>
- [ ] kb_workflow/crawler/stealth/ 模块创建完成
- [ ] StealthConfig 类包含所有 playwright-stealth 选项
- [ ] apply_stealth_sync/apply_stealth_async 函数可用
- [ ] 5 个单元测试全部通过
- [ ] 无现有测试回归
</success_criteria>
<output>
After completion, create `.planning/phases/11-stealth-fingerprinting/11-01-SUMMARY.md`
</output>