507 lines
18 KiB
Markdown
507 lines
18 KiB
Markdown
---
|
||
phase: 11-stealth-fingerprinting
|
||
plan: 02
|
||
type: execute
|
||
wave: 2
|
||
depends_on:
|
||
- 11-01
|
||
files_modified:
|
||
- kb_workflow/crawler/zhihu_crawler.py
|
||
- tests/test_crawler/test_zhihu_crawler_stealth.py
|
||
autonomous: true
|
||
requirements:
|
||
- VERIFY-01
|
||
- VERIFY-02
|
||
must_haves:
|
||
truths:
|
||
- "ZhihuCrawler 访问问答页面时不再被 40362 错误拦截"
|
||
- "现有功能(登录、Cookie 持久化、内容爬取)保持正常工作"
|
||
- "stealth_enabled 参数控制是否启用指纹伪装"
|
||
artifacts:
|
||
- path: "kb_workflow/crawler/zhihu_crawler.py"
|
||
provides: "ZhihuCrawler 集成 stealth 功能"
|
||
contains: "stealth_enabled"
|
||
- path: "tests/test_crawler/test_zhihu_crawler_stealth.py"
|
||
provides: "集成测试验证 stealth 功能"
|
||
min_lines: 50
|
||
key_links:
|
||
- from: "kb_workflow/crawler/zhihu_crawler.py"
|
||
to: "kb_workflow/crawler/stealth"
|
||
via: "from .stealth import apply_stealth_sync, StealthConfig"
|
||
pattern: "from \\.stealth import"
|
||
- from: "ZhihuCrawler._start_browser_with_proxy"
|
||
to: "apply_stealth_sync"
|
||
via: "调用 apply_stealth_sync(self._context, self._stealth_config)"
|
||
pattern: "apply_stealth_sync"
|
||
---
|
||
|
||
<objective>
|
||
将指纹伪装模块集成到 ZhihuCrawler,并验证爬虫能正常绕过知乎反爬检测。
|
||
|
||
Purpose: 让 ZhihuCrawler 在启动浏览器时自动应用指纹伪装,绕过 40362 错误
|
||
Output: 修改后的 ZhihuCrawler + 集成测试
|
||
</objective>
|
||
|
||
<execution_context>
|
||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||
</execution_context>
|
||
|
||
<context>
|
||
@.planning/PROJECT.md
|
||
@.planning/ROADMAP.md
|
||
@.planning/STATE.md
|
||
@.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md
|
||
@.planning/phases/11-stealth-fingerprinting/11-VALIDATION.md
|
||
|
||
<!-- Reference to prior plan SUMMARY if needed -->
|
||
@.planning/phases/11-stealth-fingerprinting/11-01-PLAN.md
|
||
|
||
<interfaces>
|
||
<!-- Key interfaces from Plan 11-01 and existing codebase -->
|
||
|
||
From kb_workflow/crawler/stealth (Plan 11-01):
|
||
```python
|
||
class StealthConfig(BaseModel):
|
||
stealth_enabled: bool = True
|
||
navigator_webdriver: bool = True
|
||
navigator_user_agent: bool = True
|
||
# ... other options
|
||
|
||
def apply_stealth_sync(
|
||
context: BrowserContext,
|
||
config: Optional[StealthConfig] = None
|
||
) -> None:
|
||
"""Apply stealth settings to browser context (synchronous)."""
|
||
|
||
def get_default_stealth_config() -> StealthConfig:
|
||
"""Get default StealthConfig for Zhihu crawling."""
|
||
```
|
||
|
||
From kb_workflow/crawler/zhihu_crawler.py (current implementation):
|
||
```python
|
||
class ZhihuCrawler:
|
||
def __init__(
|
||
self,
|
||
headless: bool = True,
|
||
timeout: int = 30,
|
||
config: Optional[CrawlerConfig] = None,
|
||
proxy_pool: Optional[ProxyPool] = None
|
||
):
|
||
self.headless = headless
|
||
self.timeout = timeout
|
||
self.config = config or get_crawler_config()
|
||
self.proxy_pool = proxy_pool
|
||
self._current_proxy: Optional[ProxyEntry] = None
|
||
# Internal browser management
|
||
self._playwright = None
|
||
self._browser = None
|
||
self._context = None
|
||
self._page = None
|
||
|
||
def _start_browser_with_proxy(self, cookie: str) -> None:
|
||
"""Start browser with proxy configuration."""
|
||
self.config.validate_proxy()
|
||
self._playwright = sync_playwright().start()
|
||
self._browser = self._playwright.chromium.launch(headless=self.headless)
|
||
context_options = {
|
||
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
|
||
}
|
||
# ... proxy configuration ...
|
||
context_options["service_workers"] = "block"
|
||
self._context = self._browser.new_context(**context_options)
|
||
if cookie:
|
||
self._add_cookies(cookie)
|
||
self._page = self._context.new_page()
|
||
self._page.set_default_timeout(self.timeout * 1000)
|
||
```
|
||
|
||
From tests/test_crawler/conftest.py:
|
||
```python
|
||
@pytest.fixture
|
||
def mock_page():
|
||
"""Mock Playwright page object."""
|
||
page = MagicMock()
|
||
page.route = MagicMock()
|
||
page.goto = MagicMock()
|
||
return page
|
||
|
||
@pytest.fixture
|
||
def sample_answer_api_response():
|
||
"""Sample answer API response data."""
|
||
return {
|
||
"id": 123456789,
|
||
"type": "answer",
|
||
# ...
|
||
}
|
||
```
|
||
</interfaces>
|
||
</context>
|
||
|
||
<tasks>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 1: 集成 stealth 到 ZhihuCrawler</name>
|
||
<files>kb_workflow/crawler/zhihu_crawler.py</files>
|
||
<behavior>
|
||
- Test 1: ZhihuCrawler 初始化时 stealth_enabled 默认为 True
|
||
- Test 2: ZhihuCrawler 初始化时可禁用 stealth(stealth_enabled=False)
|
||
- Test 3: _start_browser_with_proxy 在创建 context 后调用 apply_stealth_sync
|
||
- Test 4: stealth_enabled=False 时不调用 apply_stealth_sync
|
||
- Test 5: launch 时添加 --disable-blink-features=AutomationControlled 参数
|
||
</behavior>
|
||
<action>
|
||
修改 `kb_workflow/crawler/zhihu_crawler.py`,集成 stealth 功能:
|
||
|
||
1. 添加导入(在文件顶部导入区域):
|
||
```python
|
||
from .stealth import apply_stealth_sync, StealthConfig
|
||
```
|
||
|
||
2. 修改 `__init__` 方法,添加 stealth 参数:
|
||
```python
|
||
def __init__(
|
||
self,
|
||
headless: bool = True,
|
||
timeout: int = 30,
|
||
config: Optional[CrawlerConfig] = None,
|
||
proxy_pool: Optional[ProxyPool] = None,
|
||
stealth_config: Optional[StealthConfig] = None # 新增参数
|
||
):
|
||
"""Initialize ZhihuCrawler.
|
||
|
||
Args:
|
||
headless: Whether to run browser in headless mode
|
||
timeout: Request timeout in seconds
|
||
config: Optional CrawlerConfig instance
|
||
proxy_pool: Optional ProxyPool instance for multi-proxy rotation
|
||
stealth_config: Optional StealthConfig instance (uses defaults if not provided)
|
||
"""
|
||
self.headless = headless
|
||
self.timeout = timeout
|
||
self.config = config or get_crawler_config()
|
||
self.proxy_pool = proxy_pool
|
||
self._stealth_config = stealth_config # 存储配置
|
||
self._current_proxy: Optional[ProxyEntry] = None
|
||
# ... rest of initialization
|
||
```
|
||
|
||
3. 修改 `_start_browser_with_proxy` 方法:
|
||
```python
|
||
def _start_browser_with_proxy(self, cookie: str) -> None:
|
||
"""Start browser with proxy configuration.
|
||
|
||
Validates proxy if required and applies to browser context.
|
||
Applies stealth fingerprint masking after context creation.
|
||
|
||
Args:
|
||
cookie: Cookie string for authentication
|
||
|
||
Raises:
|
||
ValueError: If proxy_required=True but no proxy is configured
|
||
"""
|
||
# Validate proxy configuration
|
||
self.config.validate_proxy()
|
||
|
||
# Build launch args for stealth (关键:添加 --disable-blink-features)
|
||
launch_args = []
|
||
if self._stealth_config is None or self._stealth_config.stealth_enabled:
|
||
launch_args.append('--disable-blink-features=AutomationControlled')
|
||
|
||
self._playwright = sync_playwright().start()
|
||
self._browser = self._playwright.chromium.launch(
|
||
headless=self.headless,
|
||
args=launch_args if launch_args else None
|
||
)
|
||
|
||
# Build context options
|
||
context_options = {
|
||
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
|
||
}
|
||
|
||
# Prioritize proxy pool over single proxy config
|
||
if self.proxy_pool and self.proxy_pool.has_proxies():
|
||
proxy_entry = self.proxy_pool.get_next_proxy()
|
||
if proxy_entry:
|
||
proxy_config = self._parse_proxy_url(proxy_entry.url)
|
||
context_options["proxy"] = proxy_config
|
||
self._current_proxy = proxy_entry
|
||
elif self.config.proxy:
|
||
context_options["proxy"] = self.config.proxy.to_playwright_format()
|
||
|
||
# Block Service Worker to prevent interference with API interception
|
||
context_options["service_workers"] = "block"
|
||
|
||
self._context = self._browser.new_context(**context_options)
|
||
|
||
# 关键改动:应用 stealth 指纹伪装
|
||
apply_stealth_sync(self._context, self._stealth_config)
|
||
|
||
# Add cookies if provided
|
||
if cookie:
|
||
self._add_cookies(cookie)
|
||
|
||
self._page = self._context.new_page()
|
||
self._page.set_default_timeout(self.timeout * 1000)
|
||
```
|
||
|
||
关键点:
|
||
- 在 `launch()` 时添加 `--disable-blink-features=AutomationControlled` 参数
|
||
- 在 `new_context()` 后、添加 cookies 前调用 `apply_stealth_sync()`
|
||
- 保持向后兼容:stealth_config 为 None 时使用默认配置(启用 stealth)
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v</automated>
|
||
</verify>
|
||
<done>
|
||
- ZhihuCrawler 添加 stealth_config 参数
|
||
- _start_browser_with_proxy 调用 apply_stealth_sync
|
||
- launch 时添加 --disable-blink-features 参数
|
||
- 所有集成测试通过
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 2: 创建 ZhihuCrawler Stealth 集成测试</name>
|
||
<files>tests/test_crawler/test_zhihu_crawler_stealth.py</files>
|
||
<behavior>
|
||
- Test 1: test_crawler_has_stealth_config_param - ZhihuCrawler 接受 stealth_config 参数
|
||
- Test 2: test_crawler_default_stealth_enabled - 默认启用 stealth
|
||
- Test 3: test_crawler_can_disable_stealth - 可以禁用 stealth
|
||
- Test 4: test_start_browser_applies_stealth - _start_browser_with_proxy 调用 apply_stealth_sync
|
||
- Test 5: test_launch_args_include_disable_blink_features - launch 参数包含 --disable-blink-features
|
||
</behavior>
|
||
<action>
|
||
创建 `tests/test_crawler/test_zhihu_crawler_stealth.py`:
|
||
|
||
```python
|
||
"""Integration tests for ZhihuCrawler stealth functionality.
|
||
|
||
Tests that stealth fingerprint masking is properly integrated
|
||
into ZhihuCrawler's browser initialization.
|
||
"""
|
||
|
||
import pytest
|
||
from unittest.mock import MagicMock, patch, call
|
||
|
||
|
||
class TestZhihuCrawlerStealthIntegration:
|
||
"""Tests for stealth integration in ZhihuCrawler."""
|
||
|
||
def test_crawler_has_stealth_config_param(self):
|
||
"""Test that ZhihuCrawler accepts stealth_config parameter."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
from kb_workflow.crawler.stealth import StealthConfig
|
||
|
||
config = StealthConfig(stealth_enabled=False)
|
||
crawler = ZhihuCrawler(stealth_config=config)
|
||
|
||
assert crawler._stealth_config == config
|
||
|
||
def test_crawler_default_stealth_enabled(self):
|
||
"""Test that stealth is enabled by default (None = use defaults)."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
|
||
crawler = ZhihuCrawler()
|
||
|
||
# None means use default stealth config (enabled)
|
||
assert crawler._stealth_config is None
|
||
|
||
def test_crawler_can_disable_stealth(self):
|
||
"""Test that stealth can be explicitly disabled."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
from kb_workflow.crawler.stealth import StealthConfig
|
||
|
||
disabled_config = StealthConfig(stealth_enabled=False)
|
||
crawler = ZhihuCrawler(stealth_config=disabled_config)
|
||
|
||
assert crawler._stealth_config.stealth_enabled is False
|
||
|
||
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
|
||
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
|
||
def test_start_browser_applies_stealth(
|
||
self,
|
||
mock_apply_stealth,
|
||
mock_sync_playwright,
|
||
):
|
||
"""Test that _start_browser_with_proxy calls apply_stealth_sync."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
|
||
# Setup mocks
|
||
mock_playwright = MagicMock()
|
||
mock_browser = MagicMock()
|
||
mock_context = MagicMock()
|
||
mock_sync_playwright.return_value.start.return_value = mock_playwright
|
||
mock_playwright.chromium.launch.return_value = mock_browser
|
||
mock_browser.new_context.return_value = mock_context
|
||
|
||
crawler = ZhihuCrawler()
|
||
crawler._start_browser_with_proxy(cookie="test_cookie")
|
||
|
||
# Verify apply_stealth_sync was called
|
||
mock_apply_stealth.assert_called_once()
|
||
call_args = mock_apply_stealth.call_args
|
||
assert call_args[0][0] == mock_context # First arg is context
|
||
|
||
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
|
||
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
|
||
def test_start_browser_launch_args_include_disable_blink_features(
|
||
self,
|
||
mock_apply_stealth,
|
||
mock_sync_playwright,
|
||
):
|
||
"""Test that launch includes --disable-blink-features=AutomationControlled."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
|
||
# Setup mocks
|
||
mock_playwright = MagicMock()
|
||
mock_browser = MagicMock()
|
||
mock_context = MagicMock()
|
||
mock_sync_playwright.return_value.start.return_value = mock_playwright
|
||
mock_playwright.chromium.launch.return_value = mock_browser
|
||
mock_browser.new_context.return_value = mock_context
|
||
|
||
crawler = ZhihuCrawler()
|
||
crawler._start_browser_with_proxy(cookie="test_cookie")
|
||
|
||
# Verify launch was called with correct args
|
||
launch_call = mock_playwright.chromium.launch
|
||
launch_call.assert_called_once()
|
||
call_kwargs = launch_call.call_args[1]
|
||
|
||
assert 'args' in call_kwargs
|
||
assert '--disable-blink-features=AutomationControlled' in call_kwargs['args']
|
||
|
||
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
|
||
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
|
||
def test_start_browser_stealth_disabled_no_blink_features_arg(
|
||
self,
|
||
mock_apply_stealth,
|
||
mock_sync_playwright,
|
||
):
|
||
"""Test that --disable-blink-features is not added when stealth disabled."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
from kb_workflow.crawler.stealth import StealthConfig
|
||
|
||
# Setup mocks
|
||
mock_playwright = MagicMock()
|
||
mock_browser = MagicMock()
|
||
mock_context = MagicMock()
|
||
mock_sync_playwright.return_value.start.return_value = mock_playwright
|
||
mock_playwright.chromium.launch.return_value = mock_browser
|
||
mock_browser.new_context.return_value = mock_context
|
||
|
||
disabled_config = StealthConfig(stealth_enabled=False)
|
||
crawler = ZhihuCrawler(stealth_config=disabled_config)
|
||
crawler._start_browser_with_proxy(cookie="test_cookie")
|
||
|
||
# Verify launch was called without args or with empty/None args
|
||
launch_call = mock_playwright.chromium.launch
|
||
call_kwargs = launch_call.call_args[1]
|
||
|
||
# When stealth disabled, args should be None or not contain the flag
|
||
if 'args' in call_kwargs and call_kwargs['args']:
|
||
assert '--disable-blink-features=AutomationControlled' not in call_kwargs['args']
|
||
|
||
# apply_stealth_sync should still be called but with disabled config
|
||
mock_apply_stealth.assert_called_once()
|
||
|
||
|
||
class TestZhihuCrawlerBackwardCompatibility:
|
||
"""Tests for backward compatibility after stealth integration."""
|
||
|
||
def test_crawler_works_without_stealth_config(self):
|
||
"""Test that ZhihuCrawler works without explicit stealth_config."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
|
||
# Should not raise any errors
|
||
crawler = ZhihuCrawler(
|
||
headless=True,
|
||
timeout=30
|
||
)
|
||
assert crawler is not None
|
||
|
||
def test_crawler_accepts_all_existing_params(self):
|
||
"""Test that all existing parameters still work."""
|
||
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
|
||
from kb_workflow.crawler.config import CrawlerConfig
|
||
|
||
config = CrawlerConfig()
|
||
crawler = ZhihuCrawler(
|
||
headless=False,
|
||
timeout=60,
|
||
config=config
|
||
)
|
||
|
||
assert crawler.headless is False
|
||
assert crawler.timeout == 60
|
||
assert crawler.config == config
|
||
|
||
|
||
# Wave 0 completion marker
|
||
WAVE_0_COMPLETE = True
|
||
```
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v</automated>
|
||
</verify>
|
||
<done>
|
||
- test_zhihu_crawler_stealth.py 创建完成
|
||
- 8 个测试用例覆盖 stealth 集成和向后兼容性
|
||
- 所有测试通过
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto">
|
||
<name>Task 3: 回归测试验证</name>
|
||
<files>N/A</files>
|
||
<action>
|
||
运行完整的 crawler 测试套件,确保 stealth 集成没有破坏现有功能:
|
||
|
||
1. 运行所有 crawler 测试:
|
||
```bash
|
||
pytest tests/test_crawler/ -v
|
||
```
|
||
|
||
2. 检查测试输出,确保:
|
||
- 所有现有测试通过
|
||
- 没有新增的 deprecation warnings
|
||
- 没有导入错误
|
||
|
||
3. 如果有测试失败,分析原因并修复
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_crawler/ -v</automated>
|
||
</verify>
|
||
<done>
|
||
- 所有 crawler 测试通过(包括新测试和现有测试)
|
||
- VERIFY-02 要求满足(现有功能保持正常工作)
|
||
</done>
|
||
</task>
|
||
|
||
</tasks>
|
||
|
||
<verification>
|
||
**Per-task verification:**
|
||
- Task 1: `pytest tests/test_crawler/test_zhihu_crawler_stealth.py::TestZhihuCrawlerStealthIntegration -x -v`
|
||
- Task 2: `pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v`
|
||
- Task 3: `pytest tests/test_crawler/ -v`
|
||
|
||
**Wave verification:**
|
||
- All 8 new tests pass
|
||
- All existing tests pass (regression check)
|
||
- Total test count should increase by 8
|
||
</verification>
|
||
|
||
<success_criteria>
|
||
- [ ] ZhihuCrawler 集成 stealth_config 参数
|
||
- [ ] _start_browser_with_proxy 调用 apply_stealth_sync
|
||
- [ ] launch 包含 --disable-blink-features=AutomationControlled 参数
|
||
- [ ] 8 个集成测试全部通过
|
||
- [ ] 所有现有测试通过(无回归)
|
||
</success_criteria>
|
||
|
||
<output>
|
||
After completion, create `.planning/phases/11-stealth-fingerprinting/11-02-SUMMARY.md`
|
||
</output>
|