This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
kb-skills/.planning/phases/11-stealth-fingerprinting/11-02-PLAN.md
T

507 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
phase: 11-stealth-fingerprinting
plan: 02
type: execute
wave: 2
depends_on:
- 11-01
files_modified:
- kb_workflow/crawler/zhihu_crawler.py
- tests/test_crawler/test_zhihu_crawler_stealth.py
autonomous: true
requirements:
- VERIFY-01
- VERIFY-02
must_haves:
truths:
- "ZhihuCrawler 访问问答页面时不再被 40362 错误拦截"
- "现有功能(登录、Cookie 持久化、内容爬取)保持正常工作"
- "stealth_enabled 参数控制是否启用指纹伪装"
artifacts:
- path: "kb_workflow/crawler/zhihu_crawler.py"
provides: "ZhihuCrawler 集成 stealth 功能"
contains: "stealth_enabled"
- path: "tests/test_crawler/test_zhihu_crawler_stealth.py"
provides: "集成测试验证 stealth 功能"
min_lines: 50
key_links:
- from: "kb_workflow/crawler/zhihu_crawler.py"
to: "kb_workflow/crawler/stealth"
via: "from .stealth import apply_stealth_sync, StealthConfig"
pattern: "from \\.stealth import"
- from: "ZhihuCrawler._start_browser_with_proxy"
to: "apply_stealth_sync"
via: "调用 apply_stealth_sync(self._context, self._stealth_config)"
pattern: "apply_stealth_sync"
---
<objective>
将指纹伪装模块集成到 ZhihuCrawler,并验证爬虫能正常绕过知乎反爬检测。
Purpose: 让 ZhihuCrawler 在启动浏览器时自动应用指纹伪装,绕过 40362 错误
Output: 修改后的 ZhihuCrawler + 集成测试
</objective>
<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md
@.planning/phases/11-stealth-fingerprinting/11-VALIDATION.md
<!-- Reference to prior plan SUMMARY if needed -->
@.planning/phases/11-stealth-fingerprinting/11-01-PLAN.md
<interfaces>
<!-- Key interfaces from Plan 11-01 and existing codebase -->
From kb_workflow/crawler/stealth (Plan 11-01):
```python
class StealthConfig(BaseModel):
stealth_enabled: bool = True
navigator_webdriver: bool = True
navigator_user_agent: bool = True
# ... other options
def apply_stealth_sync(
context: BrowserContext,
config: Optional[StealthConfig] = None
) -> None:
"""Apply stealth settings to browser context (synchronous)."""
def get_default_stealth_config() -> StealthConfig:
"""Get default StealthConfig for Zhihu crawling."""
```
From kb_workflow/crawler/zhihu_crawler.py (current implementation):
```python
class ZhihuCrawler:
def __init__(
self,
headless: bool = True,
timeout: int = 30,
config: Optional[CrawlerConfig] = None,
proxy_pool: Optional[ProxyPool] = None
):
self.headless = headless
self.timeout = timeout
self.config = config or get_crawler_config()
self.proxy_pool = proxy_pool
self._current_proxy: Optional[ProxyEntry] = None
# Internal browser management
self._playwright = None
self._browser = None
self._context = None
self._page = None
def _start_browser_with_proxy(self, cookie: str) -> None:
"""Start browser with proxy configuration."""
self.config.validate_proxy()
self._playwright = sync_playwright().start()
self._browser = self._playwright.chromium.launch(headless=self.headless)
context_options = {
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
# ... proxy configuration ...
context_options["service_workers"] = "block"
self._context = self._browser.new_context(**context_options)
if cookie:
self._add_cookies(cookie)
self._page = self._context.new_page()
self._page.set_default_timeout(self.timeout * 1000)
```
From tests/test_crawler/conftest.py:
```python
@pytest.fixture
def mock_page():
"""Mock Playwright page object."""
page = MagicMock()
page.route = MagicMock()
page.goto = MagicMock()
return page
@pytest.fixture
def sample_answer_api_response():
"""Sample answer API response data."""
return {
"id": 123456789,
"type": "answer",
# ...
}
```
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: 集成 stealth 到 ZhihuCrawler</name>
<files>kb_workflow/crawler/zhihu_crawler.py</files>
<behavior>
- Test 1: ZhihuCrawler 初始化时 stealth_enabled 默认为 True
- Test 2: ZhihuCrawler 初始化时可禁用 stealth(stealth_enabled=False)
- Test 3: _start_browser_with_proxy 在创建 context 后调用 apply_stealth_sync
- Test 4: stealth_enabled=False 时不调用 apply_stealth_sync
- Test 5: launch 时添加 --disable-blink-features=AutomationControlled 参数
</behavior>
<action>
修改 `kb_workflow/crawler/zhihu_crawler.py`,集成 stealth 功能:
1. 添加导入(在文件顶部导入区域):
```python
from .stealth import apply_stealth_sync, StealthConfig
```
2. 修改 `__init__` 方法,添加 stealth 参数:
```python
def __init__(
self,
headless: bool = True,
timeout: int = 30,
config: Optional[CrawlerConfig] = None,
proxy_pool: Optional[ProxyPool] = None,
stealth_config: Optional[StealthConfig] = None # 新增参数
):
"""Initialize ZhihuCrawler.
Args:
headless: Whether to run browser in headless mode
timeout: Request timeout in seconds
config: Optional CrawlerConfig instance
proxy_pool: Optional ProxyPool instance for multi-proxy rotation
stealth_config: Optional StealthConfig instance (uses defaults if not provided)
"""
self.headless = headless
self.timeout = timeout
self.config = config or get_crawler_config()
self.proxy_pool = proxy_pool
self._stealth_config = stealth_config # 存储配置
self._current_proxy: Optional[ProxyEntry] = None
# ... rest of initialization
```
3. 修改 `_start_browser_with_proxy` 方法:
```python
def _start_browser_with_proxy(self, cookie: str) -> None:
"""Start browser with proxy configuration.
Validates proxy if required and applies to browser context.
Applies stealth fingerprint masking after context creation.
Args:
cookie: Cookie string for authentication
Raises:
ValueError: If proxy_required=True but no proxy is configured
"""
# Validate proxy configuration
self.config.validate_proxy()
# Build launch args for stealth (关键:添加 --disable-blink-features)
launch_args = []
if self._stealth_config is None or self._stealth_config.stealth_enabled:
launch_args.append('--disable-blink-features=AutomationControlled')
self._playwright = sync_playwright().start()
self._browser = self._playwright.chromium.launch(
headless=self.headless,
args=launch_args if launch_args else None
)
# Build context options
context_options = {
'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
# Prioritize proxy pool over single proxy config
if self.proxy_pool and self.proxy_pool.has_proxies():
proxy_entry = self.proxy_pool.get_next_proxy()
if proxy_entry:
proxy_config = self._parse_proxy_url(proxy_entry.url)
context_options["proxy"] = proxy_config
self._current_proxy = proxy_entry
elif self.config.proxy:
context_options["proxy"] = self.config.proxy.to_playwright_format()
# Block Service Worker to prevent interference with API interception
context_options["service_workers"] = "block"
self._context = self._browser.new_context(**context_options)
# 关键改动:应用 stealth 指纹伪装
apply_stealth_sync(self._context, self._stealth_config)
# Add cookies if provided
if cookie:
self._add_cookies(cookie)
self._page = self._context.new_page()
self._page.set_default_timeout(self.timeout * 1000)
```
关键点:
- 在 `launch()` 时添加 `--disable-blink-features=AutomationControlled` 参数
- 在 `new_context()` 后、添加 cookies 前调用 `apply_stealth_sync()`
- 保持向后兼容:stealth_config 为 None 时使用默认配置(启用 stealth)
</action>
<verify>
<automated>pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v</automated>
</verify>
<done>
- ZhihuCrawler 添加 stealth_config 参数
- _start_browser_with_proxy 调用 apply_stealth_sync
- launch 时添加 --disable-blink-features 参数
- 所有集成测试通过
</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: 创建 ZhihuCrawler Stealth 集成测试</name>
<files>tests/test_crawler/test_zhihu_crawler_stealth.py</files>
<behavior>
- Test 1: test_crawler_has_stealth_config_param - ZhihuCrawler 接受 stealth_config 参数
- Test 2: test_crawler_default_stealth_enabled - 默认启用 stealth
- Test 3: test_crawler_can_disable_stealth - 可以禁用 stealth
- Test 4: test_start_browser_applies_stealth - _start_browser_with_proxy 调用 apply_stealth_sync
- Test 5: test_launch_args_include_disable_blink_features - launch 参数包含 --disable-blink-features
</behavior>
<action>
创建 `tests/test_crawler/test_zhihu_crawler_stealth.py`:
```python
"""Integration tests for ZhihuCrawler stealth functionality.
Tests that stealth fingerprint masking is properly integrated
into ZhihuCrawler's browser initialization.
"""
import pytest
from unittest.mock import MagicMock, patch, call
class TestZhihuCrawlerStealthIntegration:
"""Tests for stealth integration in ZhihuCrawler."""
def test_crawler_has_stealth_config_param(self):
"""Test that ZhihuCrawler accepts stealth_config parameter."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.stealth import StealthConfig
config = StealthConfig(stealth_enabled=False)
crawler = ZhihuCrawler(stealth_config=config)
assert crawler._stealth_config == config
def test_crawler_default_stealth_enabled(self):
"""Test that stealth is enabled by default (None = use defaults)."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
crawler = ZhihuCrawler()
# None means use default stealth config (enabled)
assert crawler._stealth_config is None
def test_crawler_can_disable_stealth(self):
"""Test that stealth can be explicitly disabled."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.stealth import StealthConfig
disabled_config = StealthConfig(stealth_enabled=False)
crawler = ZhihuCrawler(stealth_config=disabled_config)
assert crawler._stealth_config.stealth_enabled is False
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
def test_start_browser_applies_stealth(
self,
mock_apply_stealth,
mock_sync_playwright,
):
"""Test that _start_browser_with_proxy calls apply_stealth_sync."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
# Setup mocks
mock_playwright = MagicMock()
mock_browser = MagicMock()
mock_context = MagicMock()
mock_sync_playwright.return_value.start.return_value = mock_playwright
mock_playwright.chromium.launch.return_value = mock_browser
mock_browser.new_context.return_value = mock_context
crawler = ZhihuCrawler()
crawler._start_browser_with_proxy(cookie="test_cookie")
# Verify apply_stealth_sync was called
mock_apply_stealth.assert_called_once()
call_args = mock_apply_stealth.call_args
assert call_args[0][0] == mock_context # First arg is context
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
def test_start_browser_launch_args_include_disable_blink_features(
self,
mock_apply_stealth,
mock_sync_playwright,
):
"""Test that launch includes --disable-blink-features=AutomationControlled."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
# Setup mocks
mock_playwright = MagicMock()
mock_browser = MagicMock()
mock_context = MagicMock()
mock_sync_playwright.return_value.start.return_value = mock_playwright
mock_playwright.chromium.launch.return_value = mock_browser
mock_browser.new_context.return_value = mock_context
crawler = ZhihuCrawler()
crawler._start_browser_with_proxy(cookie="test_cookie")
# Verify launch was called with correct args
launch_call = mock_playwright.chromium.launch
launch_call.assert_called_once()
call_kwargs = launch_call.call_args[1]
assert 'args' in call_kwargs
assert '--disable-blink-features=AutomationControlled' in call_kwargs['args']
@patch('kb_workflow.crawler.zhihu_crawler.sync_playwright')
@patch('kb_workflow.crawler.zhihu_crawler.apply_stealth_sync')
def test_start_browser_stealth_disabled_no_blink_features_arg(
self,
mock_apply_stealth,
mock_sync_playwright,
):
"""Test that --disable-blink-features is not added when stealth disabled."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.stealth import StealthConfig
# Setup mocks
mock_playwright = MagicMock()
mock_browser = MagicMock()
mock_context = MagicMock()
mock_sync_playwright.return_value.start.return_value = mock_playwright
mock_playwright.chromium.launch.return_value = mock_browser
mock_browser.new_context.return_value = mock_context
disabled_config = StealthConfig(stealth_enabled=False)
crawler = ZhihuCrawler(stealth_config=disabled_config)
crawler._start_browser_with_proxy(cookie="test_cookie")
# Verify launch was called without args or with empty/None args
launch_call = mock_playwright.chromium.launch
call_kwargs = launch_call.call_args[1]
# When stealth disabled, args should be None or not contain the flag
if 'args' in call_kwargs and call_kwargs['args']:
assert '--disable-blink-features=AutomationControlled' not in call_kwargs['args']
# apply_stealth_sync should still be called but with disabled config
mock_apply_stealth.assert_called_once()
class TestZhihuCrawlerBackwardCompatibility:
"""Tests for backward compatibility after stealth integration."""
def test_crawler_works_without_stealth_config(self):
"""Test that ZhihuCrawler works without explicit stealth_config."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
# Should not raise any errors
crawler = ZhihuCrawler(
headless=True,
timeout=30
)
assert crawler is not None
def test_crawler_accepts_all_existing_params(self):
"""Test that all existing parameters still work."""
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.config import CrawlerConfig
config = CrawlerConfig()
crawler = ZhihuCrawler(
headless=False,
timeout=60,
config=config
)
assert crawler.headless is False
assert crawler.timeout == 60
assert crawler.config == config
# Wave 0 completion marker
WAVE_0_COMPLETE = True
```
</action>
<verify>
<automated>pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v</automated>
</verify>
<done>
- test_zhihu_crawler_stealth.py 创建完成
- 8 个测试用例覆盖 stealth 集成和向后兼容性
- 所有测试通过
</done>
</task>
<task type="auto">
<name>Task 3: 回归测试验证</name>
<files>N/A</files>
<action>
运行完整的 crawler 测试套件,确保 stealth 集成没有破坏现有功能:
1. 运行所有 crawler 测试:
```bash
pytest tests/test_crawler/ -v
```
2. 检查测试输出,确保:
- 所有现有测试通过
- 没有新增的 deprecation warnings
- 没有导入错误
3. 如果有测试失败,分析原因并修复
</action>
<verify>
<automated>pytest tests/test_crawler/ -v</automated>
</verify>
<done>
- 所有 crawler 测试通过(包括新测试和现有测试)
- VERIFY-02 要求满足(现有功能保持正常工作)
</done>
</task>
</tasks>
<verification>
**Per-task verification:**
- Task 1: `pytest tests/test_crawler/test_zhihu_crawler_stealth.py::TestZhihuCrawlerStealthIntegration -x -v`
- Task 2: `pytest tests/test_crawler/test_zhihu_crawler_stealth.py -x -v`
- Task 3: `pytest tests/test_crawler/ -v`
**Wave verification:**
- All 8 new tests pass
- All existing tests pass (regression check)
- Total test count should increase by 8
</verification>
<success_criteria>
- [ ] ZhihuCrawler 集成 stealth_config 参数
- [ ] _start_browser_with_proxy 调用 apply_stealth_sync
- [ ] launch 包含 --disable-blink-features=AutomationControlled 参数
- [ ] 8 个集成测试全部通过
- [ ] 所有现有测试通过(无回归)
</success_criteria>
<output>
After completion, create `.planning/phases/11-stealth-fingerprinting/11-02-SUMMARY.md`
</output>