diff --git a/.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md b/.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md new file mode 100644 index 0000000..6c0bdeb --- /dev/null +++ b/.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md @@ -0,0 +1,408 @@ +# Phase 11: Stealth Fingerprinting - Research + +**Researched:** 2026-03-18 +**Domain:** Playwright 指纹伪装、反检测技术 +**Confidence:** HIGH + +## Summary + +知乎使用浏览器指纹检测来识别 Playwright 自动化脚本,触发 40362 错误。核心检测点包括 `navigator.webdriver` 属性、Playwright 特有的 JS 属性(`__playwright`、`pwInspect` 等)、以及 Chrome 对象的完整性。 + +**Primary recommendation:** 使用已安装的 `playwright-stealth` 库(v2.0.2)配合自定义 init script,在 ZhihuCrawler 启动浏览器时注入伪装脚本。 + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| playwright-stealth | 2.0.2 | 指纹伪装核心库 | 已安装,提供 18 种 evasion scripts,从 puppeteer-extra-plugin-stealth 移植 | +| playwright | (existing) | 浏览器自动化 | 项目已使用 | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| tenacity | (existing) | 重试机制 | 已在 ZhihuCrawler 中使用 | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| playwright-stealth | 手写 init script | 库更全面,维护更好;手写适合特定场景 | +| playwright-stealth | Camoufox (Firefox) | Camoufox 更隐蔽但需要切换浏览器引擎,改动大 | +| playwright-stealth | Nodriver | Nodriver 是 async Chrome,需要重构代码 | + +**Installation:** +```bash +# 已安装 +pip install playwright-stealth # v2.0.2 +``` + +## Architecture Patterns + +### Recommended Project Structure +``` +kb_workflow/crawler/ +├── zhihu_crawler.py # 现有爬虫(需集成 stealth) +├── stealth/ # 新增:指纹伪装模块 +│ ├── __init__.py +│ └── stealth_config.py # Stealth 配置 +├── config.py # 现有配置 +├── proxy_pool.py # 现有代理池 +└── models.py # 现有数据模型 +``` + +### Pattern 1: Stealth 集成模式 + +**What:** 在 `_start_browser_with_proxy()` 方法中,创建 context 后立即应用 stealth + +**When to use:** 所有需要访问知乎的场景 + +**Example:** +```python +# Source: playwright-stealth 官方 + 项目现有代码 +from playwright.sync_api import sync_playwright +from playwright_stealth import Stealth + +def _start_browser_with_proxy(self, cookie: str) -> None: + """Start browser with proxy configuration and stealth.""" + self._playwright = sync_playwright().start() + self._browser = self._playwright.chromium.launch( + headless=self.headless, + # 关键:禁用 AutomationControlled 特征 + args=['--disable-blink-features=AutomationControlled'] + ) + + context_options = { + 'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...', + 'service_workers': 'block' # 现有配置 + } + # ... 代理配置 ... + + self._context = self._browser.new_context(**context_options) + + # 关键:应用 stealth + stealth = Stealth( + navigator_webdriver=True, + navigator_user_agent=True, + navigator_plugins=True, + navigator_languages=True, + webgl_vendor=True, + chrome_app=True, + chrome_csi=True, + chrome_load_times=True, + ) + stealth.apply_stealth_sync(self._context) + + # 后续代码不变 + if cookie: + self._add_cookies(cookie) + self._page = self._context.new_page() +``` + +### Pattern 2: 最小化 Init Script(备选) + +**What:** 如果不想引入 playwright-stealth,可使用最小化脚本 + +**When to use:** 只需隐藏 webdriver 属性的简单场景 + +**Example:** +```python +# Source: Stack Overflow https://stackoverflow.com/questions/75754759 +STEALTH_SCRIPT = """ +// 隐藏 navigator.webdriver +Object.defineProperty(navigator, 'webdriver', { + get: () => undefined +}); + +// 移除 Playwright 特有属性 +delete window.__playwright; +delete window.__pw_manual; +delete window.__PW_inspect; + +// 模拟 Chrome 对象 +window.chrome = { + runtime: {}, + loadTimes: function() {}, + csi: function() {}, + app: {} +}; +""" + +# 在 context 创建后注入 +self._context.add_init_script(STEALTH_SCRIPT) +``` + +### Anti-Patterns to Avoid + +- **Anti-pattern 1: 只修改 webdriver 属性** + - 为什么不好:现代检测会检查 Chrome 对象完整性、plugins 数组、WebGL vendor 等 + - 应该:使用 playwright-stealth 的完整 evasion 套件 + +- **Anti-pattern 2: 在 page.goto() 后注入脚本** + - 为什么不好:页面加载时检测脚本已经执行,太晚了 + - 应该:使用 `context.add_init_script()` 或 `page.add_init_script()` 在页面加载前注入 + +- **Anti-pattern 3: 忽略 Chrome 启动参数** + - 为什么不好:`--disable-blink-features=AutomationControlled` 是第一层防护 + - 应该:同时使用 CLI 参数和 init script + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| navigator.webdriver 伪装 | 手写 Object.defineProperty | playwright-stealth | 库使用 Proxy 更隐蔽,绕过 descriptor 检测 | +| Chrome 对象模拟 | 自己拼凑 chrome.runtime 等 | playwright-stealth | 需要模拟 loadTimes、csi 等复杂结构 | +| Plugins 数组伪装 | 手动创建 navigator.plugins | playwright-stealth | 需要生成符合浏览器特征的 MIME types | +| WebGL 指纹 | 修改 getParameter 返回值 | playwright-stealth | 需要模拟真实的 renderer/vendor 字符串 | + +**Key insight:** playwright-stealth 包含 18 个 evasion scripts,覆盖 Chrome 环境的各个方面。自己实现容易遗漏检测点,且需要持续维护对抗新检测手段。 + +## Common Pitfalls + +### Pitfall 1: 40362 错误仍出现 + +**What goes wrong:** 集成 stealth 后仍被知乎检测到 + +**Why it happens:** +1. 没有同时使用 CLI 参数 `--disable-blink-features=AutomationControlled` +2. 只隐藏了 webdriver 但 Chrome 对象不完整 +3. 浏览器缓存了被污染的 profile + +**How to avoid:** +1. 确保 `browser.launch(args=['--disable-blink-features=AutomationControlled'])` +2. 使用完整的 stealth 配置,不只是 navigator_webdriver +3. 清除 Chrome 缓存:`rm -rf ~/.cache/chrome-*`(参考 GitHub js-reverse-mcp 项目) + +**Warning signs:** 页面加载后控制台出现 `navigator.webdriver === true` + +### Pitfall 2: Headless 模式更容易被检测 + +**What goes wrong:** headless=True 时被检测,headless=False 正常 + +**Why it happens:** Headless Chrome 有额外的指纹特征(User-Agent 中的 HeadlessChrome 字符串) + +**How to avoid:** +1. playwright-stealth 会自动将 HeadlessChrome 替换为 Chrome +2. 确保 `navigator_user_agent=True` 选项开启 + +**Warning signs:** User-Agent 包含 "HeadlessChrome" + +### Pitfall 3: Service Worker 干扰 API 拦截 + +**What goes wrong:** 现有代码已使用 `service_workers: "block"`,与 stealth 无冲突 + +**Why it happens:** 这是正确做法,继续保持 + +**How to avoid:** 无需额外处理,现有配置已正确 + +**Warning signs:** API 拦截不生效 + +## Code Examples + +### 示例 1: ZhihuCrawler 集成 Stealth + +```python +# Source: 项目代码改编 +# kb_workflow/crawler/zhihu_crawler.py + +from playwright_stealth import Stealth + +class ZhihuCrawler: + def __init__( + self, + headless: bool = True, + timeout: int = 30, + config: Optional[CrawlerConfig] = None, + proxy_pool: Optional[ProxyPool] = None, + stealth_enabled: bool = True # 新增参数 + ): + # ... 现有初始化代码 ... + self.stealth_enabled = stealth_enabled + self._stealth: Optional[Stealth] = None + + def _start_browser_with_proxy(self, cookie: str) -> None: + """Start browser with proxy configuration.""" + self.config.validate_proxy() + + # 关键改动:添加 --disable-blink-features 参数 + launch_args = [] + if self.stealth_enabled: + launch_args.append('--disable-blink-features=AutomationControlled') + + self._playwright = sync_playwright().start() + self._browser = self._playwright.chromium.launch( + headless=self.headless, + args=launch_args if launch_args else None + ) + + context_options = { + 'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' + } + # ... 代理配置逻辑不变 ... + + context_options["service_workers"] = "block" + self._context = self._browser.new_context(**context_options) + + # 关键改动:应用 stealth + if self.stealth_enabled: + self._stealth = Stealth( + navigator_webdriver=True, + navigator_user_agent=True, + navigator_plugins=True, + navigator_languages=True, + navigator_platform=True, + navigator_vendor=True, + webgl_vendor=True, + chrome_app=True, + chrome_csi=True, + chrome_load_times=True, + hairline=True, + iframe_content_window=True, + media_codecs=True, + navigator_permissions=True, + error_prototype=True, + sec_ch_ua=True, + ) + self._stealth.apply_stealth_sync(self._context) + + if cookie: + self._add_cookies(cookie) + + self._page = self._context.new_page() + self._page.set_default_timeout(self.timeout * 1000) +``` + +### 示例 2: 验证 Stealth 是否生效 + +```python +# Source: 测试用例 +def verify_stealth_applied(page) -> dict: + """验证 stealth 脚本是否正确应用""" + checks = {} + + # 检查 navigator.webdriver + checks['webdriver'] = page.evaluate('navigator.webdriver') + # 期望值: false 或 undefined + + # 检查 Chrome 对象 + checks['chrome_exists'] = page.evaluate('typeof window.chrome !== "undefined"') + # 期望值: true + + # 检查 Playwright 特有属性 + checks['playwright_removed'] = page.evaluate( + 'typeof window.__playwright === "undefined"' + ) + # 期望值: true + + # 检查 plugins 数组 + checks['plugins_count'] = page.evaluate('navigator.plugins.length') + # 期望值: > 0 + + return checks +``` + +### 示例 3: playwright-stealth 同步 API + +```python +# Source: playwright-stealth 官方文档 +from playwright.sync_api import sync_playwright +from playwright_stealth import Stealth + +with sync_playwright() as p: + browser = p.chromium.launch(headless=True) + page = browser.new_page() + + # 方式 1: 使用 Stealth 类(推荐,可配置) + stealth = Stealth(navigator_webdriver=True) + stealth.apply_stealth_sync(page) + + # 方式 2: 使用 use_sync 上下文管理器 + # with stealth.use_sync(sync_playwright()) as p: + # browser = p.chromium.launch() + # page = browser.new_page() + # # 自动应用 stealth + + page.goto('https://www.zhihu.com/') + browser.close() +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Object.defineProperty 直接覆盖 | Proxy 包装 getter | 2024+ | 绕过 descriptor 检测 | +| 单独隐藏 webdriver | 18+ evasion scripts | 2023+ | 全面指纹伪装 | +| 手动 User-Agent 替换 | 自动替换 HeadlessChrome | playwright-stealth 2.0 | 更可靠 | + +**Deprecated/outdated:** +- `navigator.webdriver = undefined` 直接赋值:现代浏览器属性是只读的,必须用 Proxy +- 只处理 webdriver:知乎检测多维度指纹,单一隐藏无效 + +## Open Questions + +1. **知乎是否使用 TLS 指纹检测?** + - What we know: 40362 错误主要与 JS 指纹相关 + - What's unclear: 是否有服务端 TLS 握手分析 + - Recommendation: 先用 JS 指纹伪装验证,如仍有问题再考虑 TLS + +2. **playwright-stealth 与代理池的交互?** + - What we know: 两者独立工作,无直接依赖 + - What's unclear: 代理 IP 变更后是否需要重新应用 stealth + - Recommendation: 不需要,stealth 只影响 JS 环境 + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | pytest | +| Config file | pytest.ini (pythonpath=., testpaths=tests) | +| Quick run command | `pytest tests/test_crawler/ -x -v` | +| Full suite command | `pytest tests/ -v` | + +### Phase Requirements -> Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| STEALTH-01 | 隐藏 navigator.webdriver 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_navigator_webdriver_hidden -x` | Wave 0 | +| STEALTH-02 | 移除 Playwright 特有 JS 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_playwright_properties_removed -x` | Wave 0 | +| STEALTH-03 | 模拟 Chrome 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_chrome_properties_mocked -x` | Wave 0 | +| STEALTH-04 | 修改 User-Agent | unit | `pytest tests/test_crawler/test_stealth.py::test_user_agent_modified -x` | Wave 0 | +| VERIFY-01 | ZhihuCrawler 集成后正常访问 | integration | `pytest tests/test_crawler/test_zhihu_crawler_stealth.py::test_crawl_answer_with_stealth -x` | Wave 0 | +| VERIFY-02 | 现有功能保持正常 | regression | `pytest tests/test_crawler/ -v` | Existing | + +### Sampling Rate +- **Per task commit:** `pytest tests/test_crawler/ -x` +- **Per wave merge:** `pytest tests/ -v` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `tests/test_crawler/test_stealth.py` - 测试 stealth 配置和属性隐藏 +- [ ] `tests/test_crawler/test_zhihu_crawler_stealth.py` - 集成测试(可选,需要真实环境) +- [ ] 共享 fixture: 在 conftest.py 中添加 `stealth_config` fixture + +*(部分测试可能需要真实浏览器环境,建议 mock 网络请求进行单元测试)* + +## Sources + +### Primary (HIGH confidence) +- [playwright-stealth GitHub](https://github.com/AtuboDad/playwright_stealth) - 源码和 API +- [Stack Overflow: Change navigator.webdriver](https://stackoverflow.com/questions/75754759) - 技术细节 +- 项目代码: `kb_workflow/crawler/zhihu_crawler.py` - 现有实现 + +### Secondary (MEDIUM confidence) +- [Scrapeless: Avoid Bot Detection](https://www.scrapeless.com/en/blog/avoid-bot-detection-with-playwright-stealth) - 10 种解决方案 +- [MDN: Navigator.webdriver](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver) - 属性规范 +- [ScrapFly: Playwright Scraping](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) - 检测机制分析 + +### Tertiary (LOW confidence) +- [CSDN: 40362 错误分析](https://ask.csdn.net/questions/8753356) - 中文社区经验 +- [GitHub js-reverse-mcp](https://github.com/zhizhuodemao/js-reverse-mcp) - 提到清除 Chrome 缓存方案 + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - playwright-stealth 已安装并广泛使用 +- Architecture: HIGH - 现有代码结构清晰,集成点明确 +- Pitfalls: MEDIUM - 知乎具体检测机制无法完全确认,基于社区经验 + +**Research date:** 2026-03-18 +**Valid until:** 2026-04-18(反检测领域变化快,建议 30 天后复查)