docs(11): research stealth fingerprinting domain

2026-03-18 23:34:59 +08:00
parent 9e30e042a8
commit 76da0f04d4
1 changed files with 408 additions and 0 deletions
@@ -0,0 +1,408 @@
+# Phase 11: Stealth Fingerprinting - Research
+
+**Researched:** 2026-03-18
+**Domain:** Playwright 指纹伪装、反检测技术
+**Confidence:** HIGH
+
+## Summary
+
+知乎使用浏览器指纹检测来识别 Playwright 自动化脚本，触发 40362 错误。核心检测点包括 `navigator.webdriver` 属性、Playwright 特有的 JS 属性（`__playwright`、`pwInspect` 等）、以及 Chrome 对象的完整性。
+
+**Primary recommendation:** 使用已安装的 `playwright-stealth` 库（v2.0.2）配合自定义 init script，在 ZhihuCrawler 启动浏览器时注入伪装脚本。
+
+## Standard Stack
+
+### Core
+| Library | Version | Purpose | Why Standard |
+|---------|---------|---------|--------------|
+| playwright-stealth | 2.0.2 | 指纹伪装核心库 | 已安装，提供 18 种 evasion scripts，从 puppeteer-extra-plugin-stealth 移植 |
+| playwright | (existing) | 浏览器自动化 | 项目已使用 |
+
+### Supporting
+| Library | Version | Purpose | When to Use |
+|---------|---------|---------|-------------|
+| tenacity | (existing) | 重试机制 | 已在 ZhihuCrawler 中使用 |
+
+### Alternatives Considered
+| Instead of | Could Use | Tradeoff |
+|------------|-----------|----------|
+| playwright-stealth | 手写 init script | 库更全面，维护更好；手写适合特定场景 |
+| playwright-stealth | Camoufox (Firefox) | Camoufox 更隐蔽但需要切换浏览器引擎，改动大 |
+| playwright-stealth | Nodriver | Nodriver 是 async Chrome，需要重构代码 |
+
+**Installation:**
+```bash
+# 已安装
+pip install playwright-stealth  # v2.0.2
+```
+
+## Architecture Patterns
+
+### Recommended Project Structure
+```
+kb_workflow/crawler/
+├── zhihu_crawler.py      # 现有爬虫（需集成 stealth）
+├── stealth/               # 新增：指纹伪装模块
+│   ├── __init__.py
+│   └── stealth_config.py  # Stealth 配置
+├── config.py              # 现有配置
+├── proxy_pool.py          # 现有代理池
+└── models.py              # 现有数据模型
+```
+
+### Pattern 1: Stealth 集成模式
+
+**What:** 在 `_start_browser_with_proxy()` 方法中，创建 context 后立即应用 stealth
+
+**When to use:** 所有需要访问知乎的场景
+
+**Example:**
+```python
+# Source: playwright-stealth 官方 + 项目现有代码
+from playwright.sync_api import sync_playwright
+from playwright_stealth import Stealth
+
+def _start_browser_with_proxy(self, cookie: str) -> None:
+    """Start browser with proxy configuration and stealth."""
+    self._playwright = sync_playwright().start()
+    self._browser = self._playwright.chromium.launch(
+        headless=self.headless,
+        # 关键：禁用 AutomationControlled 特征
+        args=['--disable-blink-features=AutomationControlled']
+    )
+
+    context_options = {
+        'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
+        'service_workers': 'block'  # 现有配置
+    }
+    # ... 代理配置 ...
+
+    self._context = self._browser.new_context(**context_options)
+
+    # 关键：应用 stealth
+    stealth = Stealth(
+        navigator_webdriver=True,
+        navigator_user_agent=True,
+        navigator_plugins=True,
+        navigator_languages=True,
+        webgl_vendor=True,
+        chrome_app=True,
+        chrome_csi=True,
+        chrome_load_times=True,
+    )
+    stealth.apply_stealth_sync(self._context)
+
+    # 后续代码不变
+    if cookie:
+        self._add_cookies(cookie)
+    self._page = self._context.new_page()
+```
+
+### Pattern 2: 最小化 Init Script（备选）
+
+**What:** 如果不想引入 playwright-stealth，可使用最小化脚本
+
+**When to use:** 只需隐藏 webdriver 属性的简单场景
+
+**Example:**
+```python
+# Source: Stack Overflow https://stackoverflow.com/questions/75754759
+STEALTH_SCRIPT = """
+// 隐藏 navigator.webdriver
+Object.defineProperty(navigator, 'webdriver', {
+    get: () => undefined
+});
+
+// 移除 Playwright 特有属性
+delete window.__playwright;
+delete window.__pw_manual;
+delete window.__PW_inspect;
+
+// 模拟 Chrome 对象
+window.chrome = {
+    runtime: {},
+    loadTimes: function() {},
+    csi: function() {},
+    app: {}
+};
+"""
+
+# 在 context 创建后注入
+self._context.add_init_script(STEALTH_SCRIPT)
+```
+
+### Anti-Patterns to Avoid
+
+- **Anti-pattern 1: 只修改 webdriver 属性**
+  - 为什么不好：现代检测会检查 Chrome 对象完整性、plugins 数组、WebGL vendor 等
+  - 应该：使用 playwright-stealth 的完整 evasion 套件
+
+- **Anti-pattern 2: 在 page.goto() 后注入脚本**
+  - 为什么不好：页面加载时检测脚本已经执行，太晚了
+  - 应该：使用 `context.add_init_script()` 或 `page.add_init_script()` 在页面加载前注入
+
+- **Anti-pattern 3: 忽略 Chrome 启动参数**
+  - 为什么不好：`--disable-blink-features=AutomationControlled` 是第一层防护
+  - 应该：同时使用 CLI 参数和 init script
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| navigator.webdriver 伪装 | 手写 Object.defineProperty | playwright-stealth | 库使用 Proxy 更隐蔽，绕过 descriptor 检测 |
+| Chrome 对象模拟 | 自己拼凑 chrome.runtime 等 | playwright-stealth | 需要模拟 loadTimes、csi 等复杂结构 |
+| Plugins 数组伪装 | 手动创建 navigator.plugins | playwright-stealth | 需要生成符合浏览器特征的 MIME types |
+| WebGL 指纹 | 修改 getParameter 返回值 | playwright-stealth | 需要模拟真实的 renderer/vendor 字符串 |
+
+**Key insight:** playwright-stealth 包含 18 个 evasion scripts，覆盖 Chrome 环境的各个方面。自己实现容易遗漏检测点，且需要持续维护对抗新检测手段。
+
+## Common Pitfalls
+
+### Pitfall 1: 40362 错误仍出现
+
+**What goes wrong:** 集成 stealth 后仍被知乎检测到
+
+**Why it happens:**
+1. 没有同时使用 CLI 参数 `--disable-blink-features=AutomationControlled`
+2. 只隐藏了 webdriver 但 Chrome 对象不完整
+3. 浏览器缓存了被污染的 profile
+
+**How to avoid:**
+1. 确保 `browser.launch(args=['--disable-blink-features=AutomationControlled'])`
+2. 使用完整的 stealth 配置，不只是 navigator_webdriver
+3. 清除 Chrome 缓存：`rm -rf ~/.cache/chrome-*`（参考 GitHub js-reverse-mcp 项目）
+
+**Warning signs:** 页面加载后控制台出现 `navigator.webdriver === true`
+
+### Pitfall 2: Headless 模式更容易被检测
+
+**What goes wrong:** headless=True 时被检测，headless=False 正常
+
+**Why it happens:** Headless Chrome 有额外的指纹特征（User-Agent 中的 HeadlessChrome 字符串）
+
+**How to avoid:**
+1. playwright-stealth 会自动将 HeadlessChrome 替换为 Chrome
+2. 确保 `navigator_user_agent=True` 选项开启
+
+**Warning signs:** User-Agent 包含 "HeadlessChrome"
+
+### Pitfall 3: Service Worker 干扰 API 拦截
+
+**What goes wrong:** 现有代码已使用 `service_workers: "block"`，与 stealth 无冲突
+
+**Why it happens:** 这是正确做法，继续保持
+
+**How to avoid:** 无需额外处理，现有配置已正确
+
+**Warning signs:** API 拦截不生效
+
+## Code Examples
+
+### 示例 1: ZhihuCrawler 集成 Stealth
+
+```python
+# Source: 项目代码改编
+# kb_workflow/crawler/zhihu_crawler.py
+
+from playwright_stealth import Stealth
+
+class ZhihuCrawler:
+    def __init__(
+        self,
+        headless: bool = True,
+        timeout: int = 30,
+        config: Optional[CrawlerConfig] = None,
+        proxy_pool: Optional[ProxyPool] = None,
+        stealth_enabled: bool = True  # 新增参数
+    ):
+        # ... 现有初始化代码 ...
+        self.stealth_enabled = stealth_enabled
+        self._stealth: Optional[Stealth] = None
+
+    def _start_browser_with_proxy(self, cookie: str) -> None:
+        """Start browser with proxy configuration."""
+        self.config.validate_proxy()
+
+        # 关键改动：添加 --disable-blink-features 参数
+        launch_args = []
+        if self.stealth_enabled:
+            launch_args.append('--disable-blink-features=AutomationControlled')
+
+        self._playwright = sync_playwright().start()
+        self._browser = self._playwright.chromium.launch(
+            headless=self.headless,
+            args=launch_args if launch_args else None
+        )
+
+        context_options = {
+            'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
+        }
+        # ... 代理配置逻辑不变 ...
+
+        context_options["service_workers"] = "block"
+        self._context = self._browser.new_context(**context_options)
+
+        # 关键改动：应用 stealth
+        if self.stealth_enabled:
+            self._stealth = Stealth(
+                navigator_webdriver=True,
+                navigator_user_agent=True,
+                navigator_plugins=True,
+                navigator_languages=True,
+                navigator_platform=True,
+                navigator_vendor=True,
+                webgl_vendor=True,
+                chrome_app=True,
+                chrome_csi=True,
+                chrome_load_times=True,
+                hairline=True,
+                iframe_content_window=True,
+                media_codecs=True,
+                navigator_permissions=True,
+                error_prototype=True,
+                sec_ch_ua=True,
+            )
+            self._stealth.apply_stealth_sync(self._context)
+
+        if cookie:
+            self._add_cookies(cookie)
+
+        self._page = self._context.new_page()
+        self._page.set_default_timeout(self.timeout * 1000)
+```
+
+### 示例 2: 验证 Stealth 是否生效
+
+```python
+# Source: 测试用例
+def verify_stealth_applied(page) -> dict:
+    """验证 stealth 脚本是否正确应用"""
+    checks = {}
+
+    # 检查 navigator.webdriver
+    checks['webdriver'] = page.evaluate('navigator.webdriver')
+    # 期望值: false 或 undefined
+
+    # 检查 Chrome 对象
+    checks['chrome_exists'] = page.evaluate('typeof window.chrome !== "undefined"')
+    # 期望值: true
+
+    # 检查 Playwright 特有属性
+    checks['playwright_removed'] = page.evaluate(
+        'typeof window.__playwright === "undefined"'
+    )
+    # 期望值: true
+
+    # 检查 plugins 数组
+    checks['plugins_count'] = page.evaluate('navigator.plugins.length')
+    # 期望值: > 0
+
+    return checks
+```
+
+### 示例 3: playwright-stealth 同步 API
+
+```python
+# Source: playwright-stealth 官方文档
+from playwright.sync_api import sync_playwright
+from playwright_stealth import Stealth
+
+with sync_playwright() as p:
+    browser = p.chromium.launch(headless=True)
+    page = browser.new_page()
+
+    # 方式 1: 使用 Stealth 类（推荐，可配置）
+    stealth = Stealth(navigator_webdriver=True)
+    stealth.apply_stealth_sync(page)
+
+    # 方式 2: 使用 use_sync 上下文管理器
+    # with stealth.use_sync(sync_playwright()) as p:
+    #     browser = p.chromium.launch()
+    #     page = browser.new_page()
+    #     # 自动应用 stealth
+
+    page.goto('https://www.zhihu.com/')
+    browser.close()
+```
+
+## State of the Art
+
+| Old Approach | Current Approach | When Changed | Impact |
+|--------------|------------------|--------------|--------|
+| Object.defineProperty 直接覆盖 | Proxy 包装 getter | 2024+ | 绕过 descriptor 检测 |
+| 单独隐藏 webdriver | 18+ evasion scripts | 2023+ | 全面指纹伪装 |
+| 手动 User-Agent 替换 | 自动替换 HeadlessChrome | playwright-stealth 2.0 | 更可靠 |
+
+**Deprecated/outdated:**
+- `navigator.webdriver = undefined` 直接赋值：现代浏览器属性是只读的，必须用 Proxy
+- 只处理 webdriver：知乎检测多维度指纹，单一隐藏无效
+
+## Open Questions
+
+1. **知乎是否使用 TLS 指纹检测？**
+   - What we know: 40362 错误主要与 JS 指纹相关
+   - What's unclear: 是否有服务端 TLS 握手分析
+   - Recommendation: 先用 JS 指纹伪装验证，如仍有问题再考虑 TLS
+
+2. **playwright-stealth 与代理池的交互？**
+   - What we know: 两者独立工作，无直接依赖
+   - What's unclear: 代理 IP 变更后是否需要重新应用 stealth
+   - Recommendation: 不需要，stealth 只影响 JS 环境
+
+## Validation Architecture
+
+### Test Framework
+| Property | Value |
+|----------|-------|
+| Framework | pytest |
+| Config file | pytest.ini (pythonpath=., testpaths=tests) |
+| Quick run command | `pytest tests/test_crawler/ -x -v` |
+| Full suite command | `pytest tests/ -v` |
+
+### Phase Requirements -> Test Map
+| Req ID | Behavior | Test Type | Automated Command | File Exists? |
+|--------|----------|-----------|-------------------|-------------|
+| STEALTH-01 | 隐藏 navigator.webdriver 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_navigator_webdriver_hidden -x` | Wave 0 |
+| STEALTH-02 | 移除 Playwright 特有 JS 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_playwright_properties_removed -x` | Wave 0 |
+| STEALTH-03 | 模拟 Chrome 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_chrome_properties_mocked -x` | Wave 0 |
+| STEALTH-04 | 修改 User-Agent | unit | `pytest tests/test_crawler/test_stealth.py::test_user_agent_modified -x` | Wave 0 |
+| VERIFY-01 | ZhihuCrawler 集成后正常访问 | integration | `pytest tests/test_crawler/test_zhihu_crawler_stealth.py::test_crawl_answer_with_stealth -x` | Wave 0 |
+| VERIFY-02 | 现有功能保持正常 | regression | `pytest tests/test_crawler/ -v` | Existing |
+
+### Sampling Rate
+- **Per task commit:** `pytest tests/test_crawler/ -x`
+- **Per wave merge:** `pytest tests/ -v`
+- **Phase gate:** Full suite green before `/gsd:verify-work`
+
+### Wave 0 Gaps
+- [ ] `tests/test_crawler/test_stealth.py` - 测试 stealth 配置和属性隐藏
+- [ ] `tests/test_crawler/test_zhihu_crawler_stealth.py` - 集成测试（可选，需要真实环境）
+- [ ] 共享 fixture: 在 conftest.py 中添加 `stealth_config` fixture
+
+*(部分测试可能需要真实浏览器环境，建议 mock 网络请求进行单元测试)*
+
+## Sources
+
+### Primary (HIGH confidence)
+- [playwright-stealth GitHub](https://github.com/AtuboDad/playwright_stealth) - 源码和 API
+- [Stack Overflow: Change navigator.webdriver](https://stackoverflow.com/questions/75754759) - 技术细节
+- 项目代码: `kb_workflow/crawler/zhihu_crawler.py` - 现有实现
+
+### Secondary (MEDIUM confidence)
+- [Scrapeless: Avoid Bot Detection](https://www.scrapeless.com/en/blog/avoid-bot-detection-with-playwright-stealth) - 10 种解决方案
+- [MDN: Navigator.webdriver](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver) - 属性规范
+- [ScrapFly: Playwright Scraping](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) - 检测机制分析
+
+### Tertiary (LOW confidence)
+- [CSDN: 40362 错误分析](https://ask.csdn.net/questions/8753356) - 中文社区经验
+- [GitHub js-reverse-mcp](https://github.com/zhizhuodemao/js-reverse-mcp) - 提到清除 Chrome 缓存方案
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH - playwright-stealth 已安装并广泛使用
+- Architecture: HIGH - 现有代码结构清晰，集成点明确
+- Pitfalls: MEDIUM - 知乎具体检测机制无法完全确认，基于社区经验
+
+**Research date:** 2026-03-18
+**Valid until:** 2026-04-18（反检测领域变化快，建议 30 天后复查）