kb-skills/.planning/phases/11-stealth-fingerprinting/11-RESEARCH.md

Phase 11: Stealth Fingerprinting - Research

Researched: 2026-03-18 Domain: Playwright 指纹伪装、反检测技术 Confidence: HIGH

Summary

知乎使用浏览器指纹检测来识别 Playwright 自动化脚本，触发 40362 错误。核心检测点包括 navigator.webdriver 属性、Playwright 特有的 JS 属性（__playwright、pwInspect 等）、以及 Chrome 对象的完整性。

Primary recommendation: 使用已安装的 playwright-stealth 库（v2.0.2）配合自定义 init script，在 ZhihuCrawler 启动浏览器时注入伪装脚本。

Standard Stack

Core

Library	Version	Purpose	Why Standard
playwright-stealth	2.0.2	指纹伪装核心库	已安装，提供 18 种 evasion scripts，从 puppeteer-extra-plugin-stealth 移植
playwright	(existing)	浏览器自动化	项目已使用

Supporting

Library	Version	Purpose	When to Use
tenacity	(existing)	重试机制	已在 ZhihuCrawler 中使用

Alternatives Considered

Instead of	Could Use	Tradeoff
playwright-stealth	手写 init script	库更全面，维护更好；手写适合特定场景
playwright-stealth	Camoufox (Firefox)	Camoufox 更隐蔽但需要切换浏览器引擎，改动大
playwright-stealth	Nodriver	Nodriver 是 async Chrome，需要重构代码

Installation:

# 已安装
pip install playwright-stealth  # v2.0.2

Architecture Patterns

Recommended Project Structure

kb_workflow/crawler/
├── zhihu_crawler.py      # 现有爬虫（需集成 stealth）
├── stealth/               # 新增：指纹伪装模块
│   ├── __init__.py
│   └── stealth_config.py  # Stealth 配置
├── config.py              # 现有配置
├── proxy_pool.py          # 现有代理池
└── models.py              # 现有数据模型

Pattern 1: Stealth 集成模式

What: 在 _start_browser_with_proxy() 方法中，创建 context 后立即应用 stealth

When to use: 所有需要访问知乎的场景

Example:

# Source: playwright-stealth 官方 + 项目现有代码
from playwright.sync_api import sync_playwright
from playwright_stealth import Stealth

def _start_browser_with_proxy(self, cookie: str) -> None:
    """Start browser with proxy configuration and stealth."""
    self._playwright = sync_playwright().start()
    self._browser = self._playwright.chromium.launch(
        headless=self.headless,
        # 关键：禁用 AutomationControlled 特征
        args=['--disable-blink-features=AutomationControlled']
    )

    context_options = {
        'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
        'service_workers': 'block'  # 现有配置
    }
    # ... 代理配置 ...

    self._context = self._browser.new_context(**context_options)

    # 关键：应用 stealth
    stealth = Stealth(
        navigator_webdriver=True,
        navigator_user_agent=True,
        navigator_plugins=True,
        navigator_languages=True,
        webgl_vendor=True,
        chrome_app=True,
        chrome_csi=True,
        chrome_load_times=True,
    )
    stealth.apply_stealth_sync(self._context)

    # 后续代码不变
    if cookie:
        self._add_cookies(cookie)
    self._page = self._context.new_page()

Pattern 2: 最小化 Init Script（备选）

What: 如果不想引入 playwright-stealth，可使用最小化脚本

When to use: 只需隐藏 webdriver 属性的简单场景

Example:

# Source: Stack Overflow https://stackoverflow.com/questions/75754759
STEALTH_SCRIPT = """
// 隐藏 navigator.webdriver
Object.defineProperty(navigator, 'webdriver', {
    get: () => undefined
});

// 移除 Playwright 特有属性
delete window.__playwright;
delete window.__pw_manual;
delete window.__PW_inspect;

// 模拟 Chrome 对象
window.chrome = {
    runtime: {},
    loadTimes: function() {},
    csi: function() {},
    app: {}
};
"""

# 在 context 创建后注入
self._context.add_init_script(STEALTH_SCRIPT)

Anti-Patterns to Avoid

• Anti-pattern 1: 只修改 webdriver 属性

  • 为什么不好：现代检测会检查 Chrome 对象完整性、plugins 数组、WebGL vendor 等
  • 应该：使用 playwright-stealth 的完整 evasion 套件

• Anti-pattern 2: 在 page.goto() 后注入脚本

  • 为什么不好：页面加载时检测脚本已经执行，太晚了
  • 应该：使用 context.add_init_script() 或 page.add_init_script() 在页面加载前注入

• Anti-pattern 3: 忽略 Chrome 启动参数

  • 为什么不好：--disable-blink-features=AutomationControlled 是第一层防护
  • 应该：同时使用 CLI 参数和 init script

Don't Hand-Roll

Problem	Don't Build	Use Instead	Why
navigator.webdriver 伪装	手写 Object.defineProperty	playwright-stealth	库使用 Proxy 更隐蔽，绕过 descriptor 检测
Chrome 对象模拟	自己拼凑 chrome.runtime 等	playwright-stealth	需要模拟 loadTimes、csi 等复杂结构
Plugins 数组伪装	手动创建 navigator.plugins	playwright-stealth	需要生成符合浏览器特征的 MIME types
WebGL 指纹	修改 getParameter 返回值	playwright-stealth	需要模拟真实的 renderer/vendor 字符串

Key insight: playwright-stealth 包含 18 个 evasion scripts，覆盖 Chrome 环境的各个方面。自己实现容易遗漏检测点，且需要持续维护对抗新检测手段。

Common Pitfalls

Pitfall 1: 40362 错误仍出现

What goes wrong: 集成 stealth 后仍被知乎检测到

Why it happens:

1. 没有同时使用 CLI 参数 --disable-blink-features=AutomationControlled
2. 只隐藏了 webdriver 但 Chrome 对象不完整
3. 浏览器缓存了被污染的 profile

How to avoid:

1. 确保 browser.launch(args=['--disable-blink-features=AutomationControlled'])
2. 使用完整的 stealth 配置，不只是 navigator_webdriver
3. 清除 Chrome 缓存：rm -rf ~/.cache/chrome-*（参考 GitHub js-reverse-mcp 项目）

Warning signs: 页面加载后控制台出现 navigator.webdriver === true

Pitfall 2: Headless 模式更容易被检测

What goes wrong: headless=True 时被检测，headless=False 正常

Why it happens: Headless Chrome 有额外的指纹特征（User-Agent 中的 HeadlessChrome 字符串）

How to avoid:

1. playwright-stealth 会自动将 HeadlessChrome 替换为 Chrome
2. 确保 navigator_user_agent=True 选项开启

Warning signs: User-Agent 包含 "HeadlessChrome"

Pitfall 3: Service Worker 干扰 API 拦截

What goes wrong: 现有代码已使用 service_workers: "block"，与 stealth 无冲突

Why it happens: 这是正确做法，继续保持

How to avoid: 无需额外处理，现有配置已正确

Warning signs: API 拦截不生效

Code Examples

示例 1: ZhihuCrawler 集成 Stealth

# Source: 项目代码改编
# kb_workflow/crawler/zhihu_crawler.py

from playwright_stealth import Stealth

class ZhihuCrawler:
    def __init__(
        self,
        headless: bool = True,
        timeout: int = 30,
        config: Optional[CrawlerConfig] = None,
        proxy_pool: Optional[ProxyPool] = None,
        stealth_enabled: bool = True  # 新增参数
    ):
        # ... 现有初始化代码 ...
        self.stealth_enabled = stealth_enabled
        self._stealth: Optional[Stealth] = None

    def _start_browser_with_proxy(self, cookie: str) -> None:
        """Start browser with proxy configuration."""
        self.config.validate_proxy()

        # 关键改动：添加 --disable-blink-features 参数
        launch_args = []
        if self.stealth_enabled:
            launch_args.append('--disable-blink-features=AutomationControlled')

        self._playwright = sync_playwright().start()
        self._browser = self._playwright.chromium.launch(
            headless=self.headless,
            args=launch_args if launch_args else None
        )

        context_options = {
            'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        }
        # ... 代理配置逻辑不变 ...

        context_options["service_workers"] = "block"
        self._context = self._browser.new_context(**context_options)

        # 关键改动：应用 stealth
        if self.stealth_enabled:
            self._stealth = Stealth(
                navigator_webdriver=True,
                navigator_user_agent=True,
                navigator_plugins=True,
                navigator_languages=True,
                navigator_platform=True,
                navigator_vendor=True,
                webgl_vendor=True,
                chrome_app=True,
                chrome_csi=True,
                chrome_load_times=True,
                hairline=True,
                iframe_content_window=True,
                media_codecs=True,
                navigator_permissions=True,
                error_prototype=True,
                sec_ch_ua=True,
            )
            self._stealth.apply_stealth_sync(self._context)

        if cookie:
            self._add_cookies(cookie)

        self._page = self._context.new_page()
        self._page.set_default_timeout(self.timeout * 1000)

示例 2: 验证 Stealth 是否生效

# Source: 测试用例
def verify_stealth_applied(page) -> dict:
    """验证 stealth 脚本是否正确应用"""
    checks = {}

    # 检查 navigator.webdriver
    checks['webdriver'] = page.evaluate('navigator.webdriver')
    # 期望值: false 或 undefined

    # 检查 Chrome 对象
    checks['chrome_exists'] = page.evaluate('typeof window.chrome !== "undefined"')
    # 期望值: true

    # 检查 Playwright 特有属性
    checks['playwright_removed'] = page.evaluate(
        'typeof window.__playwright === "undefined"'
    )
    # 期望值: true

    # 检查 plugins 数组
    checks['plugins_count'] = page.evaluate('navigator.plugins.length')
    # 期望值: > 0

    return checks

示例 3: playwright-stealth 同步 API

# Source: playwright-stealth 官方文档
from playwright.sync_api import sync_playwright
from playwright_stealth import Stealth

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # 方式 1: 使用 Stealth 类（推荐，可配置）
    stealth = Stealth(navigator_webdriver=True)
    stealth.apply_stealth_sync(page)

    # 方式 2: 使用 use_sync 上下文管理器
    # with stealth.use_sync(sync_playwright()) as p:
    #     browser = p.chromium.launch()
    #     page = browser.new_page()
    #     # 自动应用 stealth

    page.goto('https://www.zhihu.com/')
    browser.close()

State of the Art

Old Approach	Current Approach	When Changed	Impact
Object.defineProperty 直接覆盖	Proxy 包装 getter	2024+	绕过 descriptor 检测
单独隐藏 webdriver	18+ evasion scripts	2023+	全面指纹伪装
手动 User-Agent 替换	自动替换 HeadlessChrome	playwright-stealth 2.0	更可靠

Deprecated/outdated:

• navigator.webdriver = undefined 直接赋值：现代浏览器属性是只读的，必须用 Proxy
• 只处理 webdriver：知乎检测多维度指纹，单一隐藏无效

Open Questions

1. 知乎是否使用 TLS 指纹检测？

   • What we know: 40362 错误主要与 JS 指纹相关
   • What's unclear: 是否有服务端 TLS 握手分析
   • Recommendation: 先用 JS 指纹伪装验证，如仍有问题再考虑 TLS

2. playwright-stealth 与代理池的交互？

   • What we know: 两者独立工作，无直接依赖
   • What's unclear: 代理 IP 变更后是否需要重新应用 stealth
   • Recommendation: 不需要，stealth 只影响 JS 环境

Validation Architecture

Test Framework

Property	Value
Framework	pytest
Config file	pytest.ini (pythonpath=., testpaths=tests)
Quick run command	pytest tests/test_crawler/ -x -v
Full suite command	pytest tests/ -v

Phase Requirements -> Test Map

Req ID	Behavior	Test Type	Automated Command	File Exists?
STEALTH-01	隐藏 navigator.webdriver 属性	unit	pytest tests/test_crawler/test_stealth.py::test_navigator_webdriver_hidden -x	Wave 0
STEALTH-02	移除 Playwright 特有 JS 属性	unit	pytest tests/test_crawler/test_stealth.py::test_playwright_properties_removed -x	Wave 0
STEALTH-03	模拟 Chrome 属性	unit	pytest tests/test_crawler/test_stealth.py::test_chrome_properties_mocked -x	Wave 0
STEALTH-04	修改 User-Agent	unit	pytest tests/test_crawler/test_stealth.py::test_user_agent_modified -x	Wave 0
VERIFY-01	ZhihuCrawler 集成后正常访问	integration	pytest tests/test_crawler/test_zhihu_crawler_stealth.py::test_crawl_answer_with_stealth -x	Wave 0
VERIFY-02	现有功能保持正常	regression	pytest tests/test_crawler/ -v	Existing

Sampling Rate

• Per task commit: pytest tests/test_crawler/ -x
• Per wave merge: pytest tests/ -v
• Phase gate: Full suite green before /gsd:verify-work

Wave 0 Gaps

• tests/test_crawler/test_stealth.py - 测试 stealth 配置和属性隐藏
• tests/test_crawler/test_zhihu_crawler_stealth.py - 集成测试（可选，需要真实环境）
• 共享 fixture: 在 conftest.py 中添加 stealth_config fixture

(部分测试可能需要真实浏览器环境，建议 mock 网络请求进行单元测试)

Sources

Primary (HIGH confidence)

• playwright-stealth GitHub - 源码和 API
• Stack Overflow: Change navigator.webdriver - 技术细节
• 项目代码: kb_workflow/crawler/zhihu_crawler.py - 现有实现

Secondary (MEDIUM confidence)

• Scrapeless: Avoid Bot Detection - 10 种解决方案
• MDN: Navigator.webdriver - 属性规范
• ScrapFly: Playwright Scraping - 检测机制分析

Tertiary (LOW confidence)

• CSDN: 40362 错误分析 - 中文社区经验
• GitHub js-reverse-mcp - 提到清除 Chrome 缓存方案

Metadata

Confidence breakdown:

• Standard stack: HIGH - playwright-stealth 已安装并广泛使用
• Architecture: HIGH - 现有代码结构清晰，集成点明确
• Pitfalls: MEDIUM - 知乎具体检测机制无法完全确认，基于社区经验

Research date: 2026-03-18 Valid until: 2026-04-18（反检测领域变化快，建议 30 天后复查）