# Phase 11: Stealth Fingerprinting - Research **Researched:** 2026-03-18 **Domain:** Playwright 指纹伪装、反检测技术 **Confidence:** HIGH ## Summary 知乎使用浏览器指纹检测来识别 Playwright 自动化脚本,触发 40362 错误。核心检测点包括 `navigator.webdriver` 属性、Playwright 特有的 JS 属性(`__playwright`、`pwInspect` 等)、以及 Chrome 对象的完整性。 **Primary recommendation:** 使用已安装的 `playwright-stealth` 库(v2.0.2)配合自定义 init script,在 ZhihuCrawler 启动浏览器时注入伪装脚本。 ## Standard Stack ### Core | Library | Version | Purpose | Why Standard | |---------|---------|---------|--------------| | playwright-stealth | 2.0.2 | 指纹伪装核心库 | 已安装,提供 18 种 evasion scripts,从 puppeteer-extra-plugin-stealth 移植 | | playwright | (existing) | 浏览器自动化 | 项目已使用 | ### Supporting | Library | Version | Purpose | When to Use | |---------|---------|---------|-------------| | tenacity | (existing) | 重试机制 | 已在 ZhihuCrawler 中使用 | ### Alternatives Considered | Instead of | Could Use | Tradeoff | |------------|-----------|----------| | playwright-stealth | 手写 init script | 库更全面,维护更好;手写适合特定场景 | | playwright-stealth | Camoufox (Firefox) | Camoufox 更隐蔽但需要切换浏览器引擎,改动大 | | playwright-stealth | Nodriver | Nodriver 是 async Chrome,需要重构代码 | **Installation:** ```bash # 已安装 pip install playwright-stealth # v2.0.2 ``` ## Architecture Patterns ### Recommended Project Structure ``` kb_workflow/crawler/ ├── zhihu_crawler.py # 现有爬虫(需集成 stealth) ├── stealth/ # 新增:指纹伪装模块 │ ├── __init__.py │ └── stealth_config.py # Stealth 配置 ├── config.py # 现有配置 ├── proxy_pool.py # 现有代理池 └── models.py # 现有数据模型 ``` ### Pattern 1: Stealth 集成模式 **What:** 在 `_start_browser_with_proxy()` 方法中,创建 context 后立即应用 stealth **When to use:** 所有需要访问知乎的场景 **Example:** ```python # Source: playwright-stealth 官方 + 项目现有代码 from playwright.sync_api import sync_playwright from playwright_stealth import Stealth def _start_browser_with_proxy(self, cookie: str) -> None: """Start browser with proxy configuration and stealth.""" self._playwright = sync_playwright().start() self._browser = self._playwright.chromium.launch( headless=self.headless, # 关键:禁用 AutomationControlled 特征 args=['--disable-blink-features=AutomationControlled'] ) context_options = { 'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...', 'service_workers': 'block' # 现有配置 } # ... 代理配置 ... self._context = self._browser.new_context(**context_options) # 关键:应用 stealth stealth = Stealth( navigator_webdriver=True, navigator_user_agent=True, navigator_plugins=True, navigator_languages=True, webgl_vendor=True, chrome_app=True, chrome_csi=True, chrome_load_times=True, ) stealth.apply_stealth_sync(self._context) # 后续代码不变 if cookie: self._add_cookies(cookie) self._page = self._context.new_page() ``` ### Pattern 2: 最小化 Init Script(备选) **What:** 如果不想引入 playwright-stealth,可使用最小化脚本 **When to use:** 只需隐藏 webdriver 属性的简单场景 **Example:** ```python # Source: Stack Overflow https://stackoverflow.com/questions/75754759 STEALTH_SCRIPT = """ // 隐藏 navigator.webdriver Object.defineProperty(navigator, 'webdriver', { get: () => undefined }); // 移除 Playwright 特有属性 delete window.__playwright; delete window.__pw_manual; delete window.__PW_inspect; // 模拟 Chrome 对象 window.chrome = { runtime: {}, loadTimes: function() {}, csi: function() {}, app: {} }; """ # 在 context 创建后注入 self._context.add_init_script(STEALTH_SCRIPT) ``` ### Anti-Patterns to Avoid - **Anti-pattern 1: 只修改 webdriver 属性** - 为什么不好:现代检测会检查 Chrome 对象完整性、plugins 数组、WebGL vendor 等 - 应该:使用 playwright-stealth 的完整 evasion 套件 - **Anti-pattern 2: 在 page.goto() 后注入脚本** - 为什么不好:页面加载时检测脚本已经执行,太晚了 - 应该:使用 `context.add_init_script()` 或 `page.add_init_script()` 在页面加载前注入 - **Anti-pattern 3: 忽略 Chrome 启动参数** - 为什么不好:`--disable-blink-features=AutomationControlled` 是第一层防护 - 应该:同时使用 CLI 参数和 init script ## Don't Hand-Roll | Problem | Don't Build | Use Instead | Why | |---------|-------------|-------------|-----| | navigator.webdriver 伪装 | 手写 Object.defineProperty | playwright-stealth | 库使用 Proxy 更隐蔽,绕过 descriptor 检测 | | Chrome 对象模拟 | 自己拼凑 chrome.runtime 等 | playwright-stealth | 需要模拟 loadTimes、csi 等复杂结构 | | Plugins 数组伪装 | 手动创建 navigator.plugins | playwright-stealth | 需要生成符合浏览器特征的 MIME types | | WebGL 指纹 | 修改 getParameter 返回值 | playwright-stealth | 需要模拟真实的 renderer/vendor 字符串 | **Key insight:** playwright-stealth 包含 18 个 evasion scripts,覆盖 Chrome 环境的各个方面。自己实现容易遗漏检测点,且需要持续维护对抗新检测手段。 ## Common Pitfalls ### Pitfall 1: 40362 错误仍出现 **What goes wrong:** 集成 stealth 后仍被知乎检测到 **Why it happens:** 1. 没有同时使用 CLI 参数 `--disable-blink-features=AutomationControlled` 2. 只隐藏了 webdriver 但 Chrome 对象不完整 3. 浏览器缓存了被污染的 profile **How to avoid:** 1. 确保 `browser.launch(args=['--disable-blink-features=AutomationControlled'])` 2. 使用完整的 stealth 配置,不只是 navigator_webdriver 3. 清除 Chrome 缓存:`rm -rf ~/.cache/chrome-*`(参考 GitHub js-reverse-mcp 项目) **Warning signs:** 页面加载后控制台出现 `navigator.webdriver === true` ### Pitfall 2: Headless 模式更容易被检测 **What goes wrong:** headless=True 时被检测,headless=False 正常 **Why it happens:** Headless Chrome 有额外的指纹特征(User-Agent 中的 HeadlessChrome 字符串) **How to avoid:** 1. playwright-stealth 会自动将 HeadlessChrome 替换为 Chrome 2. 确保 `navigator_user_agent=True` 选项开启 **Warning signs:** User-Agent 包含 "HeadlessChrome" ### Pitfall 3: Service Worker 干扰 API 拦截 **What goes wrong:** 现有代码已使用 `service_workers: "block"`,与 stealth 无冲突 **Why it happens:** 这是正确做法,继续保持 **How to avoid:** 无需额外处理,现有配置已正确 **Warning signs:** API 拦截不生效 ## Code Examples ### 示例 1: ZhihuCrawler 集成 Stealth ```python # Source: 项目代码改编 # kb_workflow/crawler/zhihu_crawler.py from playwright_stealth import Stealth class ZhihuCrawler: def __init__( self, headless: bool = True, timeout: int = 30, config: Optional[CrawlerConfig] = None, proxy_pool: Optional[ProxyPool] = None, stealth_enabled: bool = True # 新增参数 ): # ... 现有初始化代码 ... self.stealth_enabled = stealth_enabled self._stealth: Optional[Stealth] = None def _start_browser_with_proxy(self, cookie: str) -> None: """Start browser with proxy configuration.""" self.config.validate_proxy() # 关键改动:添加 --disable-blink-features 参数 launch_args = [] if self.stealth_enabled: launch_args.append('--disable-blink-features=AutomationControlled') self._playwright = sync_playwright().start() self._browser = self._playwright.chromium.launch( headless=self.headless, args=launch_args if launch_args else None ) context_options = { 'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' } # ... 代理配置逻辑不变 ... context_options["service_workers"] = "block" self._context = self._browser.new_context(**context_options) # 关键改动:应用 stealth if self.stealth_enabled: self._stealth = Stealth( navigator_webdriver=True, navigator_user_agent=True, navigator_plugins=True, navigator_languages=True, navigator_platform=True, navigator_vendor=True, webgl_vendor=True, chrome_app=True, chrome_csi=True, chrome_load_times=True, hairline=True, iframe_content_window=True, media_codecs=True, navigator_permissions=True, error_prototype=True, sec_ch_ua=True, ) self._stealth.apply_stealth_sync(self._context) if cookie: self._add_cookies(cookie) self._page = self._context.new_page() self._page.set_default_timeout(self.timeout * 1000) ``` ### 示例 2: 验证 Stealth 是否生效 ```python # Source: 测试用例 def verify_stealth_applied(page) -> dict: """验证 stealth 脚本是否正确应用""" checks = {} # 检查 navigator.webdriver checks['webdriver'] = page.evaluate('navigator.webdriver') # 期望值: false 或 undefined # 检查 Chrome 对象 checks['chrome_exists'] = page.evaluate('typeof window.chrome !== "undefined"') # 期望值: true # 检查 Playwright 特有属性 checks['playwright_removed'] = page.evaluate( 'typeof window.__playwright === "undefined"' ) # 期望值: true # 检查 plugins 数组 checks['plugins_count'] = page.evaluate('navigator.plugins.length') # 期望值: > 0 return checks ``` ### 示例 3: playwright-stealth 同步 API ```python # Source: playwright-stealth 官方文档 from playwright.sync_api import sync_playwright from playwright_stealth import Stealth with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() # 方式 1: 使用 Stealth 类(推荐,可配置) stealth = Stealth(navigator_webdriver=True) stealth.apply_stealth_sync(page) # 方式 2: 使用 use_sync 上下文管理器 # with stealth.use_sync(sync_playwright()) as p: # browser = p.chromium.launch() # page = browser.new_page() # # 自动应用 stealth page.goto('https://www.zhihu.com/') browser.close() ``` ## State of the Art | Old Approach | Current Approach | When Changed | Impact | |--------------|------------------|--------------|--------| | Object.defineProperty 直接覆盖 | Proxy 包装 getter | 2024+ | 绕过 descriptor 检测 | | 单独隐藏 webdriver | 18+ evasion scripts | 2023+ | 全面指纹伪装 | | 手动 User-Agent 替换 | 自动替换 HeadlessChrome | playwright-stealth 2.0 | 更可靠 | **Deprecated/outdated:** - `navigator.webdriver = undefined` 直接赋值:现代浏览器属性是只读的,必须用 Proxy - 只处理 webdriver:知乎检测多维度指纹,单一隐藏无效 ## Open Questions 1. **知乎是否使用 TLS 指纹检测?** - What we know: 40362 错误主要与 JS 指纹相关 - What's unclear: 是否有服务端 TLS 握手分析 - Recommendation: 先用 JS 指纹伪装验证,如仍有问题再考虑 TLS 2. **playwright-stealth 与代理池的交互?** - What we know: 两者独立工作,无直接依赖 - What's unclear: 代理 IP 变更后是否需要重新应用 stealth - Recommendation: 不需要,stealth 只影响 JS 环境 ## Validation Architecture ### Test Framework | Property | Value | |----------|-------| | Framework | pytest | | Config file | pytest.ini (pythonpath=., testpaths=tests) | | Quick run command | `pytest tests/test_crawler/ -x -v` | | Full suite command | `pytest tests/ -v` | ### Phase Requirements -> Test Map | Req ID | Behavior | Test Type | Automated Command | File Exists? | |--------|----------|-----------|-------------------|-------------| | STEALTH-01 | 隐藏 navigator.webdriver 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_navigator_webdriver_hidden -x` | Wave 0 | | STEALTH-02 | 移除 Playwright 特有 JS 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_playwright_properties_removed -x` | Wave 0 | | STEALTH-03 | 模拟 Chrome 属性 | unit | `pytest tests/test_crawler/test_stealth.py::test_chrome_properties_mocked -x` | Wave 0 | | STEALTH-04 | 修改 User-Agent | unit | `pytest tests/test_crawler/test_stealth.py::test_user_agent_modified -x` | Wave 0 | | VERIFY-01 | ZhihuCrawler 集成后正常访问 | integration | `pytest tests/test_crawler/test_zhihu_crawler_stealth.py::test_crawl_answer_with_stealth -x` | Wave 0 | | VERIFY-02 | 现有功能保持正常 | regression | `pytest tests/test_crawler/ -v` | Existing | ### Sampling Rate - **Per task commit:** `pytest tests/test_crawler/ -x` - **Per wave merge:** `pytest tests/ -v` - **Phase gate:** Full suite green before `/gsd:verify-work` ### Wave 0 Gaps - [ ] `tests/test_crawler/test_stealth.py` - 测试 stealth 配置和属性隐藏 - [ ] `tests/test_crawler/test_zhihu_crawler_stealth.py` - 集成测试(可选,需要真实环境) - [ ] 共享 fixture: 在 conftest.py 中添加 `stealth_config` fixture *(部分测试可能需要真实浏览器环境,建议 mock 网络请求进行单元测试)* ## Sources ### Primary (HIGH confidence) - [playwright-stealth GitHub](https://github.com/AtuboDad/playwright_stealth) - 源码和 API - [Stack Overflow: Change navigator.webdriver](https://stackoverflow.com/questions/75754759) - 技术细节 - 项目代码: `kb_workflow/crawler/zhihu_crawler.py` - 现有实现 ### Secondary (MEDIUM confidence) - [Scrapeless: Avoid Bot Detection](https://www.scrapeless.com/en/blog/avoid-bot-detection-with-playwright-stealth) - 10 种解决方案 - [MDN: Navigator.webdriver](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver) - 属性规范 - [ScrapFly: Playwright Scraping](https://scrapfly.io/blog/posts/web-scraping-with-playwright-and-python) - 检测机制分析 ### Tertiary (LOW confidence) - [CSDN: 40362 错误分析](https://ask.csdn.net/questions/8753356) - 中文社区经验 - [GitHub js-reverse-mcp](https://github.com/zhizhuodemao/js-reverse-mcp) - 提到清除 Chrome 缓存方案 ## Metadata **Confidence breakdown:** - Standard stack: HIGH - playwright-stealth 已安装并广泛使用 - Architecture: HIGH - 现有代码结构清晰,集成点明确 - Pitfalls: MEDIUM - 知乎具体检测机制无法完全确认,基于社区经验 **Research date:** 2026-03-18 **Valid until:** 2026-04-18(反检测领域变化快,建议 30 天后复查)