chore: complete v1.4 Zhihu Anti-Crawler milestone

This commit is contained in:
2026-03-19 00:04:04 +08:00
parent 679b7d1440
commit a58fbfb4a7
5 changed files with 232 additions and 8 deletions
+17 -7
View File
@@ -1,15 +1,15 @@
# kb-skills 工作流重构
## Current Milestone: v1.4 Zhihu Anti-Crawler
## Current Milestone: (Planning next version)
**Goal:** 绕过知乎对 Playwright 的检测,让爬虫能正常工作
**Target features:**
- Playwright 指纹伪装 — 隐藏自动化特征
- 反检测机制 — 绕过 40362 错误
**Status:** v1.4 completed, ready for next milestone
## Previous Milestones
**v1.4 Zhihu Anti-Crawler (Shipped: 2026-03-18):**
- 浏览器指纹伪装 (Stealth) — 使用 playwright-stealth 隐藏自动化特征
- ZhihuCrawler 集成 — 默认启用 stealth,绕过反爬检测
**v1.3 Proxy Pool (Shipped: 2026-03-18):**
- 代理池管理 (ProxyPool) — 支持 YAML/环境变量配置,Round-Robin 轮换,健康检查
@@ -79,6 +79,14 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
- ✓ **PROXY-03**: 健康检查 — 代理失败检测与冷却恢复机制
- ✓ **PROXY-04**: 失败回退 — 当前代理失败时自动切换到备用代理
**v1.4 Zhihu Anti-Crawler (2026-03-18):**
- ✓ **STEALTH-01**: 隐藏 navigator.webdriver 属性
- ✓ **STEALTH-02**: 移除 Playwright 特有属性(__playwright, pwInspect 等)
- ✓ **STEALTH-03**: 模拟正常浏览器的 Chrome 属性
- ✓ **STEALTH-04**: 修改 User-Agent 为正常浏览器格式
- ✓ **VERIFY-01**: ZhihuCrawler 集成指纹伪装后能正常访问问答页面
- ✓ **VERIFY-02**: 保持现有功能不变(登录、Cookie、内容爬取)
### Active
(No active requirements — plan next milestone)
@@ -143,6 +151,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
| 代理池索引指针 | 使用 index pointer 而非 itertools.cycle | ✓ Good — 更好的可测试性 |
| 惰性冷却检查 | get_next_proxy() 中惰性检查而非后台定时器 | ✓ Good — 简化实现 |
| 代理池优先级 | ProxyPool 参数优先于 config.proxy | ✓ Good — 向后兼容 |
| playwright-stealth | 使用外部库实现指纹伪装 | ✓ Good — 成熟方案,活跃维护 |
| stealth 默认启用 | stealth_config=None 时默认启用 | ✓ Good — 向后兼容,无需修改现有代码 |
---
*Last updated: 2026-03-18 after v1.3 milestone completed*
*Last updated: 2026-03-19 after v1.4 milestone completed*