chore: complete v1.4 Zhihu Anti-Crawler milestone
This commit is contained in:
+17
-7
@@ -1,15 +1,15 @@
|
||||
# kb-skills 工作流重构
|
||||
|
||||
## Current Milestone: v1.4 Zhihu Anti-Crawler
|
||||
## Current Milestone: (Planning next version)
|
||||
|
||||
**Goal:** 绕过知乎对 Playwright 的检测,让爬虫能正常工作
|
||||
|
||||
**Target features:**
|
||||
- Playwright 指纹伪装 — 隐藏自动化特征
|
||||
- 反检测机制 — 绕过 40362 错误
|
||||
**Status:** v1.4 completed, ready for next milestone
|
||||
|
||||
## Previous Milestones
|
||||
|
||||
**v1.4 Zhihu Anti-Crawler (Shipped: 2026-03-18):**
|
||||
- 浏览器指纹伪装 (Stealth) — 使用 playwright-stealth 隐藏自动化特征
|
||||
- ZhihuCrawler 集成 — 默认启用 stealth,绕过反爬检测
|
||||
|
||||
**v1.3 Proxy Pool (Shipped: 2026-03-18):**
|
||||
- 代理池管理 (ProxyPool) — 支持 YAML/环境变量配置,Round-Robin 轮换,健康检查
|
||||
|
||||
@@ -79,6 +79,14 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
- ✓ **PROXY-03**: 健康检查 — 代理失败检测与冷却恢复机制
|
||||
- ✓ **PROXY-04**: 失败回退 — 当前代理失败时自动切换到备用代理
|
||||
|
||||
**v1.4 Zhihu Anti-Crawler (2026-03-18):**
|
||||
- ✓ **STEALTH-01**: 隐藏 navigator.webdriver 属性
|
||||
- ✓ **STEALTH-02**: 移除 Playwright 特有属性(__playwright, pwInspect 等)
|
||||
- ✓ **STEALTH-03**: 模拟正常浏览器的 Chrome 属性
|
||||
- ✓ **STEALTH-04**: 修改 User-Agent 为正常浏览器格式
|
||||
- ✓ **VERIFY-01**: ZhihuCrawler 集成指纹伪装后能正常访问问答页面
|
||||
- ✓ **VERIFY-02**: 保持现有功能不变(登录、Cookie、内容爬取)
|
||||
|
||||
### Active
|
||||
|
||||
(No active requirements — plan next milestone)
|
||||
@@ -143,6 +151,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
| 代理池索引指针 | 使用 index pointer 而非 itertools.cycle | ✓ Good — 更好的可测试性 |
|
||||
| 惰性冷却检查 | get_next_proxy() 中惰性检查而非后台定时器 | ✓ Good — 简化实现 |
|
||||
| 代理池优先级 | ProxyPool 参数优先于 config.proxy | ✓ Good — 向后兼容 |
|
||||
| playwright-stealth | 使用外部库实现指纹伪装 | ✓ Good — 成熟方案,活跃维护 |
|
||||
| stealth 默认启用 | stealth_config=None 时默认启用 | ✓ Good — 向后兼容,无需修改现有代码 |
|
||||
|
||||
---
|
||||
*Last updated: 2026-03-18 after v1.3 milestone completed*
|
||||
*Last updated: 2026-03-19 after v1.4 milestone completed*
|
||||
|
||||
Reference in New Issue
Block a user