From a58fbfb4a7eae62782ba887008d9fe2b002402d7 Mon Sep 17 00:00:00 2001 From: starmen1265 Date: Thu, 19 Mar 2026 00:04:04 +0800 Subject: [PATCH] chore: complete v1.4 Zhihu Anti-Crawler milestone --- .planning/MILESTONES.md | 9 ++ .planning/PROJECT.md | 24 +++- .planning/STATE.md | 2 +- .planning/milestones/v1.4-REQUIREMENTS.md | 55 ++++++++ .planning/milestones/v1.4-ROADMAP.md | 150 ++++++++++++++++++++++ 5 files changed, 232 insertions(+), 8 deletions(-) create mode 100644 .planning/milestones/v1.4-REQUIREMENTS.md create mode 100644 .planning/milestones/v1.4-ROADMAP.md diff --git a/.planning/MILESTONES.md b/.planning/MILESTONES.md index a998051..4a72456 100644 --- a/.planning/MILESTONES.md +++ b/.planning/MILESTONES.md @@ -1,5 +1,14 @@ # Milestones +## v1.4 Zhihu Anti-Crawler (Shipped: 2026-03-18) + +**Phases completed:** 11 phases, 30 plans, 0 tasks + +**Key accomplishments:** +- (none recorded) + +--- + ## v1.3 Proxy Pool (Shipped: 2026-03-18) **Phases completed:** 1 phase (Phase 10), 2 plans diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index eb778b3..2495b42 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -1,15 +1,15 @@ # kb-skills 工作流重构 -## Current Milestone: v1.4 Zhihu Anti-Crawler +## Current Milestone: (Planning next version) -**Goal:** 绕过知乎对 Playwright 的检测，让爬虫能正常工作 - -**Target features:** -- Playwright 指纹伪装 — 隐藏自动化特征 -- 反检测机制 — 绕过 40362 错误 +**Status:** v1.4 completed, ready for next milestone ## Previous Milestones +**v1.4 Zhihu Anti-Crawler (Shipped: 2026-03-18):** +- 浏览器指纹伪装 (Stealth) — 使用 playwright-stealth 隐藏自动化特征 +- ZhihuCrawler 集成 — 默认启用 stealth，绕过反爬检测 + **v1.3 Proxy Pool (Shipped: 2026-03-18):** - 代理池管理 (ProxyPool) — 支持 YAML/环境变量配置，Round-Robin 轮换，健康检查 @@ -79,6 +79,14 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx - ✓ **PROXY-03**: 健康检查 — 代理失败检测与冷却恢复机制 - ✓ **PROXY-04**: 失败回退 — 当前代理失败时自动切换到备用代理 +**v1.4 Zhihu Anti-Crawler (2026-03-18):** +- ✓ **STEALTH-01**: 隐藏 navigator.webdriver 属性 +- ✓ **STEALTH-02**: 移除 Playwright 特有属性（__playwright, pwInspect 等） +- ✓ **STEALTH-03**: 模拟正常浏览器的 Chrome 属性 +- ✓ **STEALTH-04**: 修改 User-Agent 为正常浏览器格式 +- ✓ **VERIFY-01**: ZhihuCrawler 集成指纹伪装后能正常访问问答页面 +- ✓ **VERIFY-02**: 保持现有功能不变（登录、Cookie、内容爬取） + ### Active (No active requirements — plan next milestone) @@ -143,6 +151,8 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx | 代理池索引指针 | 使用 index pointer 而非 itertools.cycle | ✓ Good — 更好的可测试性 | | 惰性冷却检查 | get_next_proxy() 中惰性检查而非后台定时器 | ✓ Good — 简化实现 | | 代理池优先级 | ProxyPool 参数优先于 config.proxy | ✓ Good — 向后兼容 | +| playwright-stealth | 使用外部库实现指纹伪装 | ✓ Good — 成熟方案，活跃维护 | +| stealth 默认启用 | stealth_config=None 时默认启用 | ✓ Good — 向后兼容，无需修改现有代码 | --- -*Last updated: 2026-03-18 after v1.3 milestone completed* +*Last updated: 2026-03-19 after v1.4 milestone completed* diff --git a/.planning/STATE.md b/.planning/STATE.md index a547456..0aa7bf7 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,7 +4,7 @@ milestone: v1.4 milestone_name: Zhihu Anti-Crawler status: completed stopped_at: Completed v1.4 Zhihu Anti-Crawler milestone -last_updated: "2026-03-18T15:59:24.065Z" +last_updated: "2026-03-18T16:02:40.968Z" last_activity: 2026-03-18 — Completed 11-02 stealth integration progress: total_phases: 11 diff --git a/.planning/milestones/v1.4-REQUIREMENTS.md b/.planning/milestones/v1.4-REQUIREMENTS.md new file mode 100644 index 0000000..a0e2c7f --- /dev/null +++ b/.planning/milestones/v1.4-REQUIREMENTS.md @@ -0,0 +1,55 @@ +# Requirements Archive: v1.4 Zhihu Anti-Crawler + +**Archived:** 2026-03-18 +**Status:** SHIPPED + +For current requirements, see `.planning/REQUIREMENTS.md`. + +--- + +# Requirements: v1.4 Zhihu Anti-Crawler + +**Defined:** 2026-03-18 +**Core Value:** 绕过知乎反爬检测，让爬虫正常工作 + +## v1 Requirements + +### 指纹伪装 (Stealth) + +- [x] **STEALTH-01**: 隐藏 navigator.webdriver 属性 +- [x] **STEALTH-02**: 移除 Playwright 特有的 JS 属性（__playwright, pwInspect 等） +- [x] **STEALTH-03**: 模拟正常浏览器的 Chrome 属性 +- [x] **STEALTH-04**: 修改 User-Agent 为正常浏览器格式 + +### 集成验证 (Verify) + +- [x] **VERIFY-01**: ZhihuCrawler 集成指纹伪装后能正常访问问答页面 +- [x] **VERIFY-02**: 保持现有功能不变（登录、Cookie、内容爬取） + +## Out of Scope + +| Feature | Reason | +|---------|--------| +| 其他平台适配 | 本里程碑只针对知乎 | +| 验证码识别 | 知乎主要用二维码登录，验证码场景少 | +| 代理池扩展 | v1.3 已实现，本版本不改动 | +| 行为伪装（鼠标轨迹等） | 先用最小改动验证效果，如需要再加 | + +## Traceability + +| Requirement | Phase | Status | +|-------------|-------|--------| +| STEALTH-01 | Phase 11 | Complete (11-01) | +| STEALTH-02 | Phase 11 | Complete (11-01) | +| STEALTH-03 | Phase 11 | Complete (11-01) | +| STEALTH-04 | Phase 11 | Complete (11-01) | +| VERIFY-01 | Phase 11 | Complete (11-02) | +| VERIFY-02 | Phase 11 | Complete (11-02) | + +**Coverage:** +- v1 requirements: 6 total +- Mapped to phases: 6 +- Unmapped: 0 ✓ + +--- +*Requirements defined: 2026-03-18* diff --git a/.planning/milestones/v1.4-ROADMAP.md b/.planning/milestones/v1.4-ROADMAP.md new file mode 100644 index 0000000..3a04712 --- /dev/null +++ b/.planning/milestones/v1.4-ROADMAP.md @@ -0,0 +1,150 @@ +# Roadmap: kb-skills + +## Milestones + +- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15) +- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16) +- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17) +- ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18) +- ✅ **v1.4 Zhihu Anti-Crawler** - Phase 11 (shipped 2026-03-18) + +## Phases + +
+✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2026-03-15 + +### Phase 1: Foundation +**Goal**: 项目基础设施和状态管理 +**Plans**: 3 plans + +Plans: +- [x] 01-01: WorkflowStateManager 状态管理 +- [x] 01-02: LinkParser URL 解析 +- [x] 01-03: IdempotencyChecker 幂等检查 + +### Phase 2: Adapters +**Goal**: 平台元数据适配器 +**Plans**: 2 plans + +Plans: +- [x] 02-01: BilibiliAdapter B站适配器 +- [x] 02-02: ZhihuAdapter 知乎适配器 + +### Phase 3: Processors +**Goal**: 管道处理器架构 +**Plans**: 3 plans + +Plans: +- [x] 03-01: BaseProcessor 基类 +- [x] 03-02: 具体处理器实现 +- [x] 03-03: Schema 验证 + +### Phase 4: Pipeline +**Goal**: 处理器编排和 CLI +**Plans**: 2 plans + +Plans: +- [x] 04-01: Pipeline 编排引擎 +- [x] 04-02: CLI process 命令 + +
+ +
+✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2026-03-16 + +### Phase 5: 归档与目录优化 +**Goal**: 统一中间目录结构，添加自动归档步骤，优化最终目录布局 +**Plans**: 3 plans (all complete) + +### Phase 6: 代码整合 +**Goal**: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow +**Plans**: 3 plans (all complete) + +
+ +
+✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17 + +### Phase 7: 登录认证体系 +**Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式 +**Plans**: 3 plans (all complete) + +### Phase 8: 内容爬取实现 +**Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能 +**Plans**: 3 plans (all complete) + +### Phase 9: Pipeline 整合 +**Goal**: 将新爬虫实现整合进现有 Pipeline 架构 +**Plans**: 1 plan (complete) + +
+ +
+✅ v1.3 Proxy Pool (Phase 10) - SHIPPED 2026-03-18 + +### Phase 10: Proxy Pool (代理池管理) +**Goal**: 建立代理池基础设施，支持多代理配置、自动轮换和健康检查 +**Requirements**: PROXY-01, PROXY-02, PROXY-03, PROXY-04 +**Plans**: 2 plans (all complete) + +Plans: +- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换 +- [x] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换 + +**Key Decisions:** +- 使用索引指针而非 itertools.cycle 实现轮换 +- 惰性冷却检查而非后台定时器 +- ProxyPool 参数优先于 config.proxy 配置 + +
+ +--- + +## ✅ v1.4 Zhihu Anti-Crawler (SHIPPED 2026-03-18) + +**Milestone Goal:** 绕过知乎反爬检测，让 Playwright 爬虫能正常工作 + +### Phase 11: Stealth Fingerprinting (指纹伪装) +**Goal**: 隐藏 Playwright 自动化特征，让爬虫通过知乎的浏览器指纹检测 +**Depends on**: Phase 10 +**Requirements**: STEALTH-01, STEALTH-02, STEALTH-03, STEALTH-04, VERIFY-01, VERIFY-02 +**Success Criteria** (what must be TRUE): + 1. ZhihuCrawler 访问问答页面时不再被 40362 错误拦截 + 2. 爬虫的 navigator.webdriver 属性对知乎不可见 + 3. Playwright 特有的 JS 属性（__playwright, pwInspect 等）被移除 + 4. 现有功能（登录、Cookie 持久化、内容爬取）保持正常工作 +**Plans**: 2 plans (all complete) + +Plans: +- [x] 11-01-PLAN.md — 创建 StealthConfig 模块和单元测试 +- [x] 11-02-PLAN.md — 集成 stealth 到 ZhihuCrawler 并验证 + +**Key Decisions:** +- 使用 playwright-stealth 库进行指纹伪装 +- stealth_config=None 表示默认启用 stealth (向后兼容) +- launch 时添加 --disable-blink-features=AutomationControlled 参数 + +--- + +## Progress + +**Execution Order:** +Phases execute in numeric order: 1 → 2 → ... → 10 → 11 + +| Phase | Milestone | Plans Complete | Status | Completed | +|-------|-----------|----------------|--------|-----------| +| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 | +| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 | +| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 | +| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 | +| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 | +| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 | +| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 | +| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 | +| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 | +| 10. Proxy Pool | v1.3 | 2/2 | Complete | 2026-03-18 | +| 11. Stealth Fingerprinting | v1.4 | Complete | 2026-03-18 | 2026-03-18 | + +--- + +*Last updated: 2026-03-18*