chore: remove phases 11-13 (Behavior Camouflage, CAPTCHA Detection, Error Recovery)
This commit is contained in:
+21
-69
@@ -2,15 +2,15 @@
|
|||||||
|
|
||||||
## Milestones
|
## Milestones
|
||||||
|
|
||||||
- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15)
|
- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2011-03-15)
|
||||||
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16)
|
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2011-03-16)
|
||||||
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17)
|
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2011-03-17)
|
||||||
- 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress)
|
- 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress)
|
||||||
|
|
||||||
## Phases
|
## Phases
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2026-03-15</summary>
|
<summary>✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2011-03-15</summary>
|
||||||
|
|
||||||
### Phase 1: Foundation
|
### Phase 1: Foundation
|
||||||
**Goal**: 项目基础设施和状态管理
|
**Goal**: 项目基础设施和状态管理
|
||||||
@@ -49,7 +49,7 @@ Plans:
|
|||||||
</details>
|
</details>
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2026-03-16</summary>
|
<summary>✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2011-03-16</summary>
|
||||||
|
|
||||||
### Phase 5: 归档与目录优化
|
### Phase 5: 归档与目录优化
|
||||||
**Goal**: 统一中间目录结构,添加自动归档步骤,优化最终目录布局
|
**Goal**: 统一中间目录结构,添加自动归档步骤,优化最终目录布局
|
||||||
@@ -79,7 +79,7 @@ Plans:
|
|||||||
</details>
|
</details>
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17</summary>
|
<summary>✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2011-03-17</summary>
|
||||||
|
|
||||||
### Phase 7: 登录认证体系
|
### Phase 7: 登录认证体系
|
||||||
**Goal**: 建立知乎登录认证机制,支持二维码扫码和手动 Cookie 两种方式,实现 Cookie 持久化和过期检测
|
**Goal**: 建立知乎登录认证机制,支持二维码扫码和手动 Cookie 两种方式,实现 Cookie 持久化和过期检测
|
||||||
@@ -135,54 +135,9 @@ Plans:
|
|||||||
- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理,Round-Robin 轮换
|
- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理,Round-Robin 轮换
|
||||||
- [ ] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
- [ ] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
||||||
|
|
||||||
#### Phase 11: Behavior Camouflage (请求行为伪装)
|
#### Phase 11: Platform Adaptation (平台适配)
|
||||||
**Goal**: 通过随机化请求特征模拟真实用户行为,降低被反爬系统检测的风险
|
|
||||||
**Depends on**: Phase 10
|
|
||||||
**Requirements**: BEHAV-01, BEHAV-02, BEHAV-03, BEHAV-04
|
|
||||||
**Success Criteria** (what must be TRUE):
|
|
||||||
1. 连续请求之间有随机的延迟间隔(默认 1-5 秒)
|
|
||||||
2. 每次请求使用不同的真实浏览器 User-Agent
|
|
||||||
3. 请求头中的 Accept-Language、Accept-Encoding 等字段随机化
|
|
||||||
4. 支持通过 Playwright 模拟页面滚动和鼠标移动行为
|
|
||||||
**Plans**: TBD
|
|
||||||
|
|
||||||
Plans:
|
|
||||||
- [ ] 11-01: 请求延迟与 UA 轮换 (BEHAV-01, BEHAV-02)
|
|
||||||
- [ ] 11-02: 请求头随机化 (BEHAV-03)
|
|
||||||
- [ ] 11-03: Playwright 行为模拟 (BEHAV-04)
|
|
||||||
|
|
||||||
#### Phase 12: CAPTCHA Detection (验证码检测)
|
|
||||||
**Goal**: 实现验证码页面检测机制,在触发验证码时优雅地暂停并通知用户
|
|
||||||
**Depends on**: Phase 11
|
|
||||||
**Requirements**: CAPT-01, CAPT-02, CAPT-03
|
|
||||||
**Success Criteria** (what must be TRUE):
|
|
||||||
1. 系统能自动识别知乎和 B 站的验证码页面
|
|
||||||
2. 检测到验证码时记录日志并暂停当前爬取任务
|
|
||||||
3. 用户手动处理验证码后可以继续爬取
|
|
||||||
**Plans**: TBD
|
|
||||||
|
|
||||||
Plans:
|
|
||||||
- [ ] 12-01: 验证码页面检测与告警 (CAPT-01, CAPT-02)
|
|
||||||
- [ ] 12-02: 手动恢复机制 (CAPT-03)
|
|
||||||
|
|
||||||
#### Phase 13: Error Recovery (错误恢复机制)
|
|
||||||
**Goal**: 建立智能的错误分类和重试策略,提升爬取任务的健壮性
|
|
||||||
**Depends on**: Phase 12
|
|
||||||
**Requirements**: ERR-01, ERR-02, ERR-03, ERR-04
|
|
||||||
**Success Criteria** (what must be TRUE):
|
|
||||||
1. 403/429/超时等不同错误类型采用不同的重试策略
|
|
||||||
2. 重试间隔使用指数退避并添加随机抖动,避免请求风暴
|
|
||||||
3. 永久性错误(如内容不存在)不再重试,记录原因后跳过
|
|
||||||
4. 失败位置被记录,用户可以从失败点恢复爬取
|
|
||||||
**Plans**: TBD
|
|
||||||
|
|
||||||
Plans:
|
|
||||||
- [ ] 13-01: 错误分类与重试策略 (ERR-01, ERR-02, ERR-03)
|
|
||||||
- [ ] 13-02: 断点续传增强 (ERR-04)
|
|
||||||
|
|
||||||
#### Phase 14: Platform Adaptation (平台适配)
|
|
||||||
**Goal**: 针对知乎和 B 站的反爬策略进行专项优化,整合前面所有反爬能力
|
**Goal**: 针对知乎和 B 站的反爬策略进行专项优化,整合前面所有反爬能力
|
||||||
**Depends on**: Phase 13
|
**Depends on**: Phase 11
|
||||||
**Requirements**: PLAT-01, PLAT-02
|
**Requirements**: PLAT-01, PLAT-02
|
||||||
**Success Criteria** (what must be TRUE):
|
**Success Criteria** (what must be TRUE):
|
||||||
1. 知乎爬取针对登录态检测和请求频率限制进行了优化
|
1. 知乎爬取针对登录态检测和请求频率限制进行了优化
|
||||||
@@ -191,8 +146,8 @@ Plans:
|
|||||||
**Plans**: TBD
|
**Plans**: TBD
|
||||||
|
|
||||||
Plans:
|
Plans:
|
||||||
- [ ] 14-01: 知乎反爬适配 (PLAT-01)
|
- [ ] 11-01: 知乎反爬适配 (PLAT-01)
|
||||||
- [ ] 14-02: B 站反爬适配 (PLAT-02)
|
- [ ] 11-02: B 站反爬适配 (PLAT-02)
|
||||||
|
|
||||||
## Progress
|
## Progress
|
||||||
|
|
||||||
@@ -201,17 +156,14 @@ Phases execute in numeric order: 1 → ... → 9 → 10 → 11 → 12 → 13 →
|
|||||||
|
|
||||||
| Phase | Milestone | Plans Complete | Status | Completed |
|
| Phase | Milestone | Plans Complete | Status | Completed |
|
||||||
|-------|-----------|----------------|--------|-----------|
|
|-------|-----------|----------------|--------|-----------|
|
||||||
| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 |
|
| 1. Foundation | v1.0 | 3/3 | Complete | 2011-03-15 |
|
||||||
| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 |
|
| 2. Adapters | v1.0 | 2/2 | Complete | 2011-03-15 |
|
||||||
| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 |
|
| 3. Processors | v1.0 | 3/3 | Complete | 2011-03-15 |
|
||||||
| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 |
|
| 4. Pipeline | v1.0 | 2/2 | Complete | 2011-03-15 |
|
||||||
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2011-03-16 |
|
||||||
| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
| 6. 代码整合 | v1.1 | 3/3 | Complete | 2011-03-16 |
|
||||||
| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2011-03-17 |
|
||||||
| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2011-03-17 |
|
||||||
| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 |
|
| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2011-03-17 |
|
||||||
| 10. Proxy Pool | 2/2 | Complete | 2026-03-18 | - |
|
| 10. Proxy Pool | 2/2 | Complete | 2011-03-18 | - |
|
||||||
| 11. Behavior Camouflage | v1.3 | 0/3 | Not started | - |
|
| 11. Platform Adaptation | v1.3 | 0/2 | Not started | - |
|
||||||
| 12. CAPTCHA Detection | v1.3 | 0/2 | Not started | - |
|
|
||||||
| 13. Error Recovery | v1.3 | 0/2 | Not started | - |
|
|
||||||
| 14. Platform Adaptation | v1.3 | 0/2 | Not started | - |
|
|
||||||
|
|||||||
+3
-3
@@ -4,10 +4,10 @@ milestone: v1.3
|
|||||||
milestone_name: Anti-Crawler Optimization
|
milestone_name: Anti-Crawler Optimization
|
||||||
status: executing
|
status: executing
|
||||||
stopped_at: Completed 10-02-PLAN.md
|
stopped_at: Completed 10-02-PLAN.md
|
||||||
last_updated: "2026-03-18T08:55:13.067Z"
|
last_updated: "2026-03-18T14:35:35.015Z"
|
||||||
last_activity: 2026-03-18 — 10-01 proxy pool core complete
|
last_activity: 2026-03-18 — 10-01 proxy pool core complete
|
||||||
progress:
|
progress:
|
||||||
total_phases: 14
|
total_phases: 11
|
||||||
completed_phases: 9
|
completed_phases: 9
|
||||||
total_plans: 29
|
total_plans: 29
|
||||||
completed_plans: 28
|
completed_plans: 28
|
||||||
@@ -25,7 +25,7 @@ See: .planning/PROJECT.md (updated 2026-03-18)
|
|||||||
|
|
||||||
## Current Position
|
## Current Position
|
||||||
|
|
||||||
Phase: 10 of 14 (Proxy Pool)
|
Phase: 10 of 11 (Proxy Pool)
|
||||||
Plan: 1 of 2 in current phase
|
Plan: 1 of 2 in current phase
|
||||||
Status: In progress
|
Status: In progress
|
||||||
Last activity: 2026-03-18 — 10-01 proxy pool core complete
|
Last activity: 2026-03-18 — 10-01 proxy pool core complete
|
||||||
|
|||||||
Reference in New Issue
Block a user