Compare commits
3 Commits
22fd634498
...
v1.3
| Author | SHA1 | Date | |
|---|---|---|---|
 starmen1265
|
a9ead75da7 | ||
|
starmen1265
|
d92b872b6c | ||
|
starmen1265
|
d7f32b6b93 |
@@ -1,5 +1,27 @@
|
||||
# Milestones
|
||||
|
||||
## v1.3 Proxy Pool (Shipped: 2026-03-18)
|
||||
|
||||
**Phases completed:** 1 phase (Phase 10), 2 plans
|
||||
|
||||
**Key accomplishments:**
|
||||
- ✓ ProxyPool 类 — 支持从 YAML/环境变量加载多个代理
|
||||
- ✓ Round-Robin 轮换 — 使用索引指针实现自动代理切换
|
||||
- ✓ 健康检查机制 — 代理失败检测与冷却恢复
|
||||
- ✓ ZhihuCrawler 集成 — 代理池优先于单代理配置
|
||||
|
||||
**Delivered:** 代理池基础设施 — 支持多代理配置、自动轮换、健康检查
|
||||
|
||||
**Known Gaps:**
|
||||
- 请求行为伪装 (BEHAV-01~04) — 未实现
|
||||
- 验证码检测 (CAPT-01~03) — 未实现
|
||||
- 错误恢复机制 (ERR-01~04) — 未实现
|
||||
- 平台适配 (PLAT-01~02) — 未实现
|
||||
|
||||
**Archives:** `.planning/milestones/v1.3-ROADMAP.md`, `.planning/milestones/v1.3-REQUIREMENTS.md`
|
||||
|
||||
---
|
||||
|
||||
## v1.0 Pipeline Architecture (Shipped: 2026-03-15)
|
||||
|
||||
**Phases completed:** 4 phases, 12 plans
|
||||
|
||||
+21
-16
@@ -1,15 +1,17 @@
|
||||
# kb-skills 工作流重构
|
||||
|
||||
## Current Milestone: v1.3 Anti-Crawler Optimization
|
||||
## Current State: v1.3 Shipped
|
||||
|
||||
**Goal:** 增强爬虫反检测能力，降低被封风险，提升爬取稳定性
|
||||
**Last Shipped:** 2026-03-18 — Proxy Pool 功能
|
||||
|
||||
**Target features:**
|
||||
- 代理 IP 轮换机制
|
||||
- 请求行为伪装（随机间隔、模拟人类行为）
|
||||
- 验证码检测与处理
|
||||
- 错误恢复机制（自动重试、指数退避）
|
||||
- 知乎 + B站 双平台支持
|
||||
**What's new in v1.3:**
|
||||
- 代理池管理 (ProxyPool) — 支持 YAML/环境变量配置，Round-Robin 轮换，健康检查
|
||||
|
||||
**Known Gaps:**
|
||||
- BEHAV-01~04: 请求行为伪装 — 未实现
|
||||
- CAPT-01~03: 验证码检测 — 未实现
|
||||
- ERR-01~04: 错误恢复机制 — 未实现
|
||||
- PLAT-01~02: 平台适配 — 未实现
|
||||
|
||||
## What This Is
|
||||
|
||||
@@ -71,15 +73,15 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
- ✓ **ZHIHU-07**: 代理 IP 支持 — 降低被封风险
|
||||
- ✓ **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器
|
||||
|
||||
**v1.3 Proxy Pool (2026-03-18):**
|
||||
- ✓ **PROXY-01**: 代理池配置 — 支持从 YAML 文件或环境变量加载多个代理
|
||||
- ✓ **PROXY-02**: 自动轮换 — Round-Robin 策略自动选择下一个代理
|
||||
- ✓ **PROXY-03**: 健康检查 — 代理失败检测与冷却恢复机制
|
||||
- ✓ **PROXY-04**: 失败回退 — 当前代理失败时自动切换到备用代理
|
||||
|
||||
### Active
|
||||
|
||||
**v1.3 Anti-Crawler Optimization:**
|
||||
- [ ] **AC-01**: 代理池管理 — 支持配置代理池，自动轮换 IP
|
||||
- [ ] **AC-02**: 请求行为伪装 — 随机化请求间隔，模拟人类行为
|
||||
- [ ] **AC-03**: 验证码检测 — 检测验证码页面，触发告警/暂停
|
||||
- [ ] **AC-04**: 错误恢复机制 — 自动重试、指数退避、断点续传增强
|
||||
- [ ] **AC-05**: 知乎反爬适配 — 针对知乎的反爬策略优化
|
||||
- [ ] **AC-06**: B站反爬适配 — 针对 B站的反爬策略优化
|
||||
(No active requirements — plan next milestone)
|
||||
|
||||
### Deferred
|
||||
|
||||
@@ -138,6 +140,9 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
|
||||
| 顺序执行 | Pipeline 使用顺序执行而非并行 | ✓ Good — 简单性和可调试性 |
|
||||
| CLI subparsers | 使用 argparse subparsers 组织命令 | ✓ Good — 可扩展的 CLI 结构 |
|
||||
| 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 |
|
||||
| 代理池索引指针 | 使用 index pointer 而非 itertools.cycle | ✓ Good — 更好的可测试性 |
|
||||
| 惰性冷却检查 | get_next_proxy() 中惰性检查而非后台定时器 | ✓ Good — 简化实现 |
|
||||
| 代理池优先级 | ProxyPool 参数优先于 config.proxy | ✓ Good — 向后兼容 |
|
||||
|
||||
---
|
||||
*Last updated: 2026-03-18 after v1.3 milestone started*
|
||||
*Last updated: 2026-03-18 after v1.3 milestone completed*
|
||||
|
||||
+24
-127
@@ -5,7 +5,7 @@
|
||||
- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2026-03-15)
|
||||
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2026-03-16)
|
||||
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2026-03-17)
|
||||
- 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress)
|
||||
- ✅ **v1.3 Proxy Pool** - Phase 10 (shipped 2026-03-18)
|
||||
|
||||
## Phases
|
||||
|
||||
@@ -53,27 +53,10 @@ Plans:
|
||||
|
||||
### Phase 5: 归档与目录优化
|
||||
**Goal**: 统一中间目录结构，添加自动归档步骤，优化最终目录布局
|
||||
**Depends on**: Phase 4
|
||||
**Requirements**: INT-01, INT-02, INT-03, ARC-01, ARC-02, ARC-03, ARC-04, OUT-01, OUT-02, OUT-03, OUT-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 所有分析任务的中间文件统一存放在 `~/work/intermediate/analysis/{platform}_{content_id}/`
|
||||
2. 分析完成后 md 文件自动复制到按平台分组的最终目录
|
||||
3. 最终目录结构为 `{KB_ARCHIVE_PATH}/{platform}/{YYYYMMDD}_{content_id}_analysis.md`
|
||||
4. 归档时检测重复文件，询问用户是否覆盖
|
||||
5. 不自动清理中间文件（用户手动维护）
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 6: 代码整合
|
||||
**Goal**: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow
|
||||
**Depends on**: Phase 5
|
||||
**Requirements**: MET-01, MET-02, MET-03, MET-04, MET-05, PIP-01, PIP-02, PIP-03, PIP-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. kb-bilibili-metadata 和 kb-zhihu-metadata 代码移入 kb-workflow/metadata/
|
||||
2. kb_pipeline/ 代码合并入 kb-workflow/ 保持 processors/, platforms/, schemas.py 结构
|
||||
3. 所有导入路径更新指向新位置，CLI 正常工作
|
||||
4. 原独立的 kb-bilibili-metadata、kb-zhihu-metadata、kb_pipeline 目录已删除
|
||||
5. 所有 sys.path 动态操作已删除或重构
|
||||
6. Pipeline 类可从 kb_workflow.pipeline 包级别导入
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
</details>
|
||||
@@ -82,136 +65,50 @@ Plans:
|
||||
<summary>✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17</summary>
|
||||
|
||||
### Phase 7: 登录认证体系
|
||||
**Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式，实现 Cookie 持久化和过期检测
|
||||
**Depends on**: Phase 6
|
||||
**Requirements**: AUTH-01, AUTH-02, AUTH-03, AUTH-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以通过命令行显示二维码并扫码登录知乎
|
||||
2. 扫码失败时用户可以手动设置 Cookie 进行认证
|
||||
3. Cookie 自动保存到本地文件，下次运行自动复用
|
||||
4. Cookie 过期时系统自动检测并提示用户重新登录
|
||||
**Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 8: 内容爬取实现
|
||||
**Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能，包含网络反爬措施
|
||||
**Depends on**: Phase 7
|
||||
**Requirements**: CONT-01, CONT-02, CONT-03, CONT-04, NET-01, NET-02, NET-03
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以爬取知乎问答（问题标题、描述、回答内容、元数据）
|
||||
2. 用户可以爬取知乎专栏/文章（标题、正文、元数据）
|
||||
3. 用户可以选择是否爬取内容下的评论
|
||||
4. 爬取结果包含作者信息、发布时间、点赞数等完整元数据
|
||||
5. 用户可以配置代理 IP 以降低被封风险
|
||||
**Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 9: Pipeline 整合
|
||||
**Goal**: 将新爬虫实现整合进现有 Pipeline 架构，替换原有知乎处理器
|
||||
**Depends on**: Phase 8
|
||||
**Requirements**: PIPE-01, PIPE-02, PIPE-03
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 知乎链接使用新爬虫处理器，无需手动设置 Cookie
|
||||
2. 新处理器与现有 Pipeline 架构完全兼容
|
||||
3. 支持断点续传和幂等执行（与其他处理器行为一致）
|
||||
**Goal**: 将新爬虫实现整合进现有 Pipeline 架构
|
||||
**Plans**: 1 plan (complete)
|
||||
|
||||
</details>
|
||||
|
||||
### 🚧 v1.3 Anti-Crawler Optimization (In Progress)
|
||||
<details>
|
||||
<summary>✅ v1.3 Proxy Pool (Phase 10) - SHIPPED 2026-03-18</summary>
|
||||
|
||||
**Milestone Goal:** 增强爬虫反检测能力，降低被封风险，提升爬取稳定性
|
||||
|
||||
#### Phase 10: Proxy Pool (代理池管理)
|
||||
### Phase 10: Proxy Pool (代理池管理)
|
||||
**Goal**: 建立代理池基础设施，支持多代理配置、自动轮换和健康检查
|
||||
**Depends on**: Phase 9
|
||||
**Requirements**: PROXY-01, PROXY-02, PROXY-03, PROXY-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以从配置文件或环境变量加载多个代理地址
|
||||
2. 每次请求自动选择下一个可用代理，实现负载均衡
|
||||
3. 代理不可用时自动检测并标记为失效
|
||||
4. 当前代理失败时自动切换到备用代理继续请求
|
||||
**Plans**: 2 plans
|
||||
**Plans**: 2 plans (all complete)
|
||||
|
||||
Plans:
|
||||
- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换
|
||||
- [ ] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
||||
- [x] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
||||
|
||||
#### Phase 11: Behavior Camouflage (请求行为伪装)
|
||||
**Goal**: 通过随机化请求特征模拟真实用户行为，降低被反爬系统检测的风险
|
||||
**Depends on**: Phase 10
|
||||
**Requirements**: BEHAV-01, BEHAV-02, BEHAV-03, BEHAV-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 连续请求之间有随机的延迟间隔（默认 1-5 秒）
|
||||
2. 每次请求使用不同的真实浏览器 User-Agent
|
||||
3. 请求头中的 Accept-Language、Accept-Encoding 等字段随机化
|
||||
4. 支持通过 Playwright 模拟页面滚动和鼠标移动行为
|
||||
**Plans**: TBD
|
||||
**Key Decisions:**
|
||||
- 使用索引指针而非 itertools.cycle 实现轮换
|
||||
- 惰性冷却检查而非后台定时器
|
||||
- ProxyPool 参数优先于 config.proxy 配置
|
||||
|
||||
Plans:
|
||||
- [ ] 11-01: 请求延迟与 UA 轮换 (BEHAV-01, BEHAV-02)
|
||||
- [ ] 11-02: 请求头随机化 (BEHAV-03)
|
||||
- [ ] 11-03: Playwright 行为模拟 (BEHAV-04)
|
||||
**Known Gaps (deferred):**
|
||||
- BEHAV-01~04: 请求行为伪装
|
||||
- CAPT-01~03: 验证码检测
|
||||
- ERR-01~04: 错误恢复机制
|
||||
- PLAT-01~02: 平台适配
|
||||
|
||||
#### Phase 12: CAPTCHA Detection (验证码检测)
|
||||
**Goal**: 实现验证码页面检测机制，在触发验证码时优雅地暂停并通知用户
|
||||
**Depends on**: Phase 11
|
||||
**Requirements**: CAPT-01, CAPT-02, CAPT-03
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 系统能自动识别知乎和 B 站的验证码页面
|
||||
2. 检测到验证码时记录日志并暂停当前爬取任务
|
||||
3. 用户手动处理验证码后可以继续爬取
|
||||
**Plans**: TBD
|
||||
</details>
|
||||
|
||||
Plans:
|
||||
- [ ] 12-01: 验证码页面检测与告警 (CAPT-01, CAPT-02)
|
||||
- [ ] 12-02: 手动恢复机制 (CAPT-03)
|
||||
---
|
||||
|
||||
#### Phase 13: Error Recovery (错误恢复机制)
|
||||
**Goal**: 建立智能的错误分类和重试策略，提升爬取任务的健壮性
|
||||
**Depends on**: Phase 12
|
||||
**Requirements**: ERR-01, ERR-02, ERR-03, ERR-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 403/429/超时等不同错误类型采用不同的重试策略
|
||||
2. 重试间隔使用指数退避并添加随机抖动，避免请求风暴
|
||||
3. 永久性错误（如内容不存在）不再重试，记录原因后跳过
|
||||
4. 失败位置被记录，用户可以从失败点恢复爬取
|
||||
**Plans**: TBD
|
||||
## Next Milestone
|
||||
|
||||
Plans:
|
||||
- [ ] 13-01: 错误分类与重试策略 (ERR-01, ERR-02, ERR-03)
|
||||
- [ ] 13-02: 断点续传增强 (ERR-04)
|
||||
Run `/gsd:new-milestone` to plan the next version.
|
||||
|
||||
#### Phase 14: Platform Adaptation (平台适配)
|
||||
**Goal**: 针对知乎和 B 站的反爬策略进行专项优化，整合前面所有反爬能力
|
||||
**Depends on**: Phase 13
|
||||
**Requirements**: PLAT-01, PLAT-02
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 知乎爬取针对登录态检测和请求频率限制进行了优化
|
||||
2. B 站爬取针对视频/专栏元数据获取进行了优化
|
||||
3. 两个平台的爬取成功率相比 v1.2 有明显提升
|
||||
**Plans**: TBD
|
||||
---
|
||||
|
||||
Plans:
|
||||
- [ ] 14-01: 知乎反爬适配 (PLAT-01)
|
||||
- [ ] 14-02: B 站反爬适配 (PLAT-02)
|
||||
|
||||
## Progress
|
||||
|
||||
**Execution Order:**
|
||||
Phases execute in numeric order: 1 → ... → 9 → 10 → 11 → 12 → 13 → 14
|
||||
|
||||
| Phase | Milestone | Plans Complete | Status | Completed |
|
||||
|-------|-----------|----------------|--------|-----------|
|
||||
| 1. Foundation | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||
| 2. Adapters | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 3. Processors | v1.0 | 3/3 | Complete | 2026-03-15 |
|
||||
| 4. Pipeline | v1.0 | 2/2 | Complete | 2026-03-15 |
|
||||
| 5. 归档与目录优化 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||
| 6. 代码整合 | v1.1 | 3/3 | Complete | 2026-03-16 |
|
||||
| 7. 登录认证体系 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||
| 8. 内容爬取实现 | v1.2 | 3/3 | Complete | 2026-03-17 |
|
||||
| 9. Pipeline 整合 | v1.2 | 1/1 | Complete | 2026-03-17 |
|
||||
| 10. Proxy Pool | 2/2 | Complete | 2026-03-18 | - |
|
||||
| 11. Behavior Camouflage | v1.3 | 0/3 | Not started | - |
|
||||
| 12. CAPTCHA Detection | v1.3 | 0/2 | Not started | - |
|
||||
| 13. Error Recovery | v1.3 | 0/2 | Not started | - |
|
||||
| 14. Platform Adaptation | v1.3 | 0/2 | Not started | - |
|
||||
*Last updated: 2026-03-18*
|
||||
|
||||
+3
-3
@@ -4,10 +4,10 @@ milestone: v1.3
|
||||
milestone_name: Anti-Crawler Optimization
|
||||
status: executing
|
||||
stopped_at: Completed 10-02-PLAN.md
|
||||
last_updated: "2026-03-18T08:55:13.067Z"
|
||||
last_updated: "2026-03-18T14:42:25.803Z"
|
||||
last_activity: 2026-03-18 — 10-01 proxy pool core complete
|
||||
progress:
|
||||
total_phases: 14
|
||||
total_phases: 10
|
||||
completed_phases: 9
|
||||
total_plans: 29
|
||||
completed_plans: 28
|
||||
@@ -25,7 +25,7 @@ See: .planning/PROJECT.md (updated 2026-03-18)
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 10 of 14 (Proxy Pool)
|
||||
Phase: 10 of 10 (Proxy Pool)
|
||||
Plan: 1 of 2 in current phase
|
||||
Status: In progress
|
||||
Last activity: 2026-03-18 — 10-01 proxy pool core complete
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
# Requirements Archive: v1.3 Anti-Crawler Optimization
|
||||
|
||||
**Archived:** 2026-03-18
|
||||
**Status:** SHIPPED
|
||||
|
||||
For current requirements, see `.planning/REQUIREMENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
# Requirements: kb-skills
|
||||
|
||||
**Defined:** 2026-03-17
|
||||
@@ -0,0 +1,136 @@
|
||||
# Roadmap: kb-skills
|
||||
|
||||
## Milestones
|
||||
|
||||
- ✅ **v1.0 Pipeline Architecture** - Phases 1-4 (shipped 2011-03-15)
|
||||
- ✅ **v1.1 Architecture Refactor** - Phases 5-6 (shipped 2011-03-16)
|
||||
- ✅ **v1.2 Zhihu Crawler** - Phases 7-9 (shipped 2011-03-17)
|
||||
- 🚧 **v1.3 Anti-Crawler Optimization** - Phases 10-14 (in progress)
|
||||
|
||||
## Phases
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2011-03-15</summary>
|
||||
|
||||
### Phase 1: Foundation
|
||||
**Goal**: 项目基础设施和状态管理
|
||||
**Plans**: 3 plans
|
||||
|
||||
Plans:
|
||||
- [x] 01-01: WorkflowStateManager 状态管理
|
||||
- [x] 01-02: LinkParser URL 解析
|
||||
- [x] 01-03: IdempotencyChecker 幂等检查
|
||||
|
||||
### Phase 2: Adapters
|
||||
**Goal**: 平台元数据适配器
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [x] 02-01: BilibiliAdapter B站适配器
|
||||
- [x] 02-02: ZhihuAdapter 知乎适配器
|
||||
|
||||
### Phase 3: Processors
|
||||
**Goal**: 管道处理器架构
|
||||
**Plans**: 3 plans
|
||||
|
||||
Plans:
|
||||
- [x] 03-01: BaseProcessor 基类
|
||||
- [x] 03-02: 具体处理器实现
|
||||
- [x] 03-03: Schema 验证
|
||||
|
||||
### Phase 4: Pipeline
|
||||
**Goal**: 处理器编排和 CLI
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [x] 04-01: Pipeline 编排引擎
|
||||
- [x] 04-02: CLI process 命令
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2011-03-16</summary>
|
||||
|
||||
### Phase 5: 归档与目录优化
|
||||
**Goal**: 统一中间目录结构，添加自动归档步骤，优化最终目录布局
|
||||
**Depends on**: Phase 4
|
||||
**Requirements**: INT-01, INT-02, INT-03, ARC-01, ARC-02, ARC-03, ARC-04, OUT-01, OUT-02, OUT-03, OUT-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 所有分析任务的中间文件统一存放在 `~/work/intermediate/analysis/{platform}_{content_id}/`
|
||||
2. 分析完成后 md 文件自动复制到按平台分组的最终目录
|
||||
3. 最终目录结构为 `{KB_ARCHIVE_PATH}/{platform}/{YYYYMMDD}_{content_id}_analysis.md`
|
||||
4. 归档时检测重复文件，询问用户是否覆盖
|
||||
5. 不自动清理中间文件（用户手动维护）
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 6: 代码整合
|
||||
**Goal**: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow
|
||||
**Depends on**: Phase 5
|
||||
**Requirements**: MET-01, MET-02, MET-03, MET-04, MET-05, PIP-01, PIP-02, PIP-03, PIP-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. kb-bilibili-metadata 和 kb-zhihu-metadata 代码移入 kb-workflow/metadata/
|
||||
2. kb_pipeline/ 代码合并入 kb-workflow/ 保持 processors/, platforms/, schemas.py 结构
|
||||
3. 所有导入路径更新指向新位置，CLI 正常工作
|
||||
4. 原独立的 kb-bilibili-metadata、kb-zhihu-metadata、kb_pipeline 目录已删除
|
||||
5. 所有 sys.path 动态操作已删除或重构
|
||||
6. Pipeline 类可从 kb_workflow.pipeline 包级别导入
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2011-03-17</summary>
|
||||
|
||||
### Phase 7: 登录认证体系
|
||||
**Goal**: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式，实现 Cookie 持久化和过期检测
|
||||
**Depends on**: Phase 6
|
||||
**Requirements**: AUTH-01, AUTH-02, AUTH-03, AUTH-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以通过命令行显示二维码并扫码登录知乎
|
||||
2. 扫码失败时用户可以手动设置 Cookie 进行认证
|
||||
3. Cookie 自动保存到本地文件，下次运行自动复用
|
||||
4. Cookie 过期时系统自动检测并提示用户重新登录
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 8: 内容爬取实现
|
||||
**Goal**: 实现知乎问答、专栏/文章、评论的完整爬取功能，包含网络反爬措施
|
||||
**Depends on**: Phase 7
|
||||
**Requirements**: CONT-01, CONT-02, CONT-03, CONT-04, NET-01, NET-02, NET-03
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以爬取知乎问答（问题标题、描述、回答内容、元数据）
|
||||
2. 用户可以爬取知乎专栏/文章（标题、正文、元数据）
|
||||
3. 用户可以选择是否爬取内容下的评论
|
||||
4. 爬取结果包含作者信息、发布时间、点赞数等完整元数据
|
||||
5. 用户可以配置代理 IP 以降低被封风险
|
||||
**Plans**: 3 plans (all complete)
|
||||
|
||||
### Phase 9: Pipeline 整合
|
||||
**Goal**: 将新爬虫实现整合进现有 Pipeline 架构，替换原有知乎处理器
|
||||
**Depends on**: Phase 8
|
||||
**Requirements**: PIPE-01, PIPE-02, PIPE-03
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 知乎链接使用新爬虫处理器，无需手动设置 Cookie
|
||||
2. 新处理器与现有 Pipeline 架构完全兼容
|
||||
3. 支持断点续传和幂等执行（与其他处理器行为一致）
|
||||
**Plans**: 1 plan (complete)
|
||||
|
||||
</details>
|
||||
|
||||
### 🚧 v1.3 Anti-Crawler Optimization (In Progress)
|
||||
|
||||
**Milestone Goal:** 增强爬虫反检测能力，降低被封风险，提升爬取稳定性
|
||||
|
||||
#### Phase 10: Proxy Pool (代理池管理)
|
||||
**Goal**: 建立代理池基础设施，支持多代理配置、自动轮换和健康检查
|
||||
**Depends on**: Phase 9
|
||||
**Requirements**: PROXY-01, PROXY-02, PROXY-03, PROXY-04
|
||||
**Success Criteria** (what must be TRUE):
|
||||
1. 用户可以从配置文件或环境变量加载多个代理地址
|
||||
2. 每次请求自动选择下一个可用代理，实现负载均衡
|
||||
3. 代理不可用时自动检测并标记为失效
|
||||
4. 当前代理失败时自动切换到备用代理继续请求
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [x] 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换
|
||||
- [ ] 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换
|
||||
Reference in New Issue
Block a user