starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
96dfdce59c6f6f4facb98a8379a2ac6766ba19a2
kb-skills/.planning/ROADMAP.md
T
starmen1265 96dfdce59c docs(10-01): complete proxy pool core plan 
- SUMMARY.md: proxy pool infrastructure with round-robin rotation
- STATE.md: updated position to Plan 1 of 2
- ROADMAP.md: marked 10-01 complete
- REQUIREMENTS.md: marked PROXY-01, PROXY-02 complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 16:43:34 +08:00

8.8 KiB
Raw Blame History

Roadmap: kb-skills

Milestones

  • ✅ v1.0 Pipeline Architecture - Phases 1-4 (shipped 2026-03-15)
  • ✅ v1.1 Architecture Refactor - Phases 5-6 (shipped 2026-03-16)
  • ✅ v1.2 Zhihu Crawler - Phases 7-9 (shipped 2026-03-17)
  • 🚧 v1.3 Anti-Crawler Optimization - Phases 10-14 (in progress)

Phases

✅ v1.0 Pipeline Architecture (Phases 1-4) - SHIPPED 2026-03-15

Phase 1: Foundation

Goal: 项目基础设施和状态管理 Plans: 3 plans

Plans:

  • 01-01: WorkflowStateManager 状态管理
  • 01-02: LinkParser URL 解析
  • 01-03: IdempotencyChecker 幂等检查

Phase 2: Adapters

Goal: 平台元数据适配器 Plans: 2 plans

Plans:

  • 02-01: BilibiliAdapter B站适配器
  • 02-02: ZhihuAdapter 知乎适配器

Phase 3: Processors

Goal: 管道处理器架构 Plans: 3 plans

Plans:

  • 03-01: BaseProcessor 基类
  • 03-02: 具体处理器实现
  • 03-03: Schema 验证

Phase 4: Pipeline

Goal: 处理器编排和 CLI Plans: 2 plans

Plans:

  • 04-01: Pipeline 编排引擎
  • 04-02: CLI process 命令
✅ v1.1 Architecture Refactor (Phases 5-6) - SHIPPED 2026-03-16

Phase 5: 归档与目录优化

Goal: 统一中间目录结构，添加自动归档步骤，优化最终目录布局 Depends on: Phase 4 Requirements: INT-01, INT-02, INT-03, ARC-01, ARC-02, ARC-03, ARC-04, OUT-01, OUT-02, OUT-03, OUT-04 Success Criteria (what must be TRUE):

  1. 所有分析任务的中间文件统一存放在 ~/work/intermediate/analysis/{platform}_{content_id}/
  2. 分析完成后 md 文件自动复制到按平台分组的最终目录
  3. 最终目录结构为 {KB_ARCHIVE_PATH}/{platform}/{YYYYMMDD}_{content_id}_analysis.md
  4. 归档时检测重复文件，询问用户是否覆盖
  5. 不自动清理中间文件（用户手动维护） Plans: 3 plans (all complete)

Phase 6: 代码整合

Goal: 将独立的元数据技能和 Pipeline 模块整合进 kb-workflow Depends on: Phase 5 Requirements: MET-01, MET-02, MET-03, MET-04, MET-05, PIP-01, PIP-02, PIP-03, PIP-04 Success Criteria (what must be TRUE):

  1. kb-bilibili-metadata 和 kb-zhihu-metadata 代码移入 kb-workflow/metadata/
  2. kb_pipeline/ 代码合并入 kb-workflow/ 保持 processors/, platforms/, schemas.py 结构
  3. 所有导入路径更新指向新位置，CLI 正常工作
  4. 原独立的 kb-bilibili-metadata、kb-zhihu-metadata、kb_pipeline 目录已删除
  5. 所有 sys.path 动态操作已删除或重构
  6. Pipeline 类可从 kb_workflow.pipeline 包级别导入 Plans: 3 plans (all complete)
✅ v1.2 Zhihu Crawler (Phases 7-9) - SHIPPED 2026-03-17

Phase 7: 登录认证体系

Goal: 建立知乎登录认证机制，支持二维码扫码和手动 Cookie 两种方式，实现 Cookie 持久化和过期检测 Depends on: Phase 6 Requirements: AUTH-01, AUTH-02, AUTH-03, AUTH-04 Success Criteria (what must be TRUE):

  1. 用户可以通过命令行显示二维码并扫码登录知乎
  2. 扫码失败时用户可以手动设置 Cookie 进行认证
  3. Cookie 自动保存到本地文件，下次运行自动复用
  4. Cookie 过期时系统自动检测并提示用户重新登录 Plans: 3 plans (all complete)

Phase 8: 内容爬取实现

Goal: 实现知乎问答、专栏/文章、评论的完整爬取功能，包含网络反爬措施 Depends on: Phase 7 Requirements: CONT-01, CONT-02, CONT-03, CONT-04, NET-01, NET-02, NET-03 Success Criteria (what must be TRUE):

  1. 用户可以爬取知乎问答（问题标题、描述、回答内容、元数据）
  2. 用户可以爬取知乎专栏/文章（标题、正文、元数据）
  3. 用户可以选择是否爬取内容下的评论
  4. 爬取结果包含作者信息、发布时间、点赞数等完整元数据
  5. 用户可以配置代理 IP 以降低被封风险 Plans: 3 plans (all complete)

Phase 9: Pipeline 整合

Goal: 将新爬虫实现整合进现有 Pipeline 架构，替换原有知乎处理器 Depends on: Phase 8 Requirements: PIPE-01, PIPE-02, PIPE-03 Success Criteria (what must be TRUE):

  1. 知乎链接使用新爬虫处理器，无需手动设置 Cookie
  2. 新处理器与现有 Pipeline 架构完全兼容
  3. 支持断点续传和幂等执行（与其他处理器行为一致） Plans: 1 plan (complete)

🚧 v1.3 Anti-Crawler Optimization (In Progress)

Milestone Goal: 增强爬虫反检测能力，降低被封风险，提升爬取稳定性

Phase 10: Proxy Pool (代理池管理)

Goal: 建立代理池基础设施，支持多代理配置、自动轮换和健康检查 Depends on: Phase 9 Requirements: PROXY-01, PROXY-02, PROXY-03, PROXY-04 Success Criteria (what must be TRUE):

  1. 用户可以从配置文件或环境变量加载多个代理地址
  2. 每次请求自动选择下一个可用代理，实现负载均衡
  3. 代理不可用时自动检测并标记为失效
  4. 当前代理失败时自动切换到备用代理继续请求 Plans: 2 plans

Plans:

  • 10-01: ProxyPool 核心类实现 — 从 YAML/环境变量加载代理，Round-Robin 轮换
  • 10-02: 健康检查与 ZhihuCrawler 集成 — 被动检测、冷却恢复、代理切换

Phase 11: Behavior Camouflage (请求行为伪装)

Goal: 通过随机化请求特征模拟真实用户行为，降低被反爬系统检测的风险 Depends on: Phase 10 Requirements: BEHAV-01, BEHAV-02, BEHAV-03, BEHAV-04 Success Criteria (what must be TRUE):

  1. 连续请求之间有随机的延迟间隔（默认 1-5 秒）
  2. 每次请求使用不同的真实浏览器 User-Agent
  3. 请求头中的 Accept-Language、Accept-Encoding 等字段随机化
  4. 支持通过 Playwright 模拟页面滚动和鼠标移动行为 Plans: TBD

Plans:

  • 11-01: 请求延迟与 UA 轮换 (BEHAV-01, BEHAV-02)
  • 11-02: 请求头随机化 (BEHAV-03)
  • 11-03: Playwright 行为模拟 (BEHAV-04)

Phase 12: CAPTCHA Detection (验证码检测)

Goal: 实现验证码页面检测机制，在触发验证码时优雅地暂停并通知用户 Depends on: Phase 11 Requirements: CAPT-01, CAPT-02, CAPT-03 Success Criteria (what must be TRUE):

  1. 系统能自动识别知乎和 B 站的验证码页面
  2. 检测到验证码时记录日志并暂停当前爬取任务
  3. 用户手动处理验证码后可以继续爬取 Plans: TBD

Plans:

  • 12-01: 验证码页面检测与告警 (CAPT-01, CAPT-02)
  • 12-02: 手动恢复机制 (CAPT-03)

Phase 13: Error Recovery (错误恢复机制)

Goal: 建立智能的错误分类和重试策略，提升爬取任务的健壮性 Depends on: Phase 12 Requirements: ERR-01, ERR-02, ERR-03, ERR-04 Success Criteria (what must be TRUE):

  1. 403/429/超时等不同错误类型采用不同的重试策略
  2. 重试间隔使用指数退避并添加随机抖动，避免请求风暴
  3. 永久性错误（如内容不存在）不再重试，记录原因后跳过
  4. 失败位置被记录，用户可以从失败点恢复爬取 Plans: TBD

Plans:

  • 13-01: 错误分类与重试策略 (ERR-01, ERR-02, ERR-03)
  • 13-02: 断点续传增强 (ERR-04)

Phase 14: Platform Adaptation (平台适配)

Goal: 针对知乎和 B 站的反爬策略进行专项优化，整合前面所有反爬能力 Depends on: Phase 13 Requirements: PLAT-01, PLAT-02 Success Criteria (what must be TRUE):

  1. 知乎爬取针对登录态检测和请求频率限制进行了优化
  2. B 站爬取针对视频/专栏元数据获取进行了优化
  3. 两个平台的爬取成功率相比 v1.2 有明显提升 Plans: TBD

Plans:

  • 14-01: 知乎反爬适配 (PLAT-01)
  • 14-02: B 站反爬适配 (PLAT-02)

Progress

Execution Order: Phases execute in numeric order: 1 → ... → 9 → 10 → 11 → 12 → 13 → 14

Phase Milestone Plans Complete Status Completed
1. Foundation v1.0 3/3 Complete 2026-03-15
2. Adapters v1.0 2/2 Complete 2026-03-15
3. Processors v1.0 3/3 Complete 2026-03-15
4. Pipeline v1.0 2/2 Complete 2026-03-15
5. 归档与目录优化 v1.1 3/3 Complete 2026-03-16
6. 代码整合 v1.1 3/3 Complete 2026-03-16
7. 登录认证体系 v1.2 3/3 Complete 2026-03-17
8. 内容爬取实现 v1.2 3/3 Complete 2026-03-17
9. Pipeline 整合 v1.2 1/1 Complete 2026-03-17
10. Proxy Pool v1.3 1/2 In progress -
11. Behavior Camouflage v1.3 0/3 Not started -
12. CAPTCHA Detection v1.3 0/2 Not started -
13. Error Recovery v1.3 0/2 Not started -
14. Platform Adaptation v1.3 0/2 Not started -
Reference in New Issue View Git Blame Copy Permalink