starmen1265/kb-skills
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: start milestone v1.3 Anti-Crawler Optimization

Browse Source
This commit is contained in:
starmen1265
2026-03-18 11:59:39 +08:00
parent 9c8ac34138
commit e4f96be0c4
2 changed files with 56 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/PROJECT.md
+25 -18
View File
@@ -1,16 +1,15 @@
# kb-skills 工作流重构
## Current Milestone: v1.2 Zhihu Crawler
## Current Milestone: v1.3 Anti-Crawler Optimization
**Goal:** 构建可靠的知乎内容爬取体系 — 解决 Cookie 过期问题，支持自动登录维持
**Goal:** 增强爬虫反检测能力，降低被封风险，提升爬取稳定性
**Target features:**
- 二维码扫码登录 + 手动 Cookie 降级
- Cookie 持久化存储
- 问答/专栏/文章内容爬取
- 评论爬取支持
- 代理 IP 支持
- 整合进现有 Pipeline
- 代理 IP 轮换机制
- 请求行为伪装（随机间隔、模拟人类行为）
- 验证码检测与处理
- 错误恢复机制（自动重试、指数退避）
- 知乎 + B站 双平台支持
## What This Is
@@ -62,17 +61,25 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
- ✓ OCR 文字识别 — ocr_processor.py
- ✓ AI 深度分析 — deep_analyzer.py
**v1.2 Zhihu Crawler (2026-03-17):**
- ✓ **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码，用户扫码
- ✓ **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie
- ✓ **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie，自动复用
- ✓ **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据
- ✓ **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据
- ✓ **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论
- ✓ **ZHIHU-07**: 代理 IP 支持 — 降低被封风险
- ✓ **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器
### Active
**v1.2 Zhihu Crawler:**
- [ ] **ZHIHU-01**: 二维码扫码登录 — playwright 提取二维码，用户扫码
- [ ] **ZHIHU-02**: 手动 Cookie 降级 — 扫码失败时支持手动设置 Cookie
- [ ] **ZHIHU-03**: Cookie 持久化 — 本地存储 Cookie，自动复用
- [ ] **ZHIHU-04**: 问答内容爬取 — 问题标题、回答内容、元数据
- [ ] **ZHIHU-05**: 专栏/文章爬取 — 文章标题、正文、元数据
- [ ] **ZHIHU-06**: 评论爬取 — 支持爬取内容下的评论
- [ ] **ZHIHU-07**: 代理 IP 支持 — 降低被封风险
- [ ] **ZHIHU-08**: Pipeline 整合 — 替换现有知乎处理器
**v1.3 Anti-Crawler Optimization:**
- [ ] **AC-01**: 代理池管理 — 支持配置代理池，自动轮换 IP
- [ ] **AC-02**: 请求行为伪装 — 随机化请求间隔，模拟人类行为
- [ ] **AC-03**: 验证码检测 — 检测验证码页面，触发告警/暂停
- [ ] **AC-04**: 错误恢复机制 — 自动重试、指数退避、断点续传增强
- [ ] **AC-05**: 知乎反爬适配 — 针对知乎的反爬策略优化
- [ ] **AC-06**: B站反爬适配 — 针对 B站的反爬策略优化
### Deferred
@@ -133,4 +140,4 @@ python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx
| 懒加载导入 | AnalyzeProcessor 懒加载 DeepAnalyzer | ✓ Good — 避免依赖问题 |
---
*Last updated: 2026-03-17 after v1.2 milestone started*
*Last updated: 2026-03-18 after v1.3 milestone started*
.planning/STATE.md
+31 -38
View File
@@ -1,54 +1,47 @@
---
gsd_state_version: 1.0
milestone: v1.2
milestone_name: Zhihu Crawler
status: completed
stopped_at: Completed 09-01 ZhihuAdapter integration
last_updated: "2026-03-17T15:25:04.074Z"
last_activity: 2026-03-17 — Phase 09-00 Test scaffolding complete
milestone: v1.3
milestone_name: Anti-Crawler Optimization
status: planning
stopped_at: Defining requirements
last_updated: "2026-03-18T00:00:00.000Z"
last_activity: 2026-03-18 — Milestone v1.3 started
progress:
total_phases: 9
completed_phases: 8
total_plans: 27
completed_plans: 26
percent: 100
total_phases: 0
completed_phases: 0
total_plans: 0
completed_plans: 0
percent: 0
---
# Project State
## Project Reference
See: .planning/PROJECT.md (updated 2026-03-17)
See: .planning/PROJECT.md (updated 2026-03-18)
**Core value:** 可靠、可维护的处理管道 - 每个步骤独立、可验证、可恢复
**Current focus:** v1.2 Zhihu Crawler — Phase 9 执行中
**Current focus:** v1.3 Anti-Crawler Optimization — 反爬优化
## Current Position
Phase: 9 of 9 (Pipeline Integration - 管道整合)
Plan: 01 of 04
Status: Phase 09-00 Wave 0 complete
Last activity: 2026-03-17 — Phase 09-00 Test scaffolding complete
Phase: Not started (defining requirements)
Plan: —
Status: Defining requirements
Last activity: 2026-03-18 — Milestone v1.3 started
Progress: [██████████] 100% (v1.2 milestone - Phase 9 in progress)
Progress: [░░░░░░░░░░] 0% (v1.3 milestone - requirements phase)
## Performance Metrics
**Velocity:**
- Total plans completed: 3 (v1.2)
- Average duration: 11 min
- Total execution time: 32 min
- Total plans completed: 0 (v1.3)
- Average duration: —
- Total execution time: —
| Plan | Duration | Tasks | Files | Date |
|------|----------|-------|-------|------|
| 07-01 | 15min | 2 | 5 | 2026-03-17 |
| 07-02 | 5min | 2 | 4 | 2026-03-17 |
| 07-03 | 12min | 3 | 4 | 2026-03-17 |
| Phase 08 P01 | 3min | 2 | 8 | 2026-03-17 |
| Phase 08 P02 | 5min | 2 | 3 | 2026-03-17 |
| Phase 08 P03 | 15min | 2 | 5 | 2026-03-17 |
| 09-00 | 2min | 1 | 1 | 2026-03-17 |
| Phase 09-pipeline-integration P01 | 11min | 3 tasks | 2 files |
| (no plans yet) |
## Accumulated Context
@@ -76,12 +69,12 @@ v1.2 decisions:
- 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时
- 多个 canvas 选择器降级提高 QR 码检测鲁棒性
- login CLI 命令支持三种模式：扫码登录、导入 Cookie、状态检查
- [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception
- [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies)
- [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments
- [Phase 09-01]: ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default
- [Phase 09-01]: Cookie expiration triggers automatic QR code login flow
- [Phase 09-01]: Comments appended to content in Markdown format with '## 热门评论' section
- Use Playwright request API for direct comment API calls instead of route interception
- Only fetch first-level comments (no nested replies)
- Use hot sorting (order=hot) for popular comments
- ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default
- Cookie expiration triggers automatic QR code login flow
- Comments appended to content in Markdown format with '## 热门评论' section
### Blockers/Concerns
@@ -89,8 +82,8 @@ None.
## Session Continuity
Last session: 2026-03-17T15:16:21.489Z
Stopped at: Completed 09-01 ZhihuAdapter integration
Last session: —
Stopped at: Defining requirements
Resume file: None
**Next step:** Continue with Phase 09-01 ZhihuAdapter integration
**Next step:** Define v1.3 requirements and create roadmap