docs(08-02): complete comment crawler plan
- Add 08-02-SUMMARY.md with execution details - Update STATE.md with decisions and metrics - Mark CONT-03 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -18,7 +18,7 @@
|
|||||||
|
|
||||||
- [x] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
|
- [x] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
|
||||||
- [x] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
|
- [x] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
|
||||||
- [ ] **CONT-03**: 爬取内容下的评论(可选开关)
|
- [x] **CONT-03**: 爬取内容下的评论(可选开关)
|
||||||
- [x] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
|
- [x] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
|
||||||
|
|
||||||
### 网络与反爬
|
### 网络与反爬
|
||||||
@@ -117,7 +117,7 @@
|
|||||||
| AUTH-04 | Phase 7 | Complete |
|
| AUTH-04 | Phase 7 | Complete |
|
||||||
| CONT-01 | Phase 8 | Complete |
|
| CONT-01 | Phase 8 | Complete |
|
||||||
| CONT-02 | Phase 8 | Complete |
|
| CONT-02 | Phase 8 | Complete |
|
||||||
| CONT-03 | Phase 8 | Pending |
|
| CONT-03 | Phase 8 | Complete |
|
||||||
| CONT-04 | Phase 8 | Complete |
|
| CONT-04 | Phase 8 | Complete |
|
||||||
| NET-01 | Phase 8 | Pending |
|
| NET-01 | Phase 8 | Pending |
|
||||||
| NET-02 | Phase 8 | Pending |
|
| NET-02 | Phase 8 | Pending |
|
||||||
|
|||||||
+12
-7
@@ -3,14 +3,14 @@ gsd_state_version: 1.0
|
|||||||
milestone: v1.2
|
milestone: v1.2
|
||||||
milestone_name: Zhihu Crawler
|
milestone_name: Zhihu Crawler
|
||||||
status: completed
|
status: completed
|
||||||
stopped_at: Phase 08 context gathered
|
stopped_at: Completed 08-crawler-02-PLAN.md
|
||||||
last_updated: "2026-03-17T10:57:02.248Z"
|
last_updated: "2026-03-17T13:59:35.901Z"
|
||||||
last_activity: 2026-03-17 — Phase 07 Login Auth complete
|
last_activity: 2026-03-17 — Phase 07 Login Auth complete
|
||||||
progress:
|
progress:
|
||||||
total_phases: 9
|
total_phases: 9
|
||||||
completed_phases: 7
|
completed_phases: 7
|
||||||
total_plans: 21
|
total_plans: 24
|
||||||
completed_plans: 21
|
completed_plans: 23
|
||||||
percent: 100
|
percent: 100
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -44,6 +44,8 @@ Progress: [██████████] 100% (v1.2 milestone - Phase 7 comple
|
|||||||
| 07-01 | 15min | 2 | 5 | 2026-03-17 |
|
| 07-01 | 15min | 2 | 5 | 2026-03-17 |
|
||||||
| 07-02 | 5min | 2 | 4 | 2026-03-17 |
|
| 07-02 | 5min | 2 | 4 | 2026-03-17 |
|
||||||
| 07-03 | 12min | 3 | 4 | 2026-03-17 |
|
| 07-03 | 12min | 3 | 4 | 2026-03-17 |
|
||||||
|
| Phase 08 P01 | 3min | 2 tasks | 8 files |
|
||||||
|
| Phase 08-crawler P02 | 5min | 2 tasks | 3 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -71,6 +73,9 @@ v1.2 decisions:
|
|||||||
- 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时
|
- 使用 tenacity Retrying 对象而非 @retry 装饰器支持可配置超时
|
||||||
- 多个 canvas 选择器降级提高 QR 码检测鲁棒性
|
- 多个 canvas 选择器降级提高 QR 码检测鲁棒性
|
||||||
- login CLI 命令支持三种模式:扫码登录、导入 Cookie、状态检查
|
- login CLI 命令支持三种模式:扫码登录、导入 Cookie、状态检查
|
||||||
|
- [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception
|
||||||
|
- [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies)
|
||||||
|
- [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments
|
||||||
|
|
||||||
### Blockers/Concerns
|
### Blockers/Concerns
|
||||||
|
|
||||||
@@ -78,8 +83,8 @@ None.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-03-17T10:57:02.246Z
|
Last session: 2026-03-17T13:59:35.900Z
|
||||||
Stopped at: Phase 08 context gathered
|
Stopped at: Completed 08-crawler-02-PLAN.md
|
||||||
Resume file: .planning/phases/08-crawler/08-CONTEXT.md
|
Resume file: None
|
||||||
|
|
||||||
**Next step:** Continue with Phase 08 Crawler
|
**Next step:** Continue with Phase 08 Crawler
|
||||||
|
|||||||
@@ -0,0 +1,101 @@
|
|||||||
|
---
|
||||||
|
phase: 08-crawler
|
||||||
|
plan: 02
|
||||||
|
subsystem: crawler
|
||||||
|
tags: [playwright, pydantic, zhihu, comments, api-interception]
|
||||||
|
|
||||||
|
requires:
|
||||||
|
- phase: 08-01
|
||||||
|
provides: ZhihuCrawler class + crawl_answer/crawl_article methods
|
||||||
|
provides:
|
||||||
|
- crawl_comments() method for hot comment crawling
|
||||||
|
- CommentData, CommentsResult, CommentAuthor data models
|
||||||
|
- include_comments parameter in crawl_answer()
|
||||||
|
affects: [08-03]
|
||||||
|
|
||||||
|
tech-stack:
|
||||||
|
added: []
|
||||||
|
patterns:
|
||||||
|
- Playwright page.request.get() for direct API calls
|
||||||
|
- Hot comment sorting via order=hot parameter
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- tests/test_crawler/test_comment_crawler.py
|
||||||
|
modified:
|
||||||
|
- kb_workflow/crawler/models.py
|
||||||
|
- kb_workflow/crawler/zhihu_crawler.py
|
||||||
|
- kb_workflow/crawler/__init__.py
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- Use Playwright request API for direct comment API calls instead of route interception
|
||||||
|
- Only fetch first-level comments (no nested replies)
|
||||||
|
- Use hot sorting (order=hot) for popular comments
|
||||||
|
|
||||||
|
requirements-completed: [CONT-03]
|
||||||
|
|
||||||
|
duration: 5min
|
||||||
|
completed: 2026-03-17
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 08 Plan 02: Comment Crawler Summary
|
||||||
|
|
||||||
|
**Comment crawling with hot sorting via Playwright request API, supporting optional comment fetching in crawl_answer()**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** 5 min
|
||||||
|
- **Started:** 2026-03-17T13:52:37Z
|
||||||
|
- **Completed:** 2026-03-17T13:57:00Z
|
||||||
|
- **Tasks:** 2
|
||||||
|
- **Files modified:** 3
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
- CommentData, CommentAuthor, CommentsResult data models for structured comment data
|
||||||
|
- crawl_comments() method using hot sorting API (order=hot)
|
||||||
|
- include_comments parameter in crawl_answer() for optional comment fetching
|
||||||
|
- 6 new tests for comment crawling functionality
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
Each task was committed atomically:
|
||||||
|
|
||||||
|
1. **Task 1: Add CommentData models** - `b1f5792` (feat)
|
||||||
|
2. **Task 2: Implement crawl_comments method** - `8cc4696` (test), `f41e75a` (feat)
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `kb_workflow/crawler/models.py` - Added CommentAuthor, CommentData, CommentsResult models; updated CrawlResult with comments field
|
||||||
|
- `kb_workflow/crawler/zhihu_crawler.py` - Added crawl_comments() method and COMMENT_API_TEMPLATE; updated crawl_answer() with include_comments parameter
|
||||||
|
- `kb_workflow/crawler/__init__.py` - Exported CommentAuthor, CommentData, CommentsResult
|
||||||
|
- `tests/test_crawler/test_comment_crawler.py` - 6 tests for comment crawling
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
1. **Direct API calls vs route interception:** Used Playwright's page.request.get() for comment API instead of route interception, as comments are fetched via a separate API call after the main content is loaded
|
||||||
|
2. **First-level comments only:** Only fetch first-level comments to keep the feature simple; child_count is recorded but replies are not expanded
|
||||||
|
3. **Hot sorting:** Always use order=hot parameter to get the most popular comments
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
None - plan executed exactly as written.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
- Test 6 (test_include_comments_in_crawl_answer) needed adjustment - the original test tried to patch a non-existent `_crawl_answer_internal` method. Fixed by properly mocking the interceptor and browser components.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None - no external service configuration required.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
- Comment crawling ready for integration into content pipeline
|
||||||
|
- Phase 08-03 can use crawl_comments for full content extraction with optional comments
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 08-crawler*
|
||||||
|
*Completed: 2026-03-17*
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
Reference in New Issue
Block a user