docs(08-02): complete comment crawler plan

- Add 08-02-SUMMARY.md with execution details - Update STATE.md with decisions and metrics - Mark CONT-03 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-17 21:59:49 +08:00
parent 626fb73a58
commit 6a4f13b2f6
3 changed files with 115 additions and 9 deletions
@@ -0,0 +1,101 @@
+---
+phase: 08-crawler
+plan: 02
+subsystem: crawler
+tags: [playwright, pydantic, zhihu, comments, api-interception]
+
+requires:
+  - phase: 08-01
+    provides: ZhihuCrawler class + crawl_answer/crawl_article methods
+provides:
+  - crawl_comments() method for hot comment crawling
+  - CommentData, CommentsResult, CommentAuthor data models
+  - include_comments parameter in crawl_answer()
+affects: [08-03]
+
+tech-stack:
+  added: []
+  patterns:
+    - Playwright page.request.get() for direct API calls
+    - Hot comment sorting via order=hot parameter
+
+key-files:
+  created:
+    - tests/test_crawler/test_comment_crawler.py
+  modified:
+    - kb_workflow/crawler/models.py
+    - kb_workflow/crawler/zhihu_crawler.py
+    - kb_workflow/crawler/__init__.py
+
+key-decisions:
+  - Use Playwright request API for direct comment API calls instead of route interception
+  - Only fetch first-level comments (no nested replies)
+  - Use hot sorting (order=hot) for popular comments
+
+requirements-completed: [CONT-03]
+
+duration: 5min
+completed: 2026-03-17
+---
+
+# Phase 08 Plan 02: Comment Crawler Summary
+
+**Comment crawling with hot sorting via Playwright request API, supporting optional comment fetching in crawl_answer()**
+
+## Performance
+
+- **Duration:** 5 min
+- **Started:** 2026-03-17T13:52:37Z
+- **Completed:** 2026-03-17T13:57:00Z
+- **Tasks:** 2
+- **Files modified:** 3
+
+## Accomplishments
+
+- CommentData, CommentAuthor, CommentsResult data models for structured comment data
+- crawl_comments() method using hot sorting API (order=hot)
+- include_comments parameter in crawl_answer() for optional comment fetching
+- 6 new tests for comment crawling functionality
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Add CommentData models** - `b1f5792` (feat)
+2. **Task 2: Implement crawl_comments method** - `8cc4696` (test), `f41e75a` (feat)
+
+## Files Created/Modified
+
+- `kb_workflow/crawler/models.py` - Added CommentAuthor, CommentData, CommentsResult models; updated CrawlResult with comments field
+- `kb_workflow/crawler/zhihu_crawler.py` - Added crawl_comments() method and COMMENT_API_TEMPLATE; updated crawl_answer() with include_comments parameter
+- `kb_workflow/crawler/__init__.py` - Exported CommentAuthor, CommentData, CommentsResult
+- `tests/test_crawler/test_comment_crawler.py` - 6 tests for comment crawling
+
+## Decisions Made
+
+1. **Direct API calls vs route interception:** Used Playwright's page.request.get() for comment API instead of route interception, as comments are fetched via a separate API call after the main content is loaded
+2. **First-level comments only:** Only fetch first-level comments to keep the feature simple; child_count is recorded but replies are not expanded
+3. **Hot sorting:** Always use order=hot parameter to get the most popular comments
+
+## Deviations from Plan
+
+None - plan executed exactly as written.
+
+## Issues Encountered
+
+- Test 6 (test_include_comments_in_crawl_answer) needed adjustment - the original test tried to patch a non-existent `_crawl_answer_internal` method. Fixed by properly mocking the interceptor and browser components.
+
+## User Setup Required
+
+None - no external service configuration required.
+
+## Next Phase Readiness
+
+- Comment crawling ready for integration into content pipeline
+- Phase 08-03 can use crawl_comments for full content extraction with optional comments
+
+---
+*Phase: 08-crawler*
+*Completed: 2026-03-17*
+
+## Self-Check: PASSED