starmen1265
6a4f13b2f6
- Add 08-02-SUMMARY.md with execution details - Update STATE.md with decisions and metrics - Mark CONT-03 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
102 lines
3.3 KiB
Markdown
102 lines
3.3 KiB
Markdown
---
|
|
phase: 08-crawler
|
|
plan: 02
|
|
subsystem: crawler
|
|
tags: [playwright, pydantic, zhihu, comments, api-interception]
|
|
|
|
requires:
|
|
- phase: 08-01
|
|
provides: ZhihuCrawler class + crawl_answer/crawl_article methods
|
|
provides:
|
|
- crawl_comments() method for hot comment crawling
|
|
- CommentData, CommentsResult, CommentAuthor data models
|
|
- include_comments parameter in crawl_answer()
|
|
affects: [08-03]
|
|
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- Playwright page.request.get() for direct API calls
|
|
- Hot comment sorting via order=hot parameter
|
|
|
|
key-files:
|
|
created:
|
|
- tests/test_crawler/test_comment_crawler.py
|
|
modified:
|
|
- kb_workflow/crawler/models.py
|
|
- kb_workflow/crawler/zhihu_crawler.py
|
|
- kb_workflow/crawler/__init__.py
|
|
|
|
key-decisions:
|
|
- Use Playwright request API for direct comment API calls instead of route interception
|
|
- Only fetch first-level comments (no nested replies)
|
|
- Use hot sorting (order=hot) for popular comments
|
|
|
|
requirements-completed: [CONT-03]
|
|
|
|
duration: 5min
|
|
completed: 2026-03-17
|
|
---
|
|
|
|
# Phase 08 Plan 02: Comment Crawler Summary
|
|
|
|
**Comment crawling with hot sorting via Playwright request API, supporting optional comment fetching in crawl_answer()**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** 5 min
|
|
- **Started:** 2026-03-17T13:52:37Z
|
|
- **Completed:** 2026-03-17T13:57:00Z
|
|
- **Tasks:** 2
|
|
- **Files modified:** 3
|
|
|
|
## Accomplishments
|
|
|
|
- CommentData, CommentAuthor, CommentsResult data models for structured comment data
|
|
- crawl_comments() method using hot sorting API (order=hot)
|
|
- include_comments parameter in crawl_answer() for optional comment fetching
|
|
- 6 new tests for comment crawling functionality
|
|
|
|
## Task Commits
|
|
|
|
Each task was committed atomically:
|
|
|
|
1. **Task 1: Add CommentData models** - `b1f5792` (feat)
|
|
2. **Task 2: Implement crawl_comments method** - `8cc4696` (test), `f41e75a` (feat)
|
|
|
|
## Files Created/Modified
|
|
|
|
- `kb_workflow/crawler/models.py` - Added CommentAuthor, CommentData, CommentsResult models; updated CrawlResult with comments field
|
|
- `kb_workflow/crawler/zhihu_crawler.py` - Added crawl_comments() method and COMMENT_API_TEMPLATE; updated crawl_answer() with include_comments parameter
|
|
- `kb_workflow/crawler/__init__.py` - Exported CommentAuthor, CommentData, CommentsResult
|
|
- `tests/test_crawler/test_comment_crawler.py` - 6 tests for comment crawling
|
|
|
|
## Decisions Made
|
|
|
|
1. **Direct API calls vs route interception:** Used Playwright's page.request.get() for comment API instead of route interception, as comments are fetched via a separate API call after the main content is loaded
|
|
2. **First-level comments only:** Only fetch first-level comments to keep the feature simple; child_count is recorded but replies are not expanded
|
|
3. **Hot sorting:** Always use order=hot parameter to get the most popular comments
|
|
|
|
## Deviations from Plan
|
|
|
|
None - plan executed exactly as written.
|
|
|
|
## Issues Encountered
|
|
|
|
- Test 6 (test_include_comments_in_crawl_answer) needed adjustment - the original test tried to patch a non-existent `_crawl_answer_internal` method. Fixed by properly mocking the interceptor and browser components.
|
|
|
|
## User Setup Required
|
|
|
|
None - no external service configuration required.
|
|
|
|
## Next Phase Readiness
|
|
|
|
- Comment crawling ready for integration into content pipeline
|
|
- Phase 08-03 can use crawl_comments for full content extraction with optional comments
|
|
|
|
---
|
|
*Phase: 08-crawler*
|
|
*Completed: 2026-03-17*
|
|
|
|
## Self-Check: PASSED
|