Archived

This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

starmen1265 6a4f13b2f6 docs(08-02): complete comment crawler plan

- Add 08-02-SUMMARY.md with execution details
- Update STATE.md with decisions and metrics
- Mark CONT-03 requirement complete

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-17 21:59:49 +08:00

3.3 KiB

Raw Blame History

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, duration, completed

phase

plan

subsystem

tags

requires

provides

affects

tech-stack

key-files

key-decisions

requirements-completed

duration

completed

08-crawler

crawler

playwright

pydantic

zhihu

comments

api-interception

phase	provides
08-01	ZhihuCrawler class + crawl_answer/crawl_article methods

crawl_comments() method for hot comment crawling

CommentData, CommentsResult, CommentAuthor data models

include_comments parameter in crawl_answer()

08-03

added

patterns

Playwright page.request.get() for direct API calls

Hot comment sorting via order=hot parameter

created

modified

tests/test_crawler/test_comment_crawler.py

kb_workflow/crawler/models.py

kb_workflow/crawler/zhihu_crawler.py

kb_workflow/crawler/__init__.py

Use Playwright request API for direct comment API calls instead of route interception

Only fetch first-level comments (no nested replies)

Use hot sorting (order=hot) for popular comments

CONT-03

5min

2026-03-17

Phase 08 Plan 02: Comment Crawler Summary

Comment crawling with hot sorting via Playwright request API, supporting optional comment fetching in crawl_answer()

Performance

Duration: 5 min
Started: 2026-03-17T13:52:37Z
Completed: 2026-03-17T13:57:00Z
Tasks: 2
Files modified: 3

Accomplishments

CommentData, CommentAuthor, CommentsResult data models for structured comment data
crawl_comments() method using hot sorting API (order=hot)
include_comments parameter in crawl_answer() for optional comment fetching
6 new tests for comment crawling functionality

Task Commits

Each task was committed atomically:

Task 1: Add CommentData models - b1f5792 (feat)
Task 2: Implement crawl_comments method - 8cc4696 (test), f41e75a (feat)

Files Created/Modified

kb_workflow/crawler/models.py - Added CommentAuthor, CommentData, CommentsResult models; updated CrawlResult with comments field
kb_workflow/crawler/zhihu_crawler.py - Added crawl_comments() method and COMMENT_API_TEMPLATE; updated crawl_answer() with include_comments parameter
kb_workflow/crawler/__init__.py - Exported CommentAuthor, CommentData, CommentsResult
tests/test_crawler/test_comment_crawler.py - 6 tests for comment crawling

Decisions Made

Direct API calls vs route interception: Used Playwright's page.request.get() for comment API instead of route interception, as comments are fetched via a separate API call after the main content is loaded
First-level comments only: Only fetch first-level comments to keep the feature simple; child_count is recorded but replies are not expanded
Hot sorting: Always use order=hot parameter to get the most popular comments

Deviations from Plan

None - plan executed exactly as written.

Issues Encountered

Test 6 (test_include_comments_in_crawl_answer) needed adjustment - the original test tried to patch a non-existent _crawl_answer_internal method. Fixed by properly mocking the interceptor and browser components.

User Setup Required

None - no external service configuration required.

Next Phase Readiness

Comment crawling ready for integration into content pipeline
Phase 08-03 can use crawl_comments for full content extraction with optional comments

Phase: 08-crawler Completed: 2026-03-17

3.3 KiB Raw Blame History