---
phase: 08-crawler
plan: 02
type: execute
wave: 2
depends_on: [08-01]
files_modified:
- kb_workflow/crawler/zhihu_crawler.py
- kb_workflow/crawler/models.py
- tests/test_crawler/test_comment_crawler.py
autonomous: true
requirements: [CONT-03]
user_setup: []
must_haves:
truths:
- "用户可以选择是否爬取内容下的评论"
- "爬取的评论是热门评论（非全部评论）"
- "只爬取第一层评论，不展开嵌套回复"
artifacts:
- path: "kb_workflow/crawler/models.py"
provides: "CommentData 数据模型"
contains: "class CommentData"
- path: "kb_workflow/crawler/zhihu_crawler.py"
provides: "crawl_comments() 方法"
contains: "def crawl_comments"
- path: "tests/test_crawler/test_comment_crawler.py"
provides: "评论爬取测试"
exports: ["test_crawl_comments"]
key_links:
- from: "ZhihuCrawler.crawl_comments"
to: "/api/v4/comment_v5/{type}/{id}/comments"
via: "热门评论 API"
pattern: "comment_v5.*order=hot"
---
实现知乎评论爬取功能，通过热门评论 API 获取第一层评论，作为可选功能集成到 ZhihuCrawler。
Purpose: 提供可选的评论爬取能力，丰富内容元数据
Output: crawl_comments() 方法 + CommentData 模型 + 测试
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/08-crawler/08-CONTEXT.md
@.planning/phases/08-crawler/08-RESEARCH.md
@.planning/phases/08-crawler/08-01-SUMMARY.md
评论 API 格式:
```
https://www.zhihu.com/api/v4/comment_v5/{content_type}/{content_id}/comments
?order=hot&limit=20
```
参数说明:
- content_type: "answers" (回答) 或 "articles" (文章)
- content_id: 内容 ID
- order: "hot" (热门) 或 "ts" (最新)
- limit: 每页数量，默认 20
响应结构 (预期):
```json
{
"data": [
{
"id": 12345,
"content": "评论内容",
"author": {"name": "用户名"},
"created_time": 1234567890,
"like_count": 10,
"child_comment_count": 2
}
],
"paging": {...}
}
```
From kb_workflow/crawler/models.py (created by 08-01):
```python
class AnswerData(BaseModel):
id: int
question_id: int
question_title: str
content: str
excerpt: str
metadata: ContentMetadata
class ArticleData(BaseModel):
id: int
title: str
content: str
excerpt: str
metadata: ContentMetadata
class CrawlResult(BaseModel):
success: bool
content_type: str
data: Optional[AnswerData | ArticleData] = None
error: Optional[str] = None
```
From kb_workflow/crawler/zhihu_crawler.py (created by 08-01):
```python
class ZhihuCrawler:
def crawl_answer(self, url: str) -> CrawlResult
def crawl_article(self, url: str) -> CrawlResult
def _get_cookie(self) -> str
def _parse_metadata(self, data: Dict) -> ContentMetadata
```
Task 1: 添加 CommentData 数据模型
kb_workflow/crawler/models.py
在 models.py 中添加评论相关的数据模型。
**添加到 kb_workflow/crawler/models.py:**
```python
class CommentAuthor(BaseModel):
"""评论作者信息"""
name: str = Field(..., description="作者名称")
url_token: Optional[str] = Field(None, description="作者 URL token")
class CommentData(BaseModel):
"""评论数据"""
id: int
content: str = Field(..., description="评论内容")
author: Optional[CommentAuthor] = None
created_time: Optional[int] = Field(None, description="创建时间戳")
like_count: int = Field(default=0, description="点赞数")
child_count: int = Field(default=0, description="子评论数")
is_author: bool = Field(default=False, description="是否为作者回复")
class CommentsResult(BaseModel):
"""评论爬取结果"""
content_type: str # "answers" or "articles"
content_id: int
comments: List[CommentData] = Field(default_factory=list)
total_count: int = Field(default=0, description="总评论数（可能超过实际返回）")
```
同时更新 CrawlResult 以支持包含评论:
```python
class CrawlResult(BaseModel):
"""爬取结果"""
success: bool
content_type: str # "answer" or "article"
data: Optional[AnswerData | ArticleData] = None
comments: Optional[List[CommentData]] = None # 新增
error: Optional[str] = None
```
python -c "from kb_workflow.crawler.models import CommentData, CommentsResult; print('OK')"
- CommentData 模型定义完成
- CommentsResult 模型定义完成
- CrawlResult 支持 comments 字段
Task 2: 实现 crawl_comments 方法
kb_workflow/crawler/zhihu_crawler.py, tests/test_crawler/test_comment_crawler.py
- Test 1: crawl_comments("answers", 12345) 返回 CommentsResult
- Test 2: 评论列表包含热门评论（order=hot）
- Test 3: 每条评论包含 content, author, like_count
- Test 4: 不展开嵌套回复（只有第一层评论）
- Test 5: crawl_with_comments=True 参数在 crawl_answer 中触发评论爬取
在 ZhihuCrawler 类中实现评论爬取功能。
**1. 在 ZhihuCrawler 类中添加方法:**
```python
# 在 kb_workflow/crawler/zhihu_crawler.py 中添加
from .models import CommentData, CommentsResult, CommentAuthor
class ZhihuCrawler:
# ... 现有代码 ...
COMMENT_API_TEMPLATE = (
"https://www.zhihu.com/api/v4/comment_v5/"
"{content_type}/{content_id}/comments"
"?order=hot&limit=20"
)
def crawl_comments(
self,
content_type: str, # "answers" or "articles"
content_id: int,
include_comments: bool = True
) -> CommentsResult:
"""爬取热门评论
Args:
content_type: 内容类型 ("answers" 或 "articles")
content_id: 内容 ID
include_comments: 是否包含评论（保留参数兼容性）
Returns:
CommentsResult 包含评论列表
"""
if not include_comments:
return CommentsResult(
content_type=content_type,
content_id=content_id,
comments=[],
total_count=0
)
try:
self.browser = PlaywrightBrowser(headless=self.headless, timeout=self.timeout)
self.browser.start(cookie=self._get_cookie())
# 构建评论 API URL
api_url = self.COMMENT_API_TEMPLATE.format(
content_type=content_type,
content_id=content_id
)
# 使用 Playwright 的 request API 直接调用
response = self.browser.page.request.get(api_url)
if response.status != 200:
return CommentsResult(
content_type=content_type,
content_id=content_id,
comments=[],
total_count=0
)
data = response.json()
comments = []
for item in data.get("data", []):
author_data = item.get("author", {})
comment = CommentData(
id=item.get("id", 0),
content=item.get("content", ""),
author=CommentAuthor(
name=author_data.get("name", ""),
url_token=author_data.get("url_token")
) if author_data else None,
created_time=item.get("created_time"),
like_count=item.get("like_count", 0),
child_count=item.get("child_comment_count", 0),
is_author=item.get("is_author", False)
)
comments.append(comment)
# 获取总数
total_count = data.get("paging", {}).get("totals", len(comments))
return CommentsResult(
content_type=content_type,
content_id=content_id,
comments=comments,
total_count=total_count
)
except Exception as e:
return CommentsResult(
content_type=content_type,
content_id=content_id,
comments=[],
total_count=0
)
finally:
if self.browser:
self.browser.close()
def crawl_answer(self, url: str, include_comments: bool = False) -> CrawlResult:
"""爬取知乎回答（支持可选评论）
Args:
url: 回答 URL
include_comments: 是否包含热门评论
Returns:
CrawlResult 包含回答数据和可选评论
"""
# 原有逻辑...
result = self._crawl_answer_original(url)
if result.success and include_comments:
# 爬取评论
comments_result = self.crawl_comments(
content_type="answers",
content_id=result.data.id,
include_comments=True
)
result.comments = comments_result.comments
return result
```
**2. 创建 tests/test_crawler/test_comment_crawler.py:**
```python
import pytest
from unittest.mock import Mock, patch, MagicMock
from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.models import CommentData, CommentsResult
class TestCommentCrawler:
@pytest.fixture
def crawler(self):
return ZhihuCrawler(headless=True)
@pytest.fixture
def mock_comment_response(self):
return {
"data": [
{
"id": 111,
"content": "这是一条热门评论",
"author": {"name": "用户A", "url_token": "user-a"},
"created_time": 1234567890,
"like_count": 100,
"child_comment_count": 5,
"is_author": False
},
{
"id": 222,
"content": "作者回复",
"author": {"name": "作者", "url_token": "author"},
"created_time": 1234567900,
"like_count": 50,
"child_comment_count": 0,
"is_author": True
}
],
"paging": {"totals": 100}
}
def test_crawl_comments_returns_comments_result(self, crawler, mock_comment_response):
"""crawl_comments 返回 CommentsResult"""
with patch.object(crawler, '_get_cookie', return_value="test_cookie"):
with patch('kb_workflow.crawler.zhihu_crawler.PlaywrightBrowser') as MockBrowser:
mock_browser = MockBrowser.return_value
mock_response = MagicMock()
mock_response.status = 200
mock_response.json.return_value = mock_comment_response
mock_browser.page.request.get.return_value = mock_response
result = crawler.crawl_comments("answers", 12345)
assert isinstance(result, CommentsResult)
assert result.content_type == "answers"
assert result.content_id == 12345
def test_comments_are_hot_order(self, crawler, mock_comment_response):
"""评论使用热门排序"""
with patch.object(crawler, '_get_cookie', return_value="test_cookie"):
with patch('kb_workflow.crawler.zhihu_crawler.PlaywrightBrowser') as MockBrowser:
mock_browser = MockBrowser.return_value
mock_response = MagicMock()
mock_response.status = 200
mock_response.json.return_value = mock_comment_response
mock_browser.page.request.get.return_value = mock_response
crawler.crawl_comments("answers", 12345)
# 验证 API URL 包含 order=hot
call_args = mock_browser.page.request.get.call_args
assert "order=hot" in call_args[0][0]
def test_comment_has_required_fields(self, crawler, mock_comment_response):
"""每条评论包含必要字段"""
with patch.object(crawler, '_get_cookie', return_value="test_cookie"):
with patch('kb_workflow.crawler.zhihu_crawler.PlaywrightBrowser') as MockBrowser:
mock_browser = MockBrowser.return_value
mock_response = MagicMock()
mock_response.status = 200
mock_response.json.return_value = mock_comment_response
mock_browser.page.request.get.return_value = mock_response
result = crawler.crawl_comments("answers", 12345)
assert len(result.comments) == 2
comment = result.comments[0]
assert comment.content == "这是一条热门评论"
assert comment.author.name == "用户A"
assert comment.like_count == 100
def test_no_nested_replies(self, crawler, mock_comment_response):
"""不展开嵌套回复"""
with patch.object(crawler, '_get_cookie', return_value="test_cookie"):
with patch('kb_workflow.crawler.zhihu_crawler.PlaywrightBrowser') as MockBrowser:
mock_browser = MockBrowser.return_value
mock_response = MagicMock()
mock_response.status = 200
mock_response.json.return_value = mock_comment_response
mock_browser.page.request.get.return_value = mock_response
result = crawler.crawl_comments("answers", 12345)
# 只返回第一层评论，不包含子评论
assert len(result.comments) == 2
# child_count 记录子评论数但不展开
assert result.comments[0].child_count == 5
def test_include_comments_parameter(self, crawler):
"""include_comments=False 不爬取评论"""
result = crawler.crawl_comments("answers", 12345, include_comments=False)
assert result.comments == []
assert result.total_count == 0
```
pytest tests/test_crawler/test_comment_crawler.py -x --tb=short
- crawl_comments() 方法返回 CommentsResult
- 评论使用 order=hot 排序
- 只返回第一层评论
- include_comments 参数控制是否爬取
- 所有测试通过
**自动化验证:**
```bash
pytest tests/test_crawler/test_comment_crawler.py -x -v
```
**集成验证:**
```python
from kb_workflow.crawler import ZhihuCrawler
crawler = ZhihuCrawler()
result = crawler.crawl_answer("https://www.zhihu.com/question/123/answer/456", include_comments=True)
assert result.comments is not None
```
1. CommentData 模型定义完成
2. crawl_comments() 方法实现完成
3. 评论使用热门排序 (order=hot)
4. 只爬取第一层评论，不展开嵌套
5. include_comments 参数控制评论爬取
6. 所有测试通过