starmen1265
f0ee4ca5e2
- ZhihuCrawler class with API interception - AnswerData, ArticleData models - 15 tests passing Requirements: CONT-01, CONT-02, CONT-04 completed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
87 lines
3.0 KiB
Markdown
87 lines
3.0 KiB
Markdown
---
|
||
phase: 08-crawler
|
||
plan: 01
|
||
subsystem: crawler
|
||
tags: [playwright, pydantic, zhihu, api-interception]
|
||
|
||
requires:
|
||
- phase: 07-login-auth
|
||
provides: ZhihuLogin + Cookie management
|
||
provides:
|
||
- ZhihuCrawler class with crawl_answer() and crawl_article() methods
|
||
- AnswerData, ArticleData data models
|
||
- CrawlResult wrapper
|
||
affects: [08-02, 08-03]
|
||
|
||
tech-stack:
|
||
added: []
|
||
patterns:
|
||
- Playwright route interception for content extraction
|
||
- pydantic data models for structured data
|
||
|
||
key-files:
|
||
created:
|
||
- kb_workflow/crawler/__init__.py
|
||
- kb_workflow/crawler/models.py
|
||
- kb_workflow/crawler/zhihu_crawler.py
|
||
- tests/test_crawler/__init__.py
|
||
- tests/test_crawler/conftest.py
|
||
- tests/test_crawler/test_answer_crawler.py
|
||
- tests/test_crawler/test_article_crawler.py
|
||
- tests/test_crawler/test_metadata_extraction.py
|
||
modified: []
|
||
key-decisions:
|
||
- use Playwright page.route() for API interception over direct API calls
|
||
- pydantic models for structured data
|
||
- TDD workflow with mocking for easy test maintenance
|
||
requirements-completed: [CONT-01, CONT-02, CONT-04]
|
||
duration: 3min
|
||
completed: 2026-03-17
|
||
---
|
||
|
||
## Summary
|
||
|
||
实现了知乎问答和专栏内容爬取功能，使用 Playwright 拦截 API 请求获取结构化 JSON 数据。
|
||
|
||
### What Was Built
|
||
|
||
**ZhihuCrawler 类** (`kb_workflow/crawler/zhihu_crawler.py`)
|
||
- `crawl_answer(url)` - 爬取知乎问答，返回 `AnswerData`
|
||
- `crawl_article(url)` - 爬取知乎专栏，返回 `ArticleData`
|
||
- API 拦截器 `ZhihuApiInterceptor` 捕获 `/api/v4/answers/` 和 `/api/v4/articles/` 响应
|
||
- DOM 解析备选方案（使用 `ZhihuPlaywrightExtractor`）
|
||
|
||
**数据模型** (`kb_workflow/crawler/models.py`)
|
||
- `AuthorInfo` - 作者信息（名称、URL token、签名）
|
||
- `ContentMetadata` - 内容元数据（作者、点赞数、评论数、时间戳）
|
||
- `AnswerData` - 问答数据（ID、问题ID、标题、内容、摘要、元数据）
|
||
- `ArticleData` - 专栏数据（ID、标题、内容、摘要、元数据）
|
||
- `CrawlResult` - 统一结果包装（成功/失败、内容类型、数据/错误）
|
||
|
||
### Test Coverage
|
||
|
||
**15 个测试全部通过**
|
||
- `test_answer_crawler.py` - 5 个测试
|
||
- `test_article_crawler.py` - 4 个测试
|
||
- `test_metadata_extraction.py` - 6 个测试
|
||
|
||
### Commits
|
||
|
||
1. `a102a75` feat(08-01): create crawler data models and test scaffolding
|
||
2. `912773f` test(08-01): add failing tests for ZhihuCrawler
|
||
3. `aa96bc9` feat(08-01): implement ZhihuCrawler with API interception
|
||
|
||
### Key Decisions
|
||
|
||
1. **API 拦截 vs 直接 API 调用**: 使用 Playwright `page.route()` 拦截浏览器请求，自动处理签名和 Cookie
|
||
2. **TDD 工作流**: 先写测试，使用 mock fixtures 确保测试可维护性
|
||
3. **备选方案**: 当 API 拦截失败时，回退到 DOM 解析
|
||
|
||
### Requirements Completed
|
||
|
||
- **CONT-01**: 爬取知乎问答，获取问题标题、描述、回答内容 ✓
|
||
- **CONT-02**: 爬取知乎专栏/文章，获取标题和正文 ✓
|
||
- **CONT-04**: 爬取结果包含作者信息、发布时间、点赞数等元数据 ✓
|
||
|
||
## Self-Check: PASSED
|