docs(08-01): complete zhihu content crawler plan

- ZhihuCrawler class with API interception
- AnswerData, ArticleData models
- 15 tests passing

Requirements: CONT-01, CONT-02, CONT-04 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-17 21:51:56 +08:00
parent aa96bc963c
commit f0ee4ca5e2
2 changed files with 92 additions and 6 deletions
+6 -6
View File
@@ -16,10 +16,10 @@
### 内容爬取
- [ ] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
- [ ] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
- [x] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
- [x] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
- [ ] **CONT-03**: 爬取内容下的评论(可选开关)
- [ ] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
- [x] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
### 网络与反爬
@@ -115,10 +115,10 @@
| AUTH-02 | Phase 7 | Complete |
| AUTH-03 | Phase 7 | Complete |
| AUTH-04 | Phase 7 | Complete |
| CONT-01 | Phase 8 | Pending |
| CONT-02 | Phase 8 | Pending |
| CONT-01 | Phase 8 | Complete |
| CONT-02 | Phase 8 | Complete |
| CONT-03 | Phase 8 | Pending |
| CONT-04 | Phase 8 | Pending |
| CONT-04 | Phase 8 | Complete |
| NET-01 | Phase 8 | Pending |
| NET-02 | Phase 8 | Pending |
| NET-03 | Phase 8 | Pending |
@@ -0,0 +1,86 @@
---
phase: 08-crawler
plan: 01
subsystem: crawler
tags: [playwright, pydantic, zhihu, api-interception]
requires:
- phase: 07-login-auth
provides: ZhihuLogin + Cookie management
provides:
- ZhihuCrawler class with crawl_answer() and crawl_article() methods
- AnswerData, ArticleData data models
- CrawlResult wrapper
affects: [08-02, 08-03]
tech-stack:
added: []
patterns:
- Playwright route interception for content extraction
- pydantic data models for structured data
key-files:
created:
- kb_workflow/crawler/__init__.py
- kb_workflow/crawler/models.py
- kb_workflow/crawler/zhihu_crawler.py
- tests/test_crawler/__init__.py
- tests/test_crawler/conftest.py
- tests/test_crawler/test_answer_crawler.py
- tests/test_crawler/test_article_crawler.py
- tests/test_crawler/test_metadata_extraction.py
modified: []
key-decisions:
- use Playwright page.route() for API interception over direct API calls
- pydantic models for structured data
- TDD workflow with mocking for easy test maintenance
requirements-completed: [CONT-01, CONT-02, CONT-04]
duration: 3min
completed: 2026-03-17
---
## Summary
实现了知乎问答和专栏内容爬取功能,使用 Playwright 拦截 API 请求获取结构化 JSON 数据。
### What Was Built
**ZhihuCrawler 类** (`kb_workflow/crawler/zhihu_crawler.py`)
- `crawl_answer(url)` - 爬取知乎问答,返回 `AnswerData`
- `crawl_article(url)` - 爬取知乎专栏,返回 `ArticleData`
- API 拦截器 `ZhihuApiInterceptor` 捕获 `/api/v4/answers/` 和 `/api/v4/articles/` 响应
- DOM 解析备选方案(使用 `ZhihuPlaywrightExtractor`)
**数据模型** (`kb_workflow/crawler/models.py`)
- `AuthorInfo` - 作者信息(名称、URL token、签名)
- `ContentMetadata` - 内容元数据(作者、点赞数、评论数、时间戳)
- `AnswerData` - 问答数据(ID、问题ID、标题、内容、摘要、元数据)
- `ArticleData` - 专栏数据(ID、标题、内容、摘要、元数据)
- `CrawlResult` - 统一结果包装(成功/失败、内容类型、数据/错误)
### Test Coverage
**15 个测试全部通过**
- `test_answer_crawler.py` - 5 个测试
- `test_article_crawler.py` - 4 个测试
- `test_metadata_extraction.py` - 6 个测试
### Commits
1. `a102a75` feat(08-01): create crawler data models and test scaffolding
2. `912773f` test(08-01): add failing tests for ZhihuCrawler
3. `aa96bc9` feat(08-01): implement ZhihuCrawler with API interception
### Key Decisions
1. **API 拦截 vs 直接 API 调用**: 使用 Playwright `page.route()` 拦截浏览器请求,自动处理签名和 Cookie
2. **TDD 工作流**: 先写测试,使用 mock fixtures 确保测试可维护性
3. **备选方案**: 当 API 拦截失败时,回退到 DOM 解析
### Requirements Completed
- **CONT-01**: 爬取知乎问答,获取问题标题、描述、回答内容 ✓
- **CONT-02**: 爬取知乎专栏/文章,获取标题和正文 ✓
- **CONT-04**: 爬取结果包含作者信息、发布时间、点赞数等元数据 ✓
## Self-Check: PASSED