docs(08-01): complete zhihu content crawler plan
- ZhihuCrawler class with API interception - AnswerData, ArticleData models - 15 tests passing Requirements: CONT-01, CONT-02, CONT-04 completed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -16,10 +16,10 @@
|
||||
|
||||
### 内容爬取
|
||||
|
||||
- [ ] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
|
||||
- [ ] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
|
||||
- [x] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据
|
||||
- [x] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据
|
||||
- [ ] **CONT-03**: 爬取内容下的评论(可选开关)
|
||||
- [ ] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
|
||||
- [x] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据
|
||||
|
||||
### 网络与反爬
|
||||
|
||||
@@ -115,10 +115,10 @@
|
||||
| AUTH-02 | Phase 7 | Complete |
|
||||
| AUTH-03 | Phase 7 | Complete |
|
||||
| AUTH-04 | Phase 7 | Complete |
|
||||
| CONT-01 | Phase 8 | Pending |
|
||||
| CONT-02 | Phase 8 | Pending |
|
||||
| CONT-01 | Phase 8 | Complete |
|
||||
| CONT-02 | Phase 8 | Complete |
|
||||
| CONT-03 | Phase 8 | Pending |
|
||||
| CONT-04 | Phase 8 | Pending |
|
||||
| CONT-04 | Phase 8 | Complete |
|
||||
| NET-01 | Phase 8 | Pending |
|
||||
| NET-02 | Phase 8 | Pending |
|
||||
| NET-03 | Phase 8 | Pending |
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
phase: 08-crawler
|
||||
plan: 01
|
||||
subsystem: crawler
|
||||
tags: [playwright, pydantic, zhihu, api-interception]
|
||||
|
||||
requires:
|
||||
- phase: 07-login-auth
|
||||
provides: ZhihuLogin + Cookie management
|
||||
provides:
|
||||
- ZhihuCrawler class with crawl_answer() and crawl_article() methods
|
||||
- AnswerData, ArticleData data models
|
||||
- CrawlResult wrapper
|
||||
affects: [08-02, 08-03]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- Playwright route interception for content extraction
|
||||
- pydantic data models for structured data
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- kb_workflow/crawler/__init__.py
|
||||
- kb_workflow/crawler/models.py
|
||||
- kb_workflow/crawler/zhihu_crawler.py
|
||||
- tests/test_crawler/__init__.py
|
||||
- tests/test_crawler/conftest.py
|
||||
- tests/test_crawler/test_answer_crawler.py
|
||||
- tests/test_crawler/test_article_crawler.py
|
||||
- tests/test_crawler/test_metadata_extraction.py
|
||||
modified: []
|
||||
key-decisions:
|
||||
- use Playwright page.route() for API interception over direct API calls
|
||||
- pydantic models for structured data
|
||||
- TDD workflow with mocking for easy test maintenance
|
||||
requirements-completed: [CONT-01, CONT-02, CONT-04]
|
||||
duration: 3min
|
||||
completed: 2026-03-17
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
实现了知乎问答和专栏内容爬取功能,使用 Playwright 拦截 API 请求获取结构化 JSON 数据。
|
||||
|
||||
### What Was Built
|
||||
|
||||
**ZhihuCrawler 类** (`kb_workflow/crawler/zhihu_crawler.py`)
|
||||
- `crawl_answer(url)` - 爬取知乎问答,返回 `AnswerData`
|
||||
- `crawl_article(url)` - 爬取知乎专栏,返回 `ArticleData`
|
||||
- API 拦截器 `ZhihuApiInterceptor` 捕获 `/api/v4/answers/` 和 `/api/v4/articles/` 响应
|
||||
- DOM 解析备选方案(使用 `ZhihuPlaywrightExtractor`)
|
||||
|
||||
**数据模型** (`kb_workflow/crawler/models.py`)
|
||||
- `AuthorInfo` - 作者信息(名称、URL token、签名)
|
||||
- `ContentMetadata` - 内容元数据(作者、点赞数、评论数、时间戳)
|
||||
- `AnswerData` - 问答数据(ID、问题ID、标题、内容、摘要、元数据)
|
||||
- `ArticleData` - 专栏数据(ID、标题、内容、摘要、元数据)
|
||||
- `CrawlResult` - 统一结果包装(成功/失败、内容类型、数据/错误)
|
||||
|
||||
### Test Coverage
|
||||
|
||||
**15 个测试全部通过**
|
||||
- `test_answer_crawler.py` - 5 个测试
|
||||
- `test_article_crawler.py` - 4 个测试
|
||||
- `test_metadata_extraction.py` - 6 个测试
|
||||
|
||||
### Commits
|
||||
|
||||
1. `a102a75` feat(08-01): create crawler data models and test scaffolding
|
||||
2. `912773f` test(08-01): add failing tests for ZhihuCrawler
|
||||
3. `aa96bc9` feat(08-01): implement ZhihuCrawler with API interception
|
||||
|
||||
### Key Decisions
|
||||
|
||||
1. **API 拦截 vs 直接 API 调用**: 使用 Playwright `page.route()` 拦截浏览器请求,自动处理签名和 Cookie
|
||||
2. **TDD 工作流**: 先写测试,使用 mock fixtures 确保测试可维护性
|
||||
3. **备选方案**: 当 API 拦截失败时,回退到 DOM 解析
|
||||
|
||||
### Requirements Completed
|
||||
|
||||
- **CONT-01**: 爬取知乎问答,获取问题标题、描述、回答内容 ✓
|
||||
- **CONT-02**: 爬取知乎专栏/文章,获取标题和正文 ✓
|
||||
- **CONT-04**: 爬取结果包含作者信息、发布时间、点赞数等元数据 ✓
|
||||
|
||||
## Self-Check: PASSED
|
||||
Reference in New Issue
Block a user