diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index a3e903f..4d03f26 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -16,10 +16,10 @@ ### 内容爬取 -- [ ] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据 -- [ ] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据 +- [x] **CONT-01**: 爬取知乎问答 - 问题标题、问题描述、回答内容、元数据 +- [x] **CONT-02**: 爬取知乎专栏/文章 - 标题、正文、元数据 - [ ] **CONT-03**: 爬取内容下的评论(可选开关) -- [ ] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据 +- [x] **CONT-04**: 提取作者信息、发布时间、点赞数等元数据 ### 网络与反爬 @@ -115,10 +115,10 @@ | AUTH-02 | Phase 7 | Complete | | AUTH-03 | Phase 7 | Complete | | AUTH-04 | Phase 7 | Complete | -| CONT-01 | Phase 8 | Pending | -| CONT-02 | Phase 8 | Pending | +| CONT-01 | Phase 8 | Complete | +| CONT-02 | Phase 8 | Complete | | CONT-03 | Phase 8 | Pending | -| CONT-04 | Phase 8 | Pending | +| CONT-04 | Phase 8 | Complete | | NET-01 | Phase 8 | Pending | | NET-02 | Phase 8 | Pending | | NET-03 | Phase 8 | Pending | diff --git a/.planning/phases/08-crawler/08-01-SUMMARY.md b/.planning/phases/08-crawler/08-01-SUMMARY.md new file mode 100644 index 0000000..e1742bf --- /dev/null +++ b/.planning/phases/08-crawler/08-01-SUMMARY.md @@ -0,0 +1,86 @@ +--- +phase: 08-crawler +plan: 01 +subsystem: crawler +tags: [playwright, pydantic, zhihu, api-interception] + +requires: + - phase: 07-login-auth + provides: ZhihuLogin + Cookie management +provides: + - ZhihuCrawler class with crawl_answer() and crawl_article() methods + - AnswerData, ArticleData data models + - CrawlResult wrapper +affects: [08-02, 08-03] + +tech-stack: + added: [] + patterns: + - Playwright route interception for content extraction + - pydantic data models for structured data + +key-files: + created: + - kb_workflow/crawler/__init__.py + - kb_workflow/crawler/models.py + - kb_workflow/crawler/zhihu_crawler.py + - tests/test_crawler/__init__.py + - tests/test_crawler/conftest.py + - tests/test_crawler/test_answer_crawler.py + - tests/test_crawler/test_article_crawler.py + - tests/test_crawler/test_metadata_extraction.py + modified: [] +key-decisions: + - use Playwright page.route() for API interception over direct API calls + - pydantic models for structured data + - TDD workflow with mocking for easy test maintenance +requirements-completed: [CONT-01, CONT-02, CONT-04] +duration: 3min +completed: 2026-03-17 +--- + +## Summary + +实现了知乎问答和专栏内容爬取功能,使用 Playwright 拦截 API 请求获取结构化 JSON 数据。 + +### What Was Built + +**ZhihuCrawler 类** (`kb_workflow/crawler/zhihu_crawler.py`) +- `crawl_answer(url)` - 爬取知乎问答,返回 `AnswerData` +- `crawl_article(url)` - 爬取知乎专栏,返回 `ArticleData` +- API 拦截器 `ZhihuApiInterceptor` 捕获 `/api/v4/answers/` 和 `/api/v4/articles/` 响应 +- DOM 解析备选方案(使用 `ZhihuPlaywrightExtractor`) + +**数据模型** (`kb_workflow/crawler/models.py`) +- `AuthorInfo` - 作者信息(名称、URL token、签名) +- `ContentMetadata` - 内容元数据(作者、点赞数、评论数、时间戳) +- `AnswerData` - 问答数据(ID、问题ID、标题、内容、摘要、元数据) +- `ArticleData` - 专栏数据(ID、标题、内容、摘要、元数据) +- `CrawlResult` - 统一结果包装(成功/失败、内容类型、数据/错误) + +### Test Coverage + +**15 个测试全部通过** +- `test_answer_crawler.py` - 5 个测试 +- `test_article_crawler.py` - 4 个测试 +- `test_metadata_extraction.py` - 6 个测试 + +### Commits + +1. `a102a75` feat(08-01): create crawler data models and test scaffolding +2. `912773f` test(08-01): add failing tests for ZhihuCrawler +3. `aa96bc9` feat(08-01): implement ZhihuCrawler with API interception + +### Key Decisions + +1. **API 拦截 vs 直接 API 调用**: 使用 Playwright `page.route()` 拦截浏览器请求,自动处理签名和 Cookie +2. **TDD 工作流**: 先写测试,使用 mock fixtures 确保测试可维护性 +3. **备选方案**: 当 API 拦截失败时,回退到 DOM 解析 + +### Requirements Completed + +- **CONT-01**: 爬取知乎问答,获取问题标题、描述、回答内容 ✓ +- **CONT-02**: 爬取知乎专栏/文章,获取标题和正文 ✓ +- **CONT-04**: 爬取结果包含作者信息、发布时间、点赞数等元数据 ✓ + +## Self-Check: PASSED