299 lines
11 KiB
Markdown
299 lines
11 KiB
Markdown
---
|
||
phase: 09-pipeline-integration
|
||
plan: 01
|
||
type: execute
|
||
wave: 1
|
||
depends_on: [09-00]
|
||
files_modified: [kb_workflow/pipeline/platforms/zhihu.py, tests/test_platforms/test_zhihu.py]
|
||
autonomous: true
|
||
requirements: [PIPE-01, PIPE-02, PIPE-03]
|
||
user_setup: []
|
||
|
||
must_haves:
|
||
truths:
|
||
- "知乎链接使用新爬虫处理器，无需手动设置 Cookie"
|
||
- "新处理器与现有 Pipeline 架构完全兼容"
|
||
- "支持断点续传和幂等执行（与其他处理器行为一致）"
|
||
- "评论爬取默认开启，评论追加到正文末尾"
|
||
artifacts:
|
||
- path: "kb_workflow/pipeline/platforms/zhihu.py"
|
||
provides: "ZhihuAdapter implementation"
|
||
min_lines: 30
|
||
exports: ["ZhihuAdapter"]
|
||
- path: "kb_workflow/crawler/zhihu_crawler.py"
|
||
provides: "ZhihuCrawler class"
|
||
exports: ["ZhihuCrawler", "crawl_answer", "crawl_article", "crawl_comments"]
|
||
- path: "kb_workflow/crawler/models.py"
|
||
provides: "Data models (AnswerData, ArticleData, CommentData, CrawlResult)"
|
||
exports: ["AnswerData", "ArticleData", "CommentData", "CommentAuthor", "ContentMetadata", "AuthorInfo", "CrawlResult"]
|
||
- path: "kb_workflow/auth/zhihu_login.py"
|
||
provides: "ZhihuLogin class"
|
||
exports: ["ZhihuLogin", "login_by_qrcode", "validate_cookie", "ZhihuLoginResult", "CookieValidationResult"]
|
||
- path: "tests/test_platforms/test_zhihu.py"
|
||
provides: "test coverage for ZhihuAdapter"
|
||
min_lines: 50
|
||
exports: ["mock_zhihu_answer_result", "mock_zhihu_zhuanlan_result"]
|
||
- path: "kb_workflow/pipeline/processors/download.py"
|
||
provides: "Download processor (consumer of ZhihuAdapter)"
|
||
exports: ["DownloadProcessor"]
|
||
key_links:
|
||
- from: "ZhihuAdapter.fetch_metadata"
|
||
to: "ZhihuCrawler.crawl_answer/crawl_article"
|
||
via: "asyncio.to_thread"
|
||
pattern: "asyncio\\.to_thread.*crawl"
|
||
- from: "ZhihuAdapter.fetch_metadata"
|
||
to: "ZhihuCrawler.crawl_comments"
|
||
via: "internal _crawl_comments call"
|
||
pattern: "result\\.comments"
|
||
- from: "ZhihuAdapter._format_with_frontmatter"
|
||
to: "ZhihuAdapter._format_comments_section"
|
||
via: "content append"
|
||
pattern: "\\n---\\n## 热门评论"
|
||
- from: "ZhihuAdapter.fetch_content"
|
||
to: "IdempotencyChecker"
|
||
via: "幂等检查调用"
|
||
pattern: "idempotency|already_processed"
|
||
---
|
||
|
||
<objective>
|
||
将新爬虫实现整合进现有 Pipeline 架构，替换原有知乎处理器，保持接口兼容性，支持评论爬取和 Cookie 过期自动处理。
|
||
|
||
Cookie 过期处理：Cookie 过期时自动触发扫码登录，用户扫码后流程继续。
|
||
新爬虫失败时直接报错，不降级到旧版。
|
||
评论爬取默认开启，评论追加到正文末尾。
|
||
幂等执行：通过 IdempotencyChecker 确保同一内容不会重复处理。
|
||
</objective>
|
||
|
||
<execution_context>
|
||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||
</execution_context>
|
||
|
||
<context>
|
||
@.planning/PROJECT.md
|
||
@.planning/ROADMAP.md
|
||
@.planning/STATE.md
|
||
@.planning/phases/08-crawler/08-03-SUMMARY.md
|
||
|
||
@.planning/phases/09-pipeline-integration/09-RESEARCH.md
|
||
@.planning/phases/09-pipeline-integration/09-CONTEXT.md
|
||
|
||
@.planning/phases/09-pipeline-integration/09-00-SUMMARY.md
|
||
|
||
<interfaces>
|
||
<!-- Key types and contracts the executor needs. Executor should use these directly - no codebase exploration needed. -->
|
||
|
||
From kb_workflow/pipeline/platforms/base.py:
|
||
```python
|
||
class PlatformAdapter(ABC):
|
||
@abstractmethod
|
||
def get_content_type(self, url: str) -> str:
|
||
pass
|
||
|
||
@abstractmethod
|
||
async def fetch_metadata(self, content_id: str, content_type: str, source_url: str = None) -> Dict[str, Any]:
|
||
pass
|
||
|
||
@abstractmethod
|
||
async def fetch_content(self, content_id: str, content_type: str, source_url: str = None) -> str:
|
||
pass
|
||
|
||
@abstractmethod
|
||
def get_output_filename(self, content_id: str, content_type: str, suffix: str) -> str:
|
||
pass
|
||
```
|
||
|
||
From kb_workflow/crawler/models.py:
|
||
```python
|
||
class CrawlResult(BaseModel):
|
||
success: bool
|
||
content_type: str
|
||
data: Optional[Union[AnswerData, ArticleData]]
|
||
comments: Optional[List[CommentData]]
|
||
error: Optional[str]
|
||
```
|
||
|
||
From kb_workflow/crawler/zhihu_crawler.py:
|
||
```python
|
||
class ZhihuCrawler:
|
||
def crawl_answer(self, url: str, include_comments: bool = False) -> CrawlResult:
|
||
...
|
||
|
||
def crawl_article(self, url: str) -> CrawlResult:
|
||
...
|
||
|
||
def crawl_comments(self, content_type: str, content_id: int, include_comments: bool = True) -> CommentsResult:
|
||
...
|
||
```
|
||
|
||
From kb_workflow/auth/zhihu_login.py:
|
||
```python
|
||
class ZhihuLogin:
|
||
def login_by_qrcode(self) -> ZhihuLoginResult:
|
||
...
|
||
|
||
def validate_cookie(self, cookie_str: str = None) -> CookieValidationResult:
|
||
...
|
||
```
|
||
|
||
From kb_workflow/pipeline/idempotency.py:
|
||
```python
|
||
class IdempotencyChecker:
|
||
def is_processed(self, content_id: str, content_type: str) -> bool:
|
||
"""检查内容是否已处理"""
|
||
...
|
||
|
||
def mark_processed(self, content_id: str, content_type: str) -> None:
|
||
"""标记内容已处理"""
|
||
...
|
||
```
|
||
</interfaces>
|
||
</context>
|
||
|
||
<tasks>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 1: Update ZhihuAdapter to use ZhihuCrawler</name>
|
||
<files>kb_workflow/pipeline/platforms/zhihu.py</files>
|
||
<behavior>
|
||
- Test 1: fetch_metadata 返回正确的问答元数据
|
||
- Test 2: fetch_metadata 返回正确的专栏文章元数据
|
||
- Test 3: fetch_content 返回带评论的 Markdown 内容
|
||
- Test 4: Cookie 过期时自动触发扫码登录
|
||
- Test 5: 幂等检查阻止重复处理同一内容
|
||
</behavior>
|
||
<action>
|
||
修改 ZhihuAdapter 类，内部实现使用 ZhihuCrawler 替代 ZhihuPlaywrightExtractor。
|
||
|
||
保留公共接口不变 (fetch_metadata, fetch_content, get_content_type, get_output_filename)。
|
||
|
||
**关键决策实现（来自 CONTEXT.md）：**
|
||
1. **评论爬取默认开启**：设置 include_comments=True 作为默认参数
|
||
2. **评论追加到正文**：修改 _format_with_frontmatter 在 content 末尾追加评论（格式见 CONTEXT.md specifics）
|
||
3. **幂等执行 (PIPE-03)**：在 fetch_metadata 开头调用 IdempotencyChecker.is_processed()，已处理的内容直接返回缓存结果或跳过
|
||
|
||
具体修改:
|
||
1. 移除 ZhihuPlaywrightExtractor 导入，添加 ZhihuCrawler 导入
|
||
2. 添加 `_get_crawler()` 方法创建和管理 ZhihuCrawler 实例
|
||
3. 添加 `_crawl_with_crawler()` 方法使用 ZhihuCrawler 进行内容爬取，**include_comments=True 为默认值**
|
||
4. 添加 `_convert_crawl_result_to_metadata()` 方法将 CrawlResult 转换为 metadata dict
|
||
5. 添加 `_format_comments_section()` 方法格式化评论为 Markdown
|
||
6. **修改 `_format_with_frontmatter()` 方法**：在正文末尾追加评论部分
|
||
```python
|
||
def _format_with_frontmatter(self, ...):
|
||
content = base_content
|
||
if comments:
|
||
content += "\n\n---\n\n## 热门评论\n\n" + self._format_comments_section(comments)
|
||
return content
|
||
```
|
||
7. 添加 `_is_cookie_expired()` 方法检测 Cookie 过期
|
||
8. 添加 `_trigger_qrcode_login()` 方法触发扫码登录
|
||
9. 添加幂等检查：在处理前调用 IdempotencyChecker.is_processed()
|
||
10. 移除 `_extract_with_playwright` 和 `_get_extractor` 方法（不再需要)
|
||
11. 修改 `_detect_login_page` 方法，基于 CrawlResult 检测登录页面
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_platforms/test_zhihu.py -x -v</automated>
|
||
</verify>
|
||
<done>
|
||
ZhihuAdapter 使用 ZhihuCrawler 作为内容获取
|
||
fetch_metadata 返回包含评论的 CrawlResult
|
||
fetch_content 返回 Markdown 格式内容（含评论追加，格式为 "---\n## 热门评论"）
|
||
评论爬取默认开启（include_comments=True）
|
||
幂等执行通过 IdempotencyChecker 实现
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 2: Enable test cases and verify Cookie auto login</name>
|
||
<files>tests/test_platforms/test_zhihu.py</files>
|
||
<behavior>
|
||
- Test 1: Cookie 有效时，爬取正常返回数据
|
||
- Test 2: Cookie 过期时，触发扫码登录流程
|
||
- Test 3: 触发登录后重试成功
|
||
- Test 4: mock _trigger_qrcode_login 返回登录失败时正确处理
|
||
</behavior>
|
||
<action>
|
||
移除 Wave 0 创建的测试骨架中的 pytest.mark.skip 标记，实现以下测试场景:
|
||
|
||
1. `test_cookie_valid_returns_data`: Cookie 有效时，爬取正常返回数据
|
||
2. `test_cookie_expired_triggers_login`: Cookie 过期时，触发扫码登录流程
|
||
3. `test_login_retry_success`: 触发登录后重试成功
|
||
4. `test_login_failure_raises_error`: mock _trigger_qrcode_login 返回登录失败
|
||
|
||
需要 mock:
|
||
- ZhihuCrawler.crawl_answer / crawl_article 返回 mock CrawlResult
|
||
- ZhihuLogin.validate_cookie 返回过期状态
|
||
- ZhihuLogin.login_by_qrcode 返回登录结果
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_platforms/test_zhihu.py -k "test_cookie" -v</automated>
|
||
</verify>
|
||
<done>
|
||
Cookie 过期自动触发扫码登录的测试用例全部通过
|
||
</done>
|
||
</task>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 3: Verify comment formatting and idempotency</name>
|
||
<files>tests/test_platforms/test_zhihu.py</files>
|
||
<behavior>
|
||
- Test 1: 有评论时，格式化为 "## 热门评论" 部分
|
||
- Test 2: 无评论时，不追加评论部分
|
||
- Test 3: 幂等检查阻止重复处理同一内容
|
||
- Test 4: 不同内容 ID 可以独立处理
|
||
</behavior>
|
||
<action>
|
||
移除 Wave 0 创建的测试骨架中的 pytest.mark.skip 标记，实现以下测试场景:
|
||
|
||
1. `test_comment_formatting_with_comments`: 有评论时，格式化为 "## 热门评论" 部分
|
||
2. `test_comment_formatting_no_comments`: 无评论时，不追加评论部分
|
||
3. `test_idempotent_execution_skips_processed`: 幂等检查阻止重复处理同一内容
|
||
4. `test_different_content_processes_independently`: 不同内容 ID 可以独立处理
|
||
|
||
验证评论格式:
|
||
```markdown
|
||
[原回答/文章内容]
|
||
|
||
---
|
||
|
||
## 热门评论
|
||
|
||
1. **@用户A**: 评论内容...
|
||
2. **@用户B**: 评论内容...
|
||
```
|
||
|
||
验证幂等性:
|
||
- 首次调用 fetch_metadata 应正常处理
|
||
- 再次调用相同 content_id 应跳过或返回缓存
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_platforms/test_zhihu.py -k "test_comment or test_idempotent" -v</automated>
|
||
</verify>
|
||
<done>
|
||
评论格式化测试用例全部通过
|
||
幂等执行测试用例全部通过
|
||
</done>
|
||
</task>
|
||
|
||
</tasks>
|
||
|
||
<verification>
|
||
运行完整的 Pipeline 测试验证整合是否正常工作:
|
||
```bash
|
||
pytest tests/test_platforms/test_zhihu.py -v
|
||
```
|
||
</verification>
|
||
|
||
<success_criteria>
|
||
- 知乎链接使用新爬虫处理器，无需手动设置 Cookie
|
||
- 新处理器与现有 Pipeline 架构完全兼容
|
||
- 支持断点续传和幂等执行（通过 IdempotencyChecker 实现）
|
||
- 评论爬取默认开启（include_comments=True）
|
||
- 评论追加到正文末尾（格式: "---\n## 热门评论"）
|
||
</success_criteria>
|
||
|
||
<output>
|
||
After completion, create `.planning/phases/09-pipeline-integration/09-01-SUMMARY.md`
|
||
</output>
|