7.3 KiB
phase: 09-pipeline-integration plan: 01 type: execute wave: 1 depends_on: [] files_modified: [] autonomous: true requirements: [PIPE-01, PIPE-02, PIPE-03] user_setup: []
must_haves: truths: - "知乎链接使用新爬虫处理器,无需手动设置 Cookie" - "新处理器与现有 Pipeline 架构完全兼容" - "支持断点续传和幂等执行(与其他处理器行为一致)" artifacts: - path: "kb_workflow/pipeline/platforms/zhihu.py" provides: "ZhihuAdapter implementation" min_lines: 30 exports: ["ZhihuAdapter"] - path: "kb_workflow/crawler/zhihu_crawler.py" provides: "ZhihuCrawler class" exports: ["ZhihuCrawler", "Crawl_answer", "crawl_article", "crawl_comments"] - path: "kb_workflow/crawler/models.py" provides: "Data models (AnswerData, ArticleData, CommentData, CrawlResult)" exports: ["AnswerData", "ArticleData", "CommentData", "CommentAuthor", "ContentMetadata", "AuthorInfo", "CrawlResult"] - path: "kb_workflow/auth/zhihu_login.py" provides: "ZhihuLogin class" exports: ["ZhihuLogin", "login_by_qrcode", "validate_cookie", "ZhihuLoginResult", "CookieValidationResult"] - path: "tests/test_platforms/test_zhihu.py" provides: "test coverage for ZhihuAdapter" exports: ["ZhihuAdapter", "test_get_content_type_*"] min_lines: 50 exports: ["mock_zhihu_answer_result", "mock_zhihu_zhuanlan_result"] - path: "kb_workflow/pipeline/processors/download.py" provides: "Download processor (consumer of ZhihuAdapter)" exports: ["DownloadProcessor"] key_links: - from: "ZhihuAdapter.fetch_metadata" to: "ZhihuCrawler.crawl_answer/crawl_article" via: "asyncio.to_thread" pattern: "asyncio\.to_thread.*crawl" - from: "ZhihuAdapter.fetch_metadata" to: "ZhihuCrawler.crawl_comments" via: "internal _crawl_comments call" pattern: "result\.comments" - from: "ZhihuAdapter._format_with_frontmatter" to: "ZhihuAdapter._format_comments_section" via: "content append" pattern: "\n---\n## 热门评论"
将新爬虫实现整合进现有 Pipeline 架构,替换原有知乎处理器,保持接口兼容性和,支持评论爬取和Cookie 过期自动处理。Cookie 过期处理 Cookie 过期时自动触发扫码登录,用户扫码后流程继续 新爬虫失败时直接报错,不降级到旧版。
<execution_context> @/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md @/home/starmen1265/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/08-crawler/08-03-SUMMARY.md@.planning/phases/09-pipeline-integration/09-RESEARCH.md @.planning/phases/09-pipeline-integration/09-CONTEXT.md
@.planning/phases/09-pipeline-integration/09-VALIDATION.md
From kb_workflow/pipeline/platforms/base.py:
class PlatformAdapter(ABC):
@abstractmethod
def get_content_type(self, url: str) -> str:
pass
@abstractmethod
async def fetch_metadata(self, content_id: str, content_type: str, source_url: str = None) -> Dict[str, Any]:
pass
@abstractmethod
async def fetch_content(self, content_id: str, content_type: str, source_url: str = None) -> str:
pass
@abstractmethod
def get_output_filename(self, content_id: str, content_type: str, suffix: str) -> str:
pass
From kb_workflow/crawler/models.py
class CrawlResult(BaseModel):
success: bool
content_type: str
data: Optional[Union[AnswerData, ArticleData]]
comments: Optional[List[commentData]]
error: Optional[str]
from kb_workflow/crawler/zhihu_crawler.py
class ZhihuCrawler:
def crawl_answer(self, url: str, include_comments: bool = False) -> CrawlResult
...
def crawl_article(self, url: str) -> CrawlResult:
"""
def crawl_comments(self, content_type: str, content_id: int, include_comments: bool = True) -> CommentsResult:
"""
from kb_workflow/auth/zhihu_login.py
class ZhihuLogin:
def login_by_qrcode(self) -> ZhihuLoginResult
"""
def validate_cookie(self, cookie_str: str = None) -> CookieValidationResult
"""
内部实现",使用 ZhihuCrawler 替代 ZhihuPlaywrightExtractor。
保留公共接口不变 (fetch_metadata, fetch_content, get_content_type, get_output_filename).
具体修改:
- 移除 ZhihuPlaywrightExtractor
导入,ZhihuCrawler" 2 添加_get_crawler()方法创建和管理 ZhihuCrawler 实例 添加_crawl_with_crawler()方法使用 ZhihuCrawler 进行内容爬取 添加_convert_crawl_result_to_metadata()方法将 CrawlResult 转换为 metadata dict 添加_format_comments_section()方法格式化评论为 Markdown 添加_is_cookie_expired()方法检测 Cookie 过期 添加_trigger_qrcode_login()方法触发扫码登录 移除_extract_with_playwright和_get_extractor方法(不再需要) 2 修改_detect_login_page` 方法,基于 CrawlResult 检测登录页面 pytest tests/test_platforms/test_zhihu.py -x -v Z ZhihuAdapter 使用 ZhihuCrawler 作为内容获取 fetch_metadata 返回包含评论的 CrawlResult fetch_content 返回 Markdown 格式内容(含评论追加)
<success_criteria>
- 知乎链接使用新爬虫处理器,无需手动设置 Cookie
- 新处理器与现有 Pipeline 架构完全兼容
- 支持断点续传和幂等执行(与其他处理器行为一致) </success_criteria>