docs(09): research phase 9 domain
This commit is contained in:
@@ -0,0 +1,360 @@
|
||||
# Phase 9: Pipeline 整合 - Research
|
||||
|
||||
**Researched:** 2026-03-17
|
||||
**Domain:** Pipeline Integration / Zhihu Crawler Integration
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 9 需要将 Phase 7-8 实现的 ZhihuCrawler 和 ZhihuLogin 整合进现有 Pipeline 架构，替换原有的 ZhihuPlaywrightExtractor。核心工作是修改 ZhihuAdapter 的内部实现，同时保持公共接口不变，确保对 DownloadProcessor 无感知变化。
|
||||
|
||||
**Primary recommendation:** 采用"内部替换"策略 - 修改 ZhihuAdapter 内部调用 ZhihuCrawler，保持接口稳定，评论追加到正文，Cookie 过期自动触发扫码登录。
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
#### 整合方式
|
||||
- **策略**: 修改现有 ZhihuAdapter 内部实现，切换到使用 ZhihuCrawler
|
||||
- **接口保持**: 保持 ZhihuAdapter 的公共接口不变，对 DownloadProcessor 无感知变化
|
||||
- **实现变更**: fetch_metadata() 和 fetch_content() 内部调用 ZhihuCrawler 而非 ZhihuPlaywrightExtractor
|
||||
|
||||
#### 评论数据处理
|
||||
- **默认行为**: 评论爬取默认开启
|
||||
- **存储方式**: 评论追加到正文末尾（Markdown 格式）
|
||||
- **数量上限**: 20 条热门评论（使用 hot 排序，API 默认限制）
|
||||
- **嵌套处理**: 不展开回复的回复，只爬第一层评论（继承自 Phase 8）
|
||||
|
||||
#### Cookie 过期处理
|
||||
- **策略**: 自动触发扫码登录
|
||||
- **流程**: 检测到 Cookie 失效时，调用 ZhihuLogin 触发扫码登录，用户扫码后继续执行
|
||||
- **用户体验**: 用户会看到二维码提示，扫码后流程自动继续
|
||||
|
||||
#### 失败降级策略
|
||||
- **策略**: 直接报错，不降级
|
||||
- **不降级**: 新爬虫失败时不尝试旧版 ZhihuPlaywrightExtractor
|
||||
- **错误处理**: 抛出异常，由上层 Pipeline 统一处理错误
|
||||
|
||||
### Claude's Discretion
|
||||
- 评论追加到正文的具体 Markdown 格式（标题、分隔线等）
|
||||
- 二维码显示的具体实现方式（终端/图片文件）
|
||||
- 错误提示信息的具体文案
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
None — discussion stayed within phase scope
|
||||
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|-----------------|
|
||||
| PIPE-01 | 替换现有 ZhihuProcessor，使用新的爬虫实现 | ZhihuAdapter 内部调用 ZhihuCrawler，保持接口不变 |
|
||||
| PIPE-02 | 保持与现有 Pipeline 架构的兼容性 | PlatformAdapter 接口不变，DownloadProcessor 无需修改 |
|
||||
| PIPE-03 | 支持断点续传和幂等执行 | WorkflowStateManager + is_step_completed 检查机制 |
|
||||
</phase_requirements>
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| pydantic | v2 | 数据验证和模型定义 | 项目已使用，CrawlResult/AnswerData/ArticleData 等模型 |
|
||||
| playwright | sync_api | 浏览器自动化 | ZhihuCrawler 已使用，支持代理和 Cookie |
|
||||
| tenacity | ^8.0 | 重试机制 | ZhihuCrawler 使用 Retrying 对象 |
|
||||
| asyncio | stdlib | 异步执行 | PlatformAdapter 接口使用 async/await |
|
||||
|
||||
### Supporting
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| kb_workflow.crawler.ZhihuCrawler | internal | 问答/专栏/评论爬取 | fetch_metadata/fetch_content 调用 |
|
||||
| kb_workflow.auth.ZhihuLogin | internal | QR 码登录和 Cookie 管理 | Cookie 过期时触发 |
|
||||
| kb_workflow.pipeline.errors.AuthError | internal | 认证错误处理 | Cookie 失效时抛出 |
|
||||
|
||||
### Key Classes
|
||||
|
||||
| Class | Location | Purpose |
|
||||
|-------|----------|---------|
|
||||
| ZhihuCrawler | `kb_workflow/crawler/zhihu_crawler.py` | 主爬虫，crawl_answer/crawl_article/crawl_comments |
|
||||
| ZhihuLogin | `kb_workflow/auth/zhihu_login.py` | login_by_qrcode()，Cookie 持久化 |
|
||||
| ZhihuAdapter | `kb_workflow/pipeline/platforms/zhihu.py` | **待修改** - PlatformAdapter 实现 |
|
||||
| DownloadProcessor | `kb_workflow/pipeline/processors/download.py` | 调用 Adapter，无需修改 |
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Current ZhihuAdapter Implementation
|
||||
|
||||
```python
|
||||
# 当前实现：使用 ZhihuPlaywrightExtractor (DOM 解析)
|
||||
class ZhihuAdapter(PlatformAdapter):
|
||||
def _get_extractor(self):
|
||||
return ZhihuPlaywrightExtractor(headless=self.headless, cookie=self.cookie)
|
||||
|
||||
async def fetch_metadata(self, content_id, content_type, source_url=None):
|
||||
result = await asyncio.to_thread(self._extract_with_playwright, url)
|
||||
# 返回 metadata dict
|
||||
|
||||
async def fetch_content(self, content_id, content_type, source_url=None):
|
||||
metadata = await self.fetch_metadata(...)
|
||||
return self._format_with_frontmatter(metadata, content_markdown, content_type)
|
||||
```
|
||||
|
||||
### Target Implementation Pattern
|
||||
|
||||
```python
|
||||
# 目标实现：使用 ZhihuCrawler (API 拦截)
|
||||
class ZhihuAdapter(PlatformAdapter):
|
||||
def __init__(self, ...):
|
||||
self._crawler = None
|
||||
|
||||
def _get_crawler(self) -> ZhihuCrawler:
|
||||
if self._crawler is None:
|
||||
self._crawler = ZhihuCrawler(headless=self.headless, timeout=self.timeout)
|
||||
return self._crawler
|
||||
|
||||
async def fetch_metadata(self, content_id, content_type, source_url=None):
|
||||
crawler = self._get_crawler()
|
||||
if content_type == "answer":
|
||||
result = await asyncio.to_thread(crawler.crawl_answer, url, include_comments=True)
|
||||
elif content_type == "zhuanlan":
|
||||
result = await asyncio.to_thread(crawler.crawl_article, url)
|
||||
# 转换 CrawlResult -> metadata dict
|
||||
|
||||
async def fetch_content(self, content_id, content_type, source_url=None):
|
||||
# 调用 fetch_metadata，格式化为 Markdown（含评论）
|
||||
```
|
||||
|
||||
### Cookie Expiration Handling Pattern
|
||||
|
||||
```python
|
||||
# Cookie 过期处理流程
|
||||
async def fetch_metadata(self, content_id, content_type, source_url=None):
|
||||
try:
|
||||
result = self._crawl_with_retry(url)
|
||||
if result.error and self._is_cookie_expired(result):
|
||||
# 触发扫码登录
|
||||
login_result = self._trigger_qrcode_login()
|
||||
if login_result.success:
|
||||
# 刷新配置并重试
|
||||
get_settings.cache_clear()
|
||||
result = self._crawl_with_retry(url)
|
||||
return self._convert_to_metadata(result)
|
||||
except Exception as e:
|
||||
raise AuthError(step_name="zhihu_metadata", ...)
|
||||
```
|
||||
|
||||
### Comment Append Format (Claude's Discretion)
|
||||
|
||||
```markdown
|
||||
## 正文内容
|
||||
[原回答/文章内容]
|
||||
|
||||
---
|
||||
|
||||
## 热门评论
|
||||
1. **@用户A**: 评论内容...
|
||||
2. **@用户B**: 评论内容...
|
||||
3. **@用户C**: 评论内容...
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
- **不要修改 PlatformAdapter 接口**: DownloadProcessor 依赖接口稳定
|
||||
- **不要在 ZhihuCrawler 中添加 Pipeline 逻辑**: 保持爬虫职责单一
|
||||
- **不要同步阻塞调用爬虫**: 使用 asyncio.to_thread 包装
|
||||
- **不要忽略 Cookie 过期**: 必须触发扫码登录流程
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| HTML to Markdown | 自定义转换器 | MarkdownConverter (已存在) | zhihu_extractor.py 已有 |
|
||||
| Cookie 解析 | 手动解析 cookie 字符串 | PlaywrightBrowser._add_cookies() | 已实现，处理格式正确 |
|
||||
| 重试逻辑 | @retry 装饰器 | tenacity Retrying 对象 | 支持可配置超时，Phase 8 已用 |
|
||||
| API 签名 | 逆向 x-zse-96 | 浏览器自动处理 | route.fetch() 自动包含签名 |
|
||||
|
||||
**Key insight:** ZhihuCrawler 已完整实现问答/专栏/评论爬取，只需要正确调用和结果转换。
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: CrawlResult 到 Metadata Dict 转换不完整
|
||||
**What goes wrong:** ZhihuCrawler 返回 CrawlResult(AnswerData/ArticleData)，但 DownloadProcessor 期望特定格式的 metadata dict
|
||||
**Why it happens:** 数据模型不完全匹配
|
||||
**How to avoid:** 实现 _convert_crawl_result_to_metadata() 转换函数，确保所有字段正确映射
|
||||
**Warning signs:** DownloadProcessor 提取 title/author 时返回空字符串
|
||||
|
||||
### Pitfall 2: 评论未正确追加到正文
|
||||
**What goes wrong:** fetch_content() 返回的内容不包含评论
|
||||
**Why it happens:** include_comments=True 未正确传递或评论格式化逻辑遗漏
|
||||
**How to avoid:** fetch_content() 必须调用 fetch_metadata(include_comments=True)，格式化时追加评论
|
||||
**Warning signs:** 输出 Markdown 文件无评论部分
|
||||
|
||||
### Pitfall 3: Cookie 过期检测不准确
|
||||
**What goes wrong:** 误判 Cookie 有效/无效，导致不必要的登录或无限重试
|
||||
**Why it happens:** 仅依赖 HTTP 状态码，未检查响应内容
|
||||
**How to avoid:** 检查 API 响应中的 error 字段和 401/403 状态码
|
||||
**Warning signs:** 登录循环或认证错误未触发
|
||||
|
||||
### Pitfall 4: asyncio.to_thread 调用错误
|
||||
**What goes wrong:** ZhihuCrawler 方法调用参数传递不正确
|
||||
**Why it happens:** to_thread 需要函数引用和参数分开传递
|
||||
**How to avoid:** `await asyncio.to_thread(crawler.crawl_answer, url, include_comments=True)`
|
||||
**Warning signs:** TypeError: 'CrawlResult' object is not callable
|
||||
|
||||
## Code Examples
|
||||
|
||||
### ZhihuCrawler Usage (from existing code)
|
||||
|
||||
```python
|
||||
# Source: kb_workflow/crawler/zhihu_crawler.py
|
||||
crawler = ZhihuCrawler(headless=True, timeout=30)
|
||||
|
||||
# 爬取回答（含评论）
|
||||
result = crawler.crawl_answer(
|
||||
url="https://www.zhihu.com/question/123/answer/456",
|
||||
include_comments=True
|
||||
)
|
||||
# result = CrawlResult(success=True, content_type="answer", data=AnswerData, comments=[...])
|
||||
|
||||
# 爬取专栏文章
|
||||
result = crawler.crawl_article(url="https://zhuanlan.zhihu.com/p/123")
|
||||
# result = CrawlResult(success=True, content_type="article", data=ArticleData)
|
||||
```
|
||||
|
||||
### ZhihuLogin QR Code Flow (from existing code)
|
||||
|
||||
```python
|
||||
# Source: kb_workflow/auth/zhihu_login.py
|
||||
login = ZhihuLogin(intermediate_path=str(settings.intermediate_path), headless=False)
|
||||
result = login.login_by_qrcode()
|
||||
|
||||
if result.success:
|
||||
# 保存 Cookie 到配置文件
|
||||
cookie_path = Path.home() / ".kb_env"
|
||||
# 更新 KB_ZHIHU_COOKIE 环境变量
|
||||
# 刷新 settings 缓存
|
||||
get_settings.cache_clear()
|
||||
```
|
||||
|
||||
### Metadata Conversion Pattern
|
||||
|
||||
```python
|
||||
def _convert_answer_to_metadata(self, result: CrawlResult) -> Dict[str, Any]:
|
||||
"""Convert CrawlResult[AnswerData] to metadata dict."""
|
||||
answer: AnswerData = result.data
|
||||
return {
|
||||
"type": "answer",
|
||||
"title": answer.question_title,
|
||||
"author": answer.metadata.author.name if answer.metadata.author else "",
|
||||
"content": answer.content,
|
||||
"content_markdown": self._html_to_markdown(answer.content),
|
||||
"publish_time": self._format_timestamp(answer.metadata.created_time),
|
||||
"url": f"https://www.zhihu.com/question/{answer.question_id}/answer/{answer.id}",
|
||||
"question_id": str(answer.question_id),
|
||||
"answer_id": str(answer.id),
|
||||
"vote_up": answer.metadata.voteup_count,
|
||||
"comment_count": answer.metadata.comment_count,
|
||||
}
|
||||
```
|
||||
|
||||
### Comment Formatting Pattern
|
||||
|
||||
```python
|
||||
def _format_comments_section(self, comments: List[CommentData]) -> str:
|
||||
"""Format comments as Markdown section."""
|
||||
if not comments:
|
||||
return ""
|
||||
|
||||
lines = ["\n\n---\n\n## 热门评论\n"]
|
||||
for i, comment in enumerate(comments[:20], 1): # 最多 20 条
|
||||
author = f"@{comment.author.name}" if comment.author else "匿名"
|
||||
lines.append(f"{i}. **{author}**: {comment.content}")
|
||||
|
||||
return "\n".join(lines)
|
||||
```
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| ZhihuPlaywrightExtractor (DOM 解析) | ZhihuCrawler (API 拦截) | Phase 8 | 更可靠的数据获取，支持评论 |
|
||||
| 手动 Cookie 管理 | ZhihuLogin QR 码登录 | Phase 7 | 自动化登录流程 |
|
||||
| 无评论支持 | 评论爬取 + 追加到正文 | Phase 8 | 更完整的内容存档 |
|
||||
|
||||
**Deprecated/outdated:**
|
||||
- ZhihuPlaywrightExtractor 直接调用: 将被 ZhihuCrawler 替代
|
||||
- 手动设置 KB_ZHIHU_COOKIE: 通过 login 命令自动管理
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **评论 Markdown 格式细节**
|
||||
- What we know: 用户决策评论追加到正文，分隔线 + 热门评论标题
|
||||
- What's unclear: 具体的分隔线样式、评论编号格式
|
||||
- Recommendation: 使用 `---` 分隔线，`1. **@用户**: 内容` 格式
|
||||
|
||||
2. **Cookie 过期检测的精确条件**
|
||||
- What we know: API 返回 401 或特定错误信息
|
||||
- What's unclear: 具体的错误码和响应结构
|
||||
- Recommendation: 检查 response.status == 401 或 JSON 中的 error 字段
|
||||
|
||||
3. **ZhihuLogin 在 headless 模式下的二维码展示**
|
||||
- What we know: QR 码保存到文件，路径返回给用户
|
||||
- What's unclear: 是否需要终端 ASCII 输出
|
||||
- Recommendation: 默认保存文件并打印路径，可选终端输出
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | pytest |
|
||||
| Config file | pytest.ini |
|
||||
| Quick run command | `pytest tests/test_platforms/test_zhihu.py -x -v` |
|
||||
| Full suite command | `pytest tests/ -v` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| PIPE-01 | ZhihuAdapter 使用 ZhihuCrawler | unit | `pytest tests/test_platforms/test_zhihu.py -x` | ✅ 需修改 |
|
||||
| PIPE-02 | DownloadProcessor 兼容性 | integration | `pytest tests/test_processors/test_download.py -x` | ✅ 已存在 |
|
||||
| PIPE-03 | 断点续传/幂等执行 | unit | `pytest tests/test_pipeline/test_idempotency.py -x` | ✅ 已存在 |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `pytest tests/test_platforms/test_zhihu.py -x -v`
|
||||
- **Per wave merge:** `pytest tests/test_platforms/ tests/test_processors/test_download.py -v`
|
||||
- **Phase gate:** Full suite green before `/gsd:verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `tests/test_platforms/test_zhihu.py` — 更新测试使用 ZhihuCrawler mock
|
||||
- [ ] 新增测试：Cookie 过期自动触发登录流程
|
||||
- [ ] 新增测试：评论追加到正文格式验证
|
||||
- [ ] 新增测试：CrawlResult 到 metadata 转换正确性
|
||||
|
||||
*(现有测试基础设施完善，主要是更新和新增 ZhihuAdapter 相关测试)*
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- `kb_workflow/crawler/zhihu_crawler.py` - ZhihuCrawler 实现
|
||||
- `kb_workflow/crawler/models.py` - CrawlResult, AnswerData, ArticleData, CommentData
|
||||
- `kb_workflow/auth/zhihu_login.py` - ZhihuLogin QR 码登录
|
||||
- `kb_workflow/pipeline/platforms/zhihu.py` - ZhihuAdapter (待修改)
|
||||
- `kb_workflow/pipeline/processors/download.py` - DownloadProcessor (无需修改)
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- `tests/test_platforms/test_zhihu.py` - 现有 ZhihuAdapter 测试
|
||||
- `tests/test_crawler/` - ZhihuCrawler 测试
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
- None - 所有信息均来自项目代码
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH - 项目已有完整实现，只需整合
|
||||
- Architecture: HIGH - PlatformAdapter 接口稳定，模式清晰
|
||||
- Pitfalls: HIGH - 基于现有代码分析，问题可预见
|
||||
|
||||
**Research date:** 2026-03-17
|
||||
**Valid until:** 30 days - 项目内部代码，稳定性高
|
||||
Reference in New Issue
Block a user