docs(02-03): complete zhihu adapter plan
- ZhihuAdapter supports answer and zhuanlan content extraction - Cookie-based authentication with login page detection - Markdown output with YAML frontmatter - 13 new tests, all 35 platform tests passing - Test coverage: 86% Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+10
-6
@@ -4,14 +4,14 @@ milestone: v1.0
|
||||
milestone_name: milestone
|
||||
current_plan: 3
|
||||
status: executing
|
||||
stopped_at: Completed 02-01-PLAN.md
|
||||
last_updated: "2026-03-13T10:15:54.911Z"
|
||||
stopped_at: Completed 02-03-PLAN.md
|
||||
last_updated: "2026-03-13T10:27:09.681Z"
|
||||
last_activity: 2026-03-13
|
||||
progress:
|
||||
total_phases: 4
|
||||
completed_phases: 1
|
||||
completed_phases: 2
|
||||
total_plans: 6
|
||||
completed_plans: 4
|
||||
completed_plans: 6
|
||||
percent: 8
|
||||
---
|
||||
|
||||
@@ -58,6 +58,8 @@ Progress: [█░░░░░░░░░] 8%
|
||||
| Phase 01-foundation P01 | 5 | 3 tasks | 3 files |
|
||||
| Phase 01-foundation P03 | 8 | 5 tasks | 7 files |
|
||||
| Phase 02-platform-skills P01 | 5 | 3 tasks | 5 files |
|
||||
| Phase 02-platform-skills P03 | 7min | 3 tasks | 4 files |
|
||||
| Phase 02 P02 | 8 | 4 tasks | 7 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -77,6 +79,8 @@ Recent decisions affecting current work:
|
||||
- [Phase 01-foundation]: 错误类包含 step_name, input_summary, suggestion 用于调试
|
||||
- [Phase 02-platform-skills]: 使用 dataclass 定义 ParsedLink，短链解析使用 requests.head
|
||||
- [Phase 02-platform-skills]: 无法识别的 URL 返回 None 而不是抛异常，便于调用方处理
|
||||
- [Phase 02-platform-skills]: Lazy import of kb_zhihu_metadata to avoid dependency issues
|
||||
- [Phase 02-platform-skills]: Login page detection via title keywords and content length threshold
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -92,6 +96,6 @@ None yet.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-03-13T10:15:54.910Z
|
||||
Stopped at: Completed 02-01-PLAN.md
|
||||
Last session: 2026-03-13T10:26:42.175Z
|
||||
Stopped at: Completed 02-03-PLAN.md
|
||||
Resume file: None
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
phase: 02-platform-skills
|
||||
plan: 02
|
||||
subsystem: platforms
|
||||
tags: [bilibili, adapter, metadata, video, article, retry, state-recording]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 01-foundation
|
||||
provides: NetworkError, AuthError, retry_on_network_error, WorkflowStateManager
|
||||
- phase: 02-platform-skills/01
|
||||
provides: LinkParser, ParsedLink
|
||||
provides:
|
||||
- PlatformAdapter abstract base class
|
||||
- BilibiliAdapter for video and article extraction
|
||||
- Content type identification from URL
|
||||
- Metadata extraction with retry and state recording
|
||||
- Markdown content with YAML frontmatter
|
||||
affects: [03-processing-skills, 04-analysis]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: [pytest-asyncio, pyyaml]
|
||||
patterns: [adapter-pattern, retry-wrapper, state-recording-after-retry]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- kb_pipeline/platforms/base.py
|
||||
- kb_pipeline/platforms/bilibili.py
|
||||
- tests/test_platforms/test_base.py
|
||||
- tests/test_platforms/test_bilibili.py
|
||||
modified:
|
||||
- kb_pipeline/__init__.py
|
||||
- kb_pipeline/platforms/__init__.py
|
||||
|
||||
key-decisions:
|
||||
- "State recording only after all retries exhausted (not on each retry attempt)"
|
||||
- "Video content returns empty string (handled by ASR in Phase 3)"
|
||||
- "Article content fetching as placeholder (requires additional API implementation)"
|
||||
|
||||
patterns-established:
|
||||
- "Adapter pattern: PlatformAdapter base class with abstract methods for content type, metadata, content, filename"
|
||||
- "Retry wrapper: Public method wraps retry-decorated internal method, state recording in outer catch"
|
||||
- "YAML frontmatter: title, author, source_url, platform, content_type, fetched_at"
|
||||
|
||||
requirements-completed: [PLAT-01, PLAT-02]
|
||||
|
||||
# Metrics
|
||||
duration: 8min
|
||||
completed: 2026-03-13
|
||||
---
|
||||
# Phase 02: Platform Skills Plan 02 Summary
|
||||
|
||||
**BilibiliAdapter implementing PlatformAdapter interface with video/article metadata extraction, automatic retry, state recording, and Markdown content formatting with YAML frontmatter**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 8 min
|
||||
- **Started:** 2026-03-13T10:17:26Z
|
||||
- **Completed:** 2026-03-13T10:25:49Z
|
||||
- **Tasks:** 4
|
||||
- **Files modified:** 7
|
||||
|
||||
## Accomplishments
|
||||
- PlatformAdapter abstract base class defining adapter interface contracts
|
||||
- BilibiliAdapter for B站 video (BV号) and article (CV号) metadata extraction
|
||||
- Automatic 3-retry mechanism with exponential backoff for network errors
|
||||
- State recording for failed steps after all retries exhausted
|
||||
- Markdown content with YAML frontmatter for article content
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Write interface contracts for PlatformAdapter base class** - `3f42bfa` (test)
|
||||
2. **Task 2: Implement BilibiliAdapter for video metadata with state recording** - `b74f6a0` (feat)
|
||||
3. **Task 3: Implement BilibiliAdapter for article content** - `d17fbd8` (test)
|
||||
4. **Task 4: Update package exports** - included in subsequent commits (feat)
|
||||
|
||||
_Note: TDD tasks may have multiple commits (test -> feat -> refactor)_
|
||||
|
||||
## Files Created/Modified
|
||||
- `kb_pipeline/platforms/base.py` - PlatformAdapter abstract base class with interface contracts
|
||||
- `kb_pipeline/platforms/bilibili.py` - BilibiliAdapter implementation for video and article
|
||||
- `kb_pipeline/platforms/__init__.py` - Package exports for adapters
|
||||
- `kb_pipeline/__init__.py` - Root package exports including BilibiliAdapter
|
||||
- `tests/test_platforms/__init__.py` - Test package initialization
|
||||
- `tests/test_platforms/conftest.py` - Shared fixtures for platform tests
|
||||
- `tests/test_platforms/test_base.py` - PlatformAdapter interface tests
|
||||
- `tests/test_platforms/test_bilibili.py` - BilibiliAdapter tests (15 test cases)
|
||||
|
||||
## Decisions Made
|
||||
- State recording only after all retries exhausted to avoid multiple records during retry attempts
|
||||
- Video content fetch returns empty string as ASR transcription is handled separately in Phase 3
|
||||
- Article content fetching uses placeholder implementation (full implementation requires additional API endpoints)
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 3 - Blocking] Installed pytest-asyncio for async test support**
|
||||
- **Found during:** Task 2 (running BilibiliAdapter tests)
|
||||
- **Issue:** pytest-asyncio not installed, async tests failing with "async def functions are not natively supported"
|
||||
- **Fix:** Ran `pip install pytest-asyncio`
|
||||
- **Files modified:** .venv (package installation)
|
||||
- **Verification:** Async tests run successfully
|
||||
- **Committed in:** b74f6a0 (Task 2 commit)
|
||||
|
||||
**2. [Rule 3 - Blocking] Installed PyYAML for frontmatter formatting**
|
||||
- **Found during:** Task 3 (running article content tests)
|
||||
- **Issue:** PyYAML not installed, `import yaml` failing
|
||||
- **Fix:** Ran `pip install pyyaml`
|
||||
- **Files modified:** .venv (package installation)
|
||||
- **Verification:** YAML frontmatter formatting works correctly
|
||||
- **Committed in:** d17fbd8 (Task 3 commit)
|
||||
|
||||
**3. [Rule 2 - Missing Critical] Fixed retry-state recording pattern**
|
||||
- **Found during:** Task 2 (testing state recording)
|
||||
- **Issue:** State recording called on every retry attempt, causing 3 records for single failure
|
||||
- **Fix:** Separated retry logic into internal method, state recording only in outer catch block after all retries exhausted
|
||||
- **Files modified:** kb_pipeline/platforms/bilibili.py
|
||||
- **Verification:** State recording called exactly once after 3 retries
|
||||
- **Committed in:** b74f6a0 (Task 2 commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 3 auto-fixed (2 blocking, 1 missing critical)
|
||||
**Impact on plan:** All auto-fixes necessary for correctness and test execution. No scope creep.
|
||||
|
||||
## Issues Encountered
|
||||
None - all tasks completed as planned with minor dependency installations
|
||||
|
||||
## User Setup Required
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
- BilibiliAdapter ready for integration with workflow pipeline
|
||||
- Video content extraction placeholder ready for ASR implementation in Phase 3
|
||||
- Article content extraction ready for full API implementation
|
||||
- Platform adapter pattern established for future platform implementations (Zhihu, etc.)
|
||||
|
||||
---
|
||||
*Phase: 02-platform-skills*
|
||||
*Completed: 2026-03-13*
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
phase: 02-platform-skills
|
||||
plan: 03
|
||||
subsystem: platforms
|
||||
tags: [zhihu, adapter, cookie-auth, playwright, markdown, yaml-frontmatter]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 01-foundation
|
||||
provides: errors (AuthError), config (KBSettings), schemas (StepInput/Output)
|
||||
- phase: 02-platform-skills
|
||||
plan: 02
|
||||
provides: PlatformAdapter base class, BilibiliAdapter reference implementation
|
||||
provides:
|
||||
- ZhihuAdapter - 知乎平台适配器，支持问答和专栏内容提取
|
||||
- Cookie-based authentication with login page detection
|
||||
- Markdown output with YAML frontmatter
|
||||
affects: [phase-03, phase-04]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: [pytest-asyncio]
|
||||
patterns: [lazy-import-for-optional-deps, property-based-cookie-access, login-page-detection]
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- kb_pipeline/platforms/zhihu.py
|
||||
- tests/test_platforms/test_zhihu.py
|
||||
modified:
|
||||
- kb_pipeline/__init__.py
|
||||
- kb_pipeline/platforms/__init__.py
|
||||
- tests/test_platforms/conftest.py
|
||||
|
||||
key-decisions:
|
||||
- "Lazy import of kb_zhihu_metadata to avoid dependency issues when not installed"
|
||||
- "Login page detection via title keywords and content length threshold (< 100 chars)"
|
||||
- "Set _cookie directly in tests to avoid config loading issues"
|
||||
|
||||
patterns-established:
|
||||
- "Pattern 1: ZhihuAdapter.cookie property loads from KBSettings lazily"
|
||||
- "Pattern 2: _detect_login_page() checks for login keywords and short content"
|
||||
- "Pattern 3: _format_with_frontmatter() generates YAML + Markdown output"
|
||||
|
||||
requirements-completed: [PLAT-03]
|
||||
|
||||
# Metrics
|
||||
duration: 7min
|
||||
completed: 2026-03-13
|
||||
---
|
||||
|
||||
# Phase 2 Plan 3: Zhihu Adapter Summary
|
||||
|
||||
**知乎平台适配器实现，支持问答和专栏内容提取，Cookie认证和登录页面检测**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 7 min
|
||||
- **Started:** 2026-03-13T10:17:52Z
|
||||
- **Completed:** 2026-03-13T10:25:18Z
|
||||
- **Tasks:** 3
|
||||
- **Files modified:** 4
|
||||
|
||||
## Accomplishments
|
||||
- ZhihuAdapter 支持知乎问答和专栏 URL 解析
|
||||
- Cookie 从 KBSettings 懒加载，支持 KB_ZHIHU_COOKIE 环境变量
|
||||
- 登录页面检测（标题关键词 + 内容长度阈值）
|
||||
- 内容输出为 Markdown 格式，带 YAML frontmatter
|
||||
- 所有 13 个新测试通过，总测试数 35
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: ZhihuAdapter test and implementation** - `a6f4d34` (test) + `c4beb3e` (feat)
|
||||
2. **Task 2: Zhuanlan content support** - included in above commits
|
||||
3. **Task 3: Package exports** - `c4beb3e` (feat)
|
||||
|
||||
**Plan metadata:** (pending)
|
||||
|
||||
_Note: TDD workflow - test file created first, then implementation_
|
||||
|
||||
## Files Created/Modified
|
||||
- `kb_pipeline/platforms/zhihu.py` - 知乎平台适配器，支持 answer/zhuanlan 类型
|
||||
- `tests/test_platforms/test_zhihu.py` - ZhihuAdapter 单元测试 (13 tests)
|
||||
- `kb_pipeline/platforms/__init__.py` - 添加 ZhihuAdapter 导出
|
||||
- `kb_pipeline/__init__.py` - 添加 ZhihuAdapter 到包根导出
|
||||
- `tests/test_platforms/conftest.py` - 添加知乎相关 fixtures
|
||||
|
||||
## Decisions Made
|
||||
- 使用懒加载导入 kb_zhihu_metadata，避免未安装时报错
|
||||
- 登录页面检测使用标题关键词（"登录"）和内容长度阈值（< 100 字符）
|
||||
- 测试中直接设置 `_cookie` 属性避免配置加载问题
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 3 - Blocking] Created missing PlatformAdapter base class**
|
||||
- **Found during:** Task 1 (ZhihuAdapter implementation)
|
||||
- **Issue:** Plan 02-02 not yet executed, kb_pipeline/platforms/ directory and base.py missing
|
||||
- **Fix:** Discovered base.py already existed (partial 02-02 work done), proceeded with implementation
|
||||
- **Files modified:** None needed - base class already present
|
||||
- **Verification:** Tests pass, imports work
|
||||
- **Committed in:** a6f4d34 (part of task commit)
|
||||
|
||||
**2. [Rule 3 - Blocking] Fixed zhuanlan URL pattern to support alphanumeric IDs**
|
||||
- **Found during:** Task 1 (test_get_content_type_zhuanlan)
|
||||
- **Issue:** Regex only matched numeric IDs, test used "abc123"
|
||||
- **Fix:** Changed pattern from `(\d+)` to `([\w\d]+)`
|
||||
- **Files modified:** kb_pipeline/platforms/zhihu.py
|
||||
- **Verification:** Test passes
|
||||
- **Committed in:** c4beb3e (Task commit)
|
||||
|
||||
**3. [Rule 3 - Blocking] Installed pytest-asyncio for async test support**
|
||||
- **Found during:** Task 1 (running async tests)
|
||||
- **Issue:** pytest.mark.asyncio warnings, async tests not running properly
|
||||
- **Fix:** Installed pytest-asyncio package
|
||||
- **Files modified:** venv (pytest-asyncio)
|
||||
- **Verification:** Async tests execute correctly
|
||||
- **Committed in:** a6f4d34 (part of task commit)
|
||||
|
||||
**4. [Rule 3 - Blocking] Fixed login page detection threshold**
|
||||
- **Found during:** Task 1 (test_fetch_metadata_answer failed)
|
||||
- **Issue:** Mock fixture content too short, triggering login page detection
|
||||
- **Fix:** Increased mock fixture content length to exceed 100 char threshold
|
||||
- **Files modified:** tests/test_platforms/conftest.py
|
||||
- **Verification:** Tests pass
|
||||
- **Committed in:** c4beb3e (Task commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 4 auto-fixed (4 blocking)
|
||||
**Impact on plan:** All fixes necessary for test correctness and async support. No scope creep.
|
||||
|
||||
## Issues Encountered
|
||||
- patch.object on property triggers property getter before mock setup - solved by directly setting `_cookie` attribute in tests
|
||||
|
||||
## User Setup Required
|
||||
|
||||
**External services require manual configuration.** See plan frontmatter for:
|
||||
- KB_ZHIHU_COOKIE environment variable from browser developer tools
|
||||
- Cookie must be obtained from logged-in zhihu.com session
|
||||
|
||||
## Next Phase Readiness
|
||||
- ZhihuAdapter 和 BilibiliAdapter 都已就绪
|
||||
- 平台适配器模式已建立，可扩展其他平台
|
||||
- 测试覆盖率 86%
|
||||
- 准备进入 Phase 3 (Processing Skills)
|
||||
|
||||
---
|
||||
*Phase: 02-platform-skills*
|
||||
*Completed: 2026-03-13*
|
||||
Reference in New Issue
Block a user