docs(09-01): complete ZhihuAdapter integration plan
- Integrated ZhihuCrawler into ZhihuAdapter - Added comment crawling and formatting - Implemented cookie auto-login flow - Added idempotent execution via IdempotencyChecker Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -29,9 +29,9 @@
|
|||||||
|
|
||||||
### Pipeline 整合
|
### Pipeline 整合
|
||||||
|
|
||||||
- [ ] **PIPE-01**: 替换现有 ZhihuProcessor,使用新的爬虫实现
|
- [x] **PIPE-01**: 替换现有 ZhihuProcessor,使用新的爬虫实现
|
||||||
- [ ] **PIPE-02**: 保持与现有 Pipeline 架构的兼容性
|
- [x] **PIPE-02**: 保持与现有 Pipeline 架构的兼容性
|
||||||
- [ ] **PIPE-03**: 支持断点续传和幂等执行
|
- [x] **PIPE-03**: 支持断点续传和幂等执行
|
||||||
|
|
||||||
## v1.1 Requirements (Complete)
|
## v1.1 Requirements (Complete)
|
||||||
|
|
||||||
@@ -122,9 +122,9 @@
|
|||||||
| NET-01 | Phase 8 | Complete |
|
| NET-01 | Phase 8 | Complete |
|
||||||
| NET-02 | Phase 8 | Complete |
|
| NET-02 | Phase 8 | Complete |
|
||||||
| NET-03 | Phase 8 | Complete |
|
| NET-03 | Phase 8 | Complete |
|
||||||
| PIPE-01 | Phase 9 | Pending |
|
| PIPE-01 | Phase 9 | Complete |
|
||||||
| PIPE-02 | Phase 9 | Pending |
|
| PIPE-02 | Phase 9 | Complete |
|
||||||
| PIPE-03 | Phase 9 | Pending |
|
| PIPE-03 | Phase 9 | Complete |
|
||||||
|
|
||||||
**v1.2 Coverage:**
|
**v1.2 Coverage:**
|
||||||
- Requirements: 14 total
|
- Requirements: 14 total
|
||||||
|
|||||||
+12
-8
@@ -2,15 +2,15 @@
|
|||||||
gsd_state_version: 1.0
|
gsd_state_version: 1.0
|
||||||
milestone: v1.2
|
milestone: v1.2
|
||||||
milestone_name: Zhihu Crawler
|
milestone_name: Zhihu Crawler
|
||||||
status: in_progress
|
status: completed
|
||||||
stopped_at: Phase 9 Plan 00 complete
|
stopped_at: Completed 09-01 ZhihuAdapter integration
|
||||||
last_updated: "2026-03-17T14:59:13Z"
|
last_updated: "2026-03-17T15:16:21.491Z"
|
||||||
last_activity: 2026-03-17 — Phase 09-00 Test scaffolding complete
|
last_activity: 2026-03-17 — Phase 09-00 Test scaffolding complete
|
||||||
progress:
|
progress:
|
||||||
total_phases: 9
|
total_phases: 9
|
||||||
completed_phases: 8
|
completed_phases: 8
|
||||||
total_plans: 25
|
total_plans: 27
|
||||||
completed_plans: 25
|
completed_plans: 26
|
||||||
percent: 100
|
percent: 100
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -48,6 +48,7 @@ Progress: [██████████] 100% (v1.2 milestone - Phase 9 in pro
|
|||||||
| Phase 08 P02 | 5min | 2 | 3 | 2026-03-17 |
|
| Phase 08 P02 | 5min | 2 | 3 | 2026-03-17 |
|
||||||
| Phase 08 P03 | 15min | 2 | 5 | 2026-03-17 |
|
| Phase 08 P03 | 15min | 2 | 5 | 2026-03-17 |
|
||||||
| 09-00 | 2min | 1 | 1 | 2026-03-17 |
|
| 09-00 | 2min | 1 | 1 | 2026-03-17 |
|
||||||
|
| Phase 09-pipeline-integration P01 | 11min | 3 tasks | 2 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -78,6 +79,9 @@ v1.2 decisions:
|
|||||||
- [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception
|
- [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception
|
||||||
- [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies)
|
- [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies)
|
||||||
- [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments
|
- [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments
|
||||||
|
- [Phase 09-01]: ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default
|
||||||
|
- [Phase 09-01]: Cookie expiration triggers automatic QR code login flow
|
||||||
|
- [Phase 09-01]: Comments appended to content in Markdown format with '## 热门评论' section
|
||||||
|
|
||||||
### Blockers/Concerns
|
### Blockers/Concerns
|
||||||
|
|
||||||
@@ -85,8 +89,8 @@ None.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-03-17T14:59:13Z
|
Last session: 2026-03-17T15:16:21.489Z
|
||||||
Stopped at: Phase 9 Plan 00 complete
|
Stopped at: Completed 09-01 ZhihuAdapter integration
|
||||||
Resume file: .planning/phases/09-pipeline-integration/09-00-SUMMARY.md
|
Resume file: None
|
||||||
|
|
||||||
**Next step:** Continue with Phase 09-01 ZhihuAdapter integration
|
**Next step:** Continue with Phase 09-01 ZhihuAdapter integration
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
---
|
||||||
|
phase: 09-pipeline-integration
|
||||||
|
plan: 01
|
||||||
|
subsystem: pipeline
|
||||||
|
tags: [zhihu, crawler, adapter, comments, idempotency, cookie, qrcode-login]
|
||||||
|
|
||||||
|
# Dependency graph
|
||||||
|
requires:
|
||||||
|
- phase: 08-crawler
|
||||||
|
provides: ZhihuCrawler class with crawl_answer, crawl_article, crawl_comments methods
|
||||||
|
- phase: 07-login-auth
|
||||||
|
provides: ZhihuLogin class with QR code login flow
|
||||||
|
provides:
|
||||||
|
- ZhihuAdapter integration with ZhihuCrawler
|
||||||
|
- Comment crawling and formatting
|
||||||
|
- Cookie expiration auto-detection and login
|
||||||
|
- Idempotent execution via IdempotencyChecker
|
||||||
|
affects: [download-processor, pipeline-orchestrator]
|
||||||
|
|
||||||
|
# Tech tracking
|
||||||
|
tech-stack:
|
||||||
|
added: [html2text]
|
||||||
|
patterns: [async-to-thread for sync crawlers, mock-based testing, idempotency via file existence]
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created: []
|
||||||
|
modified:
|
||||||
|
- kb_workflow/pipeline/platforms/zhihu.py
|
||||||
|
- tests/test_platforms/test_zhihu.py
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "ZhihuAdapter uses ZhihuCrawler instead of ZhihuPlaywrightExtractor"
|
||||||
|
- "Comments included by default with include_comments=True"
|
||||||
|
- "Cookie expiration triggers QR code login with 2-minute timeout"
|
||||||
|
- "Idempotency check via IdempotencyChecker when output_dir is set"
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "Pattern: async-to-thread wrapper for sync crawler methods"
|
||||||
|
- "Pattern: comment formatting with numbered list and author mentions"
|
||||||
|
- "Pattern: cookie expiration detection via error message keywords"
|
||||||
|
|
||||||
|
requirements-completed: [PIPE-01, PIPE-02, PIPE-03]
|
||||||
|
|
||||||
|
# Metrics
|
||||||
|
duration: 11min
|
||||||
|
completed: 2026-03-17
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 09 Plan 01: ZhihuAdapter Integration Summary
|
||||||
|
|
||||||
|
**ZhihuAdapter integrated with ZhihuCrawler, supporting comment crawling, cookie auto-login, and idempotent execution**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** 11 min
|
||||||
|
- **Started:** 2026-03-17T15:04:22Z
|
||||||
|
- **Completed:** 2026-03-17T15:15:11Z
|
||||||
|
- **Tasks:** 3
|
||||||
|
- **Files modified:** 2
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
- ZhihuAdapter now uses ZhihuCrawler instead of ZhihuPlaywrightExtractor
|
||||||
|
- Comments are included by default and appended to content in Markdown format
|
||||||
|
- Cookie expiration triggers automatic QR code login flow
|
||||||
|
- Idempotent execution via IdempotencyChecker when output_dir is configured
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
Each task was committed atomically:
|
||||||
|
|
||||||
|
1. **Task 1: Update ZhihuAdapter to use ZhihuCrawler** - `76f06c3` (feat)
|
||||||
|
- Includes Task 2 (Cookie auto login) and Task 3 (Comment formatting + idempotency)
|
||||||
|
|
||||||
|
**Plan metadata:** (to be committed after this summary)
|
||||||
|
|
||||||
|
_Note: All three tasks were completed in a single commit due to tight integration between the components_
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
- `kb_workflow/pipeline/platforms/zhihu.py` - Core adapter implementation with ZhihuCrawler integration
|
||||||
|
- `tests/test_platforms/test_zhihu.py` - Complete test suite with 19 passing tests
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
- Used ZhihuCrawler for content extraction (replacing ZhihuPlaywrightExtractor)
|
||||||
|
- Comments included by default (include_comments=True)
|
||||||
|
- Cookie expiration detected via error message keywords (login, expired, etc.)
|
||||||
|
- QR code login with 2-minute timeout and visible browser window
|
||||||
|
- HTML to Markdown conversion using html2text library
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
None - plan executed exactly as written. All three tasks were integrated into a single implementation because:
|
||||||
|
1. Task 1 requires TDD tests that cover Task 2 and Task 3 scenarios
|
||||||
|
2. Cookie handling and comment formatting are integral parts of the adapter implementation
|
||||||
|
3. Separating commits would require partial/incomplete code
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
**Missing dependency (Rule 3 - Blocking):** html2text package was not installed in the virtual environment. Fixed by installing via `pip install html2text`.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None - no external service configuration required beyond the existing Zhihu cookie setup.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
- ZhihuAdapter is ready for integration with DownloadProcessor
|
||||||
|
- All 19 tests passing, covering:
|
||||||
|
- Basic content type detection
|
||||||
|
- Metadata and content fetching
|
||||||
|
- Cookie handling and auto-login
|
||||||
|
- Comment formatting
|
||||||
|
- Idempotency
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 09-pipeline-integration*
|
||||||
|
*Completed: 2026-03-17*
|
||||||
Reference in New Issue
Block a user