diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 8430847..56e5d1a 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -29,9 +29,9 @@ ### Pipeline 整合 -- [ ] **PIPE-01**: 替换现有 ZhihuProcessor，使用新的爬虫实现 -- [ ] **PIPE-02**: 保持与现有 Pipeline 架构的兼容性 -- [ ] **PIPE-03**: 支持断点续传和幂等执行 +- [x] **PIPE-01**: 替换现有 ZhihuProcessor，使用新的爬虫实现 +- [x] **PIPE-02**: 保持与现有 Pipeline 架构的兼容性 +- [x] **PIPE-03**: 支持断点续传和幂等执行 ## v1.1 Requirements (Complete) @@ -122,9 +122,9 @@ | NET-01 | Phase 8 | Complete | | NET-02 | Phase 8 | Complete | | NET-03 | Phase 8 | Complete | -| PIPE-01 | Phase 9 | Pending | -| PIPE-02 | Phase 9 | Pending | -| PIPE-03 | Phase 9 | Pending | +| PIPE-01 | Phase 9 | Complete | +| PIPE-02 | Phase 9 | Complete | +| PIPE-03 | Phase 9 | Complete | **v1.2 Coverage:** - Requirements: 14 total diff --git a/.planning/STATE.md b/.planning/STATE.md index 5dfa896..4508b2f 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,15 +2,15 @@ gsd_state_version: 1.0 milestone: v1.2 milestone_name: Zhihu Crawler -status: in_progress -stopped_at: Phase 9 Plan 00 complete -last_updated: "2026-03-17T14:59:13Z" +status: completed +stopped_at: Completed 09-01 ZhihuAdapter integration +last_updated: "2026-03-17T15:16:21.491Z" last_activity: 2026-03-17 — Phase 09-00 Test scaffolding complete progress: total_phases: 9 completed_phases: 8 - total_plans: 25 - completed_plans: 25 + total_plans: 27 + completed_plans: 26 percent: 100 --- @@ -48,6 +48,7 @@ Progress: [██████████] 100% (v1.2 milestone - Phase 9 in pro | Phase 08 P02 | 5min | 2 | 3 | 2026-03-17 | | Phase 08 P03 | 15min | 2 | 5 | 2026-03-17 | | 09-00 | 2min | 1 | 1 | 2026-03-17 | +| Phase 09-pipeline-integration P01 | 11min | 3 tasks | 2 files | ## Accumulated Context @@ -78,6 +79,9 @@ v1.2 decisions: - [Phase 08-02]: Use Playwright request API for direct comment API calls instead of route interception - [Phase ?]: - [Phase 08-02]: Only fetch first-level comments (no nested replies) - [Phase ?]: - [Phase 08-02]: Use hot sorting (order=hot) for popular comments +- [Phase 09-01]: ZhihuAdapter uses ZhihuCrawler with comment crawling enabled by default +- [Phase 09-01]: Cookie expiration triggers automatic QR code login flow +- [Phase 09-01]: Comments appended to content in Markdown format with '## 热门评论' section ### Blockers/Concerns @@ -85,8 +89,8 @@ None. ## Session Continuity -Last session: 2026-03-17T14:59:13Z -Stopped at: Phase 9 Plan 00 complete -Resume file: .planning/phases/09-pipeline-integration/09-00-SUMMARY.md +Last session: 2026-03-17T15:16:21.489Z +Stopped at: Completed 09-01 ZhihuAdapter integration +Resume file: None **Next step:** Continue with Phase 09-01 ZhihuAdapter integration diff --git a/.planning/phases/09-pipeline-integration/09-01-SUMMARY.md b/.planning/phases/09-pipeline-integration/09-01-SUMMARY.md new file mode 100644 index 0000000..49fc815 --- /dev/null +++ b/.planning/phases/09-pipeline-integration/09-01-SUMMARY.md @@ -0,0 +1,115 @@ +--- +phase: 09-pipeline-integration +plan: 01 +subsystem: pipeline +tags: [zhihu, crawler, adapter, comments, idempotency, cookie, qrcode-login] + +# Dependency graph +requires: + - phase: 08-crawler + provides: ZhihuCrawler class with crawl_answer, crawl_article, crawl_comments methods + - phase: 07-login-auth + provides: ZhihuLogin class with QR code login flow +provides: + - ZhihuAdapter integration with ZhihuCrawler + - Comment crawling and formatting + - Cookie expiration auto-detection and login + - Idempotent execution via IdempotencyChecker +affects: [download-processor, pipeline-orchestrator] + +# Tech tracking +tech-stack: + added: [html2text] + patterns: [async-to-thread for sync crawlers, mock-based testing, idempotency via file existence] + +key-files: + created: [] + modified: + - kb_workflow/pipeline/platforms/zhihu.py + - tests/test_platforms/test_zhihu.py + +key-decisions: + - "ZhihuAdapter uses ZhihuCrawler instead of ZhihuPlaywrightExtractor" + - "Comments included by default with include_comments=True" + - "Cookie expiration triggers QR code login with 2-minute timeout" + - "Idempotency check via IdempotencyChecker when output_dir is set" + +patterns-established: + - "Pattern: async-to-thread wrapper for sync crawler methods" + - "Pattern: comment formatting with numbered list and author mentions" + - "Pattern: cookie expiration detection via error message keywords" + +requirements-completed: [PIPE-01, PIPE-02, PIPE-03] + +# Metrics +duration: 11min +completed: 2026-03-17 +--- + +# Phase 09 Plan 01: ZhihuAdapter Integration Summary + +**ZhihuAdapter integrated with ZhihuCrawler, supporting comment crawling, cookie auto-login, and idempotent execution** + +## Performance + +- **Duration:** 11 min +- **Started:** 2026-03-17T15:04:22Z +- **Completed:** 2026-03-17T15:15:11Z +- **Tasks:** 3 +- **Files modified:** 2 + +## Accomplishments +- ZhihuAdapter now uses ZhihuCrawler instead of ZhihuPlaywrightExtractor +- Comments are included by default and appended to content in Markdown format +- Cookie expiration triggers automatic QR code login flow +- Idempotent execution via IdempotencyChecker when output_dir is configured + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Update ZhihuAdapter to use ZhihuCrawler** - `76f06c3` (feat) + - Includes Task 2 (Cookie auto login) and Task 3 (Comment formatting + idempotency) + +**Plan metadata:** (to be committed after this summary) + +_Note: All three tasks were completed in a single commit due to tight integration between the components_ + +## Files Created/Modified +- `kb_workflow/pipeline/platforms/zhihu.py` - Core adapter implementation with ZhihuCrawler integration +- `tests/test_platforms/test_zhihu.py` - Complete test suite with 19 passing tests + +## Decisions Made +- Used ZhihuCrawler for content extraction (replacing ZhihuPlaywrightExtractor) +- Comments included by default (include_comments=True) +- Cookie expiration detected via error message keywords (login, expired, etc.) +- QR code login with 2-minute timeout and visible browser window +- HTML to Markdown conversion using html2text library + +## Deviations from Plan + +None - plan executed exactly as written. All three tasks were integrated into a single implementation because: +1. Task 1 requires TDD tests that cover Task 2 and Task 3 scenarios +2. Cookie handling and comment formatting are integral parts of the adapter implementation +3. Separating commits would require partial/incomplete code + +## Issues Encountered + +**Missing dependency (Rule 3 - Blocking):** html2text package was not installed in the virtual environment. Fixed by installing via `pip install html2text`. + +## User Setup Required + +None - no external service configuration required beyond the existing Zhihu cookie setup. + +## Next Phase Readiness +- ZhihuAdapter is ready for integration with DownloadProcessor +- All 19 tests passing, covering: + - Basic content type detection + - Metadata and content fetching + - Cookie handling and auto-login + - Comment formatting + - Idempotency + +--- +*Phase: 09-pipeline-integration* +*Completed: 2026-03-17*