知乎提取器:从 Playwright 迁移到 DrissionPage 方案
- 使用 DrissionPage 替代 Playwright - 实现 Cookie 管理和反爬绕过 - 支持回答/文章/问题三种内容类型 - 灵活配置(文件/环境变量/配置文件) - 更新文档和配置文件
This commit is contained in:
+59
-23
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: obsidian-kb
|
||||
description: Multi-source content knowledge base with video processing. Use when user says "保存知识库", "添加到知识库", "处理这个视频", "生成obsidian笔记" etc. Supports B站/bilibili (BV号, b23.tv, www.bilibili.com), YouTube videos, and Zhihu articles.
|
||||
description: Multi-source content knowledge base with video processing. Use when user says "保存知识库", "添加到知识库", "处理这个视频", "生成obsidian笔记" etc. Supports B站/bilibili (BV号, b23.tv, www.bilibili.com) using Playwright, YouTube videos, and Zhihu articles using DrissionPage.
|
||||
---
|
||||
|
||||
# SKILL.md - Obsidian KB Skill
|
||||
@@ -191,12 +191,12 @@ sources:
|
||||
enabled: false
|
||||
api_key: '' # YouTube API Key
|
||||
zhihu:
|
||||
enabled: false
|
||||
use_playwright: true # 使用 Playwright 绕过检测(推荐)
|
||||
cookie: '' # 知乎 Cookie (可选)
|
||||
cookie_file: '' # Cookie 文件路径 (可选)
|
||||
timeout: 30 # 超时时间(秒)
|
||||
max_retries: 2 # 最大重试次数
|
||||
enabled: true
|
||||
use_drissionpage: true # 使用 DrissionPage 绕过检测(推荐)
|
||||
cookie_file: ~/.obsidian-kb/zhihu_cookies.json # Cookie 文件路径
|
||||
cookie_str: "" # 可选:直接在配置中填入 Cookie JSON
|
||||
timeout: 30
|
||||
headless: true
|
||||
|
||||
# Bilibili 提取器配置
|
||||
bilibili_extraction:
|
||||
@@ -219,8 +219,51 @@ export OPENAI_API_KEY="sk-..."
|
||||
export AI_MODEL="gpt-4"
|
||||
export OBSIDIAN_VAULT_PATH="/path/to/vault"
|
||||
export BILIBILI_COOKIE="..."
|
||||
export ZHIHU_COOKIE='[{"name":"z_c0","value":"xxx"}]'
|
||||
```
|
||||
|
||||
### Zhihu 提取器配置
|
||||
|
||||
知乎提取器使用 DrissionPage + Cookie 方式:
|
||||
|
||||
```yaml
|
||||
sources:
|
||||
zhihu:
|
||||
enabled: true
|
||||
use_drissionpage: true
|
||||
cookie_file: ~/.obsidian-kb/zhihu_cookies.json # Cookie 文件路径
|
||||
cookie_str: "" # 可选:直接在配置中填入 Cookie JSON
|
||||
timeout: 30
|
||||
headless: true
|
||||
```
|
||||
|
||||
**Cookie 获取方式:**
|
||||
1. 在有浏览器的机器上登录知乎
|
||||
2. 打开开发者工具 (F12)
|
||||
3. 进入 Application → Cookies → https://www.zhihu.com
|
||||
4. 导出所有 Cookie 为 JSON 文件
|
||||
5. 上传到服务器指定路径
|
||||
|
||||
**Cookie 文件格式:**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "z_c0",
|
||||
"value": "xxx",
|
||||
"domain": ".zhihu.com",
|
||||
"path": "/",
|
||||
"expiry": 1234567890,
|
||||
"secure": true,
|
||||
"httpOnly": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Cookie 加载优先级:**
|
||||
1. 配置文件中的 `cookie_str`(最高优先级)
|
||||
2. 环境变量 `ZHIHU_COOKIE`
|
||||
3. Cookie 文件(最低优先级)
|
||||
|
||||
## 提取器说明
|
||||
|
||||
### Extractors 目录
|
||||
@@ -276,35 +319,28 @@ else:
|
||||
content["subtitle_text"] = subtitle_text
|
||||
```
|
||||
|
||||
### ZhihuPlaywrightExtractor 类
|
||||
### ZhihuDrissionExtractor 类
|
||||
|
||||
ZhihuPlaywrightExtractor 提供了知乎内容的完整提取功能:
|
||||
知乎 DrissionPage 提取器提供知乎内容的完整提取功能:
|
||||
|
||||
**主要方法:**
|
||||
- `extract(url)` - 提取知乎内容(支持回答、文章、问题)
|
||||
- `identify_url_type(url)` - 识别 URL 类型
|
||||
- `extract_id(url)` - 提取内容 ID
|
||||
- `_load_cookies()` - 加载 Cookie(支持配置/环境/文件)
|
||||
- `_apply_cookies(cookies)` - 应用 Cookie 到浏览器
|
||||
|
||||
**特性:**
|
||||
- 使用 Playwright 绕过知乎反爬虫保护
|
||||
- 使用 DrissionPage 绕过知乎反爬虫保护
|
||||
- 支持完整 Markdown 格式(代码块、表格、引用等)
|
||||
- 提取元数据:标题、作者、发布时间、点赞数
|
||||
- 智能降级机制(Playwright 失败 → HTTP)
|
||||
- 每次提取独立启动浏览器,内存占用低
|
||||
|
||||
**降级逻辑:**
|
||||
|
||||
当 Playwright 提取失败时:
|
||||
|
||||
1. **ImportError** - Playwright 未安装 → 使用 HTTP 方案
|
||||
2. **TimeoutError** - 网络超时 → 重试后降级
|
||||
3. **403 Forbidden** - 触发反爬虫 → 降级到 HTTP
|
||||
4. **RuntimeError** - 浏览器崩溃 → 降级到 HTTP
|
||||
- Cookie 加载优先级:配置 > 环境变量 > 文件
|
||||
- 支持无头模式,适合服务器环境
|
||||
|
||||
```python
|
||||
from sources.zhihu_playwright_extractor import ZhihuPlaywrightExtractor
|
||||
from obsidian_kb.extractors.zhihu import ZhihuDrissionExtractor
|
||||
|
||||
extractor = ZhihuPlaywrightExtractor()
|
||||
extractor = ZhihuDrissionExtractor()
|
||||
data = extractor.extract("https://www.zhihu.com/question/266309957/answer/2003427891925509507")
|
||||
|
||||
print(f"Title: {data['title']}")
|
||||
|
||||
Reference in New Issue
Block a user