feat: integrate bilibili-extractor into obsidian-kb
- Add extractors layer with BilibiliExtractor (Playwright stealth mode) - Add outputs layer with ObsidianWriter and GitSync - Implement subtitle/no-subtitle graceful degradation - Update config with output and bilibili extraction settings - Complete integration pipeline: extract → process → output → sync - Add comprehensive tests (100% pass rate) - Update documentation (SKILL.md, README.md, requirements.txt)
This commit is contained in:
@@ -142,6 +142,13 @@ storage:
|
||||
topics_dir: topics # 主题目录
|
||||
data_dir: data # 数据目录
|
||||
|
||||
# 输出配置
|
||||
output:
|
||||
format: markdown # 输出格式
|
||||
encoding: utf-8 # 文件编码
|
||||
include_timestamp: true # 包含时间戳
|
||||
include_source: true # 包含源链接
|
||||
|
||||
# Git 同步配置
|
||||
git:
|
||||
enabled: true # 启用 Git 同步
|
||||
@@ -169,6 +176,17 @@ sources:
|
||||
zhihu:
|
||||
enabled: false
|
||||
cookie: '' # 知乎 Cookie (可选)
|
||||
|
||||
# Bilibili 提取器配置
|
||||
bilibili_extraction:
|
||||
use_playwright: true # 使用 Playwright 绕过检测
|
||||
headless: true # 无头模式
|
||||
cookie_file: '' # Cookie 文件路径 (可选)
|
||||
enable_subtitle: true # 启用字幕提取
|
||||
subtitle_language: zh-CN # 字幕语言
|
||||
enable_danmaku: false # 启用弹幕提取
|
||||
timeout_seconds: 30 # 超时时间
|
||||
max_retries: 3 # 最大重试次数
|
||||
```
|
||||
|
||||
### 环境变量
|
||||
@@ -182,6 +200,91 @@ export OBSIDIAN_VAULT_PATH="/path/to/vault"
|
||||
export BILIBILI_COOKIE="..."
|
||||
```
|
||||
|
||||
## 提取器说明
|
||||
|
||||
### Extractors 目录
|
||||
|
||||
`extractors/` 目录包含内容提取器实现:
|
||||
|
||||
```
|
||||
extractors/
|
||||
├── __init__.py
|
||||
├── base.py # 提取器基类
|
||||
└── bilibili.py # Bilibili 提取器
|
||||
```
|
||||
|
||||
### BilibiliExtractor 类
|
||||
|
||||
BilibiliExtractor 提供了 Bilibili 视频的完整提取功能:
|
||||
|
||||
**主要方法:**
|
||||
- `extract_video(video_url)` - 提取视频信息
|
||||
- `extract_subtitle(video_url)` - 提取字幕
|
||||
- `extract_danmaku(video_url)` - 提取弹幕(可选)
|
||||
- `extract_video_info(video_url)` - 提取视频元信息
|
||||
|
||||
**特性:**
|
||||
- 使用 Playwright 绕过反爬检测
|
||||
- 支持字幕提取(CC字幕 + AI字幕)
|
||||
- 支持弹幕提取(可选)
|
||||
- 自动处理无字幕视频
|
||||
- 智能重试机制
|
||||
|
||||
**无字幕处理逻辑:**
|
||||
|
||||
当视频没有字幕时,系统会:
|
||||
|
||||
1. **检测字幕可用性** - 尝试多种字幕源(CC字幕、AI字幕)
|
||||
2. **启用备用方案** - 如果无字幕,使用视频标题和描述作为内容基础
|
||||
3. **AI 补充分析** - 基于视频元数据生成摘要和要点
|
||||
4. **标记状态** - 在生成的笔记中标注"无字幕",说明内容来源于元数据分析
|
||||
|
||||
```python
|
||||
# 示例:检测和处理无字幕视频
|
||||
if not subtitle_text:
|
||||
logger.warning(f"Video {video_url} has no subtitles")
|
||||
# 使用视频元数据生成基础内容
|
||||
content = {
|
||||
"title": video_info["title"],
|
||||
"description": video_info["desc"],
|
||||
"tags": video_info["tags"],
|
||||
"subtitle_available": False
|
||||
}
|
||||
else:
|
||||
content["subtitle_available"] = True
|
||||
content["subtitle_text"] = subtitle_text
|
||||
```
|
||||
|
||||
### 使用提取器
|
||||
|
||||
**直接使用提取器:**
|
||||
|
||||
```python
|
||||
from obsidian_kb.extractors.bilibili import BilibiliExtractor
|
||||
|
||||
# 创建提取器实例
|
||||
extractor = BilibiliExtractor(
|
||||
use_playwright=True,
|
||||
headless=True
|
||||
)
|
||||
|
||||
# 提取视频信息
|
||||
video_info = extractor.extract_video("https://www.bilibili.com/video/BV1xx411c7mD")
|
||||
|
||||
# 提取字幕
|
||||
subtitle = extractor.extract_subtitle("https://www.bilibili.com/video/BV1xx411c7mD")
|
||||
|
||||
print(f"Title: {video_info['title']}")
|
||||
print(f"Subtitle length: {len(subscription)} characters")
|
||||
```
|
||||
|
||||
**通过管道使用:**
|
||||
|
||||
```bash
|
||||
# 提取器会自动被管道调用
|
||||
/kb-process BV1xx411c7mD
|
||||
```
|
||||
|
||||
## 模板变量
|
||||
|
||||
### 通用变量
|
||||
|
||||
Reference in New Issue
Block a user