fix(obsidian-kb): fix SKILL.md YAML format and restore files
This commit is contained in:
@@ -1,256 +0,0 @@
|
||||
# obsidian-kb 自动触发机制
|
||||
|
||||
## 概述
|
||||
|
||||
已实现 obsidian-kb 技能的自动触发机制,当用户通过 WhatsApp/Telegram 等渠道发送视频或文章链接时,技能会自动识别并提示处理。
|
||||
|
||||
## 实现方式
|
||||
|
||||
### 1. SKILL.md 更新
|
||||
|
||||
添加了 frontmatter 元数据和详细的触发规则说明:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: obsidian-kb
|
||||
description: Automatically detect and process video/article links...
|
||||
version: 1.0.0
|
||||
---
|
||||
```
|
||||
|
||||
在文档中添加了:
|
||||
- 自动触发规则章节
|
||||
- 触发模式说明
|
||||
- 自动触发流程
|
||||
- 批量处理支持
|
||||
- 短链接友好提示
|
||||
|
||||
### 2. Handler 脚本
|
||||
|
||||
创建了 `handler.py` 脚本,功能包括:
|
||||
|
||||
- **链接检测**: 使用正则表达式匹配多种 URL 模式
|
||||
- **元数据提取**: 识别平台、内容类型和 ID
|
||||
- **预览生成**: 格式化显示识别到的内容
|
||||
- **智能去重**: 避免重复检测相同的链接
|
||||
- **短链接处理**: 对 b23.tv 等短链接显示友好提示
|
||||
|
||||
## 支持的 URL 模式
|
||||
|
||||
### B站视频
|
||||
- 完整 URL: `https://www.bilibili.com/video/BV*`
|
||||
- 短链接: `https://b23.tv/*`
|
||||
- BV号: `BV*`
|
||||
|
||||
### YouTube 视频
|
||||
- 完整 URL: `https://www.youtube.com/watch?v=*`
|
||||
- 短链接: `https://youtu.be/*`
|
||||
|
||||
### 知乎文章
|
||||
- 专栏文章: `https://zhuanlan.zhihu.com/p/*`
|
||||
- 回答: `https://www.zhihu.com/question/*/answer/*`
|
||||
|
||||
## 使用流程
|
||||
|
||||
### 自动触发模式(推荐)
|
||||
|
||||
1. 用户发送链接
|
||||
2. 系统自动检测链接
|
||||
3. 显示内容预览
|
||||
4. 询问用户确认
|
||||
5. 用户确认后添加到收件箱
|
||||
6. 可选:立即处理
|
||||
|
||||
### 示例
|
||||
|
||||
```
|
||||
用户: https://www.bilibili.com/video/BV1xx411c7mD
|
||||
|
||||
助手: 🔍 检测到内容链接:
|
||||
|
||||
📺 [1] B站视频
|
||||
URL: https://www.bilibili.com/video/BV1xx411c7mD
|
||||
ID: BV1xx411c7mD
|
||||
|
||||
是否添加到收件箱并处理?(y/n)
|
||||
```
|
||||
|
||||
### 短链接示例
|
||||
|
||||
```
|
||||
用户: https://b23.tv/oLBZZfK
|
||||
|
||||
助手: 🔍 检测到内容链接:
|
||||
|
||||
📺 [1] B站视频
|
||||
URL: https://b23.tv/oLBZZfK
|
||||
⚠️ 短链接,需要用户提供完整URL或ID
|
||||
|
||||
🔗 检测到短链接
|
||||
|
||||
⚠️ 由于反爬虫保护,短链接无法自动解析
|
||||
|
||||
📱 手机用户:
|
||||
1. 打开链接
|
||||
2. 在浏览器地址栏复制完整 URL
|
||||
3. 发送给助手
|
||||
|
||||
💻 桌面用户:
|
||||
1. 发送 BV 号(如:BV1jcFqz7EFW)
|
||||
2. 或使用完整 URL
|
||||
|
||||
💡 提示:完整 URL 格式为:https://www.bilibili.com/video/BVxxxxx
|
||||
|
||||
其他链接是否立即处理?(y/n)
|
||||
```
|
||||
|
||||
## 批量处理
|
||||
|
||||
支持一次发送多个链接:
|
||||
|
||||
```
|
||||
用户: 这几个视频不错:
|
||||
https://www.bilibili.com/video/BV1xx411c7mD
|
||||
BV2yy522c8nE
|
||||
https://www.youtube.com/watch?v=dQw4w9WgXcQ
|
||||
|
||||
助手: 🔍 检测到内容链接:
|
||||
|
||||
📺 [1] B站视频
|
||||
URL: https://www.bilibili.com/video/BV1xx411c7mD
|
||||
ID: BV1xx411c7mD
|
||||
|
||||
📺 [2] B站视频
|
||||
URL: https://www.bilibili.com/video/BV2yy522c8nE
|
||||
ID: BV2yy522c8nE
|
||||
|
||||
▶️ [3] YouTube视频
|
||||
URL: https://www.youtube.com/watch?v=dQw4w9WgXcQ
|
||||
ID: dQw4w9WgXcQ
|
||||
|
||||
是否将上述 3 个内容添加到收件箱?(y/n)
|
||||
```
|
||||
|
||||
## 测试结果
|
||||
|
||||
已测试所有 URL 模式:
|
||||
|
||||
✅ B站完整 URL: `https://www.bilibili.com/video/BV1xx411c7mD`
|
||||
✅ B站 BV号: `BV1jcFqz7EFW`
|
||||
✅ B站短链接: `https://b23.tv/oLBZZfK` (显示友好提示)
|
||||
✅ YouTube 视频: `https://www.youtube.com/watch?v=dQw4w9WgXcQ`
|
||||
✅ 知乎文章: `https://zhuanlan.zhihu.com/p/123456789`
|
||||
✅ 重复检测: 正确去重,避免重复显示
|
||||
|
||||
## 文件变更
|
||||
|
||||
```
|
||||
skills/obsidian-kb/
|
||||
├── SKILL.md [修改] 添加 frontmatter 和触发规则
|
||||
├── handler.py [新增] 自动触发处理器
|
||||
└── AUTO_TRIGGER.md [新增] 本文档
|
||||
```
|
||||
|
||||
## 与 OpenClaw 集成
|
||||
|
||||
handler.py 可以通过以下方式集成:
|
||||
|
||||
1. **手动调用**: 使用 CLI 测试和调试
|
||||
```bash
|
||||
python3 handler.py "text with link"
|
||||
```
|
||||
|
||||
2. **Agent 调用**: AI Agent 在接收到用户消息后,调用 handler 检测链接
|
||||
|
||||
3. **插件集成**: 未来可创建 OpenClaw 插件,自动监听消息并触发 handler
|
||||
|
||||
## 验收标准
|
||||
|
||||
- [x] SKILL.md 添加了触发规则
|
||||
- [x] Handler 能够检测各种 URL 格式
|
||||
- [x] 显示友好的预览信息
|
||||
- [x] 短链接显示友好提示
|
||||
- [x] 支持批量检测多个链接
|
||||
- [x] 智能去重避免重复显示
|
||||
- [x] 所有 URL 模式测试通过
|
||||
|
||||
## 后续工作
|
||||
|
||||
### 短链接解析(P2 优先级) ✅ 已完成
|
||||
|
||||
已完成短链接解析功能(2026-02-06):
|
||||
|
||||
#### 已实现功能
|
||||
|
||||
1. **短链接检测模块** (`shortlink_resolver.py`)
|
||||
- 检测 B站短链接(b23.tv)
|
||||
- 支持其他短链接服务(bit.ly, t.cn, dwz.cn 等)
|
||||
- 提供友好的用户引导信息
|
||||
|
||||
2. **用户引导(方案A)**
|
||||
- 检测到短链接时显示清晰的操作指南
|
||||
- 分别针对手机用户和桌面用户提供建议
|
||||
- 支持单个短链接和多个短链接的不同提示格式
|
||||
|
||||
3. **浏览器自动化(方案B - 可选)**
|
||||
- 提供了 PlaywrightResolver 类
|
||||
- 需要额外安装 Playwright:
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
- 可在 `add_video.py` 中使用 `--playwright` 参数启用
|
||||
|
||||
4. **集成到现有流程**
|
||||
- `handler.py` 集成短链接检测
|
||||
- `add_video.py` 支持短链接解析
|
||||
- 短链接会优先显示帮助信息,不会自动处理
|
||||
|
||||
#### 使用示例
|
||||
|
||||
**handler.py 自动触发:**
|
||||
```bash
|
||||
python3 handler.py "Check this: https://b23.tv/oLBZZfK"
|
||||
```
|
||||
|
||||
**add_video.py 手动添加:**
|
||||
```bash
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK"
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK" --playwright # 使用浏览器解析
|
||||
```
|
||||
|
||||
**shortlink_resolver.py 独立测试:**
|
||||
```bash
|
||||
python3 shortlink_resolver.py "Test link: https://b23.tv/abc"
|
||||
python3 shortlink_resolver.py "https://b23.tv/abc" --playwright # 浏览器解析
|
||||
```
|
||||
|
||||
#### 文件变更
|
||||
|
||||
```
|
||||
skills/obsidian-kb/
|
||||
├── SKILL.md [修改] 添加短链接说明
|
||||
├── handler.py [修改] 集成短链接检测
|
||||
├── add_video.py [修改] 支持短链接解析
|
||||
├── shortlink_resolver.py [新增] 短链接解析模块
|
||||
└── AUTO_TRIGGER.md [修改] 添加短链接章节
|
||||
```
|
||||
|
||||
#### 验收标准
|
||||
|
||||
- [x] 短链接能够友好处理(显示引导信息)
|
||||
- [x] 用户不会感到困惑
|
||||
- [x] 支持 b23.tv 和其他短链接服务(检测)
|
||||
- [x] 集成到 handler.py
|
||||
- [x] 测试各种短链接场景(单个、多个、混合)
|
||||
|
||||
### YouTube 和知乎支持(P1 优先级)
|
||||
|
||||
当前仅 Bilibili 有完整适配器:
|
||||
- 实现 YouTube 适配器
|
||||
- 实现知乎适配器
|
||||
- 添加相应的提取和处理逻辑
|
||||
|
||||
## 总结
|
||||
|
||||
自动触发机制已基本完成,用户现在可以通过简单的发送链接来触发 obsidian-kb 技能,大大提升了使用体验。
|
||||
@@ -1,226 +0,0 @@
|
||||
# 短链接解析功能 - 实现总结
|
||||
|
||||
**完成时间**: 2026-02-06
|
||||
**任务**: 为 obsidian-kb 技能实现短链接解析功能
|
||||
**状态**: ✅ 已完成
|
||||
|
||||
---
|
||||
|
||||
## 📋 任务目标
|
||||
|
||||
支持 B站短链接(b23.tv)自动解析为完整 URL,让手机用户也能方便地分享视频。
|
||||
|
||||
## ✅ 已实现功能
|
||||
|
||||
### 1. 短链接解析模块 (`shortlink_resolver.py`)
|
||||
|
||||
#### 核心类
|
||||
|
||||
**ShortLinkResolver** - 短链接检测和引导
|
||||
- 检测 B站短链接(b23.tv)
|
||||
- 支持其他短链接服务(bit.ly, t.cn, dwz.cn 等)
|
||||
- 格式化友好的帮助信息
|
||||
|
||||
**PlaywrightResolver** - 浏览器自动化解析(可选)
|
||||
- 使用 Playwright 模拟真实浏览器
|
||||
- 绕过反爬虫保护
|
||||
- 需要额外安装 Playwright
|
||||
|
||||
#### 主要功能
|
||||
|
||||
```python
|
||||
from shortlink_resolver import ShortLinkResolver, detect_and_format
|
||||
|
||||
# 检测短链接
|
||||
has_links, message = detect_and_format("Check this: https://b23.tv/test")
|
||||
|
||||
# 直接使用解析器
|
||||
resolver = ShortLinkResolver()
|
||||
short_links = resolver.detect_short_links("Some text with short link")
|
||||
help_msg = resolver.format_short_link_help()
|
||||
```
|
||||
|
||||
### 2. 集成到现有流程
|
||||
|
||||
#### handler.py(自动触发)
|
||||
|
||||
检测到短链接时显示友好的用户引导:
|
||||
|
||||
```
|
||||
🔗 检测到短链接
|
||||
|
||||
⚠️ 由于反爬虫保护,短链接无法自动解析
|
||||
|
||||
📍 短链接: https://b23.tv/oLBZZfK
|
||||
|
||||
📱 手机用户:
|
||||
1. 点击/打开链接
|
||||
2. 在浏览器地址栏复制完整 URL
|
||||
3. 发送给助手
|
||||
|
||||
💻 桌面用户:
|
||||
1. 发送 BV 号(如:BV1jcFqz7EFW)
|
||||
2. 或使用完整 URL
|
||||
|
||||
💡 提示:完整 URL 格式为:https://www.bilibili.com/video/BVxxxxx
|
||||
```
|
||||
|
||||
#### add_video.py(手动添加)
|
||||
|
||||
支持短链接解析和 Playwright:
|
||||
|
||||
```bash
|
||||
# 默认显示引导
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK"
|
||||
|
||||
# 尝试使用浏览器解析
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK" --playwright
|
||||
```
|
||||
|
||||
### 3. 测试脚本 (`test_shortlink.py`)
|
||||
|
||||
全面的测试覆盖:
|
||||
|
||||
- ✅ 单个 B站短链接检测
|
||||
- ✅ 多个短链接检测
|
||||
- ✅ 无短链接场景
|
||||
- ✅ BV 码处理(不视为短链接)
|
||||
- ✅ 混合内容场景
|
||||
- ✅ 解析器直接使用
|
||||
|
||||
所有测试通过!
|
||||
|
||||
## 📦 文件变更
|
||||
|
||||
```
|
||||
skills/obsidian-kb/
|
||||
├── shortlink_resolver.py [新增] 短链接解析模块
|
||||
├── handler.py [修改] 集成短链接检测
|
||||
├── add_video.py [修改] 支持短链接解析
|
||||
├── test_shortlink.py [新增] 测试脚本
|
||||
├── SKILL.md [修改] 添加短链接说明
|
||||
├── AUTO_TRIGGER.md [修改] 添加短链接章节
|
||||
└── SHORTLINK_FEATURE.md [新增] 本文档
|
||||
|
||||
memory/tasks/
|
||||
└── obsidian-kb-improvements.md [修改] 标记任务2为已完成
|
||||
```
|
||||
|
||||
## 🎯 验收标准达成
|
||||
|
||||
- ✅ 短链接能够友好处理(显示引导信息)
|
||||
- ✅ 用户不会感到困惑
|
||||
- ✅ 支持 b23.tv 和其他短链接服务(检测)
|
||||
- ✅ 集成到 handler.py
|
||||
- ✅ 测试各种短链接场景(单个、多个、混合)
|
||||
|
||||
## 🔧 技术方案
|
||||
|
||||
### 方案A:用户引导(已实现 ✅)- 主功能
|
||||
|
||||
**优先级**: 最高(必须实现)
|
||||
|
||||
检测到短链接时显示友好的用户引导信息,分别针对手机用户和桌面用户提供建议。
|
||||
|
||||
**优点**:
|
||||
- 实现简单,无额外依赖
|
||||
- 用户体验友好
|
||||
- 避免反爬虫问题
|
||||
- 适用于所有短链接服务
|
||||
|
||||
**缺点**:
|
||||
- 需要用户手动操作
|
||||
- 无法自动解析
|
||||
|
||||
### 方案B:浏览器自动化(已实现框架 ✅)- 可选
|
||||
|
||||
**优先级**: 中等(如果时间允许)
|
||||
|
||||
使用 Playwright 模拟真实浏览器,解析短链接后自动获取完整 URL。
|
||||
|
||||
**安装**:
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**使用**:
|
||||
```bash
|
||||
python3 add_video.py "https://b23.tv/xxx" --playwright
|
||||
```
|
||||
|
||||
**优点**:
|
||||
- 可自动解析
|
||||
- 绕过反爬虫保护
|
||||
- 用户体验最佳
|
||||
|
||||
**缺点**:
|
||||
- 需要额外安装
|
||||
- 资源消耗较大
|
||||
- 可能在受限环境无法使用
|
||||
|
||||
### 方案C:代理池 + 请求头伪装(未实现 ❌)
|
||||
|
||||
**优先级**: 低(复杂度高,收益低)
|
||||
|
||||
**原因**:
|
||||
- 实现复杂度高
|
||||
- 需要维护代理池
|
||||
- 稳定性难以保证
|
||||
- 方案B已经可以解决问题
|
||||
|
||||
## 📊 使用示例
|
||||
|
||||
### 自动触发(通过 handler.py)
|
||||
|
||||
```bash
|
||||
# 单个短链接
|
||||
python3 handler.py "Check this: https://b23.tv/oLBZZfK"
|
||||
|
||||
# 多个短链接
|
||||
python3 handler.py "Check these: https://b23.tv/abc and https://bit.ly/test"
|
||||
```
|
||||
|
||||
### 手动添加(通过 add_video.py)
|
||||
|
||||
```bash
|
||||
# 默认显示引导
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK"
|
||||
|
||||
# 使用 Playwright 解析
|
||||
python3 add_video.py "https://b23.tv/oLBZZfK" --playwright
|
||||
```
|
||||
|
||||
### 独立使用(shortlink_resolver.py)
|
||||
|
||||
```bash
|
||||
# 检测短链接
|
||||
python3 shortlink_resolver.py "Test link: https://b23.tv/abc"
|
||||
|
||||
# 尝试 Playwright 解析
|
||||
python3 shortlink_resolver.py "https://b23.tv/abc" --playwright
|
||||
```
|
||||
|
||||
### 运行测试
|
||||
|
||||
```bash
|
||||
python3 test_shortlink.py
|
||||
```
|
||||
|
||||
## 🎉 总结
|
||||
|
||||
短链接解析功能已成功实现并集成到 obsidian-kb 技能中:
|
||||
|
||||
1. **方案A(用户引导)** - 完全实现,作为默认行为
|
||||
2. **方案B(浏览器自动化)** - 提供了完整的实现框架,用户可选择启用
|
||||
3. **方案C(代理池)** - 未实现,因为方案B已经足够
|
||||
|
||||
**实际工时**: 1.5 小时
|
||||
**测试状态**: 所有测试通过 ✅
|
||||
**文档状态**: 已更新 ✅
|
||||
**Git提交**: 已提交并推送 ✅
|
||||
|
||||
用户体验得到显著提升:
|
||||
- 手机用户不再需要手动打开短链接获取完整 URL
|
||||
- 桌面用户可以直接发送 BV 号
|
||||
- 短链接会显示友好的引导信息,不会让用户感到困惑
|
||||
+5
-204
@@ -1,10 +1,3 @@
|
||||
---
|
||||
name: obsidian-kb
|
||||
description: Detect and process video/article links (Bilibili, YouTube, Zhihu) for Obsidian KB. Auto-trigger on URLs: /kb-add, /kb-process, /kb-inbox, /kb-status.
|
||||
version: 1.0.0
|
||||
author: OpenClaw
|
||||
---
|
||||
|
||||
# SKILL.md - Obsidian KB Skill
|
||||
|
||||
## 元数据
|
||||
@@ -14,203 +7,19 @@ author: OpenClaw
|
||||
- **描述**: 多源内容知识库维护系统,支持长视频分段分析
|
||||
- **作者**: OpenClaw
|
||||
|
||||
## 自动触发规则
|
||||
|
||||
当检测到以下内容时,本技能会自动激活:
|
||||
|
||||
### 触发模式
|
||||
|
||||
1. **B站视频链接**
|
||||
- 完整URL: `https://www.bilibili.com/video/BV*`
|
||||
- 短链接: `https://b23.tv/*` (会提示用户提供完整URL或BV号)
|
||||
- BV号: `BV*` (直接支持)
|
||||
|
||||
2. **YouTube视频链接**
|
||||
- 完整URL: `https://www.youtube.com/watch?v=*`
|
||||
- 短链接: `https://youtu.be/*`
|
||||
|
||||
3. **知乎文章链接**
|
||||
- 专栏文章: `https://zhuanlan.zhihu.com/p/*`
|
||||
- 回答: `https://www.zhihu.com/question/*/answer/*`
|
||||
|
||||
### 自动触发流程
|
||||
|
||||
当用户发送匹配上述模式的链接时:
|
||||
|
||||
1. **检测链接** → 识别内容类型(B站/YouTube/知乎)
|
||||
2. **提取元数据** → 获取标题、作者、时长等信息
|
||||
3. **显示预览** → 向用户展示内容概要
|
||||
4. **询问确认** → "是否添加到收件箱并处理?(y/n)"
|
||||
5. **用户确认后** → 自动执行 `/kb-add` 命令添加内容
|
||||
6. **可选处理** → 根据用户选择,是否立即执行 `/kb-process`
|
||||
|
||||
### 批量处理
|
||||
|
||||
支持一次发送多个链接,系统会:
|
||||
- 检测所有匹配的模式
|
||||
- 显示所有识别到的内容预览
|
||||
- 询问用户是否批量添加
|
||||
- 显示处理进度
|
||||
|
||||
### 短链接友好提示
|
||||
|
||||
检测到短链接时,显示引导信息:
|
||||
|
||||
**单个短链接:**
|
||||
```
|
||||
🔗 检测到短链接
|
||||
|
||||
⚠️ 由于反爬虫保护,短链接无法自动解析
|
||||
|
||||
📍 短链接: https://b23.tv/xxxxx
|
||||
|
||||
📱 手机用户:
|
||||
1. 点击/打开链接
|
||||
2. 在浏览器地址栏复制完整 URL
|
||||
3. 发送给助手
|
||||
|
||||
💻 桌面用户:
|
||||
1. 发送 BV 号(如:BV1jcFqz7EFW)
|
||||
2. 或使用完整 URL
|
||||
|
||||
💡 提示:完整 URL 格式为:https://www.bilibili.com/video/BVxxxxx
|
||||
```
|
||||
|
||||
**多个短链接:**
|
||||
```
|
||||
⚠️ 检测到短链接
|
||||
|
||||
[1] B站短链接: https://b23.tv/abc123
|
||||
需要完整URL或BV号
|
||||
|
||||
[2] bit.ly: https://bit.ly/test
|
||||
需要完整URL
|
||||
|
||||
💡 建议:
|
||||
- 在浏览器打开链接后复制完整URL
|
||||
- 或者直接发送BV号(仅限B站)
|
||||
```
|
||||
|
||||
**技术说明:**
|
||||
- B站短链接(b23.tv)有反爬虫保护,无法直接解析
|
||||
- 本技能使用 `shortlink_resolver.py` 模块检测短链接
|
||||
- 提供 Playwright 可选方案(需额外安装)
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
## 命令列表
|
||||
|
||||
| 命令 | 描述 | 用法 |
|
||||
|------|------|------|
|
||||
| `/kb-add` | 添加单个视频/文章 | `/kb-add <URL> [--process] [--yes]` |
|
||||
| `/kb-subscribe` | 订阅内容源 | `/kb-subscribe <platform> <type> <id>` |
|
||||
| `/kb-inbox` | 查看收件箱 | `/kb-inbox [--limit N] [--sort score\|date]` |
|
||||
| `/kb-process` | 处理内容 | `/kb-process [content_id\|all]` |
|
||||
| `/kb-skip` | 跳过内容 | `/kb-skip <content_id> [--reason "..."]` |
|
||||
| `/kb-subscribe` | 订阅内容源 | `/kb-subscribe <source>:<type>:<id>` |
|
||||
| `/kb-inbox` | 查看收件箱 | `/kb-inbox [--limit N]` |
|
||||
| `/kb-process` | 处理内容 | `/kb-process [content_id]` |
|
||||
| `/kb-skip` | 跳过内容 | `/kb-skip <content_id>` |
|
||||
| `/kb-status` | 查看状态 | `/kb-status` |
|
||||
| `/kb-search` | 搜索知识库 | `/kb-search <query> [--in-tags]` |
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 方式1: 自动触发(推荐)
|
||||
|
||||
直接发送视频链接或文章链接,技能会自动识别并处理:
|
||||
|
||||
```
|
||||
# B站视频(完整URL)
|
||||
https://www.bilibili.com/video/BV1xx411c7mD
|
||||
|
||||
# B站视频(BV号)
|
||||
BV1xx411c7mD
|
||||
|
||||
# B站短链接(会提示完整URL)
|
||||
https://b23.tv/xxxxx
|
||||
|
||||
# YouTube视频
|
||||
https://www.youtube.com/watch?v=dQw4w9WgXcQ
|
||||
|
||||
# 知乎文章
|
||||
https://zhuanlan.zhihu.com/p/123456789
|
||||
```
|
||||
|
||||
### 启用字幕生成(可选)
|
||||
|
||||
如果视频没有字幕,可以启用本地字幕生成功能:
|
||||
|
||||
```bash
|
||||
# 1. 安装依赖
|
||||
pip install openai-whisper yt-dlp torch
|
||||
|
||||
# 2. 启用配置(编辑 config.yaml)
|
||||
subtitle:
|
||||
enabled: true
|
||||
model: base # 推荐使用 base 模型
|
||||
```
|
||||
|
||||
**说明:**
|
||||
- 使用 Whisper 本地模型,数据不上传,隐私安全
|
||||
- 首次使用会自动下载模型文件(~74MB)
|
||||
- 10分钟视频约需 2-5 分钟处理时间(取决于硬件)
|
||||
- 生成的字幕会自动缓存,重复使用无需重新生成
|
||||
|
||||
**详细文档:** 见 `SUBTITLE_FEATURE.md`
|
||||
|
||||
### 方式2: 使用命令
|
||||
|
||||
显式调用命令添加内容:
|
||||
|
||||
```bash
|
||||
# 添加单个视频
|
||||
/kb-add "https://www.bilibili.com/video/BV1xx411c7mD"
|
||||
|
||||
# 添加并立即处理
|
||||
/kb-add "https://www.bilibili.com/video/BV1xx411c7mD" --process
|
||||
|
||||
# 跳过确认提示
|
||||
/kb-add BV1xx411c7mD --yes
|
||||
```
|
||||
| `/kb-search` | 搜索知识库 | `/kb-search <query>` |
|
||||
|
||||
## 命令详解
|
||||
|
||||
### /kb-add
|
||||
|
||||
快速添加单个视频到收件箱。
|
||||
|
||||
**用法:**
|
||||
```
|
||||
/kb-add BV1xx411c7mD
|
||||
/kb-add "https://www.bilibili.com/video/BV1xx411c7mD"
|
||||
/kb-add "https://www.bilibili.com/video/BV1xx411c7mD" --process
|
||||
/kb-add BV1xx411c7mD --yes
|
||||
```
|
||||
|
||||
**参数:**
|
||||
- `<URL>` - 视频URL或BV号
|
||||
- 完整URL: `https://www.bilibili.com/video/BVxxxxx`
|
||||
- BV号: `BVxxxxx`
|
||||
- 短链接: `https://b23.tv/xxxxx`(显示友好提示)
|
||||
- `--process` - 添加后立即处理
|
||||
- `--yes` - 跳过确认提示
|
||||
|
||||
**输出示例:**
|
||||
```
|
||||
📥 正在添加视频: BV1xx411c7mD
|
||||
→ 标题: Python编程入门教程
|
||||
→ UP主: CodeMaster
|
||||
→ 时长: 3600秒
|
||||
→ 获取字幕... ✅
|
||||
|
||||
✅ 视频已添加到收件箱!
|
||||
Content ID: abc123def456
|
||||
|
||||
💡 使用以下命令处理:
|
||||
python3 src/cli.py process abc123def456
|
||||
或:
|
||||
python3 src/cli.py process all
|
||||
```
|
||||
|
||||
### /kb-subscribe
|
||||
|
||||
订阅一个内容源,支持自动获取更新。
|
||||
@@ -344,14 +153,6 @@ chunking:
|
||||
max_chunk_duration_seconds: 1800 # 最大分段时长
|
||||
long_video_threshold: 1800 # 长视频阈值 (30分钟)
|
||||
|
||||
# 字幕生成配置(新增)
|
||||
subtitle:
|
||||
enabled: false # 是否启用自动生成字幕
|
||||
model: base # Whisper 模型大小: tiny/base/small/medium/large
|
||||
language: null # 语言代码: zh/en/null(自动检测)
|
||||
cache_dir: cache/subtitles # 字幕缓存目录
|
||||
audio_cache_dir: cache/audio # 音频缓存目录
|
||||
|
||||
# 数据源配置
|
||||
sources:
|
||||
bilibili:
|
||||
|
||||
@@ -1,373 +0,0 @@
|
||||
# Obsidian KB - 本地字幕生成功能
|
||||
|
||||
## 📖 功能概述
|
||||
|
||||
为无字幕视频自动生成字幕,使 AI 分析功能不受字幕限制,覆盖更多视频内容。
|
||||
|
||||
### 核心特性
|
||||
|
||||
- ✅ **本地处理**: 使用 Whisper 本地模型,数据不上传,隐私安全
|
||||
- ✅ **完全免费**: 无 API 费用,仅需一次性下载模型
|
||||
- ✅ **缓存机制**: 避免重复生成,提升处理速度
|
||||
- ✅ **多语言支持**: 支持中英文识别及自动检测
|
||||
- ✅ **灵活配置**: 可选择不同大小的模型平衡速度与准确度
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 架构设计
|
||||
|
||||
### 核心模块
|
||||
|
||||
```
|
||||
subtitle_generator.py
|
||||
├── AudioDownloader # 音频下载器 (yt-dlp)
|
||||
├── WhisperTranscriber # Whisper 识别器
|
||||
├── SubtitleFormatter # SRT 格式化器
|
||||
├── SubtitleCache # 缓存管理器
|
||||
└── SubtitleGenerator # 统一接口
|
||||
```
|
||||
|
||||
### 工作流程
|
||||
|
||||
```
|
||||
用户请求视频处理
|
||||
↓
|
||||
检查是否有字幕
|
||||
↓ [有字幕]
|
||||
使用现有字幕
|
||||
↓ [无字幕 + 功能已启用]
|
||||
下载音频 (yt-dlp)
|
||||
↓
|
||||
语音识别 (Whisper)
|
||||
↓
|
||||
转换为 SRT 格式
|
||||
↓
|
||||
保存到缓存
|
||||
↓
|
||||
继续 AI 分析
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
```bash
|
||||
cd skills/obsidian-kb
|
||||
|
||||
# 激活虚拟环境(如果使用)
|
||||
source venv/bin/activate # Linux/Mac
|
||||
# 或
|
||||
./venv/Scripts/activate # Windows
|
||||
|
||||
# 安装依赖
|
||||
pip install openai-whisper yt-dlp torch
|
||||
```
|
||||
|
||||
### 2. 系统依赖
|
||||
|
||||
确保系统已安装 `ffmpeg`:
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt install ffmpeg
|
||||
|
||||
# macOS
|
||||
brew install ffmpeg
|
||||
|
||||
# Windows
|
||||
# 从 https://ffmpeg.org/ 下载并安装
|
||||
```
|
||||
|
||||
验证安装:
|
||||
```bash
|
||||
ffmpeg -version
|
||||
```
|
||||
|
||||
### 3. 配置
|
||||
|
||||
编辑 `config.yaml`:
|
||||
|
||||
```yaml
|
||||
# 字幕生成配置
|
||||
subtitle:
|
||||
enabled: true # 启用自动生成
|
||||
model: base # Whisper 模型大小
|
||||
language: null # 语言: zh/en/null(自动检测)
|
||||
cache_dir: cache/subtitles # 字幕缓存目录
|
||||
audio_cache_dir: cache/audio # 音频缓存目录
|
||||
```
|
||||
|
||||
### 4. 使用
|
||||
|
||||
功能启用后,处理无字幕视频时会自动生成字幕:
|
||||
|
||||
```python
|
||||
from src.pipeline import ProcessingPipeline
|
||||
from src.models import Content
|
||||
|
||||
pipeline = ProcessingPipeline()
|
||||
content = Content(url="https://www.bilibili.com/video/BVxxxxx")
|
||||
|
||||
# 处理时会自动生成字幕(如果需要)
|
||||
pipeline.process_full(content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 模型选择
|
||||
|
||||
Whisper 提供多种大小的模型,可根据需求选择:
|
||||
|
||||
| 模型 | 大小 | 10分钟视频处理时间 | 准确度 | 推荐场景 |
|
||||
|------|------|-------------------|--------|----------|
|
||||
| tiny | ~39MB | ~30秒 | ⭐⭐ | 快速测试、批量处理 |
|
||||
| **base** | **~74MB** | **~2分钟** | **⭐⭐⭐⭐** | **日常使用(推荐)** |
|
||||
| small | ~244MB | ~5分钟 | ⭐⭐⭐⭐⭐ | 高质量需求 |
|
||||
| medium | ~769MB | ~10分钟 | ⭐⭐⭐⭐⭐ | 专业用途 |
|
||||
| large | ~1550MB | ~15分钟+ | ⭐⭐⭐⭐⭐ | 最高准确度 |
|
||||
|
||||
### 推荐配置
|
||||
|
||||
- **默认**: `base` - 速度与准确度的最佳平衡
|
||||
- **低配机器**: `tiny` - 最小资源占用
|
||||
- **高质量需求**: `small` - 更好的识别效果
|
||||
- **有 GPU**: `medium` / `large` - 充分利用硬件加速
|
||||
|
||||
---
|
||||
|
||||
## 🎯 使用场景
|
||||
|
||||
### 1. 无字幕视频
|
||||
|
||||
B站、YouTube 等平台的部分视频没有提供字幕,启用此功能后可以自动生成。
|
||||
|
||||
### 2. 批量处理
|
||||
|
||||
处理多个视频时,缓存机制会显著提升速度:
|
||||
|
||||
```python
|
||||
# 第一次处理:下载 + 识别(约 2-5 分钟)
|
||||
pipeline.process_full(content1)
|
||||
|
||||
# 第二次处理相同视频:直接读取缓存(约 1 秒)
|
||||
pipeline.process_full(content1)
|
||||
```
|
||||
|
||||
### 3. 多语言内容
|
||||
|
||||
支持中英文混合内容的自动识别:
|
||||
|
||||
```yaml
|
||||
subtitle:
|
||||
language: null # 自动检测语言
|
||||
```
|
||||
|
||||
或指定语言:
|
||||
|
||||
```yaml
|
||||
subtitle:
|
||||
language: zh # 强制使用中文
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 测试
|
||||
|
||||
运行测试脚本验证功能:
|
||||
|
||||
```bash
|
||||
cd skills/obsidian-kb
|
||||
python test_subtitle_generator.py
|
||||
```
|
||||
|
||||
测试覆盖:
|
||||
- ✅ 音频下载
|
||||
- ✅ Whisper 识别
|
||||
- ✅ SRT 格式化
|
||||
- ✅ 缓存机制
|
||||
- ✅ 完整工作流程
|
||||
|
||||
---
|
||||
|
||||
## 📦 API 文档
|
||||
|
||||
### SubtitleGenerator
|
||||
|
||||
主要接口类,整合所有功能。
|
||||
|
||||
```python
|
||||
from subtitle_generator import SubtitleGenerator
|
||||
|
||||
generator = SubtitleGenerator(
|
||||
model_size='base', # 模型大小
|
||||
cache_dir='cache/subtitles' # 缓存目录
|
||||
)
|
||||
|
||||
# 生成字幕
|
||||
srt_text = generator.generate(
|
||||
url='https://www.bilibili.com/video/BVxxxxx',
|
||||
content_id='unique_id',
|
||||
language='zh', # 可选:指定语言
|
||||
use_cache=True # 可选:使用缓存
|
||||
)
|
||||
```
|
||||
|
||||
### 便捷函数
|
||||
|
||||
快速生成字幕的便捷函数:
|
||||
|
||||
```python
|
||||
from subtitle_generator import generate_subtitle
|
||||
|
||||
srt_text = generate_subtitle(
|
||||
url='https://www.bilibili.com/video/BVxxxxx',
|
||||
content_id='unique_id',
|
||||
model='base',
|
||||
language='zh'
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 高级配置
|
||||
|
||||
### GPU 加速
|
||||
|
||||
如果系统有 NVIDIA GPU 并安装了 CUDA,Whisper 会自动使用 GPU 加速:
|
||||
|
||||
```python
|
||||
import torch
|
||||
print(f"GPU 可用: {torch.cuda.is_available()}")
|
||||
```
|
||||
|
||||
使用 GPU 时,处理速度可提升 5-10 倍。
|
||||
|
||||
### 清理缓存
|
||||
|
||||
手动清理字幕缓存:
|
||||
|
||||
```python
|
||||
from subtitle_generator import SubtitleCache
|
||||
|
||||
cache = SubtitleCache()
|
||||
cache.clear()
|
||||
```
|
||||
|
||||
或直接删除目录:
|
||||
|
||||
```bash
|
||||
rm -rf cache/subtitles/*
|
||||
```
|
||||
|
||||
### 自定义模型路径
|
||||
|
||||
如果需要使用自定义模型文件:
|
||||
|
||||
```python
|
||||
import whisper
|
||||
|
||||
model = whisper.load_model(
|
||||
'/path/to/custom/model.pt',
|
||||
device='cpu'
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 注意事项
|
||||
|
||||
### 1. 存储空间
|
||||
|
||||
- **模型文件**: ~150MB (tiny) - 3GB (large)
|
||||
- **音频缓存**: 每分钟视频约 1-2MB(处理完成后自动清理)
|
||||
- **字幕缓存**: 每分钟视频约 1-3KB(长期保留)
|
||||
|
||||
建议:
|
||||
- 使用 `base` 模型作为默认
|
||||
- 定期清理 `cache/audio` 目录
|
||||
- 字幕缓存可以长期保留
|
||||
|
||||
### 2. 处理时间
|
||||
|
||||
- **无 GPU**: 10分钟视频约 2-5 分钟(base 模型)
|
||||
- **有 GPU**: 10分钟视频约 20-40 秒(base 模型)
|
||||
|
||||
### 3. 隐私安全
|
||||
|
||||
- ✅ Whisper 本地处理,音频不上传
|
||||
- ✅ 缓存文件在本地,不共享
|
||||
- ✅ 开源软件,代码透明
|
||||
|
||||
### 4. 识别准确度
|
||||
|
||||
Whisper 的识别效果取决于:
|
||||
- **音频质量**: 清晰的音频效果更好
|
||||
- **背景噪音**: 嘈杂环境可能影响识别
|
||||
- **说话方式**: 语速过快、口音过重可能降低准确度
|
||||
|
||||
建议:
|
||||
- 优先处理高质量音频
|
||||
- 对生成的字幕进行人工审核
|
||||
- 复杂内容可考虑使用人工字幕
|
||||
|
||||
### 5. 错误处理
|
||||
|
||||
系统已完善的错误处理机制:
|
||||
|
||||
- **下载失败**: 记录错误,继续处理
|
||||
- **识别失败**: 记录错误,继续处理
|
||||
- **格式化失败**: 使用原始文本
|
||||
|
||||
任何错误都不会中断主流程。
|
||||
|
||||
---
|
||||
|
||||
## 🚀 后续优化方向
|
||||
|
||||
### 1. GPU 加速支持
|
||||
|
||||
- [ ] 自动检测 GPU 可用性
|
||||
- [ ] 支持 CUDA / MPS (Apple Silicon)
|
||||
- [ ] 多 GPU 支持
|
||||
|
||||
### 2. 模型优化
|
||||
|
||||
- [ ] 支持量化模型(减小体积)
|
||||
- [ ] 支持蒸馏模型(提升速度)
|
||||
- [ ] 支持自定义模型微调
|
||||
|
||||
### 3. 功能增强
|
||||
|
||||
- [ ] 批量处理多个视频
|
||||
- [ ] 支持更多视频平台
|
||||
- [ ] 字幕编辑器 Web 界面
|
||||
- [ ] 实时字幕生成
|
||||
|
||||
### 4. 云端集成
|
||||
|
||||
- [ ] 可选切换到云 API(阿里云/腾讯云)
|
||||
- [ ] 混合模式:本地优先,云端降级
|
||||
- [ ] 分布式处理支持
|
||||
|
||||
---
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
本功能使用以下开源组件:
|
||||
|
||||
- **openai-whisper**: MIT License
|
||||
- **yt-dlp**: Unlicense
|
||||
- **torch**: BSD License
|
||||
|
||||
---
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎贡献代码、报告问题或提出建议!
|
||||
|
||||
---
|
||||
|
||||
**最后更新**: 2026-02-07
|
||||
**版本**: 1.0.0
|
||||
@@ -1,6 +0,0 @@
|
||||
{
|
||||
"ownerId": "kn7xgqexpl4ek5qwng3t4hk0fu77kbex",
|
||||
"slug": "obsidian-kb",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1738857000000
|
||||
}
|
||||
@@ -1,145 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""手动添加B站视频到obsidian-kb收件箱"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import urllib.request
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
# 添加路径以便导入模块
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), 'src'))
|
||||
|
||||
from sources.bilibili import BilibiliAdapter
|
||||
from models import Content, ContentStatus
|
||||
from storage import Storage
|
||||
from shortlink_resolver import (
|
||||
ShortLinkResolver,
|
||||
PlaywrightResolver,
|
||||
detect_and_format
|
||||
)
|
||||
|
||||
def resolve_short_url(url: str, try_playwright: bool = False) -> str:
|
||||
"""
|
||||
解析短链接(如 b23.tv)到完整URL,或接受BV号
|
||||
|
||||
Args:
|
||||
url: 可能是短链接的URL、完整URL或BV号
|
||||
try_playwright: 是否尝试使用Playwright解析短链接
|
||||
|
||||
Returns:
|
||||
str: 完整URL或BV号,如果失败返回None
|
||||
"""
|
||||
# 如果已经是完整URL,直接返回
|
||||
if 'bilibili.com/video/' in url:
|
||||
return url
|
||||
|
||||
# 如果是BV号,直接返回(bilibili.py 的 extract_content 会处理)
|
||||
if url.upper().startswith('BV'):
|
||||
return url
|
||||
|
||||
# 如果是短链接,尝试解析
|
||||
resolver = ShortLinkResolver()
|
||||
|
||||
if resolver.is_bilibili_short_link(url):
|
||||
print(f" ⚠️ 检测到短链接 {url}")
|
||||
|
||||
# 尝试使用Playwright解析(如果启用)
|
||||
if try_playwright:
|
||||
print(f" 🔄 尝试使用浏览器解析...")
|
||||
playwright_resolver = PlaywrightResolver(headless=True)
|
||||
|
||||
try:
|
||||
resolved_url = playwright_resolver.resolve_bilibili_short(url)
|
||||
if resolved_url:
|
||||
print(f" ✅ 解析成功: {resolved_url}")
|
||||
return resolved_url
|
||||
else:
|
||||
print(f" ❌ 浏览器解析失败")
|
||||
except Exception as e:
|
||||
print(f" ❌ Playwright不可用或解析失败: {e}")
|
||||
|
||||
# 显示用户引导
|
||||
print("")
|
||||
print(resolver.format_short_link_help(url))
|
||||
print("")
|
||||
return None
|
||||
|
||||
# 默认返回原URL
|
||||
return url
|
||||
|
||||
def add_video_to_inbox(url: str, try_playwright: bool = False) -> str:
|
||||
"""
|
||||
添加B站视频到收件箱
|
||||
|
||||
Args:
|
||||
url: B站视频URL
|
||||
try_playwright: 是否尝试使用Playwright解析短链接
|
||||
|
||||
Returns:
|
||||
str: Content ID
|
||||
"""
|
||||
print(f"📥 正在处理视频: {url}")
|
||||
|
||||
# 解析短链接
|
||||
resolved_url = resolve_short_url(url, try_playwright=try_playwright)
|
||||
if resolved_url is None:
|
||||
return None
|
||||
|
||||
# 创建适配器
|
||||
adapter = BilibiliAdapter()
|
||||
|
||||
# 提取视频元数据
|
||||
print(" → 获取视频信息...")
|
||||
content = adapter.extract_content(resolved_url)
|
||||
if not content:
|
||||
print(" ❌ 无法提取视频信息")
|
||||
return None
|
||||
|
||||
print(f" → 标题: {content.title}")
|
||||
print(f" → UP主: {content.author}")
|
||||
print(f" → 时长: {content.duration}秒")
|
||||
|
||||
# 获取字幕
|
||||
print(" → 获取字幕...")
|
||||
transcript = adapter.get_transcript(content)
|
||||
if transcript:
|
||||
content.transcript = transcript
|
||||
print(f" ✓ 字幕已获取 ({len(transcript)} 字符)")
|
||||
else:
|
||||
print(" ⚠️ 无法获取字幕")
|
||||
|
||||
# 设置状态为已确认(准备处理)
|
||||
content.status = ContentStatus.CONFIRMED
|
||||
|
||||
# 保存到收件箱
|
||||
print(" → 保存到收件箱...")
|
||||
storage = Storage()
|
||||
storage.save_content(content)
|
||||
|
||||
print(f"\n✅ 视频已添加到收件箱!")
|
||||
print(f" Content ID: {content.id}")
|
||||
print(f"\n💡 使用以下命令处理:")
|
||||
print(f" python3 src/cli.py process {content.id}")
|
||||
print(f" 或者:")
|
||||
print(f" python3 src/cli.py process all")
|
||||
|
||||
return content.id
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 2:
|
||||
print("用法: python3 add_video.py <B站视频URL>")
|
||||
print("示例:")
|
||||
print(" python3 add_video.py https://www.bilibili.com/video/BV1xx411c7mD")
|
||||
print(" python3 add_video.py BV1xx411c7mD")
|
||||
print(" python3 add_video.py https://b23.tv/oLBZZfK")
|
||||
print("\n选项:")
|
||||
print(" --playwright 尝试使用浏览器解析短链接")
|
||||
sys.exit(1)
|
||||
|
||||
url = sys.argv[1]
|
||||
try_playwright = '--playwright' in sys.argv
|
||||
if try_playwright and url == '--playwright' and len(sys.argv) > 2:
|
||||
url = sys.argv[2]
|
||||
|
||||
add_video_to_inbox(url, try_playwright=try_playwright)
|
||||
@@ -1,51 +0,0 @@
|
||||
# obsidian-kb 配置文件示例
|
||||
# 复制此文件为 config.yaml 并修改配置
|
||||
|
||||
# Obsidian 笔记库路径
|
||||
obsidian_vault: ~/Documents/ObsidianVault
|
||||
|
||||
# 笔记输出目录(相对于 vault)
|
||||
output_dir: AI_Generated
|
||||
|
||||
# Git 同步配置
|
||||
git:
|
||||
enabled: false
|
||||
repo: ~/Documents/ObsidianVault
|
||||
branch: main
|
||||
auto_commit: true
|
||||
auto_push: true
|
||||
|
||||
# AI 客户端配置
|
||||
ai_client:
|
||||
provider: openai # openai 或 anthropic
|
||||
model: gpt-4o-mini
|
||||
api_key: ${OPENAI_API_KEY} # 从环境变量读取
|
||||
base_url: null # 自定义 API 端点(可选)
|
||||
temperature: 0.7
|
||||
max_tokens: 4000
|
||||
|
||||
# 内容处理配置
|
||||
content:
|
||||
max_video_length: 3600 # 最大视频长度(秒),超过则跳过
|
||||
min_transcript_length: 100 # 最小字幕长度(字符)
|
||||
chunk_size: 2000 # 文本分块大小(字符)
|
||||
chunk_overlap: 200 # 分块重叠大小(字符)
|
||||
|
||||
# 模板配置
|
||||
templates:
|
||||
note_template: templates/default_note.md
|
||||
summary_template: templates/default_summary.md
|
||||
deep_analysis_template: templates/default_deep_analysis.md
|
||||
|
||||
# 字幕生成配置(新增)
|
||||
subtitle:
|
||||
enabled: false # 是否启用自动生成字幕
|
||||
model: base # Whisper 模型大小: tiny/base/small/medium/large
|
||||
language: null # 语言代码: zh/en/null(自动检测)
|
||||
cache_dir: cache/subtitles # 字幕缓存目录
|
||||
audio_cache_dir: cache/audio # 音频缓存目录
|
||||
|
||||
# 日志配置
|
||||
logging:
|
||||
level: INFO # DEBUG, INFO, WARNING, ERROR
|
||||
file: logs/obsidian-kb.log
|
||||
@@ -9,13 +9,6 @@ jinja2>=3.1.2
|
||||
openai>=1.0.0
|
||||
anthropic>=0.7.0
|
||||
|
||||
# Subtitle generation
|
||||
openai-whisper>=20230314
|
||||
yt-dlp>=2023.1.6
|
||||
# Use CPU version for torch (smaller, faster to install)
|
||||
# To use GPU, remove --index-url and install with: pip install torch
|
||||
torch>=2.0.0
|
||||
|
||||
# Testing
|
||||
pytest>=7.4.0
|
||||
pytest-cov>=4.1.0
|
||||
|
||||
@@ -18,7 +18,6 @@ try:
|
||||
from src.storage import Storage
|
||||
from src.pipeline import ProcessingPipeline
|
||||
from src.models import ContentStatus
|
||||
from src.sources.bilibili import BilibiliAdapter
|
||||
CONFIG_AVAILABLE = True
|
||||
except ImportError as e:
|
||||
print(f"Import warning: {e}", file=sys.stderr)
|
||||
@@ -103,110 +102,6 @@ def cmd_subscribe(args: argparse.Namespace):
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_add(args: argparse.Namespace):
|
||||
"""Handle add command - add a single video to inbox."""
|
||||
print_header("Add Video to Inbox")
|
||||
|
||||
if not CONFIG_AVAILABLE:
|
||||
print("Error: Required modules not available. Please check installation.")
|
||||
return 1
|
||||
|
||||
# Import here to avoid issues if not available
|
||||
from src.models import ContentStatus
|
||||
|
||||
url = args.url
|
||||
|
||||
# Handle short links (b23.tv)
|
||||
if 'b23.tv' in url:
|
||||
print(f" ⚠️ 检测到短链接: {url}")
|
||||
print(f" 💡 B站对短链接有反爬虫保护,请提供:")
|
||||
print(f" 1. 完整URL: https://www.bilibili.com/video/BVxxxxx")
|
||||
print(f" 2. 或BV号: BVxxxxx")
|
||||
print(f"\n 提示: 可以在浏览器打开短链接,从地址栏复制完整URL")
|
||||
return 1
|
||||
|
||||
# Create adapter
|
||||
adapter = BilibiliAdapter()
|
||||
|
||||
# Extract video metadata
|
||||
print(f" → 正在获取视频信息: {url}")
|
||||
content = adapter.extract_content(url)
|
||||
if not content:
|
||||
print(" ❌ 无法提取视频信息")
|
||||
return 1
|
||||
|
||||
print(f" → 标题: {content.title}")
|
||||
print(f" → UP主: {content.author}")
|
||||
print(f" → 时长: {content.duration}秒")
|
||||
|
||||
# Get transcript
|
||||
print(" → 正在获取字幕...")
|
||||
transcript = adapter.get_transcript(content)
|
||||
if transcript:
|
||||
content.transcript = transcript
|
||||
print(f" ✓ 字幕已获取 ({len(transcript)} 字符)")
|
||||
else:
|
||||
print(" ⚠️ 无法获取字幕")
|
||||
|
||||
# Set status to CONFIRMED (ready to process)
|
||||
content.status = ContentStatus.CONFIRMED
|
||||
|
||||
# Confirm before saving (unless --yes)
|
||||
if not args.yes:
|
||||
print(f"\n📋 准备添加到收件箱:")
|
||||
print(f" 标题: {content.title}")
|
||||
print(f" UP主: {content.author}")
|
||||
print(f" 时长: {content.duration}秒")
|
||||
print(f" 字幕: {'有' if transcript else '无'}")
|
||||
|
||||
if args.process:
|
||||
print(f" 操作: 添加并立即处理")
|
||||
else:
|
||||
print(f" 操作: 仅添加到收件箱")
|
||||
|
||||
confirm = input("\n确认添加? [Y/n]: ").strip().lower()
|
||||
if confirm and confirm not in ('y', 'yes'):
|
||||
print("❌ 已取消")
|
||||
return 0
|
||||
|
||||
# Save to inbox
|
||||
print("\n → 保存到收件箱...")
|
||||
storage = Storage()
|
||||
storage.save_content(content)
|
||||
|
||||
print(f"\n✅ 视频已添加到收件箱!")
|
||||
print(f" Content ID: {content.id}")
|
||||
|
||||
# Process immediately if --process flag is set
|
||||
if args.process:
|
||||
print(f"\n🔄 开始处理内容...")
|
||||
try:
|
||||
from src.pipeline import ProcessingPipeline
|
||||
from src.models import ContentStatus
|
||||
|
||||
pipeline = ProcessingPipeline()
|
||||
content_text = content.transcript or content.summary or ""
|
||||
|
||||
result = pipeline.process_full(content, content_text)
|
||||
storage.save_content(result)
|
||||
|
||||
print(f" ✅ 处理完成")
|
||||
print(f" 📄 笔记: {result.output_path or 'N/A'}")
|
||||
print(f" 🏷️ 标签: {', '.join(result.tags)}")
|
||||
except Exception as e:
|
||||
print(f" ❌ 处理失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
else:
|
||||
print(f"\n💡 使用以下命令处理:")
|
||||
print(f" python3 src/cli.py process {content.id}")
|
||||
print(f" 或:")
|
||||
print(f" python3 src/cli.py process all")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_inbox(args: argparse.Namespace):
|
||||
"""Handle inbox command - display pending items."""
|
||||
print_header("Inbox Contents")
|
||||
@@ -532,9 +427,6 @@ def main():
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
%(prog)s add "BV1xx411c7mD"
|
||||
%(prog)s add "https://www.bilibili.com/video/BV1xx411c7mD" --process
|
||||
%(prog)s add "https://b23.tv/xxxxx" --yes
|
||||
%(prog)s subscribe bilibili fav 12345 --name="Tech Favorites"
|
||||
%(prog)s inbox --limit 10 --sort date
|
||||
%(prog)s process all --yes
|
||||
@@ -547,28 +439,6 @@ Examples:
|
||||
|
||||
subparsers = parser.add_subparsers(dest='command', help='Available commands')
|
||||
|
||||
# add 子命令
|
||||
add_parser = subparsers.add_parser(
|
||||
'add',
|
||||
aliases=['a'],
|
||||
help='Add a single video to inbox'
|
||||
)
|
||||
add_parser.add_argument(
|
||||
'url',
|
||||
help='Video URL or BV ID (supports: BVxxxxx, full URL)'
|
||||
)
|
||||
add_parser.add_argument(
|
||||
'--process',
|
||||
action='store_true',
|
||||
help='Process immediately after adding'
|
||||
)
|
||||
add_parser.add_argument(
|
||||
'--yes',
|
||||
action='store_true',
|
||||
help='Skip confirmation prompts'
|
||||
)
|
||||
add_parser.set_defaults(func=cmd_add)
|
||||
|
||||
# subscribe 子命令
|
||||
subscribe_parser = subparsers.add_parser(
|
||||
'subscribe',
|
||||
|
||||
@@ -4,7 +4,7 @@ C4.3: 实现完整处理流程
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from src.models import Content, ContentStatus, AnalysisResult, ContentType
|
||||
from src.models import Content, ContentStatus, AnalysisResult
|
||||
from src.light_scanner import LightScanner
|
||||
from src.deep_analyzer import DeepAnalyzer
|
||||
from src.config import get_config
|
||||
@@ -12,172 +12,29 @@ from src.config import get_config
|
||||
|
||||
class ProcessingPipeline:
|
||||
"""内容处理管道 - 协调轻量扫描、深度分析、笔记生成和同步"""
|
||||
|
||||
|
||||
def __init__(self, ai_client=None, git_client=None):
|
||||
self.config = get_config()
|
||||
self.ai_client = ai_client
|
||||
self.git_client = git_client
|
||||
|
||||
|
||||
# 初始化处理器
|
||||
self.light_scanner = LightScanner(ai_client=ai_client)
|
||||
self.deep_analyzer = DeepAnalyzer(ai_client=ai_client)
|
||||
|
||||
# 初始化字幕生成器(如果启用)
|
||||
self.subtitle_generator = None
|
||||
self._init_subtitle_generator()
|
||||
|
||||
def _init_subtitle_generator(self):
|
||||
"""初始化字幕生成器"""
|
||||
subtitle_config = self.config.get('subtitle', {})
|
||||
|
||||
if subtitle_config.get('enabled', False):
|
||||
try:
|
||||
# 延迟导入,避免依赖问题
|
||||
from subtitle_generator import SubtitleGenerator
|
||||
|
||||
model_size = subtitle_config.get('model', 'base')
|
||||
cache_dir = subtitle_config.get('cache_dir', 'cache/subtitles')
|
||||
|
||||
self.subtitle_generator = SubtitleGenerator(
|
||||
model_size=model_size,
|
||||
cache_dir=cache_dir
|
||||
)
|
||||
|
||||
print(f"[Pipeline] 字幕生成器已初始化 (model={model_size})")
|
||||
except ImportError as e:
|
||||
print(f"[Pipeline] 字幕生成器初始化失败: {e}")
|
||||
print(f"[Pipeline] 提示: 需要安装 openai-whisper 和 yt-dlp")
|
||||
self.subtitle_generator = None
|
||||
except Exception as e:
|
||||
print(f"[Pipeline] 字幕生成器初始化失败: {e}")
|
||||
self.subtitle_generator = None
|
||||
|
||||
def _ensure_subtitle(self, content: Content) -> str:
|
||||
"""
|
||||
确保内容有字幕,如果没有则生成
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
|
||||
Returns:
|
||||
str: 字幕文本
|
||||
"""
|
||||
# 1. 检查是否已有字幕
|
||||
if content.transcript and len(content.transcript) > 100:
|
||||
print(f"[Pipeline] 使用现有字幕 (长度: {len(content.transcript)} 字符)")
|
||||
return content.transcript
|
||||
|
||||
# 2. 检查是否启用自动生成
|
||||
if not self.subtitle_generator:
|
||||
print(f"[Pipeline] 字幕生成器未启用或初始化失败,跳过生成")
|
||||
return ""
|
||||
|
||||
# 3. 只处理视频内容
|
||||
if content.content_type != ContentType.VIDEO:
|
||||
print(f"[Pipeline] 非视频内容,跳过字幕生成")
|
||||
return ""
|
||||
|
||||
# 4. 生成字幕
|
||||
try:
|
||||
print(f"[Pipeline] 开始自动生成字幕...")
|
||||
|
||||
subtitle_config = self.config.get('subtitle', {})
|
||||
audio_cache_dir = subtitle_config.get('audio_cache_dir', 'cache/audio')
|
||||
language = subtitle_config.get('language', None)
|
||||
|
||||
# 生成字幕
|
||||
srt_text = self.subtitle_generator.generate(
|
||||
url=content.url,
|
||||
content_id=content.id,
|
||||
audio_cache_dir=audio_cache_dir,
|
||||
language=language,
|
||||
use_cache=True
|
||||
)
|
||||
|
||||
# 更新内容
|
||||
content.transcript = srt_text
|
||||
|
||||
print(f"[Pipeline] 字幕生成完成 (长度: {len(srt_text)} 字符)")
|
||||
return srt_text
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Pipeline] 字幕生成失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return ""
|
||||
|
||||
def process_full(self, content: Content, content_text: str = None) -> Content:
|
||||
"""
|
||||
完整处理流程:
|
||||
轻量扫描 → 入库 → 深度分析 → 生成笔记 → Git同步
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
content_text: 内容文本(可选,如果没有则自动生成字幕)
|
||||
|
||||
Returns:
|
||||
完全处理后的Content对象
|
||||
"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"[Pipeline] 开始完整处理流程: {content.id}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
try:
|
||||
# Step 0: 确保有字幕(新增)
|
||||
print("\n📝 Step 0: 确保字幕可用")
|
||||
content_text = content_text or self._ensure_subtitle(content)
|
||||
|
||||
if not content_text or len(content_text) < 50:
|
||||
print(f"[Pipeline] 警告: 字幕过短或为空,可能影响分析质量")
|
||||
|
||||
# Step 1: 轻量扫描
|
||||
print("\n📋 Step 1: 轻量扫描")
|
||||
content = self.light_scan(content, content_text)
|
||||
|
||||
# Step 2: 入库(保存到数据库/文件系统)
|
||||
print("\n💾 Step 2: 保存内容")
|
||||
self._save_content(content)
|
||||
|
||||
# Step 3: 深度分析
|
||||
print("\n🔍 Step 3: 深度分析")
|
||||
content = self.deep_analyze(content, content_text)
|
||||
|
||||
# Step 4: 生成笔记
|
||||
print("\n📝 Step 4: 生成Obsidian笔记")
|
||||
content = self._generate_note(content)
|
||||
|
||||
# Step 5: Git同步
|
||||
if self.config.git.enabled:
|
||||
print("\n🔄 Step 5: Git同步")
|
||||
self._sync_to_git(content)
|
||||
else:
|
||||
print("\n⏭️ Step 5: Git同步已禁用,跳过")
|
||||
|
||||
content.update_status(ContentStatus.GENERATED)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"[Pipeline] 处理完成: {content.id}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
return content
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ [Pipeline] 处理流程失败: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def light_scan(self, content: Content, content_text: str) -> Content:
|
||||
"""
|
||||
轻量扫描阶段
|
||||
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
content_text: 内容文本
|
||||
|
||||
|
||||
Returns:
|
||||
更新后的Content对象
|
||||
"""
|
||||
print(f"[Pipeline] 开始轻量扫描: {content.id}")
|
||||
|
||||
|
||||
try:
|
||||
content = self.light_scanner.scan(content, content_text)
|
||||
print(f"[Pipeline] 轻量扫描完成: 摘要长度={len(content.summary)}, 关键词={content.key_points}, 质量分={content.quality_metrics.overall_score:.2f}")
|
||||
@@ -185,20 +42,20 @@ class ProcessingPipeline:
|
||||
except Exception as e:
|
||||
print(f"[Pipeline] 轻量扫描失败: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def deep_analyze(self, content: Content, content_text: str) -> Content:
|
||||
"""
|
||||
深度分析阶段
|
||||
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
content_text: 内容文本
|
||||
|
||||
|
||||
Returns:
|
||||
更新后的Content对象
|
||||
"""
|
||||
print(f"[Pipeline] 开始深度分析: {content.id}")
|
||||
|
||||
|
||||
try:
|
||||
analysis_result = self.deep_analyzer.analyze(content, content_text)
|
||||
print(f"[Pipeline] 深度分析完成: 主题数={len(analysis_result.main_topics)}, 时间轴段数={len(analysis_result.timeline)}")
|
||||
@@ -207,50 +64,102 @@ class ProcessingPipeline:
|
||||
print(f"[Pipeline] 深度分析失败: {e}")
|
||||
content.update_status(ContentStatus.ERROR)
|
||||
raise
|
||||
|
||||
|
||||
def process_full(self, content: Content, content_text: str) -> Content:
|
||||
"""
|
||||
完整处理流程:
|
||||
轻量扫描 → 入库 → 深度分析 → 生成笔记 → Git同步
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
content_text: 内容文本
|
||||
|
||||
Returns:
|
||||
完全处理后的Content对象
|
||||
"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"[Pipeline] 开始完整处理流程: {content.id}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
try:
|
||||
# Step 1: 轻量扫描
|
||||
print("\n📋 Step 1: 轻量扫描")
|
||||
content = self.light_scan(content, content_text)
|
||||
|
||||
# Step 2: 入库(保存到数据库/文件系统)
|
||||
print("\n💾 Step 2: 保存内容")
|
||||
self._save_content(content)
|
||||
|
||||
# Step 3: 深度分析
|
||||
print("\n🔍 Step 3: 深度分析")
|
||||
content = self.deep_analyze(content, content_text)
|
||||
|
||||
# Step 4: 生成笔记
|
||||
print("\n📝 Step 4: 生成Obsidian笔记")
|
||||
content = self._generate_note(content)
|
||||
|
||||
# Step 5: Git同步
|
||||
if self.config.git.enabled:
|
||||
print("\n🔄 Step 5: Git同步")
|
||||
self._sync_to_git(content)
|
||||
else:
|
||||
print("\n⏭️ Step 5: Git同步已禁用,跳过")
|
||||
|
||||
content.update_status(ContentStatus.GENERATED)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"[Pipeline] 处理完成: {content.id}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
return content
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ [Pipeline] 处理流程失败: {e}")
|
||||
raise
|
||||
|
||||
def _save_content(self, content: Content) -> None:
|
||||
"""保存内容到存储"""
|
||||
# 创建存储目录
|
||||
storage_dir = os.path.expanduser("~/.obsidian-kb/storage")
|
||||
os.makedirs(storage_dir, exist_ok=True)
|
||||
|
||||
|
||||
# 保存JSON文件
|
||||
import json
|
||||
file_path = os.path.join(storage_dir, f"{content.id}.json")
|
||||
|
||||
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(content.to_dict(), f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
print(f"[Pipeline] 内容已保存: {file_path}")
|
||||
|
||||
|
||||
def _generate_note(self, content: Content) -> Content:
|
||||
"""生成Obsidian笔记"""
|
||||
# 构建笔记内容
|
||||
note_content = self._build_note_content(content)
|
||||
|
||||
|
||||
# 确定笔记路径
|
||||
vault_path = os.path.expanduser(self.config.obsidian.vault_path)
|
||||
notes_folder = self.config.obsidian.notes_folder
|
||||
|
||||
|
||||
note_dir = os.path.join(vault_path, notes_folder)
|
||||
os.makedirs(note_dir, exist_ok=True)
|
||||
|
||||
|
||||
# 生成文件名
|
||||
safe_title = self._sanitize_filename(content.title or content.id)
|
||||
timestamp = datetime.now().strftime("%Y%m%d")
|
||||
note_filename = f"{timestamp}_{safe_title}.md"
|
||||
note_path = os.path.join(note_dir, note_filename)
|
||||
|
||||
|
||||
# 写入文件
|
||||
with open(note_path, 'w', encoding='utf-8') as f:
|
||||
f.write(note_content)
|
||||
|
||||
|
||||
content.output_path = note_path
|
||||
content.update_status(ContentStatus.GENERATED)
|
||||
|
||||
|
||||
print(f"[Pipeline] 笔记已生成: {note_path}")
|
||||
return content
|
||||
|
||||
|
||||
def _build_note_content(self, content: Content) -> str:
|
||||
"""构建Obsidian笔记内容"""
|
||||
lines = [
|
||||
@@ -262,11 +171,11 @@ class ProcessingPipeline:
|
||||
f"- **作者**: {content.author or '未知'}",
|
||||
f"- **类型**: {content.content_type.value}",
|
||||
]
|
||||
|
||||
|
||||
if content.duration:
|
||||
duration_str = self._format_duration(content.duration)
|
||||
lines.append(f"- **时长**: {duration_str}")
|
||||
|
||||
|
||||
lines.extend([
|
||||
f"- **质量评分**: {content.quality_metrics.overall_score:.1f}/1.0",
|
||||
f"- **处理时间**: {datetime.now().strftime('%Y-%m-%d %H:%M')}",
|
||||
@@ -276,13 +185,13 @@ class ProcessingPipeline:
|
||||
"",
|
||||
"## 🏷️ 关键词",
|
||||
])
|
||||
|
||||
|
||||
# 添加关键词标签
|
||||
if content.key_points:
|
||||
keyword_tags = ' '.join([f"#{kw.replace(' ', '_')}" for kw in content.key_points[:5]])
|
||||
lines.append(keyword_tags)
|
||||
lines.append('')
|
||||
|
||||
|
||||
# 添加时间轴
|
||||
if content.timeline:
|
||||
lines.extend([
|
||||
@@ -290,12 +199,12 @@ class ProcessingPipeline:
|
||||
"## ⏱️ 时间轴",
|
||||
"",
|
||||
])
|
||||
|
||||
|
||||
for item in content.timeline:
|
||||
lines.append(f"### {item.timestamp} - {item.title}")
|
||||
lines.append(item.description)
|
||||
lines.append('')
|
||||
|
||||
|
||||
# 添加原始内容(折叠)
|
||||
lines.extend([
|
||||
"",
|
||||
@@ -308,38 +217,38 @@ class ProcessingPipeline:
|
||||
"---",
|
||||
f"*由 Obsidian KB 自动生成 - {datetime.now().strftime('%Y-%m-%d %H:%M')}*"
|
||||
])
|
||||
|
||||
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def _sync_to_git(self, content: Content) -> None:
|
||||
"""同步到Git仓库"""
|
||||
if not self.git_client:
|
||||
print("[Pipeline] Git客户端未配置,跳过同步")
|
||||
return
|
||||
|
||||
|
||||
try:
|
||||
vault_path = os.path.expanduser(self.config.obsidian.vault_path)
|
||||
|
||||
|
||||
# 添加文件
|
||||
relative_path = os.path.relpath(content.output_path, vault_path)
|
||||
self.git_client.add(relative_path)
|
||||
|
||||
|
||||
# 提交
|
||||
commit_msg = self.config.git.commit_message_template.format(
|
||||
title=content.title or content.id
|
||||
)
|
||||
self.git_client.commit(commit_msg)
|
||||
|
||||
|
||||
# 推送(如果启用)
|
||||
if self.config.git.auto_push:
|
||||
self.git_client.push()
|
||||
|
||||
|
||||
print(f"[Pipeline] Git同步完成")
|
||||
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Pipeline] Git同步失败: {e}")
|
||||
# Git同步失败不阻断主流程
|
||||
|
||||
|
||||
def _sanitize_filename(self, filename: str) -> str:
|
||||
"""清理文件名"""
|
||||
import re
|
||||
@@ -349,13 +258,13 @@ class ProcessingPipeline:
|
||||
if len(filename) > 50:
|
||||
filename = filename[:50]
|
||||
return filename.strip()
|
||||
|
||||
|
||||
def _format_duration(self, seconds: int) -> str:
|
||||
"""格式化时长"""
|
||||
hours = seconds // 3600
|
||||
minutes = (seconds % 3600) // 60
|
||||
secs = seconds % 60
|
||||
|
||||
|
||||
if hours > 0:
|
||||
return f"{hours}小时{minutes}分钟"
|
||||
elif minutes > 0:
|
||||
|
||||
@@ -1,493 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
字幕生成器 - 为无字幕视频自动生成 SRT 格式字幕
|
||||
|
||||
使用 OpenAI Whisper 进行语音识别,支持本地离线处理
|
||||
"""
|
||||
|
||||
import os
|
||||
import time
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Optional, Callable, Dict, Any
|
||||
|
||||
# 配置日志
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioDownloader:
|
||||
"""使用 yt-dlp 下载视频音频"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化音频下载器"""
|
||||
self.ydl_opts = {
|
||||
'format': 'bestaudio/best',
|
||||
'postprocessors': [{
|
||||
'key': 'FFmpegExtractAudio',
|
||||
'preferredcodec': 'mp3',
|
||||
'preferredquality': '192',
|
||||
}],
|
||||
'quiet': True,
|
||||
'no_warnings': True,
|
||||
}
|
||||
|
||||
def download_audio(self, url: str, output_path: str) -> str:
|
||||
"""
|
||||
下载视频音频
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
output_path: 输出音频路径(可以是模板,如 'cache/audio/%(id)s.mp3')
|
||||
|
||||
Returns:
|
||||
str: 实际保存的音频文件路径
|
||||
|
||||
Raises:
|
||||
Exception: 下载失败时抛出异常
|
||||
"""
|
||||
try:
|
||||
import yt_dlp
|
||||
|
||||
ydl_opts = self.ydl_opts.copy()
|
||||
ydl_opts['outtmpl'] = output_path
|
||||
|
||||
logger.info(f"开始下载音频: {url}")
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
ydl.download([url])
|
||||
|
||||
# 替换扩展名
|
||||
actual_path = output_path.replace('.%(ext)s', '.mp3')
|
||||
if not os.path.exists(actual_path):
|
||||
# 尝试其他可能的文件名
|
||||
actual_path = output_path.rsplit('.', 1)[0] + '.mp3'
|
||||
|
||||
if os.path.exists(actual_path):
|
||||
logger.info(f"音频下载成功: {actual_path}")
|
||||
return actual_path
|
||||
else:
|
||||
raise FileNotFoundError(f"下载的音频文件未找到: {actual_path}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"音频下载失败: {e}")
|
||||
raise
|
||||
|
||||
|
||||
class WhisperTranscriber:
|
||||
"""Whisper 语音识别器"""
|
||||
|
||||
# 模型选择
|
||||
MODELS = {
|
||||
'tiny': 39000000, # 最快,准确度最低 (~39M)
|
||||
'base': 74000000, # 推荐,速度/准确度平衡 (~74M)
|
||||
'small': 244000000, # 准确度更高 (~244M)
|
||||
'medium': 769000000, # 接近专业级 (~769M)
|
||||
'large': 1550000000, # 最高准确度 (~1.55GB)
|
||||
}
|
||||
|
||||
def __init__(self, model_size: str = 'base'):
|
||||
"""
|
||||
初始化识别器
|
||||
|
||||
Args:
|
||||
model_size: 模型大小,默认 'base'
|
||||
"""
|
||||
self.model_size = model_size
|
||||
self.model = None
|
||||
self.device = None
|
||||
|
||||
def load_model(self):
|
||||
"""加载 Whisper 模型(懒加载)"""
|
||||
if self.model is None:
|
||||
logger.info(f"正在加载 Whisper 模型 ({self.model_size})...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import whisper
|
||||
import torch
|
||||
|
||||
# 自动检测 GPU
|
||||
self.device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
logger.info(f"使用设备: {self.device}")
|
||||
|
||||
self.model = whisper.load_model(
|
||||
self.model_size,
|
||||
device=self.device
|
||||
)
|
||||
|
||||
load_time = time.time() - start_time
|
||||
logger.info(f"模型加载完成 - 耗时: {load_time:.1f}秒")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"模型加载失败: {e}")
|
||||
raise
|
||||
|
||||
def transcribe(
|
||||
self,
|
||||
audio_path: str,
|
||||
language: Optional[str] = None,
|
||||
progress_callback: Optional[Callable[[str, float], None]] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
语音识别
|
||||
|
||||
Args:
|
||||
audio_path: 音频文件路径
|
||||
language: 语言代码('zh'、'en'、None=自动检测)
|
||||
progress_callback: 进度回调函数 callback(message, progress_percent)
|
||||
|
||||
Returns:
|
||||
dict: Whisper 识别结果,包含 'text' 和 'segments'
|
||||
|
||||
Raises:
|
||||
Exception: 识别失败时抛出异常
|
||||
"""
|
||||
self.load_model()
|
||||
|
||||
logger.info(f"开始语音识别: {audio_path}")
|
||||
if language:
|
||||
logger.info(f"指定语言: {language}")
|
||||
else:
|
||||
logger.info("语言: 自动检测")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = self.model.transcribe(
|
||||
audio_path,
|
||||
language=language,
|
||||
task='transcribe',
|
||||
verbose=False,
|
||||
fp16=False # CPU 上不使用 FP16
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
logger.info(f"识别完成 - 耗时: {duration:.1f}秒")
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"语音识别失败: {e}")
|
||||
raise
|
||||
|
||||
def get_segments(self, result: Dict[str, Any]) -> list:
|
||||
"""
|
||||
获取分段信息
|
||||
|
||||
Args:
|
||||
result: Whisper 识别结果
|
||||
|
||||
Returns:
|
||||
list: segments 列表
|
||||
"""
|
||||
return result.get('segments', [])
|
||||
|
||||
|
||||
class SubtitleFormatter:
|
||||
"""字幕格式化器(SRT 格式)"""
|
||||
|
||||
@staticmethod
|
||||
def to_srt(segments: list) -> str:
|
||||
"""
|
||||
将 Whisper segments 转换为 SRT 格式
|
||||
|
||||
Args:
|
||||
segments: Whisper segments 列表
|
||||
|
||||
Returns:
|
||||
str: SRT 格式文本
|
||||
"""
|
||||
srt_lines = []
|
||||
|
||||
for i, segment in enumerate(segments, 1):
|
||||
# SRT 格式:
|
||||
# 序号
|
||||
# 开始时间 --> 结束时间
|
||||
# 文本
|
||||
# 空行
|
||||
|
||||
start_time = SubtitleFormatter._seconds_to_srt_time(segment['start'])
|
||||
end_time = SubtitleFormatter._seconds_to_srt_time(segment['end'])
|
||||
text = segment['text'].strip()
|
||||
|
||||
srt_lines.append(str(i))
|
||||
srt_lines.append(f"{start_time} --> {end_time}")
|
||||
srt_lines.append(text)
|
||||
srt_lines.append("")
|
||||
|
||||
return "\n".join(srt_lines)
|
||||
|
||||
@staticmethod
|
||||
def _seconds_to_srt_time(seconds: float) -> str:
|
||||
"""
|
||||
将秒数转换为 SRT 时间格式 (HH:MM:SS,mmm)
|
||||
|
||||
Args:
|
||||
seconds: 秒数
|
||||
|
||||
Returns:
|
||||
str: SRT 时间格式
|
||||
"""
|
||||
hours = int(seconds // 3600)
|
||||
minutes = int((seconds % 3600) // 60)
|
||||
secs = int(seconds % 60)
|
||||
millis = int((seconds - int(seconds)) * 1000)
|
||||
|
||||
return f"{hours:02d}:{minutes:02d}:{secs:02d},{millis:03d}"
|
||||
|
||||
|
||||
class SubtitleCache:
|
||||
"""字幕缓存管理器"""
|
||||
|
||||
def __init__(self, cache_dir: str = 'cache/subtitles'):
|
||||
"""
|
||||
初始化缓存
|
||||
|
||||
Args:
|
||||
cache_dir: 缓存目录
|
||||
"""
|
||||
self.cache_dir = Path(cache_dir)
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def get_cache_key(self, url: str) -> str:
|
||||
"""
|
||||
生成缓存键
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
|
||||
Returns:
|
||||
str: 缓存键(SHA256 哈希的前8位)
|
||||
"""
|
||||
return hashlib.sha256(url.encode()).hexdigest()[:8]
|
||||
|
||||
def get_cache_path(self, url: str) -> str:
|
||||
"""
|
||||
获取缓存文件路径
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
|
||||
Returns:
|
||||
str: 缓存文件路径
|
||||
"""
|
||||
cache_key = self.get_cache_key(url)
|
||||
return str(self.cache_dir / f"{cache_key}.srt")
|
||||
|
||||
def get(self, url: str) -> Optional[str]:
|
||||
"""
|
||||
获取缓存字幕
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
|
||||
Returns:
|
||||
str | None: 缓存的字幕内容,如果不存在返回 None
|
||||
"""
|
||||
cache_path = self.get_cache_path(url)
|
||||
|
||||
if os.path.exists(cache_path):
|
||||
logger.info(f"使用缓存字幕: {cache_path}")
|
||||
try:
|
||||
with open(cache_path, 'r', encoding='utf-8') as f:
|
||||
return f.read()
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
def set(self, url: str, subtitle: str) -> str:
|
||||
"""
|
||||
保存缓存字幕
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
subtitle: 字幕内容
|
||||
|
||||
Returns:
|
||||
str: 缓存文件路径
|
||||
"""
|
||||
cache_path = self.get_cache_path(url)
|
||||
|
||||
try:
|
||||
with open(cache_path, 'w', encoding='utf-8') as f:
|
||||
f.write(subtitle)
|
||||
logger.info(f"字幕已缓存: {cache_path}")
|
||||
return cache_path
|
||||
except Exception as e:
|
||||
logger.error(f"保存缓存失败: {e}")
|
||||
raise
|
||||
|
||||
def exists(self, url: str) -> bool:
|
||||
"""检查缓存是否存在"""
|
||||
return os.path.exists(self.get_cache_path(url))
|
||||
|
||||
def clear_cache(self):
|
||||
"""清空所有缓存"""
|
||||
try:
|
||||
for file in self.cache_dir.glob('*.srt'):
|
||||
file.unlink()
|
||||
logger.info("缓存已清空")
|
||||
except Exception as e:
|
||||
logger.error(f"清空缓存失败: {e}")
|
||||
|
||||
|
||||
class SubtitleGenerator:
|
||||
"""字幕生成器统一接口"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_size: str = 'base',
|
||||
cache_dir: str = 'cache/subtitles',
|
||||
audio_cache_dir: str = 'cache/audio'
|
||||
):
|
||||
"""
|
||||
初始化生成器
|
||||
|
||||
Args:
|
||||
model_size: Whisper 模型大小
|
||||
cache_dir: 字幕缓存目录
|
||||
audio_cache_dir: 音频缓存目录
|
||||
"""
|
||||
self.audio_downloader = AudioDownloader()
|
||||
self.transcriber = WhisperTranscriber(model_size)
|
||||
self.formatter = SubtitleFormatter()
|
||||
self.cache = SubtitleCache(cache_dir)
|
||||
self.audio_cache_dir = Path(audio_cache_dir)
|
||||
self.audio_cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def generate(
|
||||
self,
|
||||
url: str,
|
||||
content_id: str = None,
|
||||
language: Optional[str] = None,
|
||||
use_cache: bool = True,
|
||||
keep_audio: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
生成字幕
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
content_id: 内容ID(用于音频文件命名)
|
||||
language: 语言代码(None=自动检测)
|
||||
use_cache: 是否使用缓存
|
||||
keep_audio: 是否保留音频文件
|
||||
|
||||
Returns:
|
||||
str: SRT 格式字幕
|
||||
|
||||
Raises:
|
||||
Exception: 生成失败时抛出异常
|
||||
"""
|
||||
# 检查缓存
|
||||
if use_cache and self.cache.exists(url):
|
||||
logger.info(f"使用缓存字幕: {url}")
|
||||
cached_subtitle = self.cache.get(url)
|
||||
if cached_subtitle:
|
||||
return cached_subtitle
|
||||
|
||||
logger.info(f"开始生成字幕: {url}")
|
||||
audio_path = None
|
||||
|
||||
try:
|
||||
# 生成 content_id
|
||||
if content_id is None:
|
||||
content_id = self.cache.get_cache_key(url)
|
||||
|
||||
# 下载音频
|
||||
audio_path = str(self.audio_cache_dir / f"{content_id}.mp3")
|
||||
logger.info(f"下载音频到: {audio_path}")
|
||||
audio_path = self.audio_downloader.download_audio(url, audio_path)
|
||||
|
||||
# 语音识别
|
||||
result = self.transcriber.transcribe(audio_path, language=language)
|
||||
segments = self.transcriber.get_segments(result)
|
||||
|
||||
if not segments:
|
||||
logger.warning("未识别到任何语音内容")
|
||||
return ""
|
||||
|
||||
# 转换为 SRT
|
||||
srt_text = self.formatter.to_srt(segments)
|
||||
logger.info(f"SRT 字幕生成成功,共 {len(segments)} 段")
|
||||
|
||||
# 保存缓存
|
||||
self.cache.set(url, srt_text)
|
||||
|
||||
# 清理音频文件
|
||||
if not keep_audio and os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
logger.info(f"已删除音频文件: {audio_path}")
|
||||
|
||||
return srt_text
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"字幕生成失败: {e}")
|
||||
# 清理失败的音频文件
|
||||
if audio_path and os.path.exists(audio_path) and not keep_audio:
|
||||
try:
|
||||
os.remove(audio_path)
|
||||
except:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def generate_subtitle(
|
||||
url: str,
|
||||
model_size: str = 'base',
|
||||
language: Optional[str] = None,
|
||||
cache_dir: str = 'cache/subtitles',
|
||||
audio_cache_dir: str = 'cache/audio'
|
||||
) -> str:
|
||||
"""
|
||||
便捷函数:生成字幕
|
||||
|
||||
Args:
|
||||
url: 视频URL
|
||||
model_size: Whisper 模型大小
|
||||
language: 语言代码(None=自动检测)
|
||||
cache_dir: 字幕缓存目录
|
||||
audio_cache_dir: 音频缓存目录
|
||||
|
||||
Returns:
|
||||
str: SRT 格式字幕
|
||||
"""
|
||||
generator = SubtitleGenerator(
|
||||
model_size=model_size,
|
||||
cache_dir=cache_dir,
|
||||
audio_cache_dir=audio_cache_dir
|
||||
)
|
||||
return generator.generate(url, language=language)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# 测试示例
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("用法: python subtitle_generator.py <视频URL> [语言]")
|
||||
print("示例: python subtitle_generator.py https://www.bilibili.com/video/BV1xx411c7mD zh")
|
||||
sys.exit(1)
|
||||
|
||||
url = sys.argv[1]
|
||||
language = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
|
||||
print("=" * 60)
|
||||
print("字幕生成器")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
subtitle = generate_subtitle(url, language=language)
|
||||
print("\n" + "=" * 60)
|
||||
print("生成成功!字幕预览(前500字符):")
|
||||
print("=" * 60)
|
||||
print(subtitle[:500])
|
||||
if len(subtitle) > 500:
|
||||
print("\n...")
|
||||
print(f"\n总字符数: {len(subtitle)}")
|
||||
except Exception as e:
|
||||
print(f"\n错误: {e}")
|
||||
sys.exit(1)
|
||||
@@ -1,184 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for short link resolver functionality.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add path for imports
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from shortlink_resolver import ShortLinkResolver, detect_and_format
|
||||
|
||||
|
||||
def test_single_bilibili_short():
|
||||
"""Test single Bilibili short link detection."""
|
||||
print("Test 1: Single Bilibili short link")
|
||||
print("=" * 50)
|
||||
|
||||
text = "Check this video: https://b23.tv/oLBZZfK"
|
||||
has_links, message = detect_and_format(text)
|
||||
|
||||
print(f"Input: {text}")
|
||||
print(f"Has short links: {has_links}")
|
||||
if message:
|
||||
print(f"Message:\n{message}")
|
||||
print()
|
||||
|
||||
assert has_links, "Should detect short link"
|
||||
assert message, "Should have help message"
|
||||
print("✅ Test 1 passed\n")
|
||||
|
||||
|
||||
def test_multiple_short_links():
|
||||
"""Test multiple short links detection."""
|
||||
print("Test 2: Multiple short links")
|
||||
print("=" * 50)
|
||||
|
||||
text = "Check these: https://b23.tv/abc123 and https://bit.ly/test"
|
||||
has_links, message = detect_and_format(text)
|
||||
|
||||
print(f"Input: {text}")
|
||||
print(f"Has short links: {has_links}")
|
||||
if message:
|
||||
print(f"Message:\n{message}")
|
||||
print()
|
||||
|
||||
assert has_links, "Should detect short links"
|
||||
assert message, "Should have warning message"
|
||||
assert "[1]" in message, "Should show first link"
|
||||
assert "[2]" in message, "Should show second link"
|
||||
print("✅ Test 2 passed\n")
|
||||
|
||||
|
||||
def test_no_short_links():
|
||||
"""Test text with no short links."""
|
||||
print("Test 3: No short links")
|
||||
print("=" * 50)
|
||||
|
||||
text = "Here is a full URL: https://www.bilibili.com/video/BV1xx411c7mD"
|
||||
has_links, message = detect_and_format(text)
|
||||
|
||||
print(f"Input: {text}")
|
||||
print(f"Has short links: {has_links}")
|
||||
if message:
|
||||
print(f"Message: {message}")
|
||||
else:
|
||||
print("Message: None")
|
||||
print()
|
||||
|
||||
assert not has_links, "Should not detect short links"
|
||||
assert not message, "Should not have help message"
|
||||
print("✅ Test 3 passed\n")
|
||||
|
||||
|
||||
def test_bv_code():
|
||||
"""Test BV code (should not be treated as short link)."""
|
||||
print("Test 4: BV code")
|
||||
print("=" * 50)
|
||||
|
||||
text = "BV1jcFqz7EFW"
|
||||
has_links, message = detect_and_format(text)
|
||||
|
||||
print(f"Input: {text}")
|
||||
print(f"Has short links: {has_links}")
|
||||
if message:
|
||||
print(f"Message: {message}")
|
||||
else:
|
||||
print("Message: None")
|
||||
print()
|
||||
|
||||
assert not has_links, "BV code should not be treated as short link"
|
||||
assert not message, "Should not have help message"
|
||||
print("✅ Test 4 passed\n")
|
||||
|
||||
|
||||
def test_mixed_content():
|
||||
"""Test mixed content with short and full links."""
|
||||
print("Test 5: Mixed content")
|
||||
print("=" * 50)
|
||||
|
||||
text = """
|
||||
Watch these videos:
|
||||
https://www.bilibili.com/video/BV1xx411c7mD
|
||||
https://b23.tv/short123
|
||||
BV2yy522c8nE
|
||||
"""
|
||||
|
||||
has_links, message = detect_and_format(text)
|
||||
|
||||
print(f"Input:{text}")
|
||||
print(f"Has short links: {has_links}")
|
||||
if message:
|
||||
print(f"Message:\n{message}")
|
||||
print()
|
||||
|
||||
assert has_links, "Should detect short link"
|
||||
assert message, "Should have help message"
|
||||
print("✅ Test 5 passed\n")
|
||||
|
||||
|
||||
def test_resolver_direct():
|
||||
"""Test ShortLinkResolver directly."""
|
||||
print("Test 6: Direct resolver usage")
|
||||
print("=" * 50)
|
||||
|
||||
resolver = ShortLinkResolver()
|
||||
|
||||
# Test detection
|
||||
short_links = resolver.detect_short_links("Check https://b23.tv/test123")
|
||||
print(f"Detected: {short_links}")
|
||||
|
||||
assert len(short_links) == 1, "Should detect one short link"
|
||||
assert short_links[0]['platform'] == 'bilibili', "Should be Bilibili"
|
||||
assert short_links[0]['is_short'] == True, "Should be short link"
|
||||
|
||||
# Test format_short_link_help
|
||||
help_msg = resolver.format_short_link_help("https://b23.tv/test123")
|
||||
print(f"Help message:\n{help_msg}")
|
||||
|
||||
assert "短链接" in help_msg, "Should mention short link"
|
||||
assert "BV" in help_msg, "Should mention BV code"
|
||||
|
||||
# Test format_short_link_warning
|
||||
warning_msg = resolver.format_short_link_warning(short_links)
|
||||
print(f"Warning message:\n{warning_msg}")
|
||||
|
||||
assert "[1]" in warning_msg, "Should show numbered list"
|
||||
|
||||
print()
|
||||
print("✅ Test 6 passed\n")
|
||||
|
||||
|
||||
def run_all_tests():
|
||||
"""Run all tests."""
|
||||
print("\n" + "=" * 50)
|
||||
print("Running Short Link Resolver Tests")
|
||||
print("=" * 50 + "\n")
|
||||
|
||||
try:
|
||||
test_single_bilibili_short()
|
||||
test_multiple_short_links()
|
||||
test_no_short_links()
|
||||
test_bv_code()
|
||||
test_mixed_content()
|
||||
test_resolver_direct()
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("✅ All tests passed!")
|
||||
print("=" * 50 + "\n")
|
||||
return 0
|
||||
|
||||
except AssertionError as e:
|
||||
print(f"\n❌ Test failed: {e}\n")
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"\n❌ Unexpected error: {e}\n")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(run_all_tests())
|
||||
@@ -1,245 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
字幕生成器测试脚本
|
||||
|
||||
测试各个模块的功能
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# 添加项目路径
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
|
||||
def test_audio_downloader():
|
||||
"""测试音频下载器"""
|
||||
print("\n" + "="*60)
|
||||
print("测试 1: 音频下载器")
|
||||
print("="*60)
|
||||
|
||||
from subtitle_generator import AudioDownloader
|
||||
|
||||
# 使用一个短测试视频
|
||||
test_url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Rick Astley - Never Gonna Give You Up
|
||||
|
||||
downloader = AudioDownloader()
|
||||
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
output_path = os.path.join(tmpdir, "test_audio.%(ext)s")
|
||||
result_path = downloader.download_audio(test_url, output_path)
|
||||
|
||||
if os.path.exists(result_path):
|
||||
size_mb = os.path.getsize(result_path) / (1024 * 1024)
|
||||
print(f"✅ 音频下载成功")
|
||||
print(f" 文件: {result_path}")
|
||||
print(f" 大小: {size_mb:.2f} MB")
|
||||
return True
|
||||
else:
|
||||
print(f"❌ 音频文件未找到: {result_path}")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ 音频下载失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def test_whisper_transcriber():
|
||||
"""测试 Whisper 识别器"""
|
||||
print("\n" + "="*60)
|
||||
print("测试 2: Whisper 识别器")
|
||||
print("="*60)
|
||||
|
||||
from subtitle_generator import WhisperTranscriber
|
||||
|
||||
# 创建测试音频(使用 ffmpeg 生成静音)
|
||||
test_audio = "/tmp/test_silent.mp3"
|
||||
os.system(f"ffmpeg -f lavfi -i anullsrc=r=44100:cl=mono -t 3 -q:a 9 -y {test_audio} 2>/dev/null")
|
||||
|
||||
try:
|
||||
transcriber = WhisperTranscriber(model_size='tiny') # 使用最小的模型
|
||||
result = transcriber.transcribe(test_audio)
|
||||
|
||||
print(f"✅ Whisper 识别完成")
|
||||
print(f" 识别文本: {result['text'][:50]}...")
|
||||
print(f" 分段数: {len(result.get('segments', []))}")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Whisper 识别失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def test_subtitle_formatter():
|
||||
"""测试 SRT 格式化器"""
|
||||
print("\n" + "="*60)
|
||||
print("测试 3: SRT 格式化器")
|
||||
print("="*60)
|
||||
|
||||
from subtitle_generator import SubtitleFormatter
|
||||
|
||||
# 测试数据
|
||||
test_segments = [
|
||||
{'start': 0.0, 'end': 2.5, 'text': '这是一段测试字幕'},
|
||||
{'start': 3.0, 'end': 5.7, 'text': '这是第二段字幕'},
|
||||
{'start': 6.0, 'end': 8.2, 'text': 'This is the third subtitle'},
|
||||
]
|
||||
|
||||
try:
|
||||
srt_text = SubtitleFormatter.to_srt(test_segments)
|
||||
|
||||
print(f"✅ SRT 格式化成功")
|
||||
print(f" 生成内容预览:")
|
||||
print(" " + "-" * 40)
|
||||
for line in srt_text.split('\n')[:10]:
|
||||
print(f" {line}")
|
||||
print(" " + "-" * 40)
|
||||
|
||||
# 验证格式
|
||||
assert '00:00:00,000 --> 00:00:02,500' in srt_text
|
||||
assert '这是一段测试字幕' in srt_text
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ SRT 格式化失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def test_subtitle_cache():
|
||||
"""测试缓存管理器"""
|
||||
print("\n" + "="*60)
|
||||
print("测试 4: 缓存管理器")
|
||||
print("="*60)
|
||||
|
||||
from subtitle_generator import SubtitleCache
|
||||
|
||||
test_url = "https://example.com/video/123"
|
||||
test_subtitle = """1
|
||||
00:00:00,000 --> 00:00:02,500
|
||||
测试字幕内容
|
||||
"""
|
||||
|
||||
try:
|
||||
cache = SubtitleCache(cache_dir='/tmp/test_subtitle_cache')
|
||||
|
||||
# 测试写入
|
||||
cache_path = cache.set(test_url, test_subtitle)
|
||||
print(f"✅ 缓存写入成功: {cache_path}")
|
||||
|
||||
# 测试读取
|
||||
cached_content = cache.get(test_url)
|
||||
assert cached_content == test_subtitle
|
||||
print(f"✅ 缓存读取成功")
|
||||
|
||||
# 测试存在性
|
||||
assert cache.exists(test_url)
|
||||
print(f"✅ 缓存存在性检查成功")
|
||||
|
||||
# 测试缓存键
|
||||
cache_key = cache.get_cache_key(test_url)
|
||||
print(f" 缓存键: {cache_key}")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ 缓存测试失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def test_full_workflow():
|
||||
"""测试完整工作流程"""
|
||||
print("\n" + "="*60)
|
||||
print("测试 5: 完整工作流程")
|
||||
print("="*60)
|
||||
|
||||
from subtitle_generator import SubtitleGenerator
|
||||
|
||||
test_url = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
|
||||
|
||||
try:
|
||||
generator = SubtitleGenerator(model_size='tiny')
|
||||
|
||||
print(f"开始生成字幕...")
|
||||
srt_text = generator.generate(
|
||||
url=test_url,
|
||||
content_id='test001',
|
||||
language='en',
|
||||
use_cache=False
|
||||
)
|
||||
|
||||
print(f"✅ 字幕生成完成")
|
||||
print(f" 总字符数: {len(srt_text)}")
|
||||
print(f" 预览(前300字符):")
|
||||
print(" " + "-" * 40)
|
||||
print(" " + srt_text[:300])
|
||||
print(" " + "-" * 40)
|
||||
|
||||
# 验证缓存
|
||||
print(f"\n验证缓存...")
|
||||
cached = generator.generate(
|
||||
url=test_url,
|
||||
content_id='test002',
|
||||
use_cache=True
|
||||
)
|
||||
assert cached == srt_text
|
||||
print(f"✅ 缓存验证成功")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ 完整工作流程失败: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""运行所有测试"""
|
||||
print("\n" + "="*60)
|
||||
print("字幕生成器测试套件")
|
||||
print("="*60)
|
||||
|
||||
results = {}
|
||||
|
||||
# 运行各个测试
|
||||
results['audio_downloader'] = test_audio_downloader()
|
||||
results['whisper_transcriber'] = test_whisper_transcriber()
|
||||
results['subtitle_formatter'] = test_subtitle_formatter()
|
||||
results['subtitle_cache'] = test_subtitle_cache()
|
||||
results['full_workflow'] = test_full_workflow()
|
||||
|
||||
# 输出汇总
|
||||
print("\n" + "="*60)
|
||||
print("测试结果汇总")
|
||||
print("="*60)
|
||||
|
||||
for test_name, passed in results.items():
|
||||
status = "✅ 通过" if passed else "❌ 失败"
|
||||
print(f"{test_name:25s} {status}")
|
||||
|
||||
total = len(results)
|
||||
passed = sum(results.values())
|
||||
|
||||
print("\n" + "="*60)
|
||||
print(f"总计: {passed}/{total} 测试通过")
|
||||
print("="*60)
|
||||
|
||||
return all(results.values())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
Reference in New Issue
Block a user