自动同步: 2026-03-15 02:10
This commit is contained in:
+141
-639
@@ -5,698 +5,200 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
|
||||
|
||||
# KB Workflow
|
||||
|
||||
## 触发条件
|
||||
## 什么时候使用
|
||||
|
||||
当用户发送包含以下模式的消息时,此技能自动触发:
|
||||
|
||||
| 触发词 | 行为 | 示例 |
|
||||
|--------|------|------|
|
||||
| `添加到知识库` + 链接 | 添加到知识库并处理 | "添加到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
| `保存到知识库` + 链接 | 同上 | "保存到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
|
||||
---
|
||||
## 环境变量
|
||||
|
||||
## 状态机定义
|
||||
### 执行前必须检查
|
||||
|
||||
工作流是一个严格的状态机,每个状态有明确的前置条件和后置条件:
|
||||
在执行 workflow 之前,必须检查环境变量是否已经设置。
|
||||
|
||||
| 状态 | 前置条件 | 执行动作 | 成功后进入 | 失败处理 |
|
||||
|------|---------|---------|-----------|---------|
|
||||
| `init` | 用户触发 + URL 有效 | 创建 state.json,检查环境变量 | `link_parse` | 报错退出 |
|
||||
| `link_parse` | state.json 存在 | 解析链接,提取平台和ID | `metadata` | 记录错误,最多重试1次 |
|
||||
| `metadata` | platform, content_id 已知 | 提取元数据 | `content` | 记录错误,最多重试1次 |
|
||||
| `content` | 元数据文件存在 | 下载内容/字幕 | `analyze` 或 `whisper` | 根据错误类型处理 |
|
||||
| `whisper` | 视频已下载,无官方字幕 | 语音识别转文字 | `analyze` | 尝试 OCR 降级 |
|
||||
| `analyze` | 有文本内容(字幕/识别结果/文章) | AI 深度分析 | `archive` | **流程失败,报错退出** |
|
||||
| `archive` | 分析报告存在 | 归档 + Git 提交 | `done` | 记录错误,可手动处理 |
|
||||
| `done` | - | 汇报完成 | - | - |
|
||||
|
||||
### 状态判断流程图
|
||||
|
||||
```
|
||||
用户发送链接
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 0: 环境变量检查 │
|
||||
│ 缺少必需变量 → 报错退出 │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 1: 链接解析 (link_parse) │
|
||||
│ 输出: platform, content_type, id │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 2: 元数据提取 (metadata) │
|
||||
│ 输出: title, author, etc. │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ content_type 判断 │
|
||||
├─────────────────────────────────────┤
|
||||
│ video/zvideo → Step 3a (视频处理) │
|
||||
│ article/zhuanlan/question → Step 3b │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 3a: 视频处理 │
|
||||
│ 有官方字幕? → 是 → Step 5 │
|
||||
│ → 否 → Step 4 (Whisper) │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 3b: 文章处理 │
|
||||
│ 提取文字 → 直接进入 Step 5 │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 4: Whisper 语音识别 (可选) │
|
||||
│ 无字幕时执行,生成 text/json/srt │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 5: 深度分析 (analyze) │
|
||||
│ ⚠️ 必须成功,否则流程失败 │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Step 6: 归档 (archive) │
|
||||
│ 复制到归档目录 + Git 提交 │
|
||||
└─────────────────────────────────────┘
|
||||
↓
|
||||
完成
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 环境变量清单
|
||||
|
||||
### 必需变量(缺失则报错退出)
|
||||
|
||||
| 变量名 | 用途 | 示例 |
|
||||
|--------|------|------|
|
||||
| `AI_API_KEY` | AI API 密钥 | `your-api-key` |
|
||||
| `AI_PROVIDER` | AI 提供商 | `zhipu` / `openai` / `minimax` |
|
||||
| `AI_MODEL` | 模型名称 | `glm-5` / `gpt-4` |
|
||||
| `AI_BASE_URL` | API 端点 | `https://open.bigmodel.cn/api/coding/paas/v4` |
|
||||
|
||||
### 可选变量
|
||||
|
||||
| 变量名 | 用途 | 用途说明 |
|
||||
|--------|------|---------|
|
||||
| `BILIBILI_COOKIE` | B站登录态 | 下载会员视频 |
|
||||
| `ZHIHU_COOKIE` | 知乎登录态 | 访问专栏等 |
|
||||
| `HF_TOKEN` | HuggingFace Token | 说话者分离功能 |
|
||||
| `KB_ARCHIVE_PATH` | 归档目录 | 默认 `~/work/zq/30.资源/AI分析` |
|
||||
| `KB_GIT_ROOT` | Git 仓库根目录 | 默认 `~/work/zq` |
|
||||
| `KB_GIT_USER` | Git 提交用户名 | 归档时使用 |
|
||||
| `KB_GIT_EMAIL` | Git 提交邮箱 | 归档时使用 |
|
||||
|
||||
### 配置方法
|
||||
**必须的环境变量**(缺失则报错退出):
|
||||
|
||||
配置方式:将变量写入 `~/.kb_env` 文件:
|
||||
```bash
|
||||
# 创建专用配置文件
|
||||
cat > ~/.kb_env << 'EOF'
|
||||
export AI_API_KEY="your-api-key"
|
||||
export AI_PROVIDER="zhipu"
|
||||
export AI_MODEL="glm-5"
|
||||
export AI_BASE_URL="https://open.bigmodel.cn/api/coding/paas/v4"
|
||||
export BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx"
|
||||
export KB_ARCHIVE_PATH="$HOME/work/zq/30.资源/AI分析"
|
||||
export KB_GIT_ROOT="$HOME/work/zq"
|
||||
EOF
|
||||
|
||||
# 在 ~/.bashrc 中添加
|
||||
[ -f ~/.kb_env ] && source ~/.kb_env
|
||||
```
|
||||
|
||||
---
|
||||
**可选的环境变量**:
|
||||
| 变量名 | 用途 | 示例 |
|
||||
|--------|------|------|
|
||||
| `BILIBILI_COOKIE` | 下载会员视频 | `SESSDATA=xxx; bili_jct=xxx` |
|
||||
| `KB_GIT_USER` | Git提交用户名 | `Your Name` |
|
||||
| `KB_GIT_EMAIL` | Git提交邮箱 | `your@email.com` |
|
||||
| `KB_ARCHIVE_PATH` | 归档目录路径 | `~/work/zq/30.资源/AI分析` |
|
||||
| `KB_GIT_ROOT` | Git仓库根目录 | `~/work/zq` |
|
||||
| `HF_TOKEN` | HuggingFace Token(说话者分离) | `hf_xxx` |
|
||||
|
||||
## 步骤详解
|
||||
### 检查命令
|
||||
|
||||
### Step 0: 环境变量检查
|
||||
|
||||
**前置条件:** 无
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 加载环境变量
|
||||
if [ -f ~/.kb_env ]; then source ~/.kb_env; fi
|
||||
# 从 ~/.kb_env 加载环境变量
|
||||
if [ -f ~/.kb_env ]; then
|
||||
source ~/.kb_env
|
||||
fi
|
||||
|
||||
# 检查必须的环境变量
|
||||
if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ] || [ -z "$AI_BASE_URL" ]; then
|
||||
echo "❌ 错误:必须的环境变量未设置"
|
||||
echo "请配置 ~/.kb_env 文件"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查必需变量
|
||||
for var in AI_API_KEY AI_PROVIDER AI_MODEL AI_BASE_URL; do
|
||||
if [ -z "${!var}" ]; then
|
||||
echo "❌ 错误:$var 未设置"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "✅ 环境变量检查通过"
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 所有必需变量已设置
|
||||
- 输出 "✅ 环境变量检查通过"
|
||||
## Workflow 执行方式
|
||||
|
||||
**失败处理:**
|
||||
- 缺少任何必需变量 → 报错退出,提示用户配置
|
||||
### 方式一:一键处理(推荐)
|
||||
|
||||
**输出示例:**
|
||||
```
|
||||
✅ 环境变量检查通过
|
||||
AI_PROVIDER: zhipu
|
||||
AI_MODEL: glm-5
|
||||
```
|
||||
使用管道 CLI 一键完成所有步骤:
|
||||
|
||||
**错误示例:**
|
||||
```
|
||||
❌ 错误:AI_API_KEY 未设置
|
||||
请配置以下环境变量:
|
||||
- AI_API_KEY
|
||||
- AI_PROVIDER
|
||||
- AI_MODEL
|
||||
- AI_BASE_URL
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 1: 链接解析
|
||||
|
||||
**前置条件:**
|
||||
- 环境变量检查通过
|
||||
- URL 格式有效
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 获取目录路径
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py --pending
|
||||
|
||||
# 解析链接
|
||||
python3 ~/.openclaw/skills/kb-link-parser/link-parser.py --url "[URL]"
|
||||
python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- 包含字段:`platform`, `content_type`, `content_id`
|
||||
- 生成文件:`pending/{platform}_{id}.link.json`
|
||||
**该命令自动执行:**
|
||||
1. URL 解析 → 识别平台和内容 ID
|
||||
2. 元数据提取 → 获取标题、作者等信息
|
||||
3. 内容下载 → 字幕/视频/文章内容
|
||||
4. 语音识别 → Whisper/FunASR(如需要)
|
||||
5. 深度分析 → AI 生成分析报告
|
||||
6. 归档 → 移动到归档目录
|
||||
|
||||
**失败处理:**
|
||||
- 不支持的平台 → 报错退出,告知用户支持的平台
|
||||
- URL 无效 → 报错退出,请用户提供正确链接
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"platform": "bilibili",
|
||||
"content_type": "video",
|
||||
"content_id": "BV1xx411c7mD",
|
||||
"output_file": "pending/bilibili_BV1xx411c7mD.link.json"
|
||||
}
|
||||
```
|
||||
|
||||
**错误示例:**
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "不支持的平台: example.com",
|
||||
"supported_platforms": ["bilibili.com", "zhihu.com", "youtube.com"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 2: 元数据提取
|
||||
|
||||
**前置条件:**
|
||||
- Step 1 完成
|
||||
- `platform`, `content_type`, `content_id` 已知
|
||||
|
||||
**执行命令:**
|
||||
**命令参数:**
|
||||
```bash
|
||||
# 获取目录路径
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py --metadata
|
||||
# 指定输出目录
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." -o /path/to/output
|
||||
|
||||
# 根据平台选择命令
|
||||
# B站视频
|
||||
python3 ~/.openclaw/skills/kb-bilibili-metadata/bilibili_metadata.py \
|
||||
--id "[BV号]" --type video \
|
||||
--output "metadata/bilibili_[BV号].meta.json"
|
||||
|
||||
# B站专栏
|
||||
python3 ~/.openclaw/skills/kb-bilibili-metadata/bilibili_metadata.py \
|
||||
--id "[cv号]" --type article \
|
||||
--output "metadata/bilibili_cv[cv号].meta.json"
|
||||
|
||||
# 知乎问答
|
||||
python3 ~/.openclaw/skills/kb-zhihu-metadata/zhihu_metadata.py \
|
||||
--id "[问题ID]" --type question \
|
||||
--output "metadata/zhihu_[ID].meta.json"
|
||||
|
||||
# 知乎视频
|
||||
python3 ~/.openclaw/skills/kb-zhihu-metadata/zhihu_metadata.py \
|
||||
--id "[视频ID]" --type zvideo \
|
||||
--output "metadata/zhihu_[ID].meta.json"
|
||||
# 强制重新执行(忽略已完成状态)
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." --no-skip
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- 包含字段:`platform`, `content_id`, `title`
|
||||
- 生成文件:`metadata/{platform}_{id}.meta.json`
|
||||
### 方式二:手动分步执行(高级)
|
||||
|
||||
**失败处理:**
|
||||
- 网络错误 → 重试1次
|
||||
- 内容不存在 → 报错退出
|
||||
- 需要登录 → 提示配置 Cookie
|
||||
如需更细粒度控制,可手动调用各子技能:
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"platform": "bilibili",
|
||||
"content_id": "BV1xx411c7mD",
|
||||
"content_type": "video",
|
||||
"title": "视频标题",
|
||||
"author": "UP主名字",
|
||||
"duration": "10:30",
|
||||
"output_file": "metadata/bilibili_BV1xx411c7mD.meta.json"
|
||||
}
|
||||
```
|
||||
|
||||
**错误示例:**
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "视频不存在或已被删除",
|
||||
"content_id": "BV1xx411c7mD"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3a: 视频内容下载
|
||||
|
||||
**前置条件:**
|
||||
- Step 2 完成
|
||||
- `content_type` 为 `video` 或 `zvideo`
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 获取目录路径
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py --content_downloads
|
||||
# 1. 查重
|
||||
python3 kb-workflow/kb_workflow.py --check --url "https://..."
|
||||
|
||||
# B站视频 - 优先尝试字幕
|
||||
python3 ~/.openclaw/skills/kb-bilibili-content/bilibili_content.py \
|
||||
--id "[BV号]" --type subtitle-download \
|
||||
--output "content/downloads/"
|
||||
# 2. 一键处理
|
||||
python3 kb-workflow/kb_workflow.py process "https://..."
|
||||
|
||||
# 检查是否有官方字幕
|
||||
# 如果 has_subtitle=true → 进入 Step 5
|
||||
# 如果 has_subtitle=false → 下载视频,进入 Step 4
|
||||
# 3. 归档
|
||||
python3 kb-workflow/kb_workflow.py --archive-file "/path/to/report.md"
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- `has_subtitle=true`: 有字幕文件
|
||||
- `has_subtitle=false`: 视频文件下载成功
|
||||
## 支持的平台和内容类型
|
||||
|
||||
**失败处理:**
|
||||
- 403 禁止访问 → 检查 BILIBILI_COOKIE 配置
|
||||
- 网络超时 → 重试1次
|
||||
- 会员视频 → 提示需要配置 Cookie
|
||||
| 平台 | 内容类型 | URL 示例 |
|
||||
|------|----------|----------|
|
||||
| B站 | 视频 | `bilibili.com/video/BVxxx` |
|
||||
| B站 | 专栏 | `bilibili.com/read/cvxxx` |
|
||||
| 知乎 | 问答 | `zhihu.com/question/xxx` |
|
||||
| 知乎 | 专栏 | `zhuanlan.zhihu.com/p/xxx` |
|
||||
| 知乎 | 视频 | `zhihu.com/zvideo/xxx` |
|
||||
|
||||
## 处理流程
|
||||
|
||||
**输出示例(有字幕):**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"type": "subtitle",
|
||||
"has_subtitle": true,
|
||||
"subtitle_file": "content/downloads/bilibili_BV1xx411c7mD.srt",
|
||||
"message": "官方字幕下载成功"
|
||||
}
|
||||
```
|
||||
|
||||
**输出示例(无字幕):**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"type": "subtitle",
|
||||
"has_subtitle": false,
|
||||
"message": "无官方字幕,需要下载视频进行语音识别"
|
||||
}
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ kb-workflow process │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 1. Parse → LinkParser 解析 URL │
|
||||
│ 2. Download → 平台适配器获取元数据和内容 │
|
||||
│ 3. ASR → 语音识别(视频无字幕时) │
|
||||
│ 4. Analyze → DeepAnalyzer 生成分析报告 │
|
||||
│ 5. Archive → 归档到 KB_ARCHIVE_PATH │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**错误示例:**
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "403 Forbidden - 可能需要会员权限",
|
||||
"suggestion": "请配置 BILIBILI_COOKIE 环境变量"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3b: 文章内容提取
|
||||
|
||||
**前置条件:**
|
||||
- Step 2 完成
|
||||
- `content_type` 为 `article`、`zhuanlan` 或 `question`
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# B站专栏
|
||||
python3 ~/.openclaw/skills/kb-bilibili-content/bilibili_content.py \
|
||||
--id "[cv号]" --type article \
|
||||
--output "content/"
|
||||
|
||||
# 知乎专栏
|
||||
python3 ~/.openclaw/skills/kb-zhihu-content/zhihu_content.py \
|
||||
--type article-from-media \
|
||||
--output "content/"
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- 包含字段:`filepath`, `content_length`
|
||||
- 跳过 Step 4,直接进入 Step 5
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"type": "article",
|
||||
"filepath": "content/bilibili_cv43780186.txt",
|
||||
"content_length": 5234,
|
||||
"message": "文章内容提取成功"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Whisper 语音识别
|
||||
|
||||
**前置条件:**
|
||||
- Step 3a 完成
|
||||
- `has_subtitle=false`
|
||||
- 视频文件已下载
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 获取目录路径
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py --content
|
||||
|
||||
# 普通识别
|
||||
python3 ~/.openclaw/skills/kb-whisper/whisper.py \
|
||||
--input "content/downloads/bilibili_BV1xx411c7mD.mp4" \
|
||||
--model base \
|
||||
--output "content/"
|
||||
|
||||
# 带说话者分离(需要 HF_TOKEN)
|
||||
if [ -n "$HF_TOKEN" ]; then
|
||||
python3 ~/.openclaw/skills/kb-whisper-diarization/whisper_diarization.py \
|
||||
--input "content/downloads/bilibili_BV1xx411c7mD.mp4" \
|
||||
--model base --diarize \
|
||||
--output "content/"
|
||||
fi
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- 生成文件:`.json`, `.txt`, `.srt`
|
||||
|
||||
**失败处理:**
|
||||
- 模型未下载 → 提示下载模型
|
||||
- 内存不足 → 尝试更小的模型
|
||||
- Whisper 失败 → 尝试 OCR 降级
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"input_file": "content/downloads/bilibili_BV1xx411c7mD.mp4",
|
||||
"model": "base",
|
||||
"output_files": {
|
||||
"json": "content/bilibili_BV1xx411c7mD.json",
|
||||
"txt": "content/bilibili_BV1xx411c7mD.txt",
|
||||
"srt": "content/bilibili_BV1xx411c7mD.srt"
|
||||
},
|
||||
"duration_seconds": 630
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 5: 深度分析
|
||||
|
||||
**前置条件:**
|
||||
- 有文本内容(字幕/识别结果/文章)
|
||||
- 元数据文件存在
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 获取目录路径
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py --analyzed
|
||||
|
||||
# 分析视频
|
||||
AI_API_KEY="$AI_API_KEY" AI_PROVIDER="zhipu" AI_MODEL="glm-5" \
|
||||
python3 -u ~/.openclaw/skills/kb-deep-analyzer/deep_analyzer.py \
|
||||
--metadata "metadata/bilibili_BV1xx411c7mD.meta.json" \
|
||||
--content "content/bilibili_BV1xx411c7mD.json" \
|
||||
--output "analyzed/bilibili_BV1xx411c7mD.md" \
|
||||
--type video \
|
||||
--provider zhipu --model glm-5
|
||||
|
||||
# 分析文章
|
||||
AI_API_KEY="$AI_API_KEY" AI_PROVIDER="zhipu" AI_MODEL="glm-5" \
|
||||
python3 -u ~/.openclaw/skills/kb-deep-analyzer/deep_analyzer.py \
|
||||
--metadata "metadata/xxx.meta.json" \
|
||||
--content "content/xxx.txt" \
|
||||
--output "analyzed/xxx.md" \
|
||||
--type article \
|
||||
--provider zhipu --model glm-5
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 返回 JSON 包含 `success: true`
|
||||
- 生成分析报告文件
|
||||
|
||||
**失败处理:**
|
||||
- ⚠️ **深度分析失败 = 整个流程失败**
|
||||
- API 调用失败 → 记录错误,**终止流程**
|
||||
- **禁止**使用预设模板或降级内容
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"output_file": "analyzed/bilibili_BV1xx411c7mD.md",
|
||||
"model": "glm-5",
|
||||
"provider": "zhipu"
|
||||
}
|
||||
```
|
||||
|
||||
**错误示例:**
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "AI API 调用失败: rate limit exceeded",
|
||||
"message": "深度分析失败,整个流程终止"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 6: 归档
|
||||
|
||||
**前置条件:**
|
||||
- Step 5 完成
|
||||
- 分析报告文件存在
|
||||
|
||||
**执行命令:**
|
||||
```bash
|
||||
# 归档
|
||||
python3 ~/.openclaw/skills/kb-workflow/kb_workflow.py \
|
||||
--archive-file "analyzed/bilibili_BV1xx411c7mD.md"
|
||||
|
||||
# 如果配置了 Git,会自动提交
|
||||
```
|
||||
|
||||
**成功标准:**
|
||||
- 文件复制到归档目录
|
||||
- Git 提交成功(如果配置)
|
||||
|
||||
**失败处理:**
|
||||
- 归档失败 → 记录错误,文件仍可用
|
||||
- Git 失败 → 不影响主流程
|
||||
|
||||
**输出示例:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"archive_path": "~/work/zq/30.资源/AI分析/2026-03-13/bilibili_BV1xx411c7mD.md",
|
||||
"git_commit": "abc1234"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 目录结构
|
||||
|
||||
**工作目录(默认 ~/work/knowledge-base/):**
|
||||
```
|
||||
~/work/knowledge-base/
|
||||
├── pending/ # 待处理链接
|
||||
├── metadata/ # 元数据 JSON
|
||||
├── content/ # 内容文件
|
||||
│ └── downloads/ # 下载的视频/字幕
|
||||
└── analyzed/ # 分析报告
|
||||
```
|
||||
~/work/knowledge-base/ # 知识库根目录
|
||||
├── pending/ # 待处理链接
|
||||
│ └── {platform}_{id}.link.json
|
||||
├── metadata/ # 元数据
|
||||
│ └── {platform}_{id}.meta.json
|
||||
├── content/ # 内容
|
||||
│ ├── downloads/ # 视频下载
|
||||
│ │ └── {platform}_{id}.mp4
|
||||
│ ├── {platform}_{id}.txt # 文本内容
|
||||
│ └── {platform}_{id}.json # Whisper 结果
|
||||
└── analyzed/ # 深度分析报告
|
||||
└── {platform}_{id}.md
|
||||
|
||||
~/work/zq/30.资源/AI分析/ # 归档目录
|
||||
**归档目录(默认 ~/work/zq/30.资源/AI分析/):**
|
||||
```
|
||||
~/work/zq/30.资源/AI分析/
|
||||
└── YYYY-MM-DD/
|
||||
└── {platform}_{id}.md
|
||||
└── {platform}_{id}.md # 深度分析报告
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 子技能清单
|
||||
|
||||
| 步骤 | 子技能 | 命令 |
|
||||
|------|--------|------|
|
||||
| 1 | kb-link-parser | `--url "[URL]"` |
|
||||
| 2 | kb-bilibili-metadata | `--id "[ID]" --type video --output "[PATH]"` |
|
||||
| 2 | kb-zhihu-metadata | `--id "[ID]" --type question --output "[PATH]"` |
|
||||
| 3a | kb-bilibili-content | `--id "[ID]" --type subtitle-download --output "[PATH]"` |
|
||||
| 3a | kb-bilibili-content | `--id "[ID]" --type video --output "[PATH]"` |
|
||||
| 3b | kb-bilibili-content | `--id "[ID]" --type article --output "[PATH]"` |
|
||||
| 3b | kb-zhihu-content | `--type article-from-media --output "[PATH]"` |
|
||||
| 4 | kb-whisper | `--input "[VIDEO]" --model base --output "[PATH]"` |
|
||||
| 4 | kb-whisper-diarization | `--input "[VIDEO]" --model base --diarize --output "[PATH]"` |
|
||||
| 5 | kb-deep-analyzer | `--metadata "[META]" --content "[CONTENT]" --output "[OUT]" --type video` |
|
||||
|
||||
---
|
||||
|
||||
## 端到端示例
|
||||
|
||||
### 示例 1: B站视频(有官方字幕)
|
||||
|
||||
**输入:** `添加到知识库 https://www.bilibili.com/video/BV1xx411c7mD`
|
||||
|
||||
```
|
||||
Step 0: ✅ 环境变量检查通过
|
||||
Step 1: ✅ 链接解析 → bilibili, video, BV1xx411c7mD
|
||||
Step 2: ✅ 元数据提取 → 标题: "xxx", 作者: "xxx"
|
||||
Step 3a: ✅ 字幕下载 → has_subtitle=true, bilibili_BV1xx411c7mD.srt
|
||||
Step 4: ⏭️ 跳过(有官方字幕)
|
||||
Step 5: ✅ 深度分析 → bilibili_BV1xx411c7mD.md
|
||||
Step 6: ✅ 归档 → ~/work/zq/30.资源/AI分析/2026-03-13/
|
||||
```
|
||||
|
||||
### 示例 2: B站视频(无官方字幕)
|
||||
|
||||
**输入:** `添加到知识库 https://www.bilibili.com/video/BV1yy411c7mE`
|
||||
|
||||
```
|
||||
Step 0: ✅ 环境变量检查通过
|
||||
Step 1: ✅ 链接解析 → bilibili, video, BV1yy411c7mE
|
||||
Step 2: ✅ 元数据提取 → 标题: "xxx", 时长: "15:30"
|
||||
Step 3a: ⚠️ 无官方字幕 → 下载视频
|
||||
Step 3a: ✅ 视频下载 → bilibili_BV1yy411c7mE.mp4
|
||||
Step 4: ✅ Whisper 识别 → bilibili_BV1yy411c7mE.json/txt/srt
|
||||
Step 5: ✅ 深度分析 → bilibili_BV1yy411c7mE.md
|
||||
Step 6: ✅ 归档
|
||||
```
|
||||
|
||||
### 示例 3: 403 下载失败
|
||||
|
||||
**输入:** `添加到知识库 https://www.bilibili.com/video/BV1zz411c7mF`
|
||||
|
||||
```
|
||||
Step 0: ✅ 环境变量检查通过
|
||||
Step 1: ✅ 链接解析
|
||||
Step 2: ✅ 元数据提取
|
||||
Step 3a: ❌ 403 Forbidden
|
||||
→ 可能需要会员权限
|
||||
→ 请配置 BILIBILI_COOKIE 环境变量
|
||||
→ 流程暂停,等待用户处理
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见错误处理
|
||||
|
||||
### 错误 1: 环境变量未设置
|
||||
|
||||
**症状:** `❌ 错误:AI_API_KEY 未设置`
|
||||
|
||||
**解决:**
|
||||
```bash
|
||||
echo 'export AI_API_KEY="your-key"' >> ~/.kb_env
|
||||
source ~/.kb_env
|
||||
```
|
||||
|
||||
### 错误 2: 403 下载失败
|
||||
|
||||
**症状:** B站视频下载返回 403
|
||||
|
||||
**解决:**
|
||||
1. 获取 B站 Cookie(F12 → Network → Copy Cookie)
|
||||
2. 配置环境变量:
|
||||
```bash
|
||||
echo 'export BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx"' >> ~/.kb_env
|
||||
source ~/.kb_env
|
||||
```
|
||||
|
||||
### 错误 3: 深度分析失败
|
||||
|
||||
**症状:** `success: false` from kb-deep-analyzer
|
||||
|
||||
**处理:**
|
||||
- **整个流程终止**
|
||||
- 检查 AI_API_KEY 是否有效
|
||||
- 检查 AI_BASE_URL 是否正确
|
||||
- 检查模型配额是否用尽
|
||||
|
||||
### 错误 4: Whisper 模型未下载
|
||||
|
||||
**症状:** `Model file not found`
|
||||
|
||||
**解决:**
|
||||
```bash
|
||||
cd ~/whisper.cpp
|
||||
./models/download-ggml-model.sh base
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 状态管理(断点续传)
|
||||
|
||||
> ⚠️ 此功能需要完成 P3 实现
|
||||
|
||||
```bash
|
||||
# 初始化状态
|
||||
python3 kb_workflow.py --init-state "[URL]"
|
||||
|
||||
# 查询当前状态
|
||||
python3 kb_workflow.py --get-state "[URL]"
|
||||
|
||||
# 推进到下一步
|
||||
python3 kb_workflow.py --advance-step "[URL]" "metadata"
|
||||
|
||||
# 记录错误
|
||||
python3 kb_workflow.py --record-error "[URL]" "403 Forbidden"
|
||||
|
||||
# 获取恢复指令
|
||||
python3 kb_workflow.py --resume "[URL]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 严格失败处理原则
|
||||
|
||||
1. **深度分析不可跳过**:kb-deep-analyzer 是必须成功的步骤
|
||||
2. **失败即终止**:如果深度分析返回 `success: false`,必须立即停止
|
||||
3. **禁止降级处理**:不能使用预设模板或简单摘要"假装成功"
|
||||
4. **每次执行都要使用 AI**:必须有真实的 AI API 调用
|
||||
> **深度分析步骤必须成功,否则整个流程失败**
|
||||
|
||||
1. **深度分析不可跳过**:AI 分析是必须成功执行的步骤
|
||||
2. **失败即终止**:如果分析返回失败,必须立即停止并汇报
|
||||
3. **禁止降级处理**:不能使用预设模板或简单摘要来"假装成功"
|
||||
4. **每次执行都要使用 AI**:必须有真实的 AI API 调用记录
|
||||
|
||||
## 辅助命令
|
||||
|
||||
```bash
|
||||
# 查重
|
||||
python3 kb-workflow/kb_workflow.py --check --url "https://..."
|
||||
|
||||
# 获取目录路径
|
||||
python3 kb-workflow/kb_workflow.py --base-path
|
||||
python3 kb-workflow/kb_workflow.py --metadata
|
||||
python3 kb-workflow/kb_workflow.py --content
|
||||
python3 kb-workflow/kb_workflow.py --analyzed
|
||||
|
||||
# 归档分析报告
|
||||
python3 kb-workflow/kb_workflow.py --archive-file "/path/to/report.md"
|
||||
```
|
||||
|
||||
## 示例
|
||||
|
||||
```
|
||||
用户: 添加到知识库 https://www.bilibili.com/video/BV1xx
|
||||
|
||||
Claude:
|
||||
✅ 环境变量检查通过
|
||||
|
||||
◆ 处理中...
|
||||
→ 解析 URL: bilibili 视频 BV1xx
|
||||
→ 获取元数据...
|
||||
→ 下载字幕...
|
||||
→ 深度分析...
|
||||
|
||||
✅ 完成!
|
||||
|
||||
标题: 【教程】如何使用 Python 进行数据分析
|
||||
处理阶段: 全部完成
|
||||
最终文件: ~/work/zq/30.资源/AI分析/2026-03-15/bilibili_BV1xx.md
|
||||
```
|
||||
|
||||
## 技术架构
|
||||
|
||||
```
|
||||
kb-workflow/kb_workflow.py (CLI)
|
||||
└── kb_pipeline/ (核心管道包)
|
||||
├── pipeline.py — Pipeline 编排引擎
|
||||
├── processors/ — 处理器实现
|
||||
│ ├── parse.py — URL 解析
|
||||
│ ├── download.py — 内容下载
|
||||
│ ├── asr.py — 语音识别
|
||||
│ └── analyze.py — 深度分析
|
||||
└── schemas.py — 数据模型
|
||||
```
|
||||
|
||||
+128
-354
@@ -5,7 +5,7 @@ KB Workflow - Knowledge Base Workflow Orchestrator
|
||||
|
||||
知识库工作流辅助工具函数。
|
||||
|
||||
此模块提供查重、文件操作、Git 管理、状态管理等工具函数,
|
||||
此模块提供查重、文件操作、Git 管理等工具函数,
|
||||
供主代理在执行 SKILL.md 中定义的 workflow 时调用。
|
||||
"""
|
||||
|
||||
@@ -18,46 +18,7 @@ import shutil
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Any, Tuple
|
||||
|
||||
|
||||
# ==================== 状态机定义 ====================
|
||||
|
||||
# 工作流步骤顺序
|
||||
WORKFLOW_STEPS = [
|
||||
"init",
|
||||
"link_parse",
|
||||
"metadata",
|
||||
"content",
|
||||
"whisper", # 可选步骤
|
||||
"analyze",
|
||||
"archive",
|
||||
"done"
|
||||
]
|
||||
|
||||
# 每个步骤的必需输出字段
|
||||
STEP_OUTPUT_FIELDS = {
|
||||
"init": [],
|
||||
"link_parse": ["platform", "content_type", "content_id"],
|
||||
"metadata": ["title"],
|
||||
"content": ["filepath"],
|
||||
"whisper": ["output_files"],
|
||||
"analyze": ["output_file"],
|
||||
"archive": ["archive_path"],
|
||||
"done": []
|
||||
}
|
||||
|
||||
# 步骤描述(用于汇报)
|
||||
STEP_DESCRIPTIONS = {
|
||||
"init": "初始化",
|
||||
"link_parse": "链接解析",
|
||||
"metadata": "元数据提取",
|
||||
"content": "内容下载",
|
||||
"whisper": "语音识别",
|
||||
"analyze": "深度分析",
|
||||
"archive": "归档",
|
||||
"done": "完成"
|
||||
}
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
|
||||
# ==================== 子技能映射 ====================
|
||||
@@ -332,249 +293,6 @@ def archive_and_git(
|
||||
return result
|
||||
|
||||
|
||||
# ==================== 状态管理 ====================
|
||||
|
||||
def get_state_file_path(url: str, base_path: Path = None) -> Path:
|
||||
"""获取状态文件路径"""
|
||||
if base_path is None:
|
||||
base_path = get_knowledge_base_path()
|
||||
pending_dir = base_path / "pending"
|
||||
pending_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 从 URL 提取标识符
|
||||
# 尝试提取 BV 号或其他 ID
|
||||
bv_match = re.search(r'BV[a-zA-Z0-9]+', url)
|
||||
if bv_match:
|
||||
content_id = bv_match.group()
|
||||
platform = "bilibili"
|
||||
else:
|
||||
# 尝试提取知乎 ID
|
||||
zhihu_match = re.search(r'zhihu\.com/(?:question|zvideo|p)/(\d+)', url)
|
||||
if zhihu_match:
|
||||
content_id = zhihu_match.group(1)
|
||||
platform = "zhihu"
|
||||
else:
|
||||
# 使用 URL hash 作为标识
|
||||
import hashlib
|
||||
content_id = hashlib.md5(url.encode()).hexdigest()[:12]
|
||||
platform = "unknown"
|
||||
|
||||
return pending_dir / f"{platform}_{content_id}.state.json"
|
||||
|
||||
|
||||
def init_state(url: str, base_path: Path = None) -> Dict[str, Any]:
|
||||
"""初始化工作流状态"""
|
||||
state_file = get_state_file_path(url, base_path)
|
||||
|
||||
# 如果状态文件已存在,返回现有状态
|
||||
if state_file.exists():
|
||||
with open(state_file, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
|
||||
# 创建新状态
|
||||
now = datetime.now().isoformat()
|
||||
state = {
|
||||
"url": url,
|
||||
"platform": None,
|
||||
"content_id": None,
|
||||
"content_type": None,
|
||||
"current_step": "init",
|
||||
"completed_steps": [],
|
||||
"step_results": {},
|
||||
"errors": [],
|
||||
"created_at": now,
|
||||
"updated_at": now
|
||||
}
|
||||
|
||||
with open(state_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(state, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return state
|
||||
|
||||
|
||||
def get_state(url: str, base_path: Path = None) -> Dict[str, Any]:
|
||||
"""获取当前工作流状态"""
|
||||
state_file = get_state_file_path(url, base_path)
|
||||
|
||||
if not state_file.exists():
|
||||
return {
|
||||
"success": False,
|
||||
"error": "状态文件不存在",
|
||||
"url": url,
|
||||
"state_file": str(state_file)
|
||||
}
|
||||
|
||||
with open(state_file, 'r', encoding='utf-8') as f:
|
||||
state = json.load(f)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"state": state,
|
||||
"state_file": str(state_file)
|
||||
}
|
||||
|
||||
|
||||
def advance_step(
|
||||
url: str,
|
||||
step_name: str,
|
||||
result: Dict[str, Any] = None,
|
||||
base_path: Path = None
|
||||
) -> Dict[str, Any]:
|
||||
"""推进到下一步并记录结果"""
|
||||
state_file = get_state_file_path(url, base_path)
|
||||
|
||||
if not state_file.exists():
|
||||
return {
|
||||
"success": False,
|
||||
"error": "状态文件不存在,请先执行 --init-state"
|
||||
}
|
||||
|
||||
with open(state_file, 'r', encoding='utf-8') as f:
|
||||
state = json.load(f)
|
||||
|
||||
# 验证步骤名称
|
||||
if step_name not in WORKFLOW_STEPS:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"无效的步骤名称: {step_name}",
|
||||
"valid_steps": WORKFLOW_STEPS
|
||||
}
|
||||
|
||||
# 获取当前步骤索引
|
||||
try:
|
||||
current_idx = WORKFLOW_STEPS.index(state["current_step"])
|
||||
new_idx = WORKFLOW_STEPS.index(step_name)
|
||||
except ValueError:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"步骤索引错误"
|
||||
}
|
||||
|
||||
# 更新状态
|
||||
if step_name != state["current_step"]:
|
||||
# 标记当前步骤为完成
|
||||
if state["current_step"] not in state["completed_steps"]:
|
||||
state["completed_steps"].append(state["current_step"])
|
||||
|
||||
state["current_step"] = step_name
|
||||
state["updated_at"] = datetime.now().isoformat()
|
||||
|
||||
# 记录步骤结果
|
||||
if result:
|
||||
state["step_results"][step_name] = result
|
||||
|
||||
# 提取平台和 ID 信息(从 link_parse 步骤)
|
||||
if step_name == "link_parse" and result.get("success"):
|
||||
state["platform"] = result.get("platform")
|
||||
state["content_id"] = result.get("content_id")
|
||||
state["content_type"] = result.get("type")
|
||||
|
||||
# 保存状态
|
||||
with open(state_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(state, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"current_step": step_name,
|
||||
"completed_steps": state["completed_steps"],
|
||||
"state_file": str(state_file)
|
||||
}
|
||||
|
||||
|
||||
def record_error(
|
||||
url: str,
|
||||
error_msg: str,
|
||||
step: str = None,
|
||||
base_path: Path = None
|
||||
) -> Dict[str, Any]:
|
||||
"""记录错误"""
|
||||
state_file = get_state_file_path(url, base_path)
|
||||
|
||||
if not state_file.exists():
|
||||
return {
|
||||
"success": False,
|
||||
"error": "状态文件不存在,请先执行 --init-state"
|
||||
}
|
||||
|
||||
with open(state_file, 'r', encoding='utf-8') as f:
|
||||
state = json.load(f)
|
||||
|
||||
# 记录错误
|
||||
error_entry = {
|
||||
"step": step or state["current_step"],
|
||||
"error": error_msg,
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
state["errors"].append(error_entry)
|
||||
state["updated_at"] = datetime.now().isoformat()
|
||||
|
||||
# 保存状态
|
||||
with open(state_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(state, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"error_recorded": error_entry,
|
||||
"total_errors": len(state["errors"])
|
||||
}
|
||||
|
||||
|
||||
def get_resume_info(url: str, base_path: Path = None) -> Dict[str, Any]:
|
||||
"""获取恢复执行的指令"""
|
||||
state_file = get_state_file_path(url, base_path)
|
||||
|
||||
if not state_file.exists():
|
||||
return {
|
||||
"success": False,
|
||||
"error": "状态文件不存在",
|
||||
"suggestion": "请使用 --init-state 初始化工作流"
|
||||
}
|
||||
|
||||
with open(state_file, 'r', encoding='utf-8') as f:
|
||||
state = json.load(f)
|
||||
|
||||
current_step = state["current_step"]
|
||||
step_desc = STEP_DESCRIPTIONS.get(current_step, current_step)
|
||||
|
||||
# 检查是否有错误
|
||||
if state["errors"]:
|
||||
last_error = state["errors"][-1]
|
||||
return {
|
||||
"success": True,
|
||||
"status": "error",
|
||||
"current_step": current_step,
|
||||
"step_description": step_desc,
|
||||
"last_error": last_error,
|
||||
"completed_steps": state["completed_steps"],
|
||||
"suggestion": f"请先解决 {last_error['step']} 步骤的错误: {last_error['error']}"
|
||||
}
|
||||
|
||||
# 检查是否已完成
|
||||
if current_step == "done":
|
||||
return {
|
||||
"success": True,
|
||||
"status": "completed",
|
||||
"current_step": "done",
|
||||
"step_description": "已完成",
|
||||
"completed_steps": state["completed_steps"],
|
||||
"step_results": state["step_results"]
|
||||
}
|
||||
|
||||
# 返回下一步指令
|
||||
return {
|
||||
"success": True,
|
||||
"status": "in_progress",
|
||||
"current_step": current_step,
|
||||
"step_description": step_desc,
|
||||
"completed_steps": state["completed_steps"],
|
||||
"next_action": f"继续执行步骤: {step_desc}",
|
||||
"platform": state.get("platform"),
|
||||
"content_id": state.get("content_id"),
|
||||
"content_type": state.get("content_type"),
|
||||
"step_results": state.get("step_results", {})
|
||||
}
|
||||
|
||||
|
||||
def generate_summary(url: str, stages: List[Dict], final_file: str, archive_result: Dict = None) -> str:
|
||||
"""生成处理完成汇报"""
|
||||
# 提取视频标题(如果有)
|
||||
@@ -607,6 +325,83 @@ def generate_summary(url: str, stages: List[Dict], final_file: str, archive_resu
|
||||
return summary
|
||||
|
||||
|
||||
# ==================== CLI Process Command ====================
|
||||
|
||||
def process_command(
|
||||
url: str,
|
||||
output_dir: Optional[str] = None,
|
||||
skip_completed: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute full pipeline for a URL
|
||||
|
||||
Args:
|
||||
url: URL to process
|
||||
output_dir: Optional output directory for content
|
||||
skip_completed: Skip already completed steps (default True)
|
||||
|
||||
Returns:
|
||||
Dictionary with success status, content_dir, and results
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
# Add project root to path for kb_pipeline imports
|
||||
project_root = Path(__file__).parent.parent
|
||||
if str(project_root) not in sys.path:
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
try:
|
||||
from kb_pipeline.pipeline import Pipeline
|
||||
from kb_pipeline.processors import ParseProcessor, DownloadProcessor, ASRProcessor, AnalyzeProcessor
|
||||
from kb_pipeline.schemas import PipelineConfig
|
||||
|
||||
async def run_pipeline():
|
||||
# Determine output directory
|
||||
out_dir = Path(output_dir) if output_dir else Path.cwd()
|
||||
|
||||
# Create processors
|
||||
processors = [
|
||||
ParseProcessor(),
|
||||
DownloadProcessor(),
|
||||
ASRProcessor(),
|
||||
AnalyzeProcessor(),
|
||||
]
|
||||
|
||||
# Create pipeline config
|
||||
config = PipelineConfig(
|
||||
steps=["parse", "download", "asr", "analyze"],
|
||||
output_dir=out_dir,
|
||||
skip_completed=skip_completed,
|
||||
)
|
||||
|
||||
# Create and run pipeline
|
||||
pipeline = Pipeline(
|
||||
processors=processors,
|
||||
config=config,
|
||||
output_dir=out_dir,
|
||||
)
|
||||
|
||||
result = await pipeline.run(url)
|
||||
return result
|
||||
|
||||
# Run async pipeline
|
||||
result = asyncio.run(run_pipeline())
|
||||
|
||||
return {
|
||||
"success": result.success,
|
||||
"error": result.error_message,
|
||||
"content_dir": str(result.content_dir) if result.content_dir else None,
|
||||
"results": result.results,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"content_dir": None,
|
||||
"results": {},
|
||||
}
|
||||
|
||||
|
||||
# ==================== CLI ====================
|
||||
|
||||
def main():
|
||||
@@ -618,117 +413,96 @@ def main():
|
||||
使用示例:
|
||||
# 查重
|
||||
python kb_workflow.py --check --url "https://www.bilibili.com/video/BVxxx"
|
||||
|
||||
|
||||
# 归档
|
||||
python kb_workflow.py --archive-file "/path/to/analyzed.md"
|
||||
|
||||
# 状态管理
|
||||
python kb_workflow.py --init-state --url "https://..."
|
||||
python kb_workflow.py --get-state --url "https://..."
|
||||
python kb_workflow.py --advance-step --url "https://..." --step "metadata" --result '{"success":true}'
|
||||
python kb_workflow.py --record-error --url "https://..." --error-msg "403 Forbidden"
|
||||
python kb_workflow.py --resume --url "https://..."
|
||||
python kb_workflow.py --archive "/path/to/analyzed.md"
|
||||
|
||||
# 处理URL (通过完整管道)
|
||||
python kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
"""
|
||||
)
|
||||
|
||||
# 基础参数
|
||||
parser.add_argument('--url', type=str, help='URL 地址')
|
||||
parser.add_argument('--base-path', type=str, help='知识库根目录')
|
||||
|
||||
# 查重和归档
|
||||
|
||||
# Create subparsers for commands
|
||||
subparsers = parser.add_subparsers(dest='command', help='Available commands')
|
||||
|
||||
# Process subcommand
|
||||
process_parser = subparsers.add_parser('process', help='Process URL through full pipeline')
|
||||
process_parser.add_argument('url', type=str, help='URL to process')
|
||||
process_parser.add_argument('--output-dir', '-o', type=str, help='Output directory for content')
|
||||
process_parser.add_argument('--skip-completed', action='store_true', default=True,
|
||||
help='Skip completed steps (default: True)')
|
||||
process_parser.add_argument('--no-skip', action='store_false', dest='skip_completed',
|
||||
help='Re-run all steps')
|
||||
|
||||
# Legacy arguments (for backward compatibility)
|
||||
parser.add_argument('--check', action='store_true', help='查重')
|
||||
parser.add_argument('--url', type=str, help='要检查的URL')
|
||||
parser.add_argument('--archive-file', type=str, help='归档的文件路径')
|
||||
|
||||
# 目录路径查询
|
||||
parser.add_argument('--pending', action='store_true', help='获取 URL 解析目录')
|
||||
parser.add_argument('--base-path', type=str, help='获取知识库根目录')
|
||||
|
||||
parser.add_argument('--pending', action='store_true', help='获取url解析目录')
|
||||
parser.add_argument('--metadata', action='store_true', help='获取元数据目录')
|
||||
parser.add_argument('--content', action='store_true', help='获取内容目录')
|
||||
parser.add_argument('--content_downloads', action='store_true', help='获取内容下载目录')
|
||||
parser.add_argument('--content_downloads', action='store_true', help='获取主内容下载目录')
|
||||
parser.add_argument('--analyzed', action='store_true', help='获取深度分析目录')
|
||||
parser.add_argument('--archive_dir', action='store_true', help='获取归档目录')
|
||||
parser.add_argument('--archive_subscriptions', action='store_true', help='获取订阅信息目录')
|
||||
parser.add_argument('--extract_platform', type=str, help='从文件名提取平台名称')
|
||||
|
||||
# 状态管理
|
||||
parser.add_argument('--init-state', action='store_true', help='初始化工作流状态')
|
||||
parser.add_argument('--get-state', action='store_true', help='获取当前状态')
|
||||
parser.add_argument('--advance-step', action='store_true', help='推进到下一步')
|
||||
parser.add_argument('--step', type=str, help='步骤名称')
|
||||
parser.add_argument('--result', type=str, help='步骤结果(JSON 字符串)')
|
||||
parser.add_argument('--record-error', action='store_true', help='记录错误')
|
||||
parser.add_argument('--error-msg', type=str, help='错误信息')
|
||||
parser.add_argument('--resume', action='store_true', help='获取恢复执行信息')
|
||||
|
||||
parser.add_argument('--extract_platform', type=str, help='要提取平台名称的文件名')
|
||||
args = parser.parse_args()
|
||||
|
||||
|
||||
base_path = Path(args.base_path) if args.base_path else None
|
||||
dirs = ensure_directories(base_path)
|
||||
|
||||
# 查重
|
||||
|
||||
# Handle process subcommand
|
||||
if args.command == 'process':
|
||||
result = process_command(
|
||||
url=args.url,
|
||||
output_dir=args.output_dir,
|
||||
skip_completed=args.skip_completed
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
# Legacy argument handling
|
||||
if args.check and args.url:
|
||||
result = check_duplicate(base_path, args.url)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
# 归档
|
||||
|
||||
elif args.archive_file:
|
||||
result = archive_and_git(args.archive_file, base_path)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
# 目录路径查询
|
||||
|
||||
elif args.pending:
|
||||
print(json.dumps({"path": str(dirs["pending"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.metadata:
|
||||
print(json.dumps({"path": str(dirs["metadata"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.content:
|
||||
print(json.dumps({"path": str(dirs["content"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.content_downloads:
|
||||
print(json.dumps({"path": str(dirs["content_downloads"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.analyzed:
|
||||
print(json.dumps({"path": str(dirs["analyzed"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.archive_dir:
|
||||
print(json.dumps({"path": str(get_archive_path())}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.archive_subscriptions:
|
||||
print(json.dumps({"path": str(dirs["archive_subscriptions"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.extract_platform:
|
||||
platform = extract_platform(args.extract_platform)
|
||||
print(json.dumps({"platform": platform}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.base_path:
|
||||
print(json.dumps({"path": str(dirs["base"])}, ensure_ascii=False, indent=2))
|
||||
|
||||
# 状态管理
|
||||
elif args.init_state and args.url:
|
||||
result = init_state(args.url, base_path)
|
||||
print(json.dumps({"success": True, "state": result}, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.get_state and args.url:
|
||||
result = get_state(args.url, base_path)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.advance_step and args.url and args.step:
|
||||
result_data = None
|
||||
if args.result:
|
||||
try:
|
||||
result_data = json.loads(args.result)
|
||||
except json.JSONDecodeError:
|
||||
result_data = {"raw": args.result}
|
||||
result = advance_step(args.url, args.step, result_data, base_path)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.record_error and args.url and args.error_msg:
|
||||
result = record_error(args.url, args.error_msg, args.step, base_path)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
elif args.resume and args.url:
|
||||
result = get_resume_info(args.url, base_path)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
Reference in New Issue
Block a user