Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 289be86fc6 | |||
| 79cf9635c6 | |||
| bc5e02249d | |||
| 36de6cf6dd | |||
| 23254afc49 | |||
| 00b8e103c0 | |||
| 7a1b2fc244 | |||
| 61c4fd401f | |||
| 71176c9a12 | |||
| b5cd81d6e7 | |||
| 46aab1d9f4 | |||
| ed1de1a5b1 | |||
| b89627b8ce | |||
| 33c2fd4bcd | |||
| b3e40b3344 | |||
| 9f12b35e6d | |||
| 4fde411be3 |
@@ -18,10 +18,21 @@ jobs:
|
||||
SKILLS_DIR="/home/starmen1265/.openclaw/skills"
|
||||
SRC_DIR="/tmp/kb-skills-deploy"
|
||||
|
||||
# 技能列表 (目录名必须与仓库中的实际目录名匹配)
|
||||
# 技能列表
|
||||
SKILLS=(
|
||||
"dev-workflow"
|
||||
"gsd-openclaw"
|
||||
"kb_workflow"
|
||||
"kb-bilibili-content"
|
||||
"kb-bilibili-metadata"
|
||||
"kb-deep-analyzer"
|
||||
"kb-funasr"
|
||||
"kb-link-parser"
|
||||
"kb-ocr-processor"
|
||||
"kb-whisper"
|
||||
"kb-whisper-diarization"
|
||||
"kb-workflow"
|
||||
"kb-zhihu-content"
|
||||
"kb-zhihu-metadata"
|
||||
"todo-manager"
|
||||
)
|
||||
|
||||
|
||||
+57
-74
@@ -5,8 +5,6 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
|
||||
|
||||
# KB Workflow
|
||||
|
||||
知识库内容处理管道,支持 B站、知乎等平台的内容抓取、语音识别、深度分析和归档。
|
||||
|
||||
## 什么时候使用
|
||||
|
||||
当用户发送包含以下模式的消息时,此技能自动触发:
|
||||
@@ -14,7 +12,7 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
|
||||
| 触发词 | 行为 | 示例 |
|
||||
|--------|------|------|
|
||||
| `添加到知识库` + 链接 | 添加到知识库并处理 | "添加到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
| `保存到知识库` + 链接 | 同上 | "保存到知识库 https://zhuanlan.zhihu.com/p/xxx" |
|
||||
| `保存到知识库` + 链接 | 同上 | "保存到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
|
||||
## 环境变量
|
||||
|
||||
@@ -24,22 +22,23 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
|
||||
|
||||
**必须的环境变量**(缺失则报错退出):
|
||||
|
||||
配置方式:将变量写入 `~/.kb_env` 文件:
|
||||
```bash
|
||||
# ~/.kb_env 文件
|
||||
export AI_API_KEY="your-api-key"
|
||||
export AI_PROVIDER="zhipu"
|
||||
export AI_MODEL="glm-5"
|
||||
export AI_BASE_URL="https://open.bigmodel.cn/api/coding/paas/v4"
|
||||
export KB_INTERMEDIATE_PATH="~/work/intermediate/analysis"
|
||||
export KB_ARCHIVE_PATH="~/work/zq/30.资源/AI分析"
|
||||
```
|
||||
|
||||
**可选的环境变量**:
|
||||
| 变量名 | 用途 |
|
||||
|--------|------|
|
||||
| `BILIBILI_COOKIE` | 下载会员视频 |
|
||||
| `KB_GIT_USER` / `KB_GIT_EMAIL` | Git 提交信息 |
|
||||
| `HF_TOKEN` | HuggingFace Token(说话者分离) |
|
||||
| 变量名 | 用途 | 示例 |
|
||||
|--------|------|------|
|
||||
| `BILIBILI_COOKIE` | 下载会员视频 | `SESSDATA=xxx; bili_jct=xxx` |
|
||||
| `KB_GIT_USER` | Git提交用户名 | `Your Name` |
|
||||
| `KB_GIT_EMAIL` | Git提交邮箱 | `your@email.com` |
|
||||
| `KB_ARCHIVE_PATH` | 归档目录路径 | `~/work/zq/30.资源/AI分析` |
|
||||
| `KB_GIT_ROOT` | Git仓库根目录 | `~/work/zq` |
|
||||
| `HF_TOKEN` | HuggingFace Token(说话者分离) | `hf_xxx` |
|
||||
|
||||
### 检查命令
|
||||
|
||||
@@ -50,7 +49,7 @@ if [ -f ~/.kb_env ]; then
|
||||
fi
|
||||
|
||||
# 检查必须的环境变量
|
||||
if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ]; then
|
||||
if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ] || [ -z "$AI_BASE_URL" ]; then
|
||||
echo "❌ 错误:必须的环境变量未设置"
|
||||
echo "请配置 ~/.kb_env 文件"
|
||||
exit 1
|
||||
@@ -61,78 +60,72 @@ echo "✅ 环境变量检查通过"
|
||||
|
||||
## Workflow 执行方式
|
||||
|
||||
### 处理链接
|
||||
### 一键处理(推荐)
|
||||
|
||||
使用管道 CLI 一键完成所有步骤:
|
||||
|
||||
```bash
|
||||
# 处理 URL(自动执行完整管道)
|
||||
python kb_workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
```
|
||||
|
||||
**该命令自动执行:**
|
||||
1. URL 解析 → 识别平台和内容 ID
|
||||
2. 元数据提取 → 获取标题、作者等信息
|
||||
3. 内容下载 → 字幕/视频/文章内容
|
||||
4. 语音识别 → Whisper/FunASR(如需要)
|
||||
5. 深度分析 → AI 生成分析报告
|
||||
6. 归档 → 移动到归档目录
|
||||
|
||||
**命令参数:**
|
||||
```bash
|
||||
# 指定输出目录
|
||||
python kb_workflow/kb_workflow.py process "https://..." -o /path/to/output
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." -o /path/to/output
|
||||
|
||||
# 强制重新执行(忽略已完成状态)
|
||||
python kb_workflow/kb_workflow.py process "https://..." --no-skip
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." --no-skip
|
||||
```
|
||||
|
||||
### 知乎登录
|
||||
|
||||
```bash
|
||||
# 二维码扫码登录(Cookie 自动保存)
|
||||
python kb_workflow/kb_workflow.py login
|
||||
|
||||
# 检查 Cookie 状态
|
||||
python kb_workflow/kb_workflow.py login --status
|
||||
|
||||
# 从 JSON 文件导入 Cookie
|
||||
python kb_workflow/kb_workflow.py login --import-cookie cookies.json
|
||||
```
|
||||
|
||||
**注意:** 知乎链接处理时会自动检测 Cookie 是否过期,过期则自动触发扫码登录。
|
||||
|
||||
## 支持的平台和内容类型
|
||||
|
||||
| 平台 | 内容类型 | URL 示例 |
|
||||
|------|----------|----------|
|
||||
| B站 | 视频 | `bilibili.com/video/BVxxx` |
|
||||
| B站 | 专栏 | `bilibili.com/read/cvxxx` |
|
||||
| 知乎 | 问答 | `zhihu.com/question/xxx/answer/xxx` |
|
||||
| 知乎 | 问答 | `zhihu.com/question/xxx` |
|
||||
| 知乎 | 专栏 | `zhuanlan.zhihu.com/p/xxx` |
|
||||
| 知乎 | 视频 | `zhihu.com/zvideo/xxx` |
|
||||
|
||||
## 处理流程
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Pipeline 处理流程 │
|
||||
│ kb-workflow process │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 1. Parse → LinkParser 解析 URL,识别平台和 ID │
|
||||
│ 1. Parse → LinkParser 解析 URL │
|
||||
│ 2. Download → 平台适配器获取元数据和内容 │
|
||||
│ 3. ASR → 语音识别(视频无字幕时) │
|
||||
│ 4. Analyze → DeepAnalyzer 生成深度分析报告 │
|
||||
│ 4. Analyze → DeepAnalyzer 生成分析报告 │
|
||||
│ 5. Archive → 归档到 KB_ARCHIVE_PATH │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**知乎特殊处理:**
|
||||
- 评论默认爬取并追加到正文末尾
|
||||
- Cookie 过期时自动弹出二维码,扫码后继续执行
|
||||
- 无需手动设置 Cookie
|
||||
|
||||
## 目录结构
|
||||
|
||||
**工作目录(KB_INTERMEDIATE_PATH):**
|
||||
**工作目录(默认 ~/work/knowledge-base/):**
|
||||
```
|
||||
~/work/intermediate/analysis/
|
||||
└── {platform}_{content_id}/
|
||||
├── metadata.json # 元数据
|
||||
├── content.md # 内容(含评论)
|
||||
└── downloads/ # 下载的媒体文件
|
||||
~/work/knowledge-base/
|
||||
├── pending/ # 待处理链接
|
||||
├── metadata/ # 元数据 JSON
|
||||
├── content/ # 内容文件
|
||||
│ └── downloads/ # 下载的视频/字幕
|
||||
└── analyzed/ # 分析报告
|
||||
```
|
||||
|
||||
**归档目录(KB_ARCHIVE_PATH):**
|
||||
**归档目录(默认 ~/work/zq/30.资源/AI分析/):**
|
||||
```
|
||||
~/work/zq/30.资源/AI分析/
|
||||
└── YYYY-MM-DD/
|
||||
└── {platform}_{id}_analysis.md # 深度分析报告
|
||||
└── {platform}_{id}.md # 深度分析报告
|
||||
```
|
||||
|
||||
## ⚠️ 严格失败处理原则
|
||||
@@ -147,44 +140,34 @@ python kb_workflow/kb_workflow.py login --import-cookie cookies.json
|
||||
## 示例
|
||||
|
||||
```
|
||||
用户: 添加到知识库 https://zhuanlan.zhihu.com/p/123456789
|
||||
用户: 添加到知识库 https://www.bilibili.com/video/BV1xx
|
||||
|
||||
Claude:
|
||||
✅ 环境变量检查通过
|
||||
|
||||
◆ 处理中...
|
||||
→ 解析 URL: zhihu 专栏 123456789
|
||||
→ 解析 URL: bilibili 视频 BV1xx
|
||||
→ 获取元数据...
|
||||
→ 爬取内容和评论...
|
||||
→ 下载字幕...
|
||||
→ 深度分析...
|
||||
|
||||
✅ 完成!
|
||||
|
||||
标题: 深度学习在推荐系统中的应用
|
||||
内容目录: ~/work/intermediate/analysis/zhihu_123456789/
|
||||
归档文件: ~/work/zq/30.资源/AI分析/2026-03-17/zhihu_123456789_analysis.md
|
||||
标题: 【教程】如何使用 Python 进行数据分析
|
||||
处理阶段: 全部完成
|
||||
最终文件: ~/work/zq/30.资源/AI分析/2026-03-15/bilibili_BV1xx.md
|
||||
```
|
||||
|
||||
## 技术架构
|
||||
|
||||
```
|
||||
kb_workflow/
|
||||
├── kb_workflow.py — CLI 入口
|
||||
├── auth/
|
||||
│ └── zhihu_login.py — 知乎登录和 Cookie 管理
|
||||
├── crawler/
|
||||
│ ├── zhihu_crawler.py — 知乎爬虫(问答/专栏/评论)
|
||||
│ └── models.py — 数据模型
|
||||
├── pipeline/
|
||||
│ ├── pipeline.py — Pipeline 编排引擎
|
||||
│ ├── processors/ — 处理器实现
|
||||
│ │ ├── parse.py — URL 解析
|
||||
│ │ ├── download.py — 内容下载
|
||||
│ │ ├── asr.py — 语音识别
|
||||
│ │ ├── analyze.py — 深度分析
|
||||
│ │ └── archive.py — 归档
|
||||
│ ├── platforms/ — 平台适配器
|
||||
│ │ └── zhihu.py — 知乎适配器(使用 ZhihuCrawler)
|
||||
│ └── schemas.py — 数据模型
|
||||
└── deep_analyzer/ — AI 分析器
|
||||
kb-workflow/kb_workflow.py (CLI)
|
||||
└── kb_pipeline/ (核心管道包)
|
||||
├── pipeline.py — Pipeline 编排引擎
|
||||
├── processors/ — 处理器实现
|
||||
│ ├── parse.py — URL 解析
|
||||
│ ├── download.py — 内容下载
|
||||
│ ├── asr.py — 语音识别
|
||||
│ └── analyze.py — 深度分析
|
||||
└── schemas.py — 数据模型
|
||||
```
|
||||
|
||||
@@ -19,8 +19,7 @@ from playwright.sync_api import Route, Response, sync_playwright
|
||||
from tenacity import Retrying, stop_after_attempt, wait_exponential, RetryError
|
||||
|
||||
from kb_workflow.metadata.zhihu_extractor import PlaywrightBrowser
|
||||
# NOTE: get_settings is imported lazily in _get_cookie() to avoid circular import
|
||||
# Circular: zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
from .models import (
|
||||
AnswerData,
|
||||
ArticleData,
|
||||
@@ -130,9 +129,6 @@ class ZhihuCrawler:
|
||||
Returns:
|
||||
Cookie string from settings
|
||||
"""
|
||||
# Lazy import to avoid circular dependency:
|
||||
# zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler
|
||||
from kb_workflow.pipeline.config import get_settings
|
||||
settings = get_settings()
|
||||
return settings.zhihu_cookie.get_secret_value()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user