feat(06-01): create metadata directory and move bilibili modules
- Rename kb-workflow to kb_workflow for Python package compatibility - Move bilibili_extractor.py and bilibili_metadata.py to kb_workflow/metadata/ - Create metadata/__init__.py with BilibiliExporter export Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
# KB Workflow 环境变量配置
|
||||
export KB_BASE_PATH="~/work/knowledge-base"
|
||||
|
||||
# 可选: B站 Cookie(用于下载会员视频)
|
||||
# export BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx"
|
||||
|
||||
# 可选: AI 分析配置
|
||||
# export AI_API_KEY="your-api-key"
|
||||
# export AI_PROVIDER="openai" # 或 zhipu, minimax
|
||||
# export AI_MODEL="gpt-4"
|
||||
|
||||
# 可选: Git 配置
|
||||
# export KB_GIT_USER="Your Name"
|
||||
# export KB_GIT_EMAIL="your@email.com"
|
||||
@@ -0,0 +1,14 @@
|
||||
# KB Workflow 环境变量配置
|
||||
export KB_BASE_PATH="~/work/knowledge-base"
|
||||
|
||||
# 可选: B站 Cookie(用于下载会员视频)
|
||||
# export BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx"
|
||||
|
||||
# 可选: AI 分析配置
|
||||
# export AI_API_KEY="your-api-key"
|
||||
# export AI_PROVIDER="openai" # 或 zhipu, minimax
|
||||
# export AI_MODEL="gpt-4"
|
||||
|
||||
# 可选: Git 配置
|
||||
# export KB_GIT_USER="Your Name"
|
||||
# export KB_GIT_EMAIL="your@email.com"
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
name: kb-workflow
|
||||
description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链接"时使用
|
||||
---
|
||||
|
||||
# KB Workflow
|
||||
|
||||
## 什么时候使用
|
||||
|
||||
当用户发送包含以下模式的消息时,此技能自动触发:
|
||||
|
||||
| 触发词 | 行为 | 示例 |
|
||||
|--------|------|------|
|
||||
| `添加到知识库` + 链接 | 添加到知识库并处理 | "添加到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
| `保存到知识库` + 链接 | 同上 | "保存到知识库 https://www.bilibili.com/video/BVxxx" |
|
||||
|
||||
## 环境变量
|
||||
|
||||
### 执行前必须检查
|
||||
|
||||
在执行 workflow 之前,必须检查环境变量是否已经设置。
|
||||
|
||||
**必须的环境变量**(缺失则报错退出):
|
||||
|
||||
配置方式:将变量写入 `~/.kb_env` 文件:
|
||||
```bash
|
||||
export AI_API_KEY="your-api-key"
|
||||
export AI_PROVIDER="zhipu"
|
||||
export AI_MODEL="glm-5"
|
||||
export AI_BASE_URL="https://open.bigmodel.cn/api/coding/paas/v4"
|
||||
```
|
||||
|
||||
**可选的环境变量**:
|
||||
| 变量名 | 用途 | 示例 |
|
||||
|--------|------|------|
|
||||
| `BILIBILI_COOKIE` | 下载会员视频 | `SESSDATA=xxx; bili_jct=xxx` |
|
||||
| `KB_GIT_USER` | Git提交用户名 | `Your Name` |
|
||||
| `KB_GIT_EMAIL` | Git提交邮箱 | `your@email.com` |
|
||||
| `KB_ARCHIVE_PATH` | 归档目录路径 | `~/work/zq/30.资源/AI分析` |
|
||||
| `KB_GIT_ROOT` | Git仓库根目录 | `~/work/zq` |
|
||||
| `HF_TOKEN` | HuggingFace Token(说话者分离) | `hf_xxx` |
|
||||
|
||||
### 检查命令
|
||||
|
||||
```bash
|
||||
# 从 ~/.kb_env 加载环境变量
|
||||
if [ -f ~/.kb_env ]; then
|
||||
source ~/.kb_env
|
||||
fi
|
||||
|
||||
# 检查必须的环境变量
|
||||
if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ] || [ -z "$AI_BASE_URL" ]; then
|
||||
echo "❌ 错误:必须的环境变量未设置"
|
||||
echo "请配置 ~/.kb_env 文件"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ 环境变量检查通过"
|
||||
```
|
||||
|
||||
## Workflow 执行方式
|
||||
|
||||
### 一键处理(推荐)
|
||||
|
||||
使用管道 CLI 一键完成所有步骤:
|
||||
|
||||
```bash
|
||||
python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
```
|
||||
|
||||
**该命令自动执行:**
|
||||
1. URL 解析 → 识别平台和内容 ID
|
||||
2. 元数据提取 → 获取标题、作者等信息
|
||||
3. 内容下载 → 字幕/视频/文章内容
|
||||
4. 语音识别 → Whisper/FunASR(如需要)
|
||||
5. 深度分析 → AI 生成分析报告
|
||||
6. 归档 → 移动到归档目录
|
||||
|
||||
**命令参数:**
|
||||
```bash
|
||||
# 指定输出目录
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." -o /path/to/output
|
||||
|
||||
# 强制重新执行(忽略已完成状态)
|
||||
python3 kb-workflow/kb_workflow.py process "https://..." --no-skip
|
||||
```
|
||||
|
||||
## 支持的平台和内容类型
|
||||
|
||||
| 平台 | 内容类型 | URL 示例 |
|
||||
|------|----------|----------|
|
||||
| B站 | 视频 | `bilibili.com/video/BVxxx` |
|
||||
| B站 | 专栏 | `bilibili.com/read/cvxxx` |
|
||||
| 知乎 | 问答 | `zhihu.com/question/xxx` |
|
||||
| 知乎 | 专栏 | `zhuanlan.zhihu.com/p/xxx` |
|
||||
| 知乎 | 视频 | `zhihu.com/zvideo/xxx` |
|
||||
|
||||
## 处理流程
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ kb-workflow process │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 1. Parse → LinkParser 解析 URL │
|
||||
│ 2. Download → 平台适配器获取元数据和内容 │
|
||||
│ 3. ASR → 语音识别(视频无字幕时) │
|
||||
│ 4. Analyze → DeepAnalyzer 生成分析报告 │
|
||||
│ 5. Archive → 归档到 KB_ARCHIVE_PATH │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 目录结构
|
||||
|
||||
**工作目录(默认 ~/work/knowledge-base/):**
|
||||
```
|
||||
~/work/knowledge-base/
|
||||
├── pending/ # 待处理链接
|
||||
├── metadata/ # 元数据 JSON
|
||||
├── content/ # 内容文件
|
||||
│ └── downloads/ # 下载的视频/字幕
|
||||
└── analyzed/ # 分析报告
|
||||
```
|
||||
|
||||
**归档目录(默认 ~/work/zq/30.资源/AI分析/):**
|
||||
```
|
||||
~/work/zq/30.资源/AI分析/
|
||||
└── YYYY-MM-DD/
|
||||
└── {platform}_{id}.md # 深度分析报告
|
||||
```
|
||||
|
||||
## ⚠️ 严格失败处理原则
|
||||
|
||||
> **深度分析步骤必须成功,否则整个流程失败**
|
||||
|
||||
1. **深度分析不可跳过**:AI 分析是必须成功执行的步骤
|
||||
2. **失败即终止**:如果分析返回失败,必须立即停止并汇报
|
||||
3. **禁止降级处理**:不能使用预设模板或简单摘要来"假装成功"
|
||||
4. **每次执行都要使用 AI**:必须有真实的 AI API 调用记录
|
||||
|
||||
## 示例
|
||||
|
||||
```
|
||||
用户: 添加到知识库 https://www.bilibili.com/video/BV1xx
|
||||
|
||||
Claude:
|
||||
✅ 环境变量检查通过
|
||||
|
||||
◆ 处理中...
|
||||
→ 解析 URL: bilibili 视频 BV1xx
|
||||
→ 获取元数据...
|
||||
→ 下载字幕...
|
||||
→ 深度分析...
|
||||
|
||||
✅ 完成!
|
||||
|
||||
标题: 【教程】如何使用 Python 进行数据分析
|
||||
处理阶段: 全部完成
|
||||
最终文件: ~/work/zq/30.资源/AI分析/2026-03-15/bilibili_BV1xx.md
|
||||
```
|
||||
|
||||
## 技术架构
|
||||
|
||||
```
|
||||
kb-workflow/kb_workflow.py (CLI)
|
||||
└── kb_pipeline/ (核心管道包)
|
||||
├── pipeline.py — Pipeline 编排引擎
|
||||
├── processors/ — 处理器实现
|
||||
│ ├── parse.py — URL 解析
|
||||
│ ├── download.py — 内容下载
|
||||
│ ├── asr.py — 语音识别
|
||||
│ └── analyze.py — 深度分析
|
||||
└── schemas.py — 数据模型
|
||||
```
|
||||
@@ -0,0 +1,321 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# KB Workflow 环境一键安装脚本
|
||||
# =============================================================================
|
||||
# 功能: 检查并安装 kb-workflow 技能所需的全部环境依赖
|
||||
# 支持: Ubuntu/Debian 系统
|
||||
#
|
||||
# 使用方法:
|
||||
# ./install-env.sh # 检查环境并提示安装命令(检查模式)
|
||||
# ./install-env.sh --auto # 自动安装所有缺失依赖(自动模式)
|
||||
# =============================================================================
|
||||
|
||||
# 颜色定义
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
# 日志函数
|
||||
log_info() { echo -e "${BLUE}[INFO]${NC} $1"; }
|
||||
log_success() { echo -e "${GREEN}[SUCCESS]${NC} $1"; }
|
||||
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
||||
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
|
||||
log_section() {
|
||||
echo ""
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo -e "${BLUE}$1${NC}"
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
}
|
||||
|
||||
# 获取脚本目录
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
DEFAULT_KB_PATH="${HOME}/work/knowledge-base"
|
||||
|
||||
# 解析参数
|
||||
AUTO_MODE=false
|
||||
if [ "$1" = "--auto" ]; then
|
||||
AUTO_MODE=true
|
||||
log_info "自动安装模式"
|
||||
fi
|
||||
|
||||
# 检测 Linux 发行版
|
||||
detect_distro() {
|
||||
if [ -f /etc/os-release ]; then
|
||||
. /etc/os-release
|
||||
echo "$ID"
|
||||
elif [ -f /etc/debian_version ]; then
|
||||
echo "debian"
|
||||
elif [ -f /etc/redhat-release ]; then
|
||||
echo "rhel"
|
||||
fi
|
||||
}
|
||||
DISTRO=$(detect_distro)
|
||||
|
||||
# 检查命令是否存在
|
||||
command_exists() {
|
||||
command -v "$1" > /dev/null 2>&1
|
||||
}
|
||||
|
||||
# 检查 Python 包
|
||||
check_python_package() {
|
||||
python3 -c "import $1" 2>/dev/null
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 1. 检查系统依赖
|
||||
# =============================================================================
|
||||
log_section "第一步: 检查系统依赖"
|
||||
|
||||
MISSING_DEPS=()
|
||||
|
||||
# 检查 Python3
|
||||
if command_exists python3; then
|
||||
PYTHON_VERSION=$(python3 --version | cut -d' ' -f2)
|
||||
log_success "Python3 已安装: $PYTHON_VERSION"
|
||||
else
|
||||
log_warn "Python3 未安装"
|
||||
MISSING_DEPS+=("python3")
|
||||
fi
|
||||
|
||||
# 检查 pip3
|
||||
if python3 -m pip --version > /dev/null 2>&1 || command_exists pip3; then
|
||||
log_success "pip3 已安装"
|
||||
else
|
||||
log_warn "pip3 未安装"
|
||||
MISSING_DEPS+=("python3-pip")
|
||||
fi
|
||||
|
||||
# 检查 ffmpeg
|
||||
if command_exists ffmpeg; then
|
||||
log_success "ffmpeg 已安装"
|
||||
else
|
||||
log_warn "ffmpeg 未安装"
|
||||
MISSING_DEPS+=("ffmpeg")
|
||||
fi
|
||||
|
||||
# 检查 tesseract
|
||||
if command_exists tesseract; then
|
||||
log_success "Tesseract OCR 已安装"
|
||||
else
|
||||
log_warn "Tesseract OCR 未安装"
|
||||
MISSING_DEPS+=("tesseract-ocr")
|
||||
fi
|
||||
|
||||
# 检查 git
|
||||
if command_exists git; then
|
||||
log_success "Git 已安装"
|
||||
else
|
||||
log_warn "Git 未安装"
|
||||
MISSING_DEPS+=("git")
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# 2. 自动安装系统依赖(自动模式)
|
||||
# =============================================================================
|
||||
if [ ${#MISSING_DEPS[@]} -gt 0 ]; then
|
||||
if [ "$AUTO_MODE" = true ]; then
|
||||
log_section "自动安装系统依赖"
|
||||
log_info "检测到系统: $DISTRO"
|
||||
|
||||
case "$DISTRO" in
|
||||
ubuntu|debian)
|
||||
log_info "更新包列表..."
|
||||
sudo apt-get update -qq
|
||||
|
||||
log_info "安装依赖: ${MISSING_DEPS[*]}"
|
||||
sudo apt-get install -y -qq "${MISSING_DEPS[@]}" tesseract-ocr-chi-sim tesseract-ocr-eng
|
||||
;;
|
||||
fedora|rhel|centos)
|
||||
log_info "更新包列表..."
|
||||
sudo dnf check-update -y || true
|
||||
|
||||
log_info "安装依赖: ${MISSING_DEPS[*]}"
|
||||
sudo dnf install -y "${MISSING_DEPS[@]}" tesseract-langpack-chi_sim tesseract-langpack-eng
|
||||
;;
|
||||
arch)
|
||||
log_info "安装依赖: ${MISSING_DEPS[*]}"
|
||||
sudo pacman -S --noconfirm "${MISSING_DEPS[@]}" tesseract-data-chi_sim tesseract-data-eng
|
||||
;;
|
||||
*)
|
||||
log_error "不支持的系统: $DISTRO"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
log_success "系统依赖安装完成"
|
||||
MISSING_DEPS=()
|
||||
else
|
||||
log_section "需要安装的系统依赖"
|
||||
echo ""
|
||||
echo "检测到以下依赖未安装:"
|
||||
for dep in "${MISSING_DEPS[@]}"; do
|
||||
echo " - $dep"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "请运行以下命令安装:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
echo -e "${YELLOW}sudo apt-get update${NC}"
|
||||
echo -e "${YELLOW}sudo apt-get install -y ${MISSING_DEPS[*]} tesseract-ocr-chi-sim tesseract-ocr-eng${NC}"
|
||||
echo ""
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# 3. 检查 Python 依赖
|
||||
# =============================================================================
|
||||
log_section "第二步: 检查 Python 依赖"
|
||||
|
||||
PYTHON_PACKAGES=(
|
||||
"httpx:httpx"
|
||||
"aiohttp:aiohttp"
|
||||
"requests:requests"
|
||||
"Pillow:PIL"
|
||||
)
|
||||
|
||||
MISSING_PY_PKGS=()
|
||||
|
||||
for pkg_info in "${PYTHON_PACKAGES[@]}"; do
|
||||
pkg_name=$(echo "$pkg_info" | cut -d':' -f1)
|
||||
import_name=$(echo "$pkg_info" | cut -d':' -f2)
|
||||
if check_python_package "$import_name" 2>/dev/null; then
|
||||
log_success "$pkg_name 已安装"
|
||||
else
|
||||
log_warn "$pkg_name 未安装"
|
||||
MISSING_PY_PKGS+=("$pkg_name")
|
||||
fi
|
||||
done
|
||||
|
||||
# 检查特殊包
|
||||
check_python_package "playwright" 2>/dev/null && log_success "playwright 已安装" || MISSING_PY_PKGS+=("playwright")
|
||||
check_python_package "easyocr" 2>/dev/null && log_success "easyocr 已安装" || MISSING_PY_PKGS+=("easyocr")
|
||||
check_python_package "pytesseract" 2>/dev/null && log_success "pytesseract 已安装" || MISSING_PY_PKGS+=("pytesseract")
|
||||
|
||||
# =============================================================================
|
||||
# 4. 自动安装 Python 依赖(自动模式)
|
||||
# =============================================================================
|
||||
if [ ${#MISSING_PY_PKGS[@]} -gt 0 ]; then
|
||||
if [ "$AUTO_MODE" = true ]; then
|
||||
log_section "自动安装 Python 包"
|
||||
|
||||
log_info "安装: ${MISSING_PY_PKGS[*]}"
|
||||
pip3 install --user "${MISSING_PY_PKGS[@]}" -q
|
||||
|
||||
log_success "Python 包安装完成"
|
||||
|
||||
# 安装 Playwright 浏览器
|
||||
if [[ " ${MISSING_PY_PKGS[*]} " =~ " playwright " ]]; then
|
||||
log_info "安装 Playwright Chromium 浏览器..."
|
||||
python3 -m playwright install chromium
|
||||
log_success "Chromium 浏览器安装完成"
|
||||
fi
|
||||
|
||||
MISSING_PY_PKGS=()
|
||||
else
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "需要安装以下 Python 包:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
for pkg in "${MISSING_PY_PKGS[@]}"; do
|
||||
echo " - $pkg"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "请运行以下命令安装:"
|
||||
echo ""
|
||||
echo -e "${YELLOW}pip3 install --user ${MISSING_PY_PKGS[*]}${NC}"
|
||||
|
||||
if [[ " ${MISSING_PY_PKGS[*]} " =~ " playwright " ]]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}安装完 playwright 后,还需要运行:${NC}"
|
||||
echo -e "${YELLOW} python3 -m playwright install chromium${NC}"
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# 5. 创建知识库目录结构
|
||||
# =============================================================================
|
||||
log_section "第三步: 创建知识库目录结构"
|
||||
|
||||
KB_BASE_PATH="${KB_BASE_PATH:-$DEFAULT_KB_PATH}"
|
||||
log_info "知识库路径: $KB_BASE_PATH"
|
||||
|
||||
mkdir -p "$KB_BASE_PATH"/{archive/{subscriptions,pending},metadata,content/{downloads,subtitles},analyzed}
|
||||
log_success "目录结构创建完成"
|
||||
|
||||
# =============================================================================
|
||||
# 6. 环境变量配置提示
|
||||
# =============================================================================
|
||||
log_section "第四步: 环境变量配置"
|
||||
|
||||
if [ -z "$KB_BASE_PATH_SET" ]; then
|
||||
log_warn "KB_BASE_PATH 环境变量未设置"
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "建议添加到 ~/.bashrc 或 ~/.zshrc:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
echo -e "${YELLOW}export KB_BASE_PATH=\"$KB_BASE_PATH\"${NC}"
|
||||
echo ""
|
||||
else
|
||||
log_success "KB_BASE_PATH 已设置: $KB_BASE_PATH"
|
||||
fi
|
||||
|
||||
# 创建环境变量示例文件
|
||||
ENV_FILE="${SCRIPT_DIR}/.env.example"
|
||||
cat > "$ENV_FILE" << 'EOF'
|
||||
# KB Workflow 环境变量配置
|
||||
export KB_BASE_PATH="~/work/knowledge-base"
|
||||
|
||||
# 可选: B站 Cookie(用于下载会员视频)
|
||||
# export BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx"
|
||||
|
||||
# 可选: AI 分析配置
|
||||
# export AI_API_KEY="your-api-key"
|
||||
# export AI_PROVIDER="openai" # 或 zhipu, minimax
|
||||
# export AI_MODEL="gpt-4"
|
||||
|
||||
# 可选: Git 配置
|
||||
# export KB_GIT_USER="Your Name"
|
||||
# export KB_GIT_EMAIL="your@email.com"
|
||||
EOF
|
||||
|
||||
log_info "环境变量示例已保存到: $ENV_FILE"
|
||||
|
||||
# =============================================================================
|
||||
# 7. 完成总结
|
||||
# =============================================================================
|
||||
log_section "安装完成"
|
||||
|
||||
echo ""
|
||||
if [ ${#MISSING_DEPS[@]} -eq 0 ] && [ ${#MISSING_PY_PKGS[@]} -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ 所有依赖已安装完成!${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ 部分依赖需要手动安装${NC}"
|
||||
echo ""
|
||||
|
||||
if [ ${#MISSING_DEPS[@]} -gt 0 ]; then
|
||||
echo "系统依赖:"
|
||||
echo -e "${YELLOW}sudo apt-get install -y ${MISSING_DEPS[*]} tesseract-ocr-chi-sim tesseract-ocr-eng${NC}"
|
||||
fi
|
||||
|
||||
if [ ${#MISSING_PY_PKGS[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "Python 包:"
|
||||
echo -e "${YELLOW}pip3 install --user ${MISSING_PY_PKGS[*]}${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "使用方法:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " export KB_BASE_PATH=\"~/work/knowledge-base\"
|
||||
echo " python3 kb_workflow.py --check --url \"[URL]\""
|
||||
echo ""
|
||||
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
KB Workflow - Knowledge Base Workflow Orchestrator
|
||||
|
||||
知识库工作流管道入口。
|
||||
|
||||
此模块提供 process_command() 函数作为管道入口,
|
||||
内部使用 kb_pipeline 包实现完整的内容处理流程。
|
||||
|
||||
使用方式:
|
||||
python kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
python kb_workflow.py process "https://..." -o /path/to/output
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
def process_command(
|
||||
url: str,
|
||||
output_dir: Optional[str] = None,
|
||||
skip_completed: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute full pipeline for a URL
|
||||
|
||||
Args:
|
||||
url: URL to process
|
||||
output_dir: Optional output directory for content. Defaults to
|
||||
KB_INTERMEDIATE_PATH env var or ~/work/intermediate/analysis/
|
||||
skip_completed: Skip already completed steps (default True)
|
||||
|
||||
Returns:
|
||||
Dictionary with success status, content_dir, and results
|
||||
"""
|
||||
# Add project root to path for kb_pipeline imports
|
||||
project_root = Path(__file__).parent.parent
|
||||
if str(project_root) not in sys.path:
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
try:
|
||||
from kb_pipeline.config import get_settings
|
||||
from kb_pipeline.pipeline import Pipeline
|
||||
from kb_pipeline.processors import (
|
||||
ASRProcessor, AnalyzeProcessor, ArchiveProcessor,
|
||||
DownloadProcessor, ParseProcessor
|
||||
)
|
||||
from kb_pipeline.schemas import PipelineConfig
|
||||
|
||||
async def run_pipeline():
|
||||
# Determine output directory - default to intermediate_path
|
||||
settings = get_settings()
|
||||
out_dir = Path(output_dir) if output_dir else Path(settings.intermediate_path)
|
||||
|
||||
# Create processors
|
||||
processors = [
|
||||
ParseProcessor(),
|
||||
DownloadProcessor(),
|
||||
ASRProcessor(),
|
||||
AnalyzeProcessor(),
|
||||
ArchiveProcessor(),
|
||||
]
|
||||
|
||||
# Create pipeline config
|
||||
config = PipelineConfig(
|
||||
steps=["parse", "download", "asr", "analyze", "archive"],
|
||||
output_dir=out_dir,
|
||||
skip_completed=skip_completed,
|
||||
)
|
||||
|
||||
# Create and run pipeline
|
||||
pipeline = Pipeline(
|
||||
processors=processors,
|
||||
config=config,
|
||||
output_dir=out_dir,
|
||||
)
|
||||
|
||||
result = await pipeline.run(url)
|
||||
return result
|
||||
|
||||
# Run async pipeline
|
||||
result = asyncio.run(run_pipeline())
|
||||
|
||||
return {
|
||||
"success": result.success,
|
||||
"error": result.error_message,
|
||||
"content_dir": str(result.content_dir) if result.content_dir else None,
|
||||
"results": result.results,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"content_dir": None,
|
||||
"results": {},
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI入口 - 用于命令行调用"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='KB Workflow - Knowledge Base Workflow Orchestrator',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog='''
|
||||
使用示例:
|
||||
# 处理URL (通过完整管道)
|
||||
python kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
|
||||
|
||||
# 指定输出目录
|
||||
python kb_workflow.py process "https://..." -o /path/to/output
|
||||
|
||||
# 强制重新执行(忽略已完成状态)
|
||||
python kb_workflow.py process "https://..." --no-skip
|
||||
'''
|
||||
)
|
||||
|
||||
# Create subparsers for commands
|
||||
subparsers = parser.add_subparsers(dest='command', help='Available commands')
|
||||
|
||||
# Process subcommand
|
||||
process_parser = subparsers.add_parser('process', help='Process URL through full pipeline')
|
||||
process_parser.add_argument('url', type=str, help='URL to process')
|
||||
process_parser.add_argument('--output-dir', '-o', type=str, help='Output directory for content')
|
||||
process_parser.add_argument('--skip-completed', action='store_true', default=True,
|
||||
help='Skip completed steps (default: True)')
|
||||
process_parser.add_argument('--no-skip', action='store_false', dest='skip_completed',
|
||||
help='Re-run all steps')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Handle process subcommand
|
||||
if args.command == 'process':
|
||||
result = process_command(
|
||||
url=args.url,
|
||||
output_dir=args.output_dir,
|
||||
skip_completed=args.skip_completed
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,10 @@
|
||||
"""KB Workflow Metadata Module
|
||||
|
||||
提供 B站和知乎的元数据提取功能
|
||||
"""
|
||||
|
||||
from .bilibili_extractor import BilibiliExtractor
|
||||
|
||||
__all__ = [
|
||||
"BilibiliExtractor",
|
||||
]
|
||||
@@ -0,0 +1,666 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
B站元数据提取器
|
||||
|
||||
独立版本,不依赖 obsidian-kb 结构
|
||||
支持视频、专栏、UP主信息提取
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 默认请求头
|
||||
DEFAULT_HEADERS = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
||||
'Referer': 'https://www.bilibili.com/',
|
||||
'Accept': 'application/json, text/plain, */*',
|
||||
'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
|
||||
}
|
||||
|
||||
try:
|
||||
from playwright.async_api import async_playwright
|
||||
PLAYWRIGHT_AVAILABLE = True
|
||||
except ImportError:
|
||||
PLAYWRIGHT_AVAILABLE = False
|
||||
|
||||
try:
|
||||
from undetected_playwright import stealth_async
|
||||
STEALTH_AVAILABLE = True
|
||||
except ImportError:
|
||||
STEALTH_AVAILABLE = False
|
||||
|
||||
try:
|
||||
import httpx
|
||||
HTTPX_AVAILABLE = True
|
||||
except ImportError:
|
||||
HTTPX_AVAILABLE = False
|
||||
|
||||
|
||||
class BilibiliExtractor:
|
||||
"""
|
||||
B站元数据提取器
|
||||
|
||||
支持:
|
||||
- 视频信息提取(标题、作者、统计数据、字幕等)
|
||||
- 专栏文章提取
|
||||
- UP主信息提取
|
||||
"""
|
||||
|
||||
# 正则模式
|
||||
BV_PATTERN = re.compile(r'BV[a-zA-Z0-9]{10}')
|
||||
CV_PATTERN = re.compile(r'cv(\d+)')
|
||||
URL_PATTERN = re.compile(r'https?://www\.bilibili\.com/video/(BV[a-zA-Z0-9]{10})')
|
||||
ARTICLE_URL_PATTERN = re.compile(r'https?://www\.bilibili\.com/read/(cv\d+)')
|
||||
UP_URL_PATTERN = re.compile(r'https?://space\.bilibili\.com/(\d+)')
|
||||
|
||||
def __init__(self, headless: bool = True, timeout: int = 30):
|
||||
"""
|
||||
初始化提取器
|
||||
|
||||
Args:
|
||||
headless: 是否使用无头模式(浏览器)
|
||||
timeout: 超时时间(秒)
|
||||
"""
|
||||
self.headless = headless
|
||||
self.timeout = timeout * 1000 # 转换为毫秒
|
||||
|
||||
def extract_video_metadata(self, bvid: str) -> Dict[str, Any]:
|
||||
"""
|
||||
提取视频元数据
|
||||
|
||||
Args:
|
||||
bvid: B站视频BV号(如 BV1xx411c7mD)
|
||||
|
||||
Returns:
|
||||
Dict containing metadata, stats, description, subtitle
|
||||
"""
|
||||
# 验证并解析BV号
|
||||
bvid = self._resolve_bvid(bvid)
|
||||
if not bvid:
|
||||
raise ValueError(f"无效的BV号: {bvid}")
|
||||
|
||||
# 使用异步方法提取
|
||||
return asyncio.run(self._extract_video_async(bvid))
|
||||
|
||||
def extract_article_metadata(self, cvid: str) -> Dict[str, Any]:
|
||||
"""
|
||||
提取专栏元数据
|
||||
|
||||
Args:
|
||||
cvid: B站专栏CV号(如 cv43780186)
|
||||
|
||||
Returns:
|
||||
Dict containing article metadata
|
||||
"""
|
||||
# 解析CV号
|
||||
cvid = self._resolve_cvid(cvid)
|
||||
if not cvid:
|
||||
raise ValueError(f"无效的CV号: {cvid}")
|
||||
|
||||
return asyncio.run(self._extract_article_async(cvid))
|
||||
|
||||
def extract_up_metadata(self, mid: str) -> Dict[str, Any]:
|
||||
"""
|
||||
提取UP主信息
|
||||
|
||||
Args:
|
||||
mid: B站用户ID(如 7770724)
|
||||
|
||||
Returns:
|
||||
Dict containing UP主信息
|
||||
"""
|
||||
# 解析MID
|
||||
mid = self._resolve_mid(mid)
|
||||
if not mid:
|
||||
raise ValueError(f"无效的MID: {mid}")
|
||||
|
||||
return asyncio.run(self._extract_up_async(mid))
|
||||
|
||||
def _resolve_bvid(self, source_id: str) -> Optional[str]:
|
||||
"""解析并返回BV号"""
|
||||
source = source_id.strip()
|
||||
|
||||
# 直接BV号
|
||||
if self.BV_PATTERN.fullmatch(source):
|
||||
return source
|
||||
|
||||
# 完整URL
|
||||
match = self.URL_PATTERN.match(source)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
def _resolve_cvid(self, source_id: str) -> Optional[str]:
|
||||
"""解析并返回CV号"""
|
||||
source = source_id.strip()
|
||||
|
||||
# 直接CV号(带cv前缀)
|
||||
if source.startswith('cv'):
|
||||
match = self.CV_PATTERN.match(source)
|
||||
if match:
|
||||
return source
|
||||
|
||||
# 纯数字
|
||||
if source.isdigit():
|
||||
return f"cv{source}"
|
||||
|
||||
# 完整URL
|
||||
match = self.ARTICLE_URL_PATTERN.match(source)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
def _resolve_mid(self, source_id: str) -> Optional[str]:
|
||||
"""解析并返回MID"""
|
||||
source = source_id.strip()
|
||||
|
||||
# 纯数字
|
||||
if source.isdigit():
|
||||
return source
|
||||
|
||||
# 完整URL
|
||||
match = self.UP_URL_PATTERN.match(source)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
async def _extract_video_async(self, bvid: str) -> Dict[str, Any]:
|
||||
"""异步提取视频信息"""
|
||||
url = f"https://www.bilibili.com/video/{bvid}"
|
||||
result = {
|
||||
'metadata': {},
|
||||
'stats': {},
|
||||
'subtitle': '',
|
||||
'description': ''
|
||||
}
|
||||
|
||||
if not PLAYWRIGHT_AVAILABLE:
|
||||
# 使用 HTTP API
|
||||
return await self._extract_video_by_api(bvid)
|
||||
|
||||
async with async_playwright() as playwright:
|
||||
try:
|
||||
browser = await playwright.chromium.launch(
|
||||
headless=self.headless,
|
||||
args=[
|
||||
'--no-sandbox',
|
||||
'--disable-setuid-sandbox',
|
||||
'--disable-blink-features=AutomationControlled',
|
||||
'--disable-dev-shm-usage',
|
||||
]
|
||||
)
|
||||
|
||||
context = await browser.new_context(
|
||||
viewport={'width': 1920, 'height': 1080},
|
||||
user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
||||
locale='zh-CN',
|
||||
)
|
||||
|
||||
if STEALTH_AVAILABLE:
|
||||
await stealth_async(context)
|
||||
|
||||
page = await context.new_page()
|
||||
await page.goto(url, wait_until='domcontentloaded', timeout=self.timeout)
|
||||
await asyncio.sleep(2)
|
||||
|
||||
content = await page.content()
|
||||
video_data = await self._try_extract_video_methods(page, content, bvid)
|
||||
|
||||
if video_data:
|
||||
result['metadata'] = {
|
||||
'title': video_data.get('title', ''),
|
||||
'author': video_data.get('author', ''),
|
||||
'bvid': video_data.get('bvid', ''),
|
||||
'aid': video_data.get('aid', ''),
|
||||
'mid': video_data.get('mid', ''),
|
||||
'pubtime': video_data.get('pubtime', ''),
|
||||
'duration': video_data.get('duration', ''),
|
||||
'url': url,
|
||||
'tags': video_data.get('tags', ''),
|
||||
}
|
||||
result['stats'] = {
|
||||
'view': video_data.get('view', 0),
|
||||
'danmaku': video_data.get('danmaku', 0),
|
||||
'reply': video_data.get('reply', 0),
|
||||
'favorite': video_data.get('favorite', 0),
|
||||
'coin': video_data.get('coin', 0),
|
||||
'share': video_data.get('share', 0),
|
||||
'like': video_data.get('like', 0),
|
||||
}
|
||||
result['description'] = video_data.get('desc', '')
|
||||
|
||||
await browser.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"视频提取错误: {e}")
|
||||
# 回退到 API
|
||||
return await self._extract_video_by_api(bvid)
|
||||
|
||||
return result
|
||||
|
||||
async def _extract_video_by_api(self, bvid: str) -> Dict[str, Any]:
|
||||
"""使用B站API提取视频信息"""
|
||||
result = {
|
||||
'metadata': {},
|
||||
'stats': {},
|
||||
'subtitle': '',
|
||||
'description': ''
|
||||
}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取视频信息
|
||||
response = await client.get('https://api.bilibili.com/x/web-interface/view', params={'bvid': bvid})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
video_data = data.get('data', {})
|
||||
|
||||
result['metadata'] = {
|
||||
'title': video_data.get('title', ''),
|
||||
'author': video_data.get('owner', {}).get('name', ''),
|
||||
'bvid': video_data.get('bvid', ''),
|
||||
'aid': video_data.get('aid', ''),
|
||||
'mid': video_data.get('owner', {}).get('mid', ''),
|
||||
'pubtime': datetime.fromtimestamp(video_data.get('pubdate', 0)).strftime('%Y-%m-%d %H:%M:%S') if video_data.get('pubdate') else '',
|
||||
'duration': self._format_duration(video_data.get('duration', 0)),
|
||||
'url': f"https://www.bilibili.com/video/{bvid}",
|
||||
'tags': ', '.join([t.get('tag_name', '') for t in video_data.get('tag', [])]),
|
||||
}
|
||||
|
||||
stat = video_data.get('stat', {})
|
||||
result['stats'] = {
|
||||
'view': stat.get('view', 0),
|
||||
'danmaku': stat.get('danmaku', 0),
|
||||
'reply': stat.get('reply', 0),
|
||||
'favorite': stat.get('favorite', 0),
|
||||
'coin': stat.get('coin', 0),
|
||||
'share': stat.get('share', 0),
|
||||
'like': stat.get('like', 0),
|
||||
}
|
||||
result['description'] = video_data.get('desc', '')
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"API提取错误: {e}")
|
||||
|
||||
return result
|
||||
|
||||
async def _try_extract_video_methods(self, page, content: str, bvid: str) -> Dict[str, Any]:
|
||||
"""尝试多种方法提取视频信息"""
|
||||
video_data = {}
|
||||
|
||||
# 方法1: 从 __INITIAL_STATE__
|
||||
try:
|
||||
initial_state = await page.evaluate('() => typeof window.__INITIAL_STATE__ !== "undefined" ? window.__INITIAL_STATE__ : null')
|
||||
if initial_state:
|
||||
video_data = self._parse_video_state(initial_state)
|
||||
if video_data and video_data.get('title'):
|
||||
return video_data
|
||||
except:
|
||||
pass
|
||||
|
||||
# 方法2: 使用 API(更可靠)
|
||||
return await self._extract_video_by_api(bvid)
|
||||
|
||||
def _parse_video_state(self, data: Dict) -> Dict[str, Any]:
|
||||
"""解析 __INITIAL_STATE__ 数据"""
|
||||
video_data = {}
|
||||
try:
|
||||
video = data.get('videoData', {})
|
||||
video_data['title'] = video.get('title', '')
|
||||
video_data['bvid'] = video.get('bvid', '')
|
||||
video_data['aid'] = video.get('aid', '')
|
||||
owner = video.get('owner', {})
|
||||
video_data['author'] = owner.get('name', '')
|
||||
video_data['mid'] = owner.get('mid', '')
|
||||
stat = video.get('stat', {})
|
||||
video_data['view'] = stat.get('view', 0)
|
||||
video_data['danmaku'] = stat.get('danmaku', 0)
|
||||
video_data['reply'] = stat.get('reply', 0)
|
||||
video_data['favorite'] = stat.get('favorite', 0)
|
||||
video_data['coin'] = stat.get('coin', 0)
|
||||
video_data['share'] = stat.get('share', 0)
|
||||
video_data['like'] = stat.get('like', 0)
|
||||
pubdate = video.get('pubdate', 0)
|
||||
video_data['pubtime'] = datetime.fromtimestamp(pubdate).strftime('%Y-%m-%d %H:%M:%S') if pubdate else ''
|
||||
video_data['duration'] = self._format_duration(video.get('duration', 0))
|
||||
video_data['desc'] = video.get('desc', '')
|
||||
video_data['tags'] = ', '.join([t.get('tag_name', '') for t in video.get('tag', [])])
|
||||
except:
|
||||
pass
|
||||
return video_data
|
||||
|
||||
async def _extract_article_async(self, cvid: str) -> Dict[str, Any]:
|
||||
"""异步提取专栏信息"""
|
||||
result = {}
|
||||
|
||||
if not HTTPX_AVAILABLE:
|
||||
return {'error': 'httpx 未安装'}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取专栏信息
|
||||
response = await client.get('https://api.bilibili.com/x/article/viewinfo', params={'id': cvid[2:]})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
article = data.get('data', {})
|
||||
result = {
|
||||
'cvid': cvid,
|
||||
'title': article.get('title', ''),
|
||||
'author': article.get('author_name', ''),
|
||||
'mid': article.get('author_mid', ''),
|
||||
'publish_time': datetime.fromtimestamp(article.get('publish_time', 0)).strftime('%Y-%m-%d %H:%M:%S') if article.get('publish_time') else '',
|
||||
'view_count': article.get('stats', {}).get('view', 0),
|
||||
'like_count': article.get('stats', {}).get('like', 0),
|
||||
'coin_count': article.get('stats', {}).get('coin', 0),
|
||||
'share_count': article.get('stats', {}).get('share', 0),
|
||||
'reply_count': article.get('stats', {}).get('reply', 0),
|
||||
'url': f"https://www.bilibili.com/read/{cvid}",
|
||||
'summary': article.get('summary', ''),
|
||||
'word_count': article.get('words', 0),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"专栏提取错误: {e}")
|
||||
result = {'error': str(e)}
|
||||
|
||||
return result
|
||||
|
||||
async def _extract_up_async(self, mid: str) -> Dict[str, Any]:
|
||||
"""
|
||||
异步提取UP主信息(支持订阅跟踪)
|
||||
|
||||
Args:
|
||||
mid: UP主ID
|
||||
|
||||
Returns:
|
||||
Dict containing UP主完整信息
|
||||
"""
|
||||
result = {
|
||||
'type': 'up',
|
||||
'mid': mid,
|
||||
'subscribed': False,
|
||||
'basic_info': {},
|
||||
'subscription_info': {}
|
||||
}
|
||||
|
||||
if not HTTPX_AVAILABLE:
|
||||
return {'error': 'httpx 未安装'}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取UP主基本信息
|
||||
response = await client.get('https://api.bilibili.com/x/space/acc/info', params={'mid': mid})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
up = data.get('data', {})
|
||||
result['basic_info'] = {
|
||||
'name': up.get('name', ''),
|
||||
'sex': up.get('sex', ''),
|
||||
'face': up.get('face', ''),
|
||||
'sign': up.get('sign', ''),
|
||||
'level': up.get('level', 0),
|
||||
'vip_type': up.get('vip', {}).get('type', 0),
|
||||
'vip_status': up.get('vip', {}).get('status', 0),
|
||||
'following_count': up.get('following', 0),
|
||||
'follower_count': up.get('follower', 0),
|
||||
'url': f"https://space.bilibili.com/{mid}",
|
||||
}
|
||||
|
||||
# 获取UP主视频统计
|
||||
stats_response = await client.get('https://api.bilibili.com/x/space/navnum', params={'mid': mid})
|
||||
if stats_response.status_code == 200:
|
||||
stats_data = stats_response.json()
|
||||
if stats_data.get('code') == 0:
|
||||
result['basic_info']['video_count'] = stats_data.get('data', {}).get('video', 0)
|
||||
|
||||
# 获取UP主专栏数量
|
||||
article_response = await client.get('https://api.bilibili.com/x/space/article', params={'mid': mid, 'pn': 1, 'ps': 1})
|
||||
if article_response.status_code == 200:
|
||||
article_data = article_response.json()
|
||||
if article_data.get('code') == 0:
|
||||
result['basic_info']['article_count'] = article_data.get('data', {}).get('count', 0)
|
||||
|
||||
# 获取UP主最近视频(用于跟踪更新)
|
||||
video_response = await client.get('https://api.bilibili.com/x/space/arc/search',
|
||||
params={'mid': mid, 'pn': 1, 'ps': 1, 'order': 'pubtime'})
|
||||
if video_response.status_code == 200:
|
||||
video_data = video_response.json()
|
||||
if video_data.get('code') == 0:
|
||||
videos = video_data.get('data', {}).get('list', {}).get('vlist', [])
|
||||
if videos:
|
||||
latest_video = videos[0]
|
||||
result['subscription_info'] = {
|
||||
'last_video_id': latest_video.get('bvid', ''),
|
||||
'last_video_title': latest_video.get('title', ''),
|
||||
'last_video_time': datetime.fromtimestamp(latest_video.get('created', 0)).strftime('%Y-%m-%d %H:%M:%S') if latest_video.get('created') else '',
|
||||
}
|
||||
|
||||
# 获取UP主最近专栏(用于跟踪更新)
|
||||
article_response = await client.get('https://api.bilibili.com/x/space/article',
|
||||
params={'mid': mid, 'pn': 1, 'ps': 1, 'sort': 'publish_time'})
|
||||
if article_response.status_code == 200:
|
||||
article_data = article_response.json()
|
||||
if article_data.get('code') == 0:
|
||||
articles = article_data.get('data', {}).get('articles', [])
|
||||
if articles:
|
||||
latest_article = articles[0]
|
||||
if 'last_article' not in result['subscription_info']:
|
||||
result['subscription_info']['last_article'] = {}
|
||||
result['subscription_info']['last_article'] = {
|
||||
'id': latest_article.get('id', 0),
|
||||
'title': latest_article.get('title', ''),
|
||||
'time': datetime.fromtimestamp(latest_article.get('publish_time', 0)).strftime('%Y-%m-%d %H:%M:%S') if latest_article.get('publish_time') else '',
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"UP主提取错误: {e}")
|
||||
result = {'error': str(e)}
|
||||
|
||||
return result
|
||||
|
||||
def _format_duration(self, seconds: int) -> str:
|
||||
"""格式化时长(秒 -> HH:MM:SS)"""
|
||||
if not seconds:
|
||||
return ''
|
||||
hours = seconds // 3600
|
||||
minutes = (seconds % 3600) // 60
|
||||
secs = seconds % 60
|
||||
if hours > 0:
|
||||
return f"{hours:02d}:{minutes:02d}:{secs:02d}"
|
||||
else:
|
||||
return f"{minutes:02d}:{secs:02d}"
|
||||
|
||||
async def _extract_series_async(self, mid: str, series_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
异步提取系列合集信息(不包含列表内容)
|
||||
|
||||
Args:
|
||||
mid: UP主ID
|
||||
series_id: 系列ID
|
||||
|
||||
Returns:
|
||||
Dict containing series basic info
|
||||
"""
|
||||
result = {
|
||||
'type': 'series',
|
||||
'series_id': series_id,
|
||||
'mid': mid,
|
||||
'subscribed': False,
|
||||
'basic_info': {}
|
||||
}
|
||||
|
||||
if not HTTPX_AVAILABLE:
|
||||
return {'error': 'httpx 未安装'}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取系列信息
|
||||
response = await client.get('https://api.bilibili.com/x/space/series',
|
||||
params={'mid': mid, 'series_id': series_id, 'pn': 1, 'ps': 1})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
series_info = data.get('data', {}).get('list', {})
|
||||
result['basic_info'] = {
|
||||
'title': series_info.get('title', ''),
|
||||
'description': series_info.get('desc', ''),
|
||||
'cover': series_info.get('cover', ''),
|
||||
'mid': mid,
|
||||
'creator_name': '',
|
||||
'total_count': series_info.get('cnt', 0),
|
||||
'last_updated': '',
|
||||
'update_count': 0
|
||||
}
|
||||
|
||||
# 获取创作者信息
|
||||
up_info = await client.get('https://api.bilibili.com/x/space/acc/info',
|
||||
params={'mid': mid})
|
||||
if up_info.status_code == 200:
|
||||
up_data = up_info.json()
|
||||
if up_data.get('code') == 0:
|
||||
result['basic_info']['creator_name'] = up_data.get('data', {}).get('name', '')
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"系列合集提取错误: {e}")
|
||||
result = {'error': str(e)}
|
||||
|
||||
return result
|
||||
|
||||
async def _extract_readlist_async(self, rl_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
异步提取专栏合集信息(不包含列表内容)
|
||||
|
||||
Args:
|
||||
rl_id: 专栏合集ID (rl开头)
|
||||
|
||||
Returns:
|
||||
Dict containing readlist basic info
|
||||
"""
|
||||
result = {
|
||||
'type': 'readlist',
|
||||
'readlist_id': rl_id,
|
||||
'subscribed': False,
|
||||
'basic_info': {}
|
||||
}
|
||||
|
||||
if not HTTPX_AVAILABLE:
|
||||
return {'error': 'httpx 未安装'}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取专栏合集信息
|
||||
response = await client.get('https://api.bilibili.com/x/article/readlist',
|
||||
params={'id': rl_id.replace('rl', '')})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
list_info = data.get('data', {})
|
||||
result['basic_info'] = {
|
||||
'title': list_info.get('name', ''),
|
||||
'description': list_info.get('description', ''),
|
||||
'cover': list_info.get('cover', ''),
|
||||
'mid': list_info.get('mid', ''),
|
||||
'creator_name': '',
|
||||
'total_count': list_info.get('articles_count', 0),
|
||||
'last_updated': '',
|
||||
'is_public': list_info.get('public', True)
|
||||
}
|
||||
|
||||
# 获取创作者信息
|
||||
mid = list_info.get('mid', '')
|
||||
if mid:
|
||||
up_info = await client.get('https://api.bilibili.com/x/space/acc/info',
|
||||
params={'mid': mid})
|
||||
if up_info.status_code == 200:
|
||||
up_data = up_info.json()
|
||||
if up_data.get('code') == 0:
|
||||
result['basic_info']['creator_name'] = up_data.get('data', {}).get('name', '')
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"专栏合集提取错误: {e}")
|
||||
result = {'error': str(e)}
|
||||
|
||||
return result
|
||||
|
||||
async def _extract_favlist_async(self, fid: str) -> Dict[str, Any]:
|
||||
"""
|
||||
异步提取收藏夹信息(不包含列表内容)
|
||||
|
||||
Args:
|
||||
fid: 收藏夹ID
|
||||
|
||||
Returns:
|
||||
Dict containing favlist basic info
|
||||
"""
|
||||
result = {
|
||||
'type': 'favlist',
|
||||
'favlist_id': fid,
|
||||
'subscribed': False,
|
||||
'basic_info': {}
|
||||
}
|
||||
|
||||
if not HTTPX_AVAILABLE:
|
||||
return {'error': 'httpx 未安装'}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30.0, headers=DEFAULT_HEADERS) as client:
|
||||
# 获取收藏夹信息
|
||||
response = await client.get('https://api.bilibili.com/x/space/favlist',
|
||||
params={'fid': fid, 'pn': 1, 'ps': 1})
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
if data.get('code') == 0:
|
||||
fav_info = data.get('data', {})
|
||||
result['basic_info'] = {
|
||||
'title': fav_info.get('title', ''),
|
||||
'description': fav_info.get('intro', ''),
|
||||
'cover': fav_info.get('cover', ''),
|
||||
'mid': fav_info.get('mid', ''),
|
||||
'creator_name': '',
|
||||
'total_count': fav_info.get('media_count', 0),
|
||||
'is_public': fav_info.get('public', True),
|
||||
'like_count': fav_info.get('like', 0)
|
||||
}
|
||||
|
||||
# 获取创作者信息
|
||||
mid = fav_info.get('mid', '')
|
||||
if mid:
|
||||
up_info = await client.get('https://api.bilibili.com/x/space/acc/info',
|
||||
params={'mid': mid})
|
||||
if up_info.status_code == 200:
|
||||
up_data = up_info.json()
|
||||
if up_data.get('code') == 0:
|
||||
result['basic_info']['creator_name'] = up_data.get('data', {}).get('name', '')
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"收藏夹提取错误: {e}")
|
||||
result = {'error': str(e)}
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# 测试代码
|
||||
if __name__ == '__main__':
|
||||
extractor = BilibiliExtractor()
|
||||
|
||||
print("=" * 60)
|
||||
print("B站元数据提取器测试")
|
||||
print("=" * 60)
|
||||
|
||||
# 测试视频
|
||||
print("\n测试视频提取: BV1MnFNz5EDL")
|
||||
result = extractor.extract_video_metadata('BV1MnFNz5EDL')
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,228 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
B站元数据提取 CLI 工具
|
||||
|
||||
支持命令行调用,输出 JSON 格式数据
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
|
||||
from bilibili_extractor import BilibiliExtractor
|
||||
|
||||
|
||||
def format_number(num: int) -> str:
|
||||
"""格式化数字"""
|
||||
if num >= 100000000:
|
||||
return f"{num / 100000000:.1f}亿"
|
||||
elif num >= 10000:
|
||||
return f"{num / 10000:.1f}万"
|
||||
else:
|
||||
return f"{num:,}"
|
||||
|
||||
|
||||
def extract_video(extractor: BilibiliExtractor, bvid: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取视频元数据"""
|
||||
result = extractor.extract_video_metadata(bvid)
|
||||
|
||||
# 如果需要输出到文件
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def extract_article(extractor: BilibiliExtractor, cvid: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取专栏元数据"""
|
||||
result = extractor.extract_article_metadata(cvid)
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def extract_up(extractor: BilibiliExtractor, mid: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取UP主信息(包含订阅跟踪所需信息)"""
|
||||
result = asyncio.run(extractor._extract_up_async(mid))
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def extract_series(extractor: BilibiliExtractor, mid: str, series_id: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取系列合集元数据"""
|
||||
result = asyncio.run(extractor._extract_series_async(mid, series_id))
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def extract_readlist(extractor: BilibiliExtractor, rl_id: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取专栏合集元数据"""
|
||||
result = asyncio.run(extractor._extract_readlist_async(rl_id))
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def extract_favlist(extractor: BilibiliExtractor, fid: str, output_file: str = None) -> Dict[str, Any]:
|
||||
"""提取收藏夹元数据"""
|
||||
result = asyncio.run(extractor._extract_favlist_async(fid))
|
||||
|
||||
if output_file:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
"""主函数"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='B站元数据提取工具',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
示例:
|
||||
# 提取视频元数据
|
||||
python bilibili_metadata.py --id "BV1xx411c7mD" --type video
|
||||
|
||||
# 提取专栏元数据
|
||||
python bilibili_metadata.py --id "cv43780186" --type article
|
||||
|
||||
# 提取UP主信息
|
||||
python bilibili_metadata.py --id "7770724" --type up
|
||||
|
||||
# 提取系列合集信息
|
||||
python bilibili_metadata.py --id "123456" --type series --mid "7770724"
|
||||
|
||||
# 提取专栏合集信息
|
||||
python bilibili_metadata.py --id "rl123456" --type readlist
|
||||
|
||||
# 提取收藏夹信息
|
||||
python bilibili_metadata.py --id "123456" --type favlist
|
||||
|
||||
# 输出到文件
|
||||
python bilibili_metadata.py --id "BV1xx411c7mD" --type video --output result.json
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--id',
|
||||
required=True,
|
||||
help='B站资源ID(BV号、CV号或UP主ID)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--type',
|
||||
required=True,
|
||||
choices=['video', 'article', 'up', 'series', 'readlist', 'favlist'],
|
||||
help='资源类型'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--mid',
|
||||
help='UP主ID (用于series类型)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
'-o',
|
||||
help='输出文件路径(JSON格式)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--headless',
|
||||
action='store_true',
|
||||
default=True,
|
||||
help='使用无头模式(浏览器,默认开启,使用API)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--timeout',
|
||||
type=int,
|
||||
default=30,
|
||||
help='超时时间(秒,默认30)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--pretty',
|
||||
action='store_true',
|
||||
default=True,
|
||||
help='格式化输出(默认开启)'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 创建提取器
|
||||
extractor = BilibiliExtractor(headless=args.headless, timeout=args.timeout)
|
||||
|
||||
try:
|
||||
# 根据类型提取数据
|
||||
if args.type == 'video':
|
||||
result = extract_video(extractor, args.id, args.output)
|
||||
elif args.type == 'article':
|
||||
result = extract_article(extractor, args.id, args.output)
|
||||
elif args.type == 'up':
|
||||
result = extract_up(extractor, args.id, args.output)
|
||||
elif args.type == 'series':
|
||||
if not args.mid:
|
||||
print(json.dumps({
|
||||
"success": False,
|
||||
"error": "series类型需要提供 --mid 参数"
|
||||
}, ensure_ascii=False, indent=2))
|
||||
sys.exit(1)
|
||||
result = extract_series(extractor, args.mid, args.id, args.output)
|
||||
elif args.type == 'readlist':
|
||||
result = extract_readlist(extractor, args.id, args.output)
|
||||
elif args.type == 'favlist':
|
||||
result = extract_favlist(extractor, args.id, args.output)
|
||||
else:
|
||||
print(json.dumps({
|
||||
"success": False,
|
||||
"error": f"不支持的类型 {args.type}"
|
||||
}, ensure_ascii=False, indent=2))
|
||||
sys.exit(1)
|
||||
|
||||
# 生成文件名
|
||||
platform = "bilibili"
|
||||
filename = f"{platform}_{args.id}.meta.json"
|
||||
filepath = args.output or filename
|
||||
|
||||
# 统一输出格式
|
||||
output = {
|
||||
"success": True,
|
||||
"filename": filename,
|
||||
"filepath": str(Path(filepath).resolve()) if filepath else None,
|
||||
"message": f"{args.type}元数据提取成功"
|
||||
}
|
||||
|
||||
print(json.dumps(output, ensure_ascii=False, indent=2))
|
||||
|
||||
# 如果指定了输出路径,则写入文件
|
||||
if args.output:
|
||||
with open(args.output, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
except Exception as e:
|
||||
print(json.dumps({
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"detail": type(e).__name__
|
||||
}, ensure_ascii=False, indent=2))
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user