5507b1d986
- kb-link-parser: Multi-platform URL parser (Bilibili, YouTube, Zhihu) - kb-bilibili-metadata: Bilibili video/article/up主 metadata extractor - kb-zhihu-metadata: Zhihu question/answer/user metadata extractor - kb-bilibili-content: Bilibili video download and content extraction - kb-ocr-processor: OCR text extraction from video frames - kb-deep-analyzer: AI-powered deep content analysis - kb-workflow: Workflow orchestration (inbox, archive, subscriptions, report) Also add: - ARCHITECTURE_FLOWCHART.md: System architecture diagrams - DETAILED_DESIGN.md: Complete design documentation
24 KiB
24 KiB
知识库维护系统 - 详细设计
版本: 1.0.0
状态: 草稿
目录
- 技能接口总览
- 技能1-link-parser
- 技能2-bilibili-metadata
- 技能2-zhihu-metadata
- 技能3-bilibili-content
- 技能3-ocr-processor
- 技能4-deep-analyzer
- 技能5-kb-workflow
- 数据格式定义
- 配置文件规范
- 扩展指南
1. 技能接口总览
1.1 调用约定
所有技能遵循以下调用约定:
输入:
- 命令行参数
- stdin JSON (可选)
- 环境变量
输出:
- stdout JSON
- 0: 成功
- 1: 失败 (错误信息输出到 stderr)
状态码:
- 0: 成功
- 1: 一般错误
- 2: 参数错误
- 3: 网络错误
- 4: 权限错误
1.2 通用错误响应
{
"success": false,
"error": "错误描述",
"error_code": "ERROR_CODE",
"details": {}
}
2. 技能1: link-parser
功能: 解析链接,提取平台+类型+ID,处理短链跳转
2.1 命令行接口
# 基本用法
python link-parser.py --url "https://www.bilibili.com/video/BV1xx411c7mD"
# 支持短链
python link-parser.py --url "https://b23.tv/xxxxx"
# 批量处理
python link-parser.py --batch input.txt
# 输出文件
python link-parser.py --url "..." --output result.json
2.2 输入参数
参数:
--url, -u # 必需,单个URL
--batch, -b # 可选,批量文件(每行一个URL)
--output, -o # 可选,输出文件路径
--format # 输出格式: json/yaml (默认: json)
2.3 输出格式
成功:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "video",
"id": "BV1xx411c7mD",
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
"canonical_url": "https://www.bilibili.com/video/BV1xx411c7mD",
"short_link_resolved": false,
"title": null,
"metadata": {}
},
"metadata": {
"processed_at": "2024-01-15T10:00:00",
"parser_version": "1.0.0"
}
}
列表类型:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "up",
"id": "7770724",
"source_url": "https://space.bilibili.com/7770724",
"is_list_type": true,
"list_info": {
"total_items": 156,
"item_type": "video"
}
},
"metadata": {
"requires_user_input": true,
"input_options": {
"scope": ["subscribe_only", "new_only", "all"],
"subscription": ["daily", "weekly", "custom", "none"]
}
}
}
2.4 支持的链接格式
| 类型 | 平台 | 正则示例 | 输出 |
|---|---|---|---|
| 短链 | B站 | b23.tv/[a-zA-Z0-9]+ |
resolved_url + 解析 |
| 视频 | B站 | bilibili.com/video/BV[a-zA-Z0-9]+ |
{type: video, id: BV...} |
| 专栏 | B站 | bilibili.com/read/cv[0-9]+ |
{type: article, id: cv...} |
| 动态 | B站 | bilibili.com/opus/[0-9]+ |
{type: opus, id: ...} |
| UP主 | B站 | space.bilibili.com/[0-9]+ |
{type: up, id: ...} |
| 收藏夹 | B站 | space.bilibili.com/[0-9]+/favlist?fid=[0-9]+ |
{type: favlist, id: ...} |
| 合集 | B站 | space.bilibili.com/[0-9]+/lists/[0-9]+?type=season |
{type: playlist, id: ...} |
| YouTube | YouTube | `(youtu.be/[a-zA-Z0-9]+ | youtube.com/watch?v=[a-zA-Z0-9]+)` |
| 知乎 | 知乎 | zhihu.com/question/[0-9]+ |
{type: question, id: ...} |
| 知乎 | 知乎 | zhihu.com/answer/[0-9]+ |
{type: answer, id: ...} |
| 知乎 | 知乎 | zhihu.com/people/[a-zA-Z0-9-]+ |
{type: user, id: ...} |
| 知乎 | 知乎 | zhihu.com/collection/[0-9]+ |
{type: collection, id: ...} |
2.5 复用代码
# 复用现有代码
from shortlink_resolver import resolve_shortlink, extract_bilibili_id
3. 技能2: bilibili-metadata
功能: 反爬获取B站视频/专栏元数据
3.1 命令行接口
# 基本用法
python bilibili-metadata.py --id "BV1xx411c7mD"
# 指定输出目录
python bilibili-metadata.py --id "BV1xx" --output ./metadata/
# 批量处理
python bilibili-metadata.py --batch ids.txt --output ./
# 视频+专栏支持
python bilibili-metadata.py --id "cv43780186" --type "article"
3.2 输入参数
参数:
--id, -i # 必需,内容ID
--type, -t # 内容类型: video/article/opus (默认: video)
--output, -o # 输出目录 (默认: ./)
--batch, -b # 批量文件 (每行: id [type])
--config, -c # 配置文件路径
3.3 输出格式
视频元数据:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "video",
"id": "BV1xx411c7mD",
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
"title": "Python完整教程",
"description": "这是一个Python教程...",
"author": {
"id": 7770724,
"name": "UP主名字",
"face": "https://i0.hdslb.com/xxx.jpg"
},
"duration": 9000,
"duration_text": "2:30:00",
"publish_time": "2024-01-01T12:00:00",
"tags": ["python", "tutorial", "编程"],
"stats": {
"view": 100000,
"like": 5000,
"coin": 2000,
"favorite": 1000,
"share": 500
},
"subtitle": {
"has_subtitle": true,
"languages": ["zh-CN"],
"url": "https://api.bilibili.com/..."
},
"pic": "https://i0.hdslb.com/xxx.jpg",
"quality_score": null
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00",
"source": "bilibili_api"
}
}
专栏文章元数据:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "article",
"id": "cv43780186",
"source_url": "https://www.bilibili.com/read/cv43780186",
"title": "文章标题",
"description": "文章摘要...",
"author": {
"id": 123456,
"name": "作者名字",
"face": "https://i0.hdslb.com/xxx.jpg"
},
"publish_time": "2024-01-01T12:00:00",
"stats": {
"view": 50000,
"like": 1000,
"comment": 200
},
"word_count": 5000,
"images_count": 10,
"category": "技术"
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00",
"source": "bilibili_api"
}
}
UP主/收藏夹列表:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "up",
"id": "7770724",
"source_url": "https://space.bilibili.com/7770724",
"name": "UP主名字",
"description": "UP主介绍...",
"follower": 100000,
"following": 100,
"archive_count": 156,
"content_stats": {
"total_views": 10000000,
"total_likes": 500000
},
"subscription": null
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00"
}
}
3.4 复用代码
# 复用现有代码
from bilibili_extractor import BilibiliExtractor
4. 技能2: zhihu-metadata
功能: 获取知乎元数据
4.1 命令行接口
python zhihu-metadata.py --id "266309957" --type "question"
python zhihu-metadata.py --id "2003427891925509507" --type "answer"
4.2 输入参数
参数:
--id, -i # 必需,内容ID
--type, -t # 内容类型: question/answer/article/user/collection (默认: question)
--output, -o # 输出目录
--config, -c # 配置文件路径
4.3 输出格式
问题元数据:
{
"success": true,
"data": {
"platform": "zhihu",
"type": "question",
"id": "266309957",
"source_url": "https://www.zhihu.com/question/266309957",
"title": "问题标题",
"description": "问题描述...",
"author": {
"id": "user-xxx",
"name": "作者名字",
"url_token": "user-xxx"
},
"answer_count": 50,
"followers": 1000,
"browse_count": 50000,
"created_time": "2024-01-01T12:00:00",
"updated_time": "2024-01-15T10:00:00",
"tags": ["标签1", "标签2"]
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00"
}
}
回答元数据:
{
"success": true,
"data": {
"platform": "zhihu",
"type": "answer",
"id": "2003427891925509507",
"source_url": "https://www.zhihu.com/question/266309957/answer/2003427891925509507",
"question_id": "266309957",
"question_title": "问题标题",
"author": {
"id": "user-xxx",
"name": "作者名字",
"headline": "作者简介"
},
"vote_up": 1000,
"comment_count": 50,
"created_time": "2024-01-01T12:00:00",
"updated_time": "2024-01-15T10:00:00",
"content_summary": "回答摘要..."
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00"
}
}
4.4 复用代码
# 复用现有代码
from zhihu_extractor import ZhihuDrissionExtractor
5. 技能3: bilibili-content
功能: 下载低分视频 + 提取字幕/专栏全文
5.1 命令行接口
# 基本用法(视频)
python bilibili-content.py --id "BV1xx411c7mD" --output ./content/
# 视频+字幕
python bilibili-content.py --id "BV1xx" --output ./ --with-subtitle
# 专栏全文
python bilibili-content.py --id "cv43780186" --type article --output ./
# 低分辨率(用于OCR)
python bilibili-content.py --id "BV1xx" --quality 480 --output ./
5.2 输入参数
参数:
--id, -i # 必需,内容ID
--type, -t # 内容类型: video/article/opus (默认: video)
--output, -o # 输出目录 (默认: ./)
--quality # 视频质量: 360/480/720/1080 (默认: 480)
--with-subtitle # 是否提取字幕 (默认: true)
--subtitle-lang # 字幕语言 (默认: zh-CN)
--format # 输出格式: json/ndjson (默认: json)
5.3 输出格式
视频内容:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "video",
"id": "BV1xx411c7mD",
"video_file": {
"path": "content/BV1xx_video.mp4",
"size_bytes": 52428800,
"duration": 9000,
"quality": "480p"
},
"subtitle": {
"format": "srt",
"path": "content/BV1xx_subtitle.srt",
"language": "zh-CN",
"text_length": 50000,
"lines_count": 1500
},
"danmaku": {
"path": "content/BV1xx_danmaku.xml",
"count": 5000
},
"frames": {
"output_dir": "content/BV1xx_frames",
"count": 900,
"fps": 0.1
}
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00",
"content_quality": "preview"
}
}
专栏内容:
{
"success": true,
"data": {
"platform": "bilibili",
"type": "article",
"id": "cv43780186",
"content": {
"markdown": "# 文章标题\n\n内容...",
"markdown_path": "content/cv43780186.md",
"html": "<html>...</html>",
"word_count": 5000,
"images": [
{
"url": "https://...",
"path": "content/images/img_001.jpg",
"description": "图片描述"
}
]
},
"author": {
"name": "作者名字"
},
"publish_time": "2024-01-01T12:00:00"
},
"metadata": {
"fetched_at": "2024-01-15T10:00:00"
}
}
5.4 复用代码
# 复用现有代码
from video_stream_extractor import VideoStreamExtractor
from bilibili_extractor import BilibiliExtractor
6. 技能3: ocr-processor
功能: 通用字幕OCR(被技能3调用)
6.1 命令行接口
# 基本用法
python ocr-processor.py --input ./frames/ --output ./ocr.json
# 指定引擎
python ocr-processor.py --input ./frames/ --engine paddleocr --output ./
# 只处理部分帧
python ocr-processor.py --input ./frames/ --start 0 --end 100 --output ./
6.2 输入参数
参数:
--input, -i # 必需,帧图片目录或单个图片
--output, -o # 输出文件 (默认: ocr.json)
--engine # OCR引擎: paddleocr/tesseract (默认: paddleocr)
--region # 字幕区域: bottom/full/auto (默认: bottom)
--fps # 抽帧频率 (默认: 0.1,即每10秒1帧)
--start # 开始帧号 (默认: 0)
--end # 结束帧号 (默认: -1,即全部)
--confidence # 置信度阈值 (默认: 0.7)
6.3 输出格式
{
"success": true,
"data": {
"frames": {
"frame_0001.jpg": {
"texts": [
{
"text": "字幕文本",
"confidence": 0.95,
"bbox": [x1, y1, x2, y2]
}
],
"timestamp": 0.0
}
},
"merged_text": "合并的字幕文本...",
"statistics": {
"total_frames": 100,
"frames_with_text": 80,
"total_texts": 500,
"unique_texts": 200
}
},
"metadata": {
"engine": "paddleocr",
"processed_at": "2024-01-15T10:00:00"
}
}
6.4 复用代码
# 复用现有代码
from ocr_processor import OCRProcessor
7. 技能4: deep-analyzer
功能: AI深度分析,按内容类型使用不同模板
7.1 命令行接口
# 基本用法
python deep-analyzer.py --metadata meta.json --content content.json --output ./analysis.md
# 指定类型
python deep-analyzer.py --metadata meta.json --type tutorial --output ./
# 使用自定义模板
python deep-analyzer.py --metadata meta.json --template my_template.md --output ./
# 带OCR结果
python deep-analyzer.py --metadata meta.json --content content.json --ocr ocr.json --output ./
7.2 输入参数
参数:
--metadata, -m # 必需,元数据JSON文件
--content, -c # 必需,内容JSON文件
--output, -o # 必需,输出Markdown文件
--type, -t # 内容类型: auto/tutorial/documentation/news (默认: auto)
--template # 自定义模板路径
--ocr # OCR结果JSON文件 (可选)
--model # AI模型 (默认: 配置中指定)
--tags # 预设标签 (可选)
7.3 模板变量
通用变量:
{{ title }} # 标题
{{ source }} # 来源 (bilibili/zhihu)
{{ url }} # 原始URL
{{ author }} # 作者/UP主
{{ duration }} # 时长
{{ date }} # 发布日期
{{ tags }} # 标签列表
{{ summary }} # 详细摘要
{{ key_points }} # 核心要点
{{ quality_score }} # 质量评分 (1-100)
教程视频额外变量:
{{ prerequisites }} # 前置知识
{{ outline }} # 分段大纲
{{ pitfalls }} # 常见坑点
{{ practice_advice }} # 实践建议
7.4 输出格式
教程类输出:
---
title: "Python完整教程"
source: "bilibili"
url: "https://..."
author: "CodeMaster"
duration: "2:30:00"
date: "2024-01-01T00:00:00"
tags:
- python
- tutorial
- 编程入门
quality_score: 95
---
## 🎯 一句话总结
本教程系统讲解Python基础语法,适合零基础学习者。
## 📚 前置知识
- 无需编程基础
- 电脑基本操作能力
## 📝 步骤详解
### 环境配置 [00:00-15:00]
首先下载Python...

```bash
# 安装Python
pip install python
基础语法 [15:00-45:00]
...
💡 关键要点
- Python使用缩进定义代码块
- 变量不需要声明类型
- ...
⚠️ 常见坑点
- Python 2 与 3 的区别
- 缩进问题导致报错
🤔 实践建议
- 跟随视频动手编码
- 完成课后练习
📚 相关资源
### 7.5 复用代码
```python
# 复用现有代码
from deep_analyzer import DeepAnalyzer
from ai_client import AIClient
8. 技能5: kb-workflow
功能: 工作流编排 + OpenClaw cron定时 + 手动触发 + 归档管理
8.1 命令行接口
# 手动触发处理
python kb-workflow.py --trigger manual
# 定时任务触发
python kb-workflow.py --trigger scheduled
# 只处理待处理列表
python kb-workflow.py --trigger manual --scope pending
# 强制刷新订阅
python kb-workflow.py --trigger manual --refresh-subscriptions
# 生成汇报
python kb-workflow.py --report
# 查看状态
python kb-workflow.py --status
8.2 输入参数
参数:
--trigger, -t # 触发方式: manual/scheduled (默认: manual)
--scope # 处理范围: all/pending/metadata/content (默认: all)
--limit # 最大处理数量 (默认: 5)
--refresh-subs # 刷新订阅
--report # 生成汇报
--status # 显示状态
--config, -c # 配置文件路径
8.3 汇报文件格式
{
"run_id": "run_20240115_030000",
"run_time": "2024-01-15T03:00:00",
"trigger": "scheduled",
"scope": "all",
"processed": 5,
"success": 4,
"failed": 1,
"details": {
"total_inbox": 12,
"pending": 2,
"metadata_ready": 3,
"content_ready": 5,
"analyzed": 2,
"archived": 10
},
"items": [
{
"id": "BV1xx411c7mD",
"type": "video",
"status": "success",
"duration_seconds": 45
},
{
"id": "BV2yy522c8nE",
"type": "video",
"status": "failed",
"error": "download timeout"
}
],
"subscriptions": {
"checked": 3,
"updates_found": 5,
"new_items_added": 2
},
"storage": {
"inbox_size_mb": 128,
"archive_size_mb": 512
},
"generated_at": "2024-01-15T03:05:00"
}
8.4 工作流步骤
def run_workflow(trigger, scope, limit):
# 1. 扫描收件箱
pending_items = scan_pending()
metadata_items = scan_metadata()
content_items = scan_content()
# 2. 检查订阅更新
if trigger == "scheduled":
check_subscriptions()
# 3. 按优先级处理
all_items = pending_items + metadata_items + content_items
for item in all_items[:limit]:
try:
if item.status == "pending":
metadata = run_skill2(item)
update_item(item, metadata, status="metadata_ready")
elif item.status == "metadata_ready":
content = run_skill3(item)
update_item(item, content, status="content_ready")
elif item.status == "content_ready":
analysis = run_skill4(item)
archive_item(item, analysis)
update_item(item, status="archived")
except Exception as e:
record_failure(item, str(e))
# 4. 生成汇报
generate_report()
8.5 复用代码
# 复用现有代码
from pipeline import Pipeline
from git_sync import GitSync
9. 数据格式定义
9.1 收件箱目录结构
inbox/
├── pending/
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
├── metadata/
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
├── content/
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
└── analyzed/
└── bilibili_video_BV1xx_2024-01-15T100000.json
9.2 元数据文件格式
{
"platform": "bilibili",
"type": "video",
"id": "BV1xx411c7mD",
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
"status": "pending",
"scope": "all",
"title": "Python完整教程",
"author": {"id": 7770724, "name": "UP主名字"},
"duration": 9000,
"publish_time": "2024-01-01T12:00:00",
"tags": ["python", "tutorial"],
"stats": {"view": 100000, "like": 5000},
"subscription": {
"enabled": true,
"scope": "new_only",
"frequency": "daily",
"last_checked": "2024-01-15T00:00:00",
"items_count": 156,
"processed_count": 45
},
"created_at": "2024-01-15T00:00:00",
"updated_at": "2024-01-15T10:00:00",
"retry_count": 0
}
9.3 归档目录结构
archive/
├── 2024/
│ ├── 01/
│ │ ├── video/
│ │ │ ├── bilibili_BV1xx_2024-01-15.md
│ │ │ └── index.json
│ │ ├── article/
│ │ │ └── index.json
│ │ └── index.json
│ └── index.json
└── index.json
10. 配置文件规范
10.1 配置文件位置
优先级:
./config.yaml(当前目录)~/.config/obsidian-kb/config.yaml~/.obsidian-kb/config.yaml
10.2 完整配置示例
# AI 配置
ai:
provider: "openai"
model: "gpt-4"
api_key: "${OPENAI_API_KEY}"
temperature: 0.3
max_tokens: 8192
# 存储配置
storage:
inbox_dir: "~/work/inbox"
archive_dir: "~/ObsidianVault/knowledge-base"
temp_dir: "/tmp/obsidian-kb"
max_concurrent: 3
cleanup_after_days: 7
# 输出配置
output:
format: "markdown"
encoding: "utf-8"
include_timestamp: true
include_source: true
# Git 同步配置
git:
enabled: true
vault_path: "~/ObsidianVault"
remote: "origin"
branch: "main"
auto_commit: true
# Bilibili 提取器配置
bilibili:
enabled: true
cookie: "${BILIBILI_COOKIE}"
use_playwright: true
headless: true
timeout: 30
subtitle:
enabled: true
language: "zh-CN"
# 知乎提取器配置
zhihu:
enabled: true
use_drissionpage: true
cookie_file: "~/.obsidian-kb/zhihu_cookies.json"
# OCR 配置
ocr:
engine: "paddleocr"
confidence_threshold: 0.7
region: "bottom"
fps: 0.1
max_frames: 500
# 工作流配置
workflow:
schedule: "0 3 * * *"
max_items_per_run: 5
subscription_check_interval: 24
# 汇报配置
reporting:
enabled: true
output_dir: "~/work/reports"
format: "json"
retention_days: 30
# 日志配置
logging:
level: "INFO"
format: "[%(asctime)s] %(levelname)s - %(message)s"
file: "~/work/logs/obsidian-kb.log"
11. 扩展指南
11.1 新增平台步骤
以新增「小红书」为例:
步骤1: 创建链接解析器扩展
# link-parser 的 platform_registry 中添加
PLATFORM_REGISTRY = {
"xiaohongshu": {
"name": "小红书",
"patterns": [
r"xiaohongshu.com/notes/([a-zA-Z0-9]+)",
],
"types": ["note", "user", "collection"]
}
}
步骤2: 创建元数据获取器
# 创建文件: skills/xiaohongshu-metadata/
class XiaohongshuMetadata:
def extract(self, url, config):
# 实现获取逻辑
pass
步骤3: 更新工作流配置
xiaohongshu:
enabled: true
cookie: "${XIAOHONGSHU_COOKIE}"
11.2 新增分析模板
# templates/podcast.md
---
title: "{{ title }}"
source: "{{ source }}"
url: "{{ url }}"
author: "{{ author }}"
duration: "{{ duration }}"
date: "{{ date }}"
tags:
{{- for tag in tags }}
- {{ tag }}
{{- endfor }}
---
## 🎧 播客摘要
{{ summary }}
## 💡 核心要点
{{ key_points }}
附录A: 技能清单
| # | 技能名 | 主要文件 | 复用代码 |
|---|---|---|---|
| 1 | link-parser | link-parser.py |
shortlink_resolver.py |
| 2 | bilibili-metadata | bilibili_metadata.py |
bilibili_extractor.py |
| 2 | zhihu-metadata | zhihu_metadata.py |
zhihu_extractor.py |
| 3 | bilibili-content | bilibili_content.py |
video_stream_extractor.py |
| 3 | ocr-processor | ocr_processor.py |
ocr_processor.py |
| 4 | deep-analyzer | deep_analyzer.py |
deep_analyzer.py |
| 5 | kb-workflow | kb_workflow.py |
pipeline.py, git_sync.py |
附录B: OpenClaw Cron 配置示例
# 定时任务配置
schedule: "0 3 * * *"
command: "python /path/to/skills/kb-workflow/kb_workflow.py --trigger scheduled"
蒋先生,详细设计文档已完成!
包含内容:
- ✅ 7个技能的完整接口规范(命令行参数、输入输出格式)
- ✅ 数据格式详细定义(收件箱、归档、汇报)
- ✅ 配置文件规范
- ✅ 复用代码对照表
- ✅ 扩展指南(新增平台/模板步骤)
下一步: 可以进入 write-plan 阶段,生成具体实现计划吗?