OpenClaw Agent
5507b1d986
- kb-link-parser: Multi-platform URL parser (Bilibili, YouTube, Zhihu) - kb-bilibili-metadata: Bilibili video/article/up主 metadata extractor - kb-zhihu-metadata: Zhihu question/answer/user metadata extractor - kb-bilibili-content: Bilibili video download and content extraction - kb-ocr-processor: OCR text extraction from video frames - kb-deep-analyzer: AI-powered deep content analysis - kb-workflow: Workflow orchestration (inbox, archive, subscriptions, report) Also add: - ARCHITECTURE_FLOWCHART.md: System architecture diagrams - DETAILED_DESIGN.md: Complete design documentation
1108 lines
24 KiB
Markdown
1108 lines
24 KiB
Markdown
# 知识库维护系统 - 详细设计
|
||
|
||
> 版本: 1.0.0
|
||
> 状态: 草稿
|
||
|
||
---
|
||
|
||
## 目录
|
||
|
||
1. [技能接口总览](#1-技能接口总览)
|
||
2. [技能1-link-parser](#2-技能1-link-parser)
|
||
3. [技能2-bilibili-metadata](#3-技能2-bilibili-metadata)
|
||
4. [技能2-zhihu-metadata](#4-技能2-zhihu-metadata)
|
||
5. [技能3-bilibili-content](#5-技能3-bilibili-content)
|
||
6. [技能3-ocr-processor](#6-技能3-ocr-processor)
|
||
7. [技能4-deep-analyzer](#7-技能4-deep-analyzer)
|
||
8. [技能5-kb-workflow](#8-技能5-kb-workflow)
|
||
9. [数据格式定义](#9-数据格式定义)
|
||
10. [配置文件规范](#10-配置文件规范)
|
||
11. [扩展指南](#11-扩展指南)
|
||
|
||
---
|
||
|
||
## 1. 技能接口总览
|
||
|
||
### 1.1 调用约定
|
||
|
||
所有技能遵循以下调用约定：
|
||
|
||
```yaml
|
||
输入:
|
||
- 命令行参数
|
||
- stdin JSON (可选)
|
||
- 环境变量
|
||
|
||
输出:
|
||
- stdout JSON
|
||
- 0: 成功
|
||
- 1: 失败 (错误信息输出到 stderr)
|
||
|
||
状态码:
|
||
- 0: 成功
|
||
- 1: 一般错误
|
||
- 2: 参数错误
|
||
- 3: 网络错误
|
||
- 4: 权限错误
|
||
```
|
||
|
||
### 1.2 通用错误响应
|
||
|
||
```json
|
||
{
|
||
"success": false,
|
||
"error": "错误描述",
|
||
"error_code": "ERROR_CODE",
|
||
"details": {}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 2. 技能1: link-parser
|
||
|
||
**功能**: 解析链接，提取平台+类型+ID，处理短链跳转
|
||
|
||
### 2.1 命令行接口
|
||
|
||
```bash
|
||
# 基本用法
|
||
python link-parser.py --url "https://www.bilibili.com/video/BV1xx411c7mD"
|
||
|
||
# 支持短链
|
||
python link-parser.py --url "https://b23.tv/xxxxx"
|
||
|
||
# 批量处理
|
||
python link-parser.py --batch input.txt
|
||
|
||
# 输出文件
|
||
python link-parser.py --url "..." --output result.json
|
||
```
|
||
|
||
### 2.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--url, -u # 必需，单个URL
|
||
--batch, -b # 可选，批量文件（每行一个URL）
|
||
--output, -o # 可选，输出文件路径
|
||
--format # 输出格式: json/yaml (默认: json)
|
||
```
|
||
|
||
### 2.3 输出格式
|
||
|
||
**成功**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "video",
|
||
"id": "BV1xx411c7mD",
|
||
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
|
||
"canonical_url": "https://www.bilibili.com/video/BV1xx411c7mD",
|
||
"short_link_resolved": false,
|
||
"title": null,
|
||
"metadata": {}
|
||
},
|
||
"metadata": {
|
||
"processed_at": "2024-01-15T10:00:00",
|
||
"parser_version": "1.0.0"
|
||
}
|
||
}
|
||
```
|
||
|
||
**列表类型**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "up",
|
||
"id": "7770724",
|
||
"source_url": "https://space.bilibili.com/7770724",
|
||
"is_list_type": true,
|
||
"list_info": {
|
||
"total_items": 156,
|
||
"item_type": "video"
|
||
}
|
||
},
|
||
"metadata": {
|
||
"requires_user_input": true,
|
||
"input_options": {
|
||
"scope": ["subscribe_only", "new_only", "all"],
|
||
"subscription": ["daily", "weekly", "custom", "none"]
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 2.4 支持的链接格式
|
||
|
||
| 类型 | 平台 | 正则示例 | 输出 |
|
||
|------|------|----------|------|
|
||
| 短链 | B站 | `b23.tv/[a-zA-Z0-9]+` | resolved_url + 解析 |
|
||
| 视频 | B站 | `bilibili.com/video/BV[a-zA-Z0-9]+` | `{type: video, id: BV...}` |
|
||
| 专栏 | B站 | `bilibili.com/read/cv[0-9]+` | `{type: article, id: cv...}` |
|
||
| 动态 | B站 | `bilibili.com/opus/[0-9]+` | `{type: opus, id: ...}` |
|
||
| UP主 | B站 | `space.bilibili.com/[0-9]+` | `{type: up, id: ...}` |
|
||
| 收藏夹 | B站 | `space.bilibili.com/[0-9]+/favlist?fid=[0-9]+` | `{type: favlist, id: ...}` |
|
||
| 合集 | B站 | `space.bilibili.com/[0-9]+/lists/[0-9]+?type=season` | `{type: playlist, id: ...}` |
|
||
| YouTube | YouTube | `(youtu.be/[a-zA-Z0-9]+|youtube.com/watch?v=[a-zA-Z0-9]+)` | `{type: video, id: ...}` |
|
||
| 知乎 | 知乎 | `zhihu.com/question/[0-9]+` | `{type: question, id: ...}` |
|
||
| 知乎 | 知乎 | `zhihu.com/answer/[0-9]+` | `{type: answer, id: ...}` |
|
||
| 知乎 | 知乎 | `zhihu.com/people/[a-zA-Z0-9-]+` | `{type: user, id: ...}` |
|
||
| 知乎 | 知乎 | `zhihu.com/collection/[0-9]+` | `{type: collection, id: ...}` |
|
||
|
||
### 2.5 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from shortlink_resolver import resolve_shortlink, extract_bilibili_id
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 技能2: bilibili-metadata
|
||
|
||
**功能**: 反爬获取B站视频/专栏元数据
|
||
|
||
### 3.1 命令行接口
|
||
|
||
```bash
|
||
# 基本用法
|
||
python bilibili-metadata.py --id "BV1xx411c7mD"
|
||
|
||
# 指定输出目录
|
||
python bilibili-metadata.py --id "BV1xx" --output ./metadata/
|
||
|
||
# 批量处理
|
||
python bilibili-metadata.py --batch ids.txt --output ./
|
||
|
||
# 视频+专栏支持
|
||
python bilibili-metadata.py --id "cv43780186" --type "article"
|
||
```
|
||
|
||
### 3.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--id, -i # 必需，内容ID
|
||
--type, -t # 内容类型: video/article/opus (默认: video)
|
||
--output, -o # 输出目录 (默认: ./)
|
||
--batch, -b # 批量文件 (每行: id [type])
|
||
--config, -c # 配置文件路径
|
||
```
|
||
|
||
### 3.3 输出格式
|
||
|
||
**视频元数据**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "video",
|
||
"id": "BV1xx411c7mD",
|
||
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
|
||
"title": "Python完整教程",
|
||
"description": "这是一个Python教程...",
|
||
"author": {
|
||
"id": 7770724,
|
||
"name": "UP主名字",
|
||
"face": "https://i0.hdslb.com/xxx.jpg"
|
||
},
|
||
"duration": 9000,
|
||
"duration_text": "2:30:00",
|
||
"publish_time": "2024-01-01T12:00:00",
|
||
"tags": ["python", "tutorial", "编程"],
|
||
"stats": {
|
||
"view": 100000,
|
||
"like": 5000,
|
||
"coin": 2000,
|
||
"favorite": 1000,
|
||
"share": 500
|
||
},
|
||
"subtitle": {
|
||
"has_subtitle": true,
|
||
"languages": ["zh-CN"],
|
||
"url": "https://api.bilibili.com/..."
|
||
},
|
||
"pic": "https://i0.hdslb.com/xxx.jpg",
|
||
"quality_score": null
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00",
|
||
"source": "bilibili_api"
|
||
}
|
||
}
|
||
```
|
||
|
||
**专栏文章元数据**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "article",
|
||
"id": "cv43780186",
|
||
"source_url": "https://www.bilibili.com/read/cv43780186",
|
||
"title": "文章标题",
|
||
"description": "文章摘要...",
|
||
"author": {
|
||
"id": 123456,
|
||
"name": "作者名字",
|
||
"face": "https://i0.hdslb.com/xxx.jpg"
|
||
},
|
||
"publish_time": "2024-01-01T12:00:00",
|
||
"stats": {
|
||
"view": 50000,
|
||
"like": 1000,
|
||
"comment": 200
|
||
},
|
||
"word_count": 5000,
|
||
"images_count": 10,
|
||
"category": "技术"
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00",
|
||
"source": "bilibili_api"
|
||
}
|
||
}
|
||
```
|
||
|
||
**UP主/收藏夹列表**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "up",
|
||
"id": "7770724",
|
||
"source_url": "https://space.bilibili.com/7770724",
|
||
"name": "UP主名字",
|
||
"description": "UP主介绍...",
|
||
"follower": 100000,
|
||
"following": 100,
|
||
"archive_count": 156,
|
||
"content_stats": {
|
||
"total_views": 10000000,
|
||
"total_likes": 500000
|
||
},
|
||
"subscription": null
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3.4 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from bilibili_extractor import BilibiliExtractor
|
||
```
|
||
|
||
---
|
||
|
||
## 4. 技能2: zhihu-metadata
|
||
|
||
**功能**: 获取知乎元数据
|
||
|
||
### 4.1 命令行接口
|
||
|
||
```bash
|
||
python zhihu-metadata.py --id "266309957" --type "question"
|
||
python zhihu-metadata.py --id "2003427891925509507" --type "answer"
|
||
```
|
||
|
||
### 4.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--id, -i # 必需，内容ID
|
||
--type, -t # 内容类型: question/answer/article/user/collection (默认: question)
|
||
--output, -o # 输出目录
|
||
--config, -c # 配置文件路径
|
||
```
|
||
|
||
### 4.3 输出格式
|
||
|
||
**问题元数据**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "zhihu",
|
||
"type": "question",
|
||
"id": "266309957",
|
||
"source_url": "https://www.zhihu.com/question/266309957",
|
||
"title": "问题标题",
|
||
"description": "问题描述...",
|
||
"author": {
|
||
"id": "user-xxx",
|
||
"name": "作者名字",
|
||
"url_token": "user-xxx"
|
||
},
|
||
"answer_count": 50,
|
||
"followers": 1000,
|
||
"browse_count": 50000,
|
||
"created_time": "2024-01-01T12:00:00",
|
||
"updated_time": "2024-01-15T10:00:00",
|
||
"tags": ["标签1", "标签2"]
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
**回答元数据**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "zhihu",
|
||
"type": "answer",
|
||
"id": "2003427891925509507",
|
||
"source_url": "https://www.zhihu.com/question/266309957/answer/2003427891925509507",
|
||
"question_id": "266309957",
|
||
"question_title": "问题标题",
|
||
"author": {
|
||
"id": "user-xxx",
|
||
"name": "作者名字",
|
||
"headline": "作者简介"
|
||
},
|
||
"vote_up": 1000,
|
||
"comment_count": 50,
|
||
"created_time": "2024-01-01T12:00:00",
|
||
"updated_time": "2024-01-15T10:00:00",
|
||
"content_summary": "回答摘要..."
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4.4 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from zhihu_extractor import ZhihuDrissionExtractor
|
||
```
|
||
|
||
---
|
||
|
||
## 5. 技能3: bilibili-content
|
||
|
||
**功能**: 下载低分视频 + 提取字幕/专栏全文
|
||
|
||
### 5.1 命令行接口
|
||
|
||
```bash
|
||
# 基本用法（视频）
|
||
python bilibili-content.py --id "BV1xx411c7mD" --output ./content/
|
||
|
||
# 视频+字幕
|
||
python bilibili-content.py --id "BV1xx" --output ./ --with-subtitle
|
||
|
||
# 专栏全文
|
||
python bilibili-content.py --id "cv43780186" --type article --output ./
|
||
|
||
# 低分辨率（用于OCR）
|
||
python bilibili-content.py --id "BV1xx" --quality 480 --output ./
|
||
```
|
||
|
||
### 5.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--id, -i # 必需，内容ID
|
||
--type, -t # 内容类型: video/article/opus (默认: video)
|
||
--output, -o # 输出目录 (默认: ./)
|
||
--quality # 视频质量: 360/480/720/1080 (默认: 480)
|
||
--with-subtitle # 是否提取字幕 (默认: true)
|
||
--subtitle-lang # 字幕语言 (默认: zh-CN)
|
||
--format # 输出格式: json/ndjson (默认: json)
|
||
```
|
||
|
||
### 5.3 输出格式
|
||
|
||
**视频内容**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "video",
|
||
"id": "BV1xx411c7mD",
|
||
"video_file": {
|
||
"path": "content/BV1xx_video.mp4",
|
||
"size_bytes": 52428800,
|
||
"duration": 9000,
|
||
"quality": "480p"
|
||
},
|
||
"subtitle": {
|
||
"format": "srt",
|
||
"path": "content/BV1xx_subtitle.srt",
|
||
"language": "zh-CN",
|
||
"text_length": 50000,
|
||
"lines_count": 1500
|
||
},
|
||
"danmaku": {
|
||
"path": "content/BV1xx_danmaku.xml",
|
||
"count": 5000
|
||
},
|
||
"frames": {
|
||
"output_dir": "content/BV1xx_frames",
|
||
"count": 900,
|
||
"fps": 0.1
|
||
}
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00",
|
||
"content_quality": "preview"
|
||
}
|
||
}
|
||
```
|
||
|
||
**专栏内容**:
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"platform": "bilibili",
|
||
"type": "article",
|
||
"id": "cv43780186",
|
||
"content": {
|
||
"markdown": "# 文章标题\n\n内容...",
|
||
"markdown_path": "content/cv43780186.md",
|
||
"html": "<html>...</html>",
|
||
"word_count": 5000,
|
||
"images": [
|
||
{
|
||
"url": "https://...",
|
||
"path": "content/images/img_001.jpg",
|
||
"description": "图片描述"
|
||
}
|
||
]
|
||
},
|
||
"author": {
|
||
"name": "作者名字"
|
||
},
|
||
"publish_time": "2024-01-01T12:00:00"
|
||
},
|
||
"metadata": {
|
||
"fetched_at": "2024-01-15T10:00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.4 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from video_stream_extractor import VideoStreamExtractor
|
||
from bilibili_extractor import BilibiliExtractor
|
||
```
|
||
|
||
---
|
||
|
||
## 6. 技能3: ocr-processor
|
||
|
||
**功能**: 通用字幕OCR（被技能3调用）
|
||
|
||
### 6.1 命令行接口
|
||
|
||
```bash
|
||
# 基本用法
|
||
python ocr-processor.py --input ./frames/ --output ./ocr.json
|
||
|
||
# 指定引擎
|
||
python ocr-processor.py --input ./frames/ --engine paddleocr --output ./
|
||
|
||
# 只处理部分帧
|
||
python ocr-processor.py --input ./frames/ --start 0 --end 100 --output ./
|
||
```
|
||
|
||
### 6.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--input, -i # 必需，帧图片目录或单个图片
|
||
--output, -o # 输出文件 (默认: ocr.json)
|
||
--engine # OCR引擎: paddleocr/tesseract (默认: paddleocr)
|
||
--region # 字幕区域: bottom/full/auto (默认: bottom)
|
||
--fps # 抽帧频率 (默认: 0.1，即每10秒1帧)
|
||
--start # 开始帧号 (默认: 0)
|
||
--end # 结束帧号 (默认: -1，即全部)
|
||
--confidence # 置信度阈值 (默认: 0.7)
|
||
```
|
||
|
||
### 6.3 输出格式
|
||
|
||
```json
|
||
{
|
||
"success": true,
|
||
"data": {
|
||
"frames": {
|
||
"frame_0001.jpg": {
|
||
"texts": [
|
||
{
|
||
"text": "字幕文本",
|
||
"confidence": 0.95,
|
||
"bbox": [x1, y1, x2, y2]
|
||
}
|
||
],
|
||
"timestamp": 0.0
|
||
}
|
||
},
|
||
"merged_text": "合并的字幕文本...",
|
||
"statistics": {
|
||
"total_frames": 100,
|
||
"frames_with_text": 80,
|
||
"total_texts": 500,
|
||
"unique_texts": 200
|
||
}
|
||
},
|
||
"metadata": {
|
||
"engine": "paddleocr",
|
||
"processed_at": "2024-01-15T10:00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 6.4 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from ocr_processor import OCRProcessor
|
||
```
|
||
|
||
---
|
||
|
||
## 7. 技能4: deep-analyzer
|
||
|
||
**功能**: AI深度分析，按内容类型使用不同模板
|
||
|
||
### 7.1 命令行接口
|
||
|
||
```bash
|
||
# 基本用法
|
||
python deep-analyzer.py --metadata meta.json --content content.json --output ./analysis.md
|
||
|
||
# 指定类型
|
||
python deep-analyzer.py --metadata meta.json --type tutorial --output ./
|
||
|
||
# 使用自定义模板
|
||
python deep-analyzer.py --metadata meta.json --template my_template.md --output ./
|
||
|
||
# 带OCR结果
|
||
python deep-analyzer.py --metadata meta.json --content content.json --ocr ocr.json --output ./
|
||
```
|
||
|
||
### 7.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--metadata, -m # 必需，元数据JSON文件
|
||
--content, -c # 必需，内容JSON文件
|
||
--output, -o # 必需，输出Markdown文件
|
||
--type, -t # 内容类型: auto/tutorial/documentation/news (默认: auto)
|
||
--template # 自定义模板路径
|
||
--ocr # OCR结果JSON文件 (可选)
|
||
--model # AI模型 (默认: 配置中指定)
|
||
--tags # 预设标签 (可选)
|
||
```
|
||
|
||
### 7.3 模板变量
|
||
|
||
**通用变量**:
|
||
```jinja2
|
||
{{ title }} # 标题
|
||
{{ source }} # 来源 (bilibili/zhihu)
|
||
{{ url }} # 原始URL
|
||
{{ author }} # 作者/UP主
|
||
{{ duration }} # 时长
|
||
{{ date }} # 发布日期
|
||
{{ tags }} # 标签列表
|
||
{{ summary }} # 详细摘要
|
||
{{ key_points }} # 核心要点
|
||
{{ quality_score }} # 质量评分 (1-100)
|
||
```
|
||
|
||
**教程视频额外变量**:
|
||
```jinja2
|
||
{{ prerequisites }} # 前置知识
|
||
{{ outline }} # 分段大纲
|
||
{{ pitfalls }} # 常见坑点
|
||
{{ practice_advice }} # 实践建议
|
||
```
|
||
|
||
### 7.4 输出格式
|
||
|
||
**教程类输出**:
|
||
```markdown
|
||
---
|
||
title: "Python完整教程"
|
||
source: "bilibili"
|
||
url: "https://..."
|
||
author: "CodeMaster"
|
||
duration: "2:30:00"
|
||
date: "2024-01-01T00:00:00"
|
||
tags:
|
||
- python
|
||
- tutorial
|
||
- 编程入门
|
||
quality_score: 95
|
||
---
|
||
|
||
## 🎯 一句话总结
|
||
|
||
本教程系统讲解Python基础语法，适合零基础学习者。
|
||
|
||
## 📚 前置知识
|
||
|
||
- 无需编程基础
|
||
- 电脑基本操作能力
|
||
|
||
## 📝 步骤详解
|
||
|
||
### 环境配置 [00:00-15:00]
|
||
|
||
首先下载Python...
|
||
|
||
|
||
|
||
```bash
|
||
# 安装Python
|
||
pip install python
|
||
```
|
||
|
||
### 基础语法 [15:00-45:00]
|
||
|
||
...
|
||
|
||
## 💡 关键要点
|
||
|
||
1. Python使用缩进定义代码块
|
||
2. 变量不需要声明类型
|
||
3. ...
|
||
|
||
## ⚠️ 常见坑点
|
||
|
||
- Python 2 与 3 的区别
|
||
- 缩进问题导致报错
|
||
|
||
## 🤔 实践建议
|
||
|
||
- 跟随视频动手编码
|
||
- 完成课后练习
|
||
|
||
## 📚 相关资源
|
||
|
||
- [Python官网](https://www.python.org/)
|
||
- [官方文档](https://docs.python.org/)
|
||
```
|
||
|
||
### 7.5 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from deep_analyzer import DeepAnalyzer
|
||
from ai_client import AIClient
|
||
```
|
||
|
||
---
|
||
|
||
## 8. 技能5: kb-workflow
|
||
|
||
**功能**: 工作流编排 + OpenClaw cron定时 + 手动触发 + 归档管理
|
||
|
||
### 8.1 命令行接口
|
||
|
||
```bash
|
||
# 手动触发处理
|
||
python kb-workflow.py --trigger manual
|
||
|
||
# 定时任务触发
|
||
python kb-workflow.py --trigger scheduled
|
||
|
||
# 只处理待处理列表
|
||
python kb-workflow.py --trigger manual --scope pending
|
||
|
||
# 强制刷新订阅
|
||
python kb-workflow.py --trigger manual --refresh-subscriptions
|
||
|
||
# 生成汇报
|
||
python kb-workflow.py --report
|
||
|
||
# 查看状态
|
||
python kb-workflow.py --status
|
||
```
|
||
|
||
### 8.2 输入参数
|
||
|
||
```yaml
|
||
参数:
|
||
--trigger, -t # 触发方式: manual/scheduled (默认: manual)
|
||
--scope # 处理范围: all/pending/metadata/content (默认: all)
|
||
--limit # 最大处理数量 (默认: 5)
|
||
--refresh-subs # 刷新订阅
|
||
--report # 生成汇报
|
||
--status # 显示状态
|
||
--config, -c # 配置文件路径
|
||
```
|
||
|
||
### 8.3 汇报文件格式
|
||
|
||
```json
|
||
{
|
||
"run_id": "run_20240115_030000",
|
||
"run_time": "2024-01-15T03:00:00",
|
||
"trigger": "scheduled",
|
||
"scope": "all",
|
||
"processed": 5,
|
||
"success": 4,
|
||
"failed": 1,
|
||
"details": {
|
||
"total_inbox": 12,
|
||
"pending": 2,
|
||
"metadata_ready": 3,
|
||
"content_ready": 5,
|
||
"analyzed": 2,
|
||
"archived": 10
|
||
},
|
||
"items": [
|
||
{
|
||
"id": "BV1xx411c7mD",
|
||
"type": "video",
|
||
"status": "success",
|
||
"duration_seconds": 45
|
||
},
|
||
{
|
||
"id": "BV2yy522c8nE",
|
||
"type": "video",
|
||
"status": "failed",
|
||
"error": "download timeout"
|
||
}
|
||
],
|
||
"subscriptions": {
|
||
"checked": 3,
|
||
"updates_found": 5,
|
||
"new_items_added": 2
|
||
},
|
||
"storage": {
|
||
"inbox_size_mb": 128,
|
||
"archive_size_mb": 512
|
||
},
|
||
"generated_at": "2024-01-15T03:05:00"
|
||
}
|
||
```
|
||
|
||
### 8.4 工作流步骤
|
||
|
||
```python
|
||
def run_workflow(trigger, scope, limit):
|
||
# 1. 扫描收件箱
|
||
pending_items = scan_pending()
|
||
metadata_items = scan_metadata()
|
||
content_items = scan_content()
|
||
|
||
# 2. 检查订阅更新
|
||
if trigger == "scheduled":
|
||
check_subscriptions()
|
||
|
||
# 3. 按优先级处理
|
||
all_items = pending_items + metadata_items + content_items
|
||
|
||
for item in all_items[:limit]:
|
||
try:
|
||
if item.status == "pending":
|
||
metadata = run_skill2(item)
|
||
update_item(item, metadata, status="metadata_ready")
|
||
elif item.status == "metadata_ready":
|
||
content = run_skill3(item)
|
||
update_item(item, content, status="content_ready")
|
||
elif item.status == "content_ready":
|
||
analysis = run_skill4(item)
|
||
archive_item(item, analysis)
|
||
update_item(item, status="archived")
|
||
except Exception as e:
|
||
record_failure(item, str(e))
|
||
|
||
# 4. 生成汇报
|
||
generate_report()
|
||
```
|
||
|
||
### 8.5 复用代码
|
||
|
||
```python
|
||
# 复用现有代码
|
||
from pipeline import Pipeline
|
||
from git_sync import GitSync
|
||
```
|
||
|
||
---
|
||
|
||
## 9. 数据格式定义
|
||
|
||
### 9.1 收件箱目录结构
|
||
|
||
```
|
||
inbox/
|
||
├── pending/
|
||
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
|
||
├── metadata/
|
||
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
|
||
├── content/
|
||
│ └── bilibili_video_BV1xx_2024-01-15T100000.json
|
||
└── analyzed/
|
||
└── bilibili_video_BV1xx_2024-01-15T100000.json
|
||
```
|
||
|
||
### 9.2 元数据文件格式
|
||
|
||
```json
|
||
{
|
||
"platform": "bilibili",
|
||
"type": "video",
|
||
"id": "BV1xx411c7mD",
|
||
"source_url": "https://www.bilibili.com/video/BV1xx411c7mD",
|
||
"status": "pending",
|
||
"scope": "all",
|
||
"title": "Python完整教程",
|
||
"author": {"id": 7770724, "name": "UP主名字"},
|
||
"duration": 9000,
|
||
"publish_time": "2024-01-01T12:00:00",
|
||
"tags": ["python", "tutorial"],
|
||
"stats": {"view": 100000, "like": 5000},
|
||
"subscription": {
|
||
"enabled": true,
|
||
"scope": "new_only",
|
||
"frequency": "daily",
|
||
"last_checked": "2024-01-15T00:00:00",
|
||
"items_count": 156,
|
||
"processed_count": 45
|
||
},
|
||
"created_at": "2024-01-15T00:00:00",
|
||
"updated_at": "2024-01-15T10:00:00",
|
||
"retry_count": 0
|
||
}
|
||
```
|
||
|
||
### 9.3 归档目录结构
|
||
|
||
```
|
||
archive/
|
||
├── 2024/
|
||
│ ├── 01/
|
||
│ │ ├── video/
|
||
│ │ │ ├── bilibili_BV1xx_2024-01-15.md
|
||
│ │ │ └── index.json
|
||
│ │ ├── article/
|
||
│ │ │ └── index.json
|
||
│ │ └── index.json
|
||
│ └── index.json
|
||
└── index.json
|
||
```
|
||
|
||
---
|
||
|
||
## 10. 配置文件规范
|
||
|
||
### 10.1 配置文件位置
|
||
|
||
优先级:
|
||
1. `./config.yaml` (当前目录)
|
||
2. `~/.config/obsidian-kb/config.yaml`
|
||
3. `~/.obsidian-kb/config.yaml`
|
||
|
||
### 10.2 完整配置示例
|
||
|
||
```yaml
|
||
# AI 配置
|
||
ai:
|
||
provider: "openai"
|
||
model: "gpt-4"
|
||
api_key: "${OPENAI_API_KEY}"
|
||
temperature: 0.3
|
||
max_tokens: 8192
|
||
|
||
# 存储配置
|
||
storage:
|
||
inbox_dir: "~/work/inbox"
|
||
archive_dir: "~/ObsidianVault/knowledge-base"
|
||
temp_dir: "/tmp/obsidian-kb"
|
||
max_concurrent: 3
|
||
cleanup_after_days: 7
|
||
|
||
# 输出配置
|
||
output:
|
||
format: "markdown"
|
||
encoding: "utf-8"
|
||
include_timestamp: true
|
||
include_source: true
|
||
|
||
# Git 同步配置
|
||
git:
|
||
enabled: true
|
||
vault_path: "~/ObsidianVault"
|
||
remote: "origin"
|
||
branch: "main"
|
||
auto_commit: true
|
||
|
||
# Bilibili 提取器配置
|
||
bilibili:
|
||
enabled: true
|
||
cookie: "${BILIBILI_COOKIE}"
|
||
use_playwright: true
|
||
headless: true
|
||
timeout: 30
|
||
subtitle:
|
||
enabled: true
|
||
language: "zh-CN"
|
||
|
||
# 知乎提取器配置
|
||
zhihu:
|
||
enabled: true
|
||
use_drissionpage: true
|
||
cookie_file: "~/.obsidian-kb/zhihu_cookies.json"
|
||
|
||
# OCR 配置
|
||
ocr:
|
||
engine: "paddleocr"
|
||
confidence_threshold: 0.7
|
||
region: "bottom"
|
||
fps: 0.1
|
||
max_frames: 500
|
||
|
||
# 工作流配置
|
||
workflow:
|
||
schedule: "0 3 * * *"
|
||
max_items_per_run: 5
|
||
subscription_check_interval: 24
|
||
|
||
# 汇报配置
|
||
reporting:
|
||
enabled: true
|
||
output_dir: "~/work/reports"
|
||
format: "json"
|
||
retention_days: 30
|
||
|
||
# 日志配置
|
||
logging:
|
||
level: "INFO"
|
||
format: "[%(asctime)s] %(levelname)s - %(message)s"
|
||
file: "~/work/logs/obsidian-kb.log"
|
||
```
|
||
|
||
---
|
||
|
||
## 11. 扩展指南
|
||
|
||
### 11.1 新增平台步骤
|
||
|
||
以新增「小红书」为例：
|
||
|
||
#### 步骤1: 创建链接解析器扩展
|
||
|
||
```python
|
||
# link-parser 的 platform_registry 中添加
|
||
PLATFORM_REGISTRY = {
|
||
"xiaohongshu": {
|
||
"name": "小红书",
|
||
"patterns": [
|
||
r"xiaohongshu.com/notes/([a-zA-Z0-9]+)",
|
||
],
|
||
"types": ["note", "user", "collection"]
|
||
}
|
||
}
|
||
```
|
||
|
||
#### 步骤2: 创建元数据获取器
|
||
|
||
```bash
|
||
# 创建文件: skills/xiaohongshu-metadata/
|
||
```
|
||
|
||
```python
|
||
class XiaohongshuMetadata:
|
||
def extract(self, url, config):
|
||
# 实现获取逻辑
|
||
pass
|
||
```
|
||
|
||
#### 步骤3: 更新工作流配置
|
||
|
||
```yaml
|
||
xiaohongshu:
|
||
enabled: true
|
||
cookie: "${XIAOHONGSHU_COOKIE}"
|
||
```
|
||
|
||
### 11.2 新增分析模板
|
||
|
||
```jinja2
|
||
# templates/podcast.md
|
||
---
|
||
title: "{{ title }}"
|
||
source: "{{ source }}"
|
||
url: "{{ url }}"
|
||
author: "{{ author }}"
|
||
duration: "{{ duration }}"
|
||
date: "{{ date }}"
|
||
tags:
|
||
{{- for tag in tags }}
|
||
- {{ tag }}
|
||
{{- endfor }}
|
||
---
|
||
|
||
## 🎧 播客摘要
|
||
|
||
{{ summary }}
|
||
|
||
## 💡 核心要点
|
||
|
||
{{ key_points }}
|
||
```
|
||
|
||
---
|
||
|
||
## 附录A: 技能清单
|
||
|
||
| # | 技能名 | 主要文件 | 复用代码 |
|
||
|---|--------|----------|----------|
|
||
| 1 | link-parser | `link-parser.py` | shortlink_resolver.py |
|
||
| 2 | bilibili-metadata | `bilibili_metadata.py` | bilibili_extractor.py |
|
||
| 2 | zhihu-metadata | `zhihu_metadata.py` | zhihu_extractor.py |
|
||
| 3 | bilibili-content | `bilibili_content.py` | video_stream_extractor.py |
|
||
| 3 | ocr-processor | `ocr_processor.py` | ocr_processor.py |
|
||
| 4 | deep-analyzer | `deep_analyzer.py` | deep_analyzer.py |
|
||
| 5 | kb-workflow | `kb_workflow.py` | pipeline.py, git_sync.py |
|
||
|
||
---
|
||
|
||
## 附录B: OpenClaw Cron 配置示例
|
||
|
||
```yaml
|
||
# 定时任务配置
|
||
schedule: "0 3 * * *"
|
||
command: "python /path/to/skills/kb-workflow/kb_workflow.py --trigger scheduled"
|
||
```
|
||
|
||
---
|
||
|
||
蒋先生，详细设计文档已完成！
|
||
|
||
**包含内容：**
|
||
- ✅ 7个技能的完整接口规范（命令行参数、输入输出格式）
|
||
- ✅ 数据格式详细定义（收件箱、归档、汇报）
|
||
- ✅ 配置文件规范
|
||
- ✅ 复用代码对照表
|
||
- ✅ 扩展指南（新增平台/模板步骤）
|
||
|
||
**下一步：**
|
||
可以进入 **write-plan** 阶段，生成具体实现计划吗？ |