feat(kb-workflow): Add Git version control and cleanup
## Git Version Management - Git repository in archive/ directory - Environment variables for configuration (KB_GIT_ROOT, KB_GIT_REMOTE, etc.) - Auto commit after archive with retry (3 attempts) - Status command shows git status ## Subscription Management - Subscriptions stored in archive/subscriptions/ - Track processed items to avoid duplicates - Git tracked for history ## Automatic Cleanup - Cleanup strategy based on file type and age - video/frame retention: 7 days - metadata/reports: permanent - archive/: Git managed ## New Commands - --trigger git-init: Initialize git repository - --trigger git-status: Show git status - --trigger cleanup: Clean expired temporary files
This commit is contained in:
OpenClaw Agent
+167
-225
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: kb-workflow
|
||||
description: 知识库工作流编排技能，提供收件箱扫描、内容处理、归档、订阅检测和汇报生成功能。
|
||||
description: 知识库工作流编排技能，提供收件箱扫描、内容处理、归档、订阅检测、Git版本管理和自动清理功能。
|
||||
metadata:
|
||||
short-description: 知识库工作流编排
|
||||
---
|
||||
@@ -9,45 +9,124 @@ metadata:
|
||||
|
||||
## Description
|
||||
|
||||
KB Workflow 是一个知识库工作流编排技能，负责协调和管理知识库内容的完整生命周期：
|
||||
KB Workflow 是一个完整的知识库管理系统，包含：
|
||||
|
||||
**核心功能**：
|
||||
- 📥 **收件箱扫描** - 扫描四个子目录 (pending, metadata, content, analyzed)
|
||||
- 🔄 **内容处理** - 调用技能2-4处理内容
|
||||
- 📦 **智能归档** - 将处理完成的内容归档到 archive/
|
||||
- 🔔 **订阅检测** - 检测订阅源的更新
|
||||
- 📊 **汇报生成** - 生成处理统计和推荐
|
||||
- 📦 **Git版本管理** - 归档后自动提交并推送
|
||||
- 🔔 **订阅管理** - 跟踪订阅源更新
|
||||
- 🧹 **自动清理** - 定期清理临时文件
|
||||
- 📊 **汇报生成** - 生成处理统计
|
||||
|
||||
## Workflow Stages
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐
|
||||
│ pending │ → │ metadata│ → │ content │ → │ analyzed│ → │ archive/ │
|
||||
└─────────┘ └─────────┘ └─────────┘ └─────────┘ └──────────┘
|
||||
↓ ↓ ↓ ↓
|
||||
Skill 2 Skill 3 Skill 4 ↓
|
||||
(Link解析) (元数据提取) (内容分析) 最终归档
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
KB Workflow 作为 OpenClaw 技能套件的一部分，无需额外安装。
|
||||
|
||||
确保以下依赖已配置：
|
||||
- Python 3.8+
|
||||
- 知识库目录结构 (knowledge-base/)
|
||||
|
||||
## Directory Structure
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
knowledge-base/
|
||||
├── pending/ # 待处理的新内容
|
||||
├── metadata/ # 已提取元数据
|
||||
├── content/ # 已提取内容
|
||||
├── analyzed/ # 已完成分析
|
||||
├── archive/ # 归档内容
|
||||
│ └── YYYY-MM-DD/ # 按日期归档
|
||||
└── reports/ # 生成的汇报
|
||||
├── kb.config.yaml # 配置文件
|
||||
├── .gitignore # Git忽略规则
|
||||
├── pending/ # 待处理内容 (*.md)
|
||||
├── metadata/ # 链接解析结果 (*.link.json)
|
||||
├── content/
|
||||
│ ├── json/ # 元数据 (*.meta.json)
|
||||
│ ├── downloads/ # 视频文件 (*.mp4/*.flv) - 7天后清理
|
||||
│ └── subtitles/ # 字幕文件 (*.srt/*.lrc) - 7天后清理
|
||||
├── frames/ # 视频截帧 (*.jpg) - 7天后清理
|
||||
├── analyzed/ # 分析报告 (*.md)
|
||||
└── archive/ # Git归档目录
|
||||
├── YYYY-MM-DD/ # 按日期归档
|
||||
└── subscriptions/ # 订阅数据
|
||||
├── subscriptions.json # 订阅配置
|
||||
└── processed.json # 已处理项目
|
||||
```
|
||||
|
||||
## Git版本管理
|
||||
|
||||
### 配置
|
||||
|
||||
设置环境变量：
|
||||
```bash
|
||||
export KB_GIT_ROOT="/path/to/archive" # Git仓库目录
|
||||
export KB_GIT_REMOTE="git@github.com:user/repo.git" # 远程仓库
|
||||
export KB_GIT_USER="Username" # 用户名
|
||||
export KB_GIT_EMAIL="user@example.com" # 邮箱
|
||||
```
|
||||
|
||||
### 工作流程
|
||||
|
||||
```
|
||||
归档项目 → Git add → Git commit → Git push → 清理临时文件
|
||||
```
|
||||
|
||||
- 每次归档生成一个commit
|
||||
- 自动推送到远程仓库（重试3次）
|
||||
- 汇报中体现Git状态
|
||||
|
||||
## 订阅管理
|
||||
|
||||
### 数据结构
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"subscriptions": [
|
||||
{
|
||||
"id": "sub_001",
|
||||
"platform": "bilibili",
|
||||
"type": "series",
|
||||
"identifier": "123456",
|
||||
"name": "UE5教程",
|
||||
"subscribed_at": "2026-02-10T12:00:00",
|
||||
"last_checked": null,
|
||||
"last_update": null,
|
||||
"status": "active"
|
||||
}
|
||||
],
|
||||
"processed_items": [
|
||||
{
|
||||
"id": "BVxxx",
|
||||
"title": "视频标题",
|
||||
"processed_at": "2026-02-10T14:00:00",
|
||||
"archived": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 命令
|
||||
|
||||
```bash
|
||||
# 添加订阅
|
||||
python kb_workflow.py --trigger subscriptions --add "bilibili" "series" "123456" "教程名称"
|
||||
|
||||
# 检查更新
|
||||
python kb_workflow.py --trigger subscriptions --check
|
||||
|
||||
# 查看状态
|
||||
python kb_workflow.py --trigger subscriptions --status
|
||||
```
|
||||
|
||||
## 自动清理策略
|
||||
|
||||
### 清理规则
|
||||
|
||||
| 类型 | 保留 | 清理周期 |
|
||||
|------|------|---------|
|
||||
| 分析报告 | ✅ 永久 | - |
|
||||
| 链接信息 | ✅ 永久 | - |
|
||||
| 元数据 | ✅ 永久 | - |
|
||||
| 订阅数据 | ✅ Git管理 | - |
|
||||
| 视频文件 | ❌ | 7天 |
|
||||
| 截帧文件 | ❌ | 7天 |
|
||||
|
||||
### 命令
|
||||
|
||||
```bash
|
||||
# 清理30天前的临时文件
|
||||
python kb_workflow.py --trigger cleanup --days 30
|
||||
|
||||
# 清理单个项目
|
||||
python kb_workflow.py --trigger cleanup --project "项目名称"
|
||||
```
|
||||
|
||||
## Usage
|
||||
@@ -55,49 +134,30 @@ knowledge-base/
|
||||
### 命令行用法
|
||||
|
||||
```bash
|
||||
# 查看工作流状态
|
||||
# 查看状态
|
||||
python kb_workflow.py --trigger status
|
||||
|
||||
# 手动处理待处理内容
|
||||
python kb_workflow.py --trigger manual --scope pending
|
||||
# 处理待处理内容
|
||||
python kb_workflow.py --trigger manual --scope pending --limit 5
|
||||
|
||||
# 处理所有新内容（限制5个）
|
||||
python kb_workflow.py --trigger manual --limit 5
|
||||
# 初始化Git
|
||||
python kb_workflow.py --trigger git-init
|
||||
|
||||
# 生成日报
|
||||
python kb_workflow.py --trigger scheduled --report
|
||||
# 查看Git状态
|
||||
python kb_workflow.py --trigger git-status
|
||||
|
||||
# 检查订阅更新
|
||||
python kb_workflow.py --trigger subscriptions
|
||||
# 清理过期文件
|
||||
python kb_workflow.py --trigger cleanup --days 30
|
||||
```
|
||||
|
||||
### 参数说明
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
|------|------|--------|
|
||||
| `--trigger` | 触发类型 | `manual` |
|
||||
| `--scope` | 处理范围 | `all` |
|
||||
| `--trigger` | 触发类型 | `status` |
|
||||
| `--scope` | 处理范围 | `pending` |
|
||||
| `--limit` | 处理数量限制 | 无限制 |
|
||||
| `--report` | 生成汇报 | False |
|
||||
| `--output` | 输出格式 | `json` |
|
||||
|
||||
### 触发类型
|
||||
|
||||
- `manual` - 手动触发，立即处理
|
||||
- `scheduled` - 定时任务触发
|
||||
- `webhook` - Webhook 触发
|
||||
- `status` - 仅查看状态
|
||||
- `report` - 生成汇报
|
||||
- `subscriptions` - 检查订阅
|
||||
|
||||
### 作用域
|
||||
|
||||
- `all` - 所有目录
|
||||
- `pending` - 仅待处理
|
||||
- `metadata` - 仅元数据阶段
|
||||
- `content` - 仅内容阶段
|
||||
- `analyzed` - 仅已完成分析
|
||||
- `archive` - 归档目录
|
||||
| `--days` | 清理天数 | 30 |
|
||||
|
||||
## Integration
|
||||
|
||||
@@ -105,181 +165,63 @@ python kb_workflow.py --trigger subscriptions
|
||||
|
||||
KB Workflow 协调以下子技能：
|
||||
|
||||
1. **Skill 2: Link Parser** - 解析 URL/链接
|
||||
2. **Skill 3: Zhihu Metadata** - 提取元数据
|
||||
3. **Skill 4: Bilibili Analyzer** - 内容分析
|
||||
|
||||
### 调用示例
|
||||
|
||||
```python
|
||||
from kb_workflow import KBWorkflow
|
||||
|
||||
# 初始化
|
||||
workflow = KBWorkflow(base_path="/path/to/knowledge-base")
|
||||
|
||||
# 扫描收件箱
|
||||
inbox = workflow.scan_inbox(scope="pending")
|
||||
print(f"待处理: {len(inbox['pending'])} 项")
|
||||
|
||||
# 处理项目
|
||||
for item in inbox['pending'][:5]:
|
||||
result = workflow.process_item(item)
|
||||
print(f"处理结果: {result['success']}")
|
||||
|
||||
# 查看状态
|
||||
status = workflow.get_status()
|
||||
print(json.dumps(status, indent=2))
|
||||
|
||||
# 生成汇报
|
||||
report = workflow.generate_report(period="daily")
|
||||
```
|
||||
|
||||
## Input/Output Formats
|
||||
|
||||
### Input
|
||||
|
||||
#### 扫描收件箱输入
|
||||
```json
|
||||
{
|
||||
"scope": "all", // 或 "pending", "metadata", "content", "analyzed"
|
||||
"limit": 10 // 可选，限制返回数量
|
||||
}
|
||||
```
|
||||
|
||||
#### 处理项目输入
|
||||
```json
|
||||
{
|
||||
"name": "example.md",
|
||||
"path": "/path/to/knowledge-base/pending/example.md",
|
||||
"extension": ".md",
|
||||
"size": 1024,
|
||||
"modified": "2026-02-10T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### Output
|
||||
|
||||
#### 状态输出
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"timestamp": "2026-02-10T12:00:00",
|
||||
"inbox": {
|
||||
"pending": {"count": 5, "path": "/path/pending"},
|
||||
"metadata": {"count": 3, "path": "/path/metadata"},
|
||||
"content": {"count": 2, "path": "/path/content"},
|
||||
"analyzed": {"count": 8, "path": "/path/analyzed"}
|
||||
},
|
||||
"archive": {
|
||||
"count": 120,
|
||||
"path": "/path/archive"
|
||||
},
|
||||
"total_items": 18
|
||||
}
|
||||
```
|
||||
|
||||
#### 处理结果输出
|
||||
```json
|
||||
{
|
||||
"item": {"name": "example.md", ...},
|
||||
"success": true,
|
||||
"stages_completed": ["link_parsing", "metadata_extraction", "content_analysis"],
|
||||
"output_path": "/path/analyzed/example.md",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
#### 汇报输出
|
||||
```json
|
||||
{
|
||||
"generated_at": "2026-02-10T12:00:00",
|
||||
"period": "daily",
|
||||
"summary": {
|
||||
"total_items_inbox": 18,
|
||||
"total_archived": 120,
|
||||
"items_by_stage": {...}
|
||||
},
|
||||
"processing_stats": {...},
|
||||
"recommendations": [
|
||||
"⚠️ 待处理内容较多 (15 项)，建议优先处理",
|
||||
"📦 已分析内容较多 (25 项)，建议归档"
|
||||
]
|
||||
}
|
||||
```
|
||||
1. **Skill 1: Link Parser** - 解析 URL/链接
|
||||
2. **Skill 2: Metadata** - 提取元数据
|
||||
3. **Skill 3: Content** - 获取内容
|
||||
4. **Skill 4: Deep Analyzer** - 深度分析
|
||||
|
||||
## Configuration
|
||||
|
||||
KB Workflow 无需额外配置，自动使用以下路径：
|
||||
- 知识库根目录: `skills/kb-workflow/knowledge-base/`
|
||||
- 归档目录: `skills/kb-workflow/knowledge-base/archive/`
|
||||
- 汇报目录: `skills/kb-workflow/knowledge-base/reports/`
|
||||
### 配置文件 (kb.config.yaml)
|
||||
|
||||
### 自定义路径
|
||||
```yaml
|
||||
git:
|
||||
enabled: true
|
||||
remote: "git@github.com:user/repo.git"
|
||||
user: "Username"
|
||||
email: "user@example.com"
|
||||
|
||||
```python
|
||||
from kb_workflow import KBWorkflow
|
||||
cleanup:
|
||||
video_retention_days: 7
|
||||
frame_retention_days: 7
|
||||
auto_cleanup: true
|
||||
|
||||
# 使用自定义路径
|
||||
workflow = KBWorkflow(
|
||||
base_path="/custom/knowledge-base",
|
||||
archive_path="/custom/archive",
|
||||
output_format="json"
|
||||
)
|
||||
```
|
||||
|
||||
## Workflow Steps
|
||||
|
||||
### 完整处理流程
|
||||
|
||||
1. **扫描收件箱**
|
||||
- 扫描 `pending/` 目录获取新内容
|
||||
- 扫描其他阶段目录获取待继续处理的内容
|
||||
|
||||
2. **分类处理**
|
||||
- 新内容 → 提取链接/URL (Skill 2)
|
||||
- 待提取元数据 → 提取元数据 (Skill 3)
|
||||
- 待分析内容 → 深度分析 (Skill 4)
|
||||
|
||||
3. **自动流转**
|
||||
- 处理完成后自动移动到下一阶段目录
|
||||
|
||||
4. **归档**
|
||||
- 分析完成后可归档到 `archive/YYYY-MM-DD/`
|
||||
|
||||
5. **汇报**
|
||||
- 生成处理统计
|
||||
- 提供处理建议
|
||||
|
||||
### 定时任务示例
|
||||
|
||||
```bash
|
||||
# 每天早上9点检查并处理新内容
|
||||
0 9 * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger scheduled --scope pending --report
|
||||
|
||||
# 每小时检查订阅更新
|
||||
0 * * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger subscriptions
|
||||
subscriptions:
|
||||
check_interval_hours: 24
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### 常见问题
|
||||
### Git未初始化
|
||||
|
||||
**Q: 扫描不到内容？**
|
||||
A: 确保知识库目录存在且包含正确的子目录 (pending, metadata, content, analyzed)
|
||||
```bash
|
||||
# 检查配置
|
||||
python kb_workflow.py --trigger git-status
|
||||
|
||||
**Q: 处理失败？**
|
||||
A: 检查日志输出，使用 `--verbose` 参数获取详细调试信息
|
||||
# 初始化
|
||||
export KB_GIT_ROOT="/path/to/archive"
|
||||
python kb_workflow.py --trigger git-init
|
||||
```
|
||||
|
||||
**Q: 如何查看处理进度？**
|
||||
A: 使用 `--trigger status` 查看各阶段的项目数量
|
||||
### 推送失败
|
||||
|
||||
- 检查网络连接
|
||||
- 验证远程仓库权限
|
||||
- 查看错误日志
|
||||
|
||||
### 存储空间过大
|
||||
|
||||
```bash
|
||||
# 清理过期文件
|
||||
python kb_workflow.py --trigger cleanup --days 7
|
||||
|
||||
# 查看存储使用情况
|
||||
python kb_workflow.py --trigger status
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **obsidian-kb** - Obsidian 知识库管理
|
||||
- **link-parser** - 链接解析
|
||||
- **zhihu-metadata** - 知乎元数据提取
|
||||
- **bilibili-analyzer** - B站视频分析
|
||||
|
||||
---
|
||||
|
||||
*KB Workflow v1.0 - Knowledge Base Workflow Orchestration*
|
||||
- **kb-link-parser** - 链接解析
|
||||
- **kb-bilibili-metadata** - B站元数据
|
||||
- **kb-bilibili-content** - B站内容获取
|
||||
- **kb-deep-analyzer** - 深度分析
|
||||
|
||||
+610
-350
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user