feat: Add KB knowledge base series skills

- kb-link-parser: Multi-platform URL parser (Bilibili, YouTube, Zhihu)
- kb-bilibili-metadata: Bilibili video/article/up主 metadata extractor
- kb-zhihu-metadata: Zhihu question/answer/user metadata extractor
- kb-bilibili-content: Bilibili video download and content extraction
- kb-ocr-processor: OCR text extraction from video frames
- kb-deep-analyzer: AI-powered deep content analysis
- kb-workflow: Workflow orchestration (inbox, archive, subscriptions, report)

Also add:
- ARCHITECTURE_FLOWCHART.md: System architecture diagrams
- DETAILED_DESIGN.md: Complete design documentation
This commit is contained in:
OpenClaw Agent
2026-02-10 12:18:52 +08:00
parent 2f5a269866
commit 5507b1d986
28 changed files with 7715 additions and 0 deletions
+285
View File
@@ -0,0 +1,285 @@
---
name: kb-workflow
description: 知识库工作流编排技能,提供收件箱扫描、内容处理、归档、订阅检测和汇报生成功能。
metadata:
short-description: 知识库工作流编排
---
# KB Workflow
## Description
KB Workflow 是一个知识库工作流编排技能,负责协调和管理知识库内容的完整生命周期:
**核心功能**:
- 📥 **收件箱扫描** - 扫描四个子目录 (pending, metadata, content, analyzed)
- 🔄 **内容处理** - 调用技能2-4处理内容
- 📦 **智能归档** - 将处理完成的内容归档到 archive/
- 🔔 **订阅检测** - 检测订阅源的更新
- 📊 **汇报生成** - 生成处理统计和推荐
## Workflow Stages
```
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐
│ pending │ → │ metadata│ → │ content │ → │ analyzed│ → │ archive/ │
└─────────┘ └─────────┘ └─────────┘ └─────────┘ └──────────┘
↓ ↓ ↓ ↓
Skill 2 Skill 3 Skill 4 ↓
(Link解析) (元数据提取) (内容分析) 最终归档
```
## Installation
KB Workflow 作为 OpenClaw 技能套件的一部分,无需额外安装。
确保以下依赖已配置:
- Python 3.8+
- 知识库目录结构 (knowledge-base/)
## Directory Structure
```
knowledge-base/
├── pending/ # 待处理的新内容
├── metadata/ # 已提取元数据
├── content/ # 已提取内容
├── analyzed/ # 已完成分析
├── archive/ # 归档内容
│ └── YYYY-MM-DD/ # 按日期归档
└── reports/ # 生成的汇报
```
## Usage
### 命令行用法
```bash
# 查看工作流状态
python kb_workflow.py --trigger status
# 手动处理待处理内容
python kb_workflow.py --trigger manual --scope pending
# 处理所有新内容(限制5个)
python kb_workflow.py --trigger manual --limit 5
# 生成日报
python kb_workflow.py --trigger scheduled --report
# 检查订阅更新
python kb_workflow.py --trigger subscriptions
```
### 参数说明
| 参数 | 说明 | 默认值 |
|------|------|--------|
| `--trigger` | 触发类型 | `manual` |
| `--scope` | 处理范围 | `all` |
| `--limit` | 处理数量限制 | 无限制 |
| `--report` | 生成汇报 | False |
| `--output` | 输出格式 | `json` |
### 触发类型
- `manual` - 手动触发,立即处理
- `scheduled` - 定时任务触发
- `webhook` - Webhook 触发
- `status` - 仅查看状态
- `report` - 生成汇报
- `subscriptions` - 检查订阅
### 作用域
- `all` - 所有目录
- `pending` - 仅待处理
- `metadata` - 仅元数据阶段
- `content` - 仅内容阶段
- `analyzed` - 仅已完成分析
- `archive` - 归档目录
## Integration
### 技能2-4 集成
KB Workflow 协调以下子技能:
1. **Skill 2: Link Parser** - 解析 URL/链接
2. **Skill 3: Zhihu Metadata** - 提取元数据
3. **Skill 4: Bilibili Analyzer** - 内容分析
### 调用示例
```python
from kb_workflow import KBWorkflow
# 初始化
workflow = KBWorkflow(base_path="/path/to/knowledge-base")
# 扫描收件箱
inbox = workflow.scan_inbox(scope="pending")
print(f"待处理: {len(inbox['pending'])} 项")
# 处理项目
for item in inbox['pending'][:5]:
result = workflow.process_item(item)
print(f"处理结果: {result['success']}")
# 查看状态
status = workflow.get_status()
print(json.dumps(status, indent=2))
# 生成汇报
report = workflow.generate_report(period="daily")
```
## Input/Output Formats
### Input
#### 扫描收件箱输入
```json
{
"scope": "all", // 或 "pending", "metadata", "content", "analyzed"
"limit": 10 // 可选,限制返回数量
}
```
#### 处理项目输入
```json
{
"name": "example.md",
"path": "/path/to/knowledge-base/pending/example.md",
"extension": ".md",
"size": 1024,
"modified": "2026-02-10T12:00:00"
}
```
### Output
#### 状态输出
```json
{
"status": "ready",
"timestamp": "2026-02-10T12:00:00",
"inbox": {
"pending": {"count": 5, "path": "/path/pending"},
"metadata": {"count": 3, "path": "/path/metadata"},
"content": {"count": 2, "path": "/path/content"},
"analyzed": {"count": 8, "path": "/path/analyzed"}
},
"archive": {
"count": 120,
"path": "/path/archive"
},
"total_items": 18
}
```
#### 处理结果输出
```json
{
"item": {"name": "example.md", ...},
"success": true,
"stages_completed": ["link_parsing", "metadata_extraction", "content_analysis"],
"output_path": "/path/analyzed/example.md",
"error": null
}
```
#### 汇报输出
```json
{
"generated_at": "2026-02-10T12:00:00",
"period": "daily",
"summary": {
"total_items_inbox": 18,
"total_archived": 120,
"items_by_stage": {...}
},
"processing_stats": {...},
"recommendations": [
"⚠️ 待处理内容较多 (15 项),建议优先处理",
"📦 已分析内容较多 (25 项),建议归档"
]
}
```
## Configuration
KB Workflow 无需额外配置,自动使用以下路径:
- 知识库根目录: `skills/kb-workflow/knowledge-base/`
- 归档目录: `skills/kb-workflow/knowledge-base/archive/`
- 汇报目录: `skills/kb-workflow/knowledge-base/reports/`
### 自定义路径
```python
from kb_workflow import KBWorkflow
# 使用自定义路径
workflow = KBWorkflow(
base_path="/custom/knowledge-base",
archive_path="/custom/archive",
output_format="json"
)
```
## Workflow Steps
### 完整处理流程
1. **扫描收件箱**
- 扫描 `pending/` 目录获取新内容
- 扫描其他阶段目录获取待继续处理的内容
2. **分类处理**
- 新内容 → 提取链接/URL (Skill 2)
- 待提取元数据 → 提取元数据 (Skill 3)
- 待分析内容 → 深度分析 (Skill 4)
3. **自动流转**
- 处理完成后自动移动到下一阶段目录
4. **归档**
- 分析完成后可归档到 `archive/YYYY-MM-DD/`
5. **汇报**
- 生成处理统计
- 提供处理建议
### 定时任务示例
```bash
# 每天早上9点检查并处理新内容
0 9 * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger scheduled --scope pending --report
# 每小时检查订阅更新
0 * * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger subscriptions
```
## Troubleshooting
### 常见问题
**Q: 扫描不到内容?**
A: 确保知识库目录存在且包含正确的子目录 (pending, metadata, content, analyzed)
**Q: 处理失败?**
A: 检查日志输出,使用 `--verbose` 参数获取详细调试信息
**Q: 如何查看处理进度?**
A: 使用 `--trigger status` 查看各阶段的项目数量
## Related Skills
- **obsidian-kb** - Obsidian 知识库管理
- **link-parser** - 链接解析
- **zhihu-metadata** - 知乎元数据提取
- **bilibili-analyzer** - B站视频分析
---
*KB Workflow v1.0 - Knowledge Base Workflow Orchestration*
+427
View File
@@ -0,0 +1,427 @@
#!/usr/bin/env python3
"""
Auto-trigger handler for obsidian-kb skill.
Detects video/article links in user messages and automatically processes them.
Supports Bilibili, YouTube, and Zhihu.
"""
import re
import sys
import os
from typing import List, Dict, Optional, Tuple
from urllib.parse import urlparse
# Add path for imports
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), 'src'))
# Import short link resolver
from shortlink_resolver import (
ShortLinkResolver,
detect_and_format,
PlaywrightResolver
)
# 导入主流程
try:
from main_pipeline import MainPipeline
MAIN_PIPELINE_AVAILABLE = True
except ImportError:
MAIN_PIPELINE_AVAILABLE = False
# Pattern definitions for auto-detection
PATTERNS = {
'bilibili_full': re.compile(r'https?://(?:www\.)?bilibili\.com/video/(BV\w+)'),
'bilibili_short': re.compile(r'https?://b23\.tv/(\w+)'),
'bilibili_bv': re.compile(r'\b(BV[a-zA-Z0-9]{10})\b'),
'youtube_full': re.compile(r'https?://(?:www\.)?youtube\.com/watch\?v=([a-zA-Z0-9_-]{11})'),
'youtube_short': re.compile(r'https?://youtu\.be/([a-zA-Z0-9_-]{11})'),
'zhihu_article': re.compile(r'https?://zhuanlan\.zhihu\.com/p/(\d+)'),
'zhihu_answer': re.compile(r'https?://(?:www\.)?zhihu\.com/question/\d+/answer/(\d+)'),
}
def detect_links(text: str) -> List[Dict[str, str]]:
"""
Detect video/article links in text.
Args:
text: Input text to search
Returns:
List of detected links with metadata:
[
{
'url': 'https://...',
'platform': 'bilibili',
'type': 'video',
'id': 'BVxxx',
'is_short': False
},
...
]
"""
detected = []
seen_ids = set() # Track IDs to avoid duplicates
for pattern_name, pattern in PATTERNS.items():
matches = pattern.finditer(text)
for match in matches:
# Determine platform and type
if 'bilibili' in pattern_name:
platform = 'bilibili'
content_type = 'video'
if 'short' in pattern_name:
is_short = True
url = match.group(0)
content_id = None # Need to resolve short URL
else:
is_short = False
if pattern_name == 'bilibili_bv':
url = f"https://www.bilibili.com/video/{match.group(1)}"
else:
url = match.group(0)
content_id = match.group(1)
elif 'youtube' in pattern_name:
platform = 'youtube'
content_type = 'video'
is_short = 'short' in pattern_name
url = match.group(0)
content_id = match.group(1)
elif 'zhihu' in pattern_name:
platform = 'zhihu'
content_type = 'article'
is_short = False
url = match.group(0)
content_id = match.group(1)
else:
continue
# Skip duplicates by URL or ID
unique_key = url if (not content_id or is_short) else content_id
if unique_key in seen_ids:
continue
seen_ids.add(unique_key)
detected.append({
'url': url,
'platform': platform,
'type': content_type,
'id': content_id,
'is_short': is_short,
'pattern': pattern_name
})
return detected
def format_preview(detected: List[Dict[str, str]]) -> str:
"""
Format a preview message for detected links.
Args:
detected: List of detected links
Returns:
Formatted preview message
"""
if not detected:
return ""
lines = ["🔍 检测到内容链接:\n"]
for i, item in enumerate(detected, 1):
platform_emoji = {
'bilibili': '📺',
'youtube': '▶️',
'zhihu': '📝'
}.get(item['platform'], '🔗')
platform_name = {
'bilibili': 'B站视频',
'youtube': 'YouTube视频',
'zhihu': '知乎文章'
}.get(item['platform'], item['platform'])
lines.append(f"{platform_emoji} [{i}] {platform_name}")
lines.append(f" URL: {item['url']}")
if item['is_short']:
lines.append(f" ⚠️ 短链接,需要用户提供完整URL或ID")
elif item['id']:
lines.append(f" ID: {item['id']}")
lines.append("")
return "\n".join(lines)
def format_short_link_help(short_link: Optional[str] = None) -> str:
"""
Format help message for short links.
Args:
short_link: Optional short link URL to mention
Returns:
Help message
"""
return ShortLinkResolver.format_short_link_help(short_link)
def get_action_prompt(detected: List[Dict[str, str]]) -> str:
"""
Generate prompt asking user what to do.
Args:
detected: List of detected links
Returns:
Action prompt
"""
has_short = any(item['is_short'] for item in detected)
if has_short:
return f"{format_short_link_help()}\n\n其他链接是否立即处理?(y/n)"
count = len(detected)
if count == 1:
return "是否添加到收件箱并处理?(y/n)"
else:
return f"是否将上述 {count} 个内容添加到收件箱?(y/n)"
def should_auto_process(text: str) -> bool:
"""
Check if user wants to auto-process based on their message.
Args:
text: User's message
Returns:
True if user indicates they want immediate processing
"""
auto_keywords = ['立即处理', '处理', 'process', '--process', '马上']
return any(keyword in text.lower() for keyword in auto_keywords)
def process_link(url: str, platform: str, auto_process: bool = False) -> Tuple[bool, str]:
"""
Process a single link (extract metadata, add to inbox).
Args:
url: Content URL
platform: Platform name (bilibili/youtube/zhihu)
auto_process: Whether to immediately process after adding
Returns:
(success, message) tuple
"""
try:
# Import here to avoid issues if dependencies not available
from sources.bilibili import BilibiliAdapter
from models import Content, ContentStatus
from storage import Storage
# Only support Bilibili for now
if platform != 'bilibili':
return False, f"暂不支持 {platform},请手动添加"
# Handle short links
if 'b23.tv' in url:
return False, "短链接无法自动处理,请提供完整URL或BV号"
# Create adapter
adapter = BilibiliAdapter()
# Extract metadata
content = adapter.extract_content(url)
if not content:
return False, "无法提取视频信息"
# Get transcript
transcript = adapter.get_transcript(content)
if transcript:
content.transcript = transcript
# Save to inbox
content.status = ContentStatus.CONFIRMED
storage = Storage()
storage.save_content(content)
# Build success message
msg = f"✅ 已添加到收件箱\n"
msg += f" 标题: {content.title}\n"
msg += f" UP主: {content.author}\n"
msg += f" 时长: {content.duration}秒\n"
if auto_process:
# Process immediately
try:
from src.pipeline import ProcessingPipeline
pipeline = ProcessingPipeline()
content_text = content.transcript or content.summary or ""
result = pipeline.process_full(content, content_text)
storage.save_content(result)
msg += f"\n🔄 处理完成\n"
msg += f" 📄 笔记: {result.output_path or 'N/A'}\n"
msg += f" 🏷️ 标签: {', '.join(result.tags)}"
except Exception as e:
msg += f"\n⚠️ 处理失败: {e}"
return True, msg
except ImportError as e:
return False, f"依赖模块未安装: {e}"
except Exception as e:
return False, f"处理失败: {e}"
def process_video_full(source_id: str, enable_git_sync: bool = False) -> Tuple[bool, str]:
"""
使用主流程完整处理视频(新接口)
完整流程: 提取 → 处理 → 输出 → 同步
Args:
source_id: 视频ID(支持BV号、URL)
enable_git_sync: 是否启用Git同步
Returns:
(success, message) tuple
"""
if not MAIN_PIPELINE_AVAILABLE:
return False, "主流程模块未可用,请检查依赖"
try:
# 导入配置
from src.config import get_config
# 获取配置
config = get_config()
# 创建主流程
pipeline = MainPipeline(config=config, enable_git_sync=enable_git_sync)
# 处理视频
result = pipeline.process_video(source_id)
# 返回结果
if result['success']:
return True, result['message']
else:
return False, result.get('error', 'Unknown error')
except Exception as e:
return False, f"处理失败: {e}"
def handle_message(text: str) -> Optional[str]:
"""
Main entry point: handle incoming message and return response.
Args:
text: User's message text
Returns:
Response message, or None if no action needed
"""
# First, check for short links and provide help
has_short, short_msg = detect_and_format(text)
# Detect all links (including short ones)
detected = detect_links(text)
if not detected:
return None
# If we have short links, show help message first
if has_short:
return short_msg
# Check if this is a confirmation (y/yes)
text_lower = text.lower().strip()
if text_lower in ('y', 'yes', '是', '好的', 'ok'):
# This would need state management - for now return a message
# about using explicit command
return "请使用 /kb-add <URL> 命令添加内容,或重新发送链接"
# Show preview
preview = format_preview(detected)
prompt = get_action_prompt(detected)
return f"{preview}\n{prompt}"
# CLI interface for testing
def main():
"""CLI for testing the handler."""
if len(sys.argv) < 2:
print("Usage: python3 handler.py <command> [options]")
print("\nCommands:")
print(" detect 'text' - 检测文本中的内容链接")
print(" process <BV/URL> - 使用主流程处理视频")
print(" process <BV/URL> --git - 处理视频并同步到Git")
print("\nExamples:")
print(' python3 handler.py detect "https://www.bilibili.com/video/BV1xx411c7mD"')
print(' python3 handler.py process BV1xx411c7mD')
print(' python3 handler.py process BV1xx411c7mD --git')
sys.exit(1)
command = sys.argv[1].lower()
if command == 'detect':
# 检测链接
if len(sys.argv) < 3:
print("Error: Please provide text to detect")
sys.exit(1)
text = ' '.join(sys.argv[2:])
response = handle_message(text)
if response:
print(response)
else:
print("No content links detected")
elif command == 'process':
# 使用主流程处理视频
if len(sys.argv) < 3:
print("Error: Please provide BV number or URL")
sys.exit(1)
source_id = sys.argv[2]
enable_git = '--git' in sys.argv
print(f"Processing video: {source_id}")
if enable_git:
print("Git sync: enabled")
else:
print("Git sync: disabled")
print()
success, message = process_video_full(source_id, enable_git_sync=enable_git)
if success:
print(message)
sys.exit(0)
else:
print(f"\n❌ Failed: {message}")
sys.exit(1)
else:
# 兼容旧格式:直接传入文本进行检测
text = sys.argv[1]
response = handle_message(text)
if response:
print(response)
else:
print("No content links detected")
if __name__ == '__main__':
main()
+489
View File
@@ -0,0 +1,489 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Obsidian KB - 主流程集成
整合提取层、处理层、输出层,实现完整的数据流:
提取(BilibiliExtractor) → 处理(DeepAnalyzer/ContentChunker) → 输出(ObsidianWriter) → 同步(GitSync)
"""
import os
import sys
import logging
from typing import Dict, Any, Optional
from datetime import datetime
from pathlib import Path
# 添加项目路径
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from src.extractors.bilibili_extractor import BilibiliExtractor, ExtractionError
from src.deep_analyzer import DeepAnalyzer
from src.content_chunker import ContentChunker, Chunk
from src.outputs.obsidian_writer import ObsidianWriter
from src.outputs.git_sync import GitSync
from src.models import Content, ContentType, ContentStatus, TimelineEntry
from src.config import get_config
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.StreamHandler(sys.stdout),
logging.FileHandler('obsidian_kb.log', encoding='utf-8')
]
)
logger = logging.getLogger('ObsidianKB')
class MainPipeline:
"""主流程集成器"""
def __init__(self, config=None, enable_git_sync: bool = False):
"""
初始化主流程
Args:
config: 配置对象
enable_git_sync: 是否启用Git同步
"""
self.config = config or get_config()
self.enable_git_sync = enable_git_sync
logger.info("初始化主流程...")
logger.info(f"Obsidian vault 路径: {self.config.vault_path}")
logger.info(f"Git 同步: {'启用' if enable_git_sync else '禁用'}")
# 初始化各层组件
self.extractor = BilibiliExtractor(config=self.config)
self.analyzer = DeepAnalyzer(ai_client=None) # 暂时不传入ai_client
self.chunker = ContentChunker()
self.writer = ObsidianWriter(config=self.config)
# 初始化Git同步器(仅在需要时)
self.git_sync = None
if enable_git_sync and self._is_git_enabled():
self.git_sync = GitSync(config=self.config)
logger.info("Git 同步器已初始化")
def _is_git_enabled(self) -> bool:
"""检查Git同步是否在配置中启用"""
return self.config.get('git.enabled', False)
def process_video(self, source_id: str) -> Dict[str, Any]:
"""
处理单个视频的完整流程
Args:
source_id: 视频ID(支持BV号、URL)
Returns:
处理结果字典,包含:
- success: bool - 是否成功
- content: Content对象 - 处理的内容
- message: str - 处理消息
- error: str - 错误信息(如果失败)
"""
result = {
'success': False,
'content': None,
'message': '',
'error': ''
}
try:
logger.info("=" * 80)
logger.info(f"开始处理视频: {source_id}")
logger.info("=" * 80)
# Step 1: 提取层 - 提取视频信息
logger.info("\n[Step 1/5] 提取视频信息...")
extractor_data = self._extract_video(source_id)
# Step 2: 构建Content对象
logger.info("\n[Step 2/5] 构建内容对象...")
content = self._build_content(extractor_data)
# Step 3: 处理层 - 深度分析
logger.info("\n[Step 3/5] 深度分析...")
analysis_result = self._analyze_content(extractor_data)
# Step 4: 处理层 - 内容分片
logger.info("\n[Step 4/5] 内容分片...")
chunks = self._chunk_content(extractor_data)
# 合并分析结果和分片信息到content
self._merge_analysis_to_content(content, analysis_result, chunks)
# Step 5: 输出层 - 写入Obsidian
logger.info("\n[Step 5/5] 写入Obsidian...")
output_path = self._write_obsidian_note(content)
# Step 6: 同步层 - Git同步(可选)
if self.enable_git_sync and self.git_sync:
logger.info("\n[Step 6/6] Git同步...")
self._sync_to_git(content)
# 更新状态
content.update_status(ContentStatus.GENERATED)
content.output_path = output_path
# 构建成功结果
result['success'] = True
result['content'] = content
result['message'] = self._format_success_message(content, chunks)
logger.info("\n" + "=" * 80)
logger.info(f"✅ 处理完成: {content.title}")
logger.info(f" 📄 笔记路径: {output_path}")
logger.info(f" 📊 字幕条数: {len(chunks) if chunks else 0}")
logger.info("=" * 80 + "\n")
except ExtractionError as e:
logger.error(f"❌ 提取失败: {e}")
result['error'] = f"提取失败: {e}"
result['message'] = f"提取视频信息失败: {e}"
except Exception as e:
logger.error(f"❌ 处理失败: {e}", exc_info=True)
result['error'] = str(e)
result['message'] = f"处理过程中发生错误: {e}"
return result
def _extract_video(self, source_id: str) -> Dict[str, Any]:
"""
提取视频信息
Args:
source_id: 视频ID或URL
Returns:
提取器返回的数据结构
Raises:
ExtractionError: 提取失败
"""
logger.info(f" 正在提取: {source_id}")
try:
extractor_data = self.extractor.extract(source_id)
logger.info(f" ✅ 提取成功: {extractor_data.get('metadata', {}).get('title', '未知标题')}")
logger.info(f" 作者: {extractor_data.get('metadata', {}).get('author', '未知')}")
logger.info(f" 时长: {extractor_data.get('metadata', {}).get('duration', '0')}")
# 检查字幕
subtitle = extractor_data.get('subtitle', '')
if subtitle:
subtitle_lines = subtitle.split('\n')
logger.info(f" 字幕: {len(subtitle_lines)} 条")
else:
logger.warning(f" ⚠️ 无字幕")
return extractor_data
except ExtractionError as e:
logger.error(f" ❌ 提取失败: {e}")
raise
except Exception as e:
logger.error(f" ❌ 提取异常: {e}", exc_info=True)
raise ExtractionError(f"提取过程异常: {e}", source_id=source_id)
def _build_content(self, extractor_data: Dict[str, Any]) -> Content:
"""
从提取器数据构建Content对象
Args:
extractor_data: 提取器返回的数据
Returns:
Content对象
"""
metadata = extractor_data.get('metadata', {})
content = Content(
source='bilibili',
source_id=metadata.get('bvid', ''),
url=metadata.get('url', ''),
title=metadata.get('title', ''),
author=metadata.get('author', ''),
description=extractor_data.get('description', ''),
duration=self._parse_duration(metadata.get('duration', '0')),
content_type=ContentType.VIDEO,
transcript=extractor_data.get('subtitle', ''),
status=ContentStatus.SCANNED
)
logger.info(f" ✅ Content对象已创建: {content.id}")
return content
def _parse_duration(self, duration_str: str) -> Optional[int]:
"""
解析时长字符串为秒数
Args:
duration_str: 时长字符串(如 "10:30")
Returns:
秒数
"""
try:
if ':' in duration_str:
parts = duration_str.split(':')
if len(parts) == 2:
# MM:SS
return int(parts[0]) * 60 + int(parts[1])
elif len(parts) == 3:
# HH:MM:SS
return int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])
except:
pass
return None
def _analyze_content(self, extractor_data: Dict[str, Any]) -> Dict[str, Any]:
"""
深度分析内容
Args:
extractor_data: 提取器返回的数据
Returns:
分析结果字典
"""
try:
logger.info(" 正在进行深度分析...")
# 使用DeepAnalyzer进行分析
# DeepAnalyzer接收extractor_data作为输入
analysis_result = self.analyzer.analyze(extractor_data)
# 将AnalysisResult转换为字典以便合并
result_dict = {
'main_topics': analysis_result.main_topics,
'key_insights': analysis_result.key_insights,
'action_items': analysis_result.action_items,
'summary': analysis_result.summary,
'timeline': analysis_result.timeline,
'recommended_tags': analysis_result.recommended_tags,
'full_analysis': analysis_result.full_analysis
}
logger.info(f" ✅ 分析完成: {len(result_dict['main_topics'])} 个主题")
return result_dict
except Exception as e:
logger.error(f" ❌ 分析失败: {e}", exc_info=True)
# 返回默认结果,不中断流程
return {
'main_topics': ['未分类'],
'key_insights': [],
'action_items': [],
'summary': '分析失败,无法生成摘要',
'timeline': [],
'recommended_tags': ['未分类'],
'full_analysis': ''
}
def _chunk_content(self, extractor_data: Dict[str, Any]) -> list:
"""
内容分片
Args:
extractor_data: 提取器返回的数据
Returns:
Chunk对象列表
"""
try:
logger.info(" 正在进行内容分片...")
# 使用ContentChunker进行分片
chunks = self.chunker.chunk_from_extractor(extractor_data)
logger.info(f" ✅ 分片完成: {len(chunks)} 段")
return chunks
except Exception as e:
logger.error(f" ❌ 分片失败: {e}", exc_info=True)
# 返回空列表,不中断流程
return []
def _merge_analysis_to_content(self, content: Content, analysis_result: Dict[str, Any], chunks: list):
"""
将分析结果和分片信息合并到Content对象
Args:
content: Content对象
analysis_result: 分析结果字典
chunks: Chunk对象列表
"""
# 合并基本信息
content.summary = analysis_result.get('summary', '')
content.key_points = analysis_result.get('main_topics', [])
content.tags = analysis_result.get('recommended_tags', [])
# 合并时间轴
timeline_data = analysis_result.get('timeline', [])
if timeline_data:
content.timeline = [
TimelineEntry(
timestamp=item.get('start_time', ''),
title=item.get('title', ''),
description=item.get('summary', '')
)
for item in timeline_data
]
else:
# 从chunks生成时间轴
if chunks:
content.timeline = [
TimelineEntry(
timestamp=chunk.time_range,
title=f"片段 {chunk.index + 1}",
description=f"时长 {int(chunk.duration)}秒"
)
for chunk in chunks
]
logger.info(f" ✅ 数据合并完成: 时间轴 {len(content.timeline)} 条")
def _write_obsidian_note(self, content: Content) -> Optional[str]:
"""
写入Obsidian笔记
Args:
content: Content对象
Returns:
文件路径,失败返回None
"""
try:
logger.info(f" 正在写入笔记...")
# 确保输出目录存在
self.config.vault_path.mkdir(parents=True, exist_ok=True)
# 调用ObsidianWriter写入
output_path = self.writer.write(content, self.config)
if output_path:
logger.info(f" ✅ 笔记已写入: {output_path}")
else:
logger.warning(f" ⚠️ 笔记写入失败")
return output_path
except Exception as e:
logger.error(f" ❌ 写入失败: {e}", exc_info=True)
return None
def _sync_to_git(self, content: Content):
"""
同步到Git仓库
Args:
content: Content对象
"""
if not self.git_sync:
logger.warning(" ⚠️ Git同步器未初始化,跳过同步")
return
try:
logger.info(" 正在同步到Git...")
success = self.git_sync.sync(self.config)
if success:
logger.info(" ✅ Git同步成功")
else:
logger.warning(" ⚠️ Git同步失败")
except Exception as e:
logger.error(f" ❌ Git同步异常: {e}", exc_info=True)
# Git同步失败不视为整体失败
def _format_success_message(self, content: Content, chunks: list) -> str:
"""
格式化成功消息
Args:
content: Content对象
chunks: Chunk列表
Returns:
格式化的消息字符串
"""
lines = [
"✅ 处理完成!",
"",
f"📺 标题: {content.title}",
f"👤 作者: {content.author}",
f"📄 笔记: {content.output_path or 'N/A'}",
]
if content.duration:
lines.append(f"⏱️ 时长: {self._format_duration(content.duration)}")
if chunks:
lines.append(f"📊 分片: {len(chunks)} 段")
if content.tags:
lines.append(f"🏷️ 标签: {', '.join(content.tags[:5])}")
lines.append("")
return "\n".join(lines)
def _format_duration(self, seconds: int) -> str:
"""格式化时长"""
if not seconds:
return "N/A"
hours = seconds // 3600
minutes = (seconds % 3600) // 60
secs = seconds % 60
if hours > 0:
return f"{hours}小时{minutes}分钟"
elif minutes > 0:
return f"{minutes}分钟{secs}秒"
else:
return f"{secs}秒"
def main():
"""命令行入口"""
import argparse
parser = argparse.ArgumentParser(description='Obsidian KB - 主流程集成')
parser.add_argument('source_id', help='视频ID(BV号或URL)')
parser.add_argument('--git', action='store_true', help='启用Git同步')
parser.add_argument('--config', help='配置文件路径')
args = parser.parse_args()
# 加载配置
config = get_config(args.config) if args.config else get_config()
# 创建主流程
pipeline = MainPipeline(config=config, enable_git_sync=args.git)
# 处理视频
result = pipeline.process_video(args.source_id)
# 输出结果
print(result['message'])
if not result['success']:
print(f"\n❌ 失败: {result['error']}")
sys.exit(1)
else:
sys.exit(0)
if __name__ == '__main__':
main()
+674
View File
@@ -0,0 +1,674 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
KB Workflow - Knowledge Base Workflow Orchestration
工作流编排技能,整合以下功能:
- 扫描收件箱目录 (pending, metadata, content, analyzed)
- 调用技能2-4处理内容
- 归档处理完成的条目
- 订阅更新检测
- 生成汇报
Usage:
python kb_workflow.py --trigger manual
python kb_workflow.py --trigger scheduled
python kb_workflow.py --status
python kb_workflow.py --report
python kb_workflow.py --scope pending --limit 5
"""
import os
import sys
import json
import argparse
import logging
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, List, Optional, Any
from enum import Enum
# Setup logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger('KBWorkflow')
class TriggerType(Enum):
MANUAL = "manual"
SCHEDULED = "scheduled"
WEBHOOK = "webhook"
class Scope(Enum):
ALL = "all"
PENDING = "pending"
METADATA = "metadata"
CONTENT = "content"
ANALYZED = "analyzed"
ARCHIVE = "archive"
class KBWorkflow:
"""Knowledge Base Workflow Orchestrator"""
# Base directories for the inbox structure
INBOX_DIRS = ['pending', 'metadata', 'content', 'analyzed']
def __init__(self,
base_path: str = None,
archive_path: str = None,
output_format: str = "json"):
"""
Initialize KB Workflow
Args:
base_path: Base path for knowledge base (default: workspace/knowledge-base)
archive_path: Path for archived items (default: archive/)
output_format: Output format (json, text)
"""
self.base_path = Path(base_path) if base_path else Path(__file__).parent / "knowledge-base"
self.archive_path = Path(archive_path) if archive_path else self.base_path / "archive"
self.output_format = output_format
# Ensure directories exist
self._ensure_directories()
# Initialize subsystems
self._init_subsystems()
def _ensure_directories(self):
"""Create necessary directories"""
# Create main inbox directories
for subdir in self.INBOX_DIRS:
dir_path = self.base_path / subdir
if not dir_path.exists():
dir_path.mkdir(parents=True, exist_ok=True)
logger.info(f"Created directory: {dir_path}")
# Create archive directory
if not self.archive_path.exists():
self.archive_path.mkdir(parents=True, exist_ok=True)
logger.info(f"Created archive directory: {self.archive_path}")
def _init_subsystems(self):
"""Initialize subsystem references"""
# Placeholder for skill integrations
self.subscriptions_file = self.base_path / "subscriptions.json"
self.report_file = self.base_path / "reports" / f"report_{datetime.now().strftime('%Y%m%d')}.json"
# ==================== Inbox Scanning ====================
def scan_inbox(self, scope: str = "all", limit: int = None) -> Dict[str, List[Dict]]:
"""
Scan inbox directories for items
Args:
scope: Which directories to scan (all, pending, metadata, content, analyzed)
limit: Maximum number of items to return per directory
Returns:
Dictionary with directory names as keys and item lists as values
"""
result = {}
if scope == "all":
dirs_to_scan = self.INBOX_DIRS
elif scope in self.INBOX_DIRS:
dirs_to_scan = [scope]
else:
logger.warning(f"Unknown scope: {scope}, scanning all")
dirs_to_scan = self.INBOX_DIRS
for dir_name in dirs_to_scan:
dir_path = self.base_path / dir_name
items = self._scan_directory(dir_path)
if limit:
items = items[:limit]
result[dir_name] = items
logger.info(f"Scanned {dir_name}: {len(items)} items")
return result
def _scan_directory(self, path: Path) -> List[Dict]:
"""
Scan a directory for items
Args:
path: Directory path to scan
Returns:
List of item dictionaries
"""
items = []
if not path.exists():
return items
for item_path in path.iterdir():
if item_path.is_file():
item_info = {
'name': item_path.name,
'path': str(item_path),
'extension': item_path.suffix,
'size': item_path.stat().st_size,
'modified': datetime.fromtimestamp(item_path.stat().st_mtime).isoformat(),
'type': self._classify_file_type(item_path)
}
items.append(item_info)
elif item_path.is_dir():
# Treat subdirectories as items too
item_info = {
'name': item_path.name,
'path': str(item_path),
'type': 'directory',
'modified': datetime.fromtimestamp(item_path.stat().st_mtime).isoformat()
}
items.append(item_info)
return items
def _classify_file_type(self, path: Path) -> str:
"""Classify file type by extension"""
suffix = path.suffix.lower()
type_mapping = {
'.md': 'markdown',
'.txt': 'text',
'.json': 'json',
'.yaml': 'yaml',
'.yml': 'yaml',
'.pdf': 'pdf',
'.jpg': 'image',
'.jpeg': 'image',
'.png': 'image',
'.mp4': 'video',
'.mp3': 'audio',
'.zip': 'archive',
'.url': 'link'
}
return type_mapping.get(suffix, 'unknown')
def get_status(self) -> Dict[str, Any]:
"""
Get current workflow status
Returns:
Status dictionary
"""
inbox_status = {}
total_items = 0
for dir_name in self.INBOX_DIRS:
dir_path = self.base_path / dir_name
items = self._scan_directory(dir_path)
inbox_status[dir_name] = {
'count': len(items),
'path': str(dir_path)
}
total_items += len(items)
# Check archive
archive_items = self._scan_directory(self.archive_path)
return {
'status': 'ready',
'timestamp': datetime.now().isoformat(),
'inbox': inbox_status,
'archive': {
'count': len(archive_items),
'path': str(self.archive_path)
},
'total_items': total_items
}
# ==================== Item Processing ====================
def process_item(self, item: Dict, target_stage: str = None) -> Dict[str, Any]:
"""
Process a single item through the workflow
This method calls skills 2-4 to process items:
- Skill 2: Link parsing / URL extraction
- Skill 3: Metadata extraction
- Skill 4: Content analysis
Args:
item: Item dictionary with path and metadata
target_stage: Target stage to process to
Returns:
Processing result dictionary
"""
result = {
'item': item,
'success': False,
'stages_completed': [],
'output_path': None,
'error': None
}
try:
item_path = Path(item['path'])
# Determine processing stage based on current location
current_stage = self._get_stage_from_path(item_path)
# Skill 2: Link/URL Parsing
if current_stage in ['pending', None] or target_stage == 'metadata':
logger.info(f"Stage 1 → 2: Processing link/URL for {item['name']}")
# TODO: Integrate with link-parser skill
result['stages_completed'].append('link_parsing')
# Skill 3: Metadata Extraction
if current_stage in ['metadata', 'pending'] or target_stage == 'content':
logger.info(f"Stage 2 → 3: Extracting metadata for {item['name']}")
# TODO: Integrate with zhihu-metadata skill
result['stages_completed'].append('metadata_extraction')
# Skill 4: Content Analysis
if current_stage in ['content', 'metadata'] or target_stage == 'analyzed':
logger.info(f"Stage 3 → 4: Analyzing content for {item['name']}")
# TODO: Integrate with bilibili-analyzer skill
result['stages_completed'].append('content_analysis')
# Move to next stage if successful
if result['stages_completed']:
self._move_to_next_stage(item_path, result['stages_completed'][-1])
result['success'] = True
except Exception as e:
result['error'] = str(e)
logger.error(f"Processing failed: {e}")
return result
def _get_stage_from_path(self, path: Path) -> Optional[str]:
"""Determine which stage a file is in based on its path"""
for stage in self.INBOX_DIRS:
if stage in str(path):
return stage
return None
def _move_to_next_stage(self, item_path: Path, completed_stage: str) -> None:
"""Move item to the next stage directory"""
stage_order = ['pending', 'metadata', 'content', 'analyzed']
try:
current_idx = stage_order.index(completed_stage)
if current_idx < len(stage_order) - 1:
next_stage = stage_order[current_idx + 1]
next_path = self.base_path / next_stage / item_path.name
# Move file
if item_path.exists():
item_path.rename(next_path)
logger.info(f"Moved {item_path.name} → {next_stage}/")
except ValueError:
logger.warning(f"Unknown stage: {completed_stage}")
def process_all(self, scope: str = "pending", limit: int = None) -> Dict[str, Any]:
"""
Process all items in specified scope
Args:
scope: Which directories to process
limit: Maximum items to process
Returns:
Processing summary
"""
summary = {
'scope': scope,
'processed': 0,
'failed': 0,
'results': []
}
inbox = self.scan_inbox(scope=scope)
for dir_name, items in inbox.items():
if limit and summary['processed'] >= limit:
break
for item in items:
if limit and summary['processed'] >= limit:
break
result = self.process_item(item)
summary['results'].append(result)
if result['success']:
summary['processed'] += 1
else:
summary['failed'] += 1
return summary
# ==================== Archiving ====================
def archive_item(self, item: Dict, analysis: Dict = None) -> Dict[str, Any]:
"""
Archive a processed item to the archive directory
Args:
item: Item to archive
analysis: Optional analysis metadata
Returns:
Archiving result
"""
result = {
'success': False,
'original_path': item['path'],
'archive_path': None,
'error': None
}
try:
item_path = Path(item['path'])
if not item_path.exists():
result['error'] = f"Item not found: {item['path']}"
return result
# Create archive subdirectory by date
date_dir = self.archive_path / datetime.now().strftime('%Y-%m-%d')
if not date_dir.exists():
date_dir.mkdir(parents=True, exist_ok=True)
# Move item to archive
archive_path = date_dir / item_path.name
item_path.rename(archive_path)
# Save analysis metadata if provided
if analysis:
meta_path = date_dir / f"{item_path.stem}_meta.json"
with open(meta_path, 'w', encoding='utf-8') as f:
json.dump({
'archived_at': datetime.now().isoformat(),
'analysis': analysis
}, f, ensure_ascii=False, indent=2)
result['success'] = True
result['archive_path'] = str(archive_path)
logger.info(f"Archived: {item['name']} → {archive_path}")
except Exception as e:
result['error'] = str(e)
logger.error(f"Archive failed: {e}")
return result
# ==================== Subscriptions ====================
def check_subscriptions(self) -> Dict[str, Any]:
"""
Check for updates from subscribed sources
Returns:
Subscription check results
"""
result = {
'checked_at': datetime.now().isoformat(),
'sources': {},
'new_items': 0,
'errors': []
}
# Load subscription list
subscriptions = self._load_subscriptions()
for sub in subscriptions:
try:
source_name = sub.get('name', 'unknown')
source_type = sub.get('type', 'rss')
# Placeholder: Check for new content
logger.info(f"Checking subscription: {source_name}")
# TODO: Implement actual subscription checking
# - RSS feeds
# - YouTube channel updates
# - Bilibili user updates
# - Custom web hooks
result['sources'][source_name] = {
'type': source_type,
'status': 'checked',
'new_items': 0
}
except Exception as e:
result['errors'].append({
'source': sub.get('name', 'unknown'),
'error': str(e)
})
return result
def _load_subscriptions(self) -> List[Dict]:
"""Load subscriptions from file"""
if self.subscriptions_file.exists():
try:
with open(self.subscriptions_file, 'r', encoding='utf-8') as f:
return json.load(f)
except Exception as e:
logger.warning(f"Failed to load subscriptions: {e}")
return []
# ==================== Reporting ====================
def generate_report(self, period: str = "daily") -> Dict[str, Any]:
"""
Generate workflow report
Args:
period: Report period (daily, weekly, monthly)
Returns:
Report dictionary
"""
# Calculate date range
now = datetime.now()
if period == "daily":
start_date = now - timedelta(days=1)
elif period == "weekly":
start_date = now - timedelta(weeks=1)
elif period == "monthly":
start_date = now - timedelta(days=30)
else:
start_date = now - timedelta(days=1)
# Gather statistics
status = self.get_status()
inbox = self.scan_inbox(scope="all")
# Build report
report = {
'generated_at': now.isoformat(),
'period': period,
'start_date': start_date.isoformat(),
'summary': {
'total_items_inbox': status['total_items'],
'total_archived': status['archive']['count'],
'items_by_stage': status['inbox']
},
'processing_stats': {
'pending': len(inbox.get('pending', [])),
'metadata': len(inbox.get('metadata', [])),
'content': len(inbox.get('content', [])),
'analyzed': len(inbox.get('analyzed', []))
},
'recommendations': self._generate_recommendations(status, inbox)
}
# Save report
if self.report_file:
self.report_file.parent.mkdir(parents=True, exist_ok=True)
with open(self.report_file, 'w', encoding='utf-8') as f:
json.dump(report, f, ensure_ascii=False, indent=2)
logger.info(f"Report saved: {self.report_file}")
return report
def _generate_recommendations(self, status: Dict, inbox: Dict) -> List[str]:
"""Generate processing recommendations"""
recommendations = []
pending_count = status['inbox'].get('pending', {}).get('count', 0)
analyzed_count = status['inbox'].get('analyzed', {}).get('count', 0)
if pending_count > 10:
recommendations.append(f"⚠️ 待处理内容较多 ({pending_count} 项),建议优先处理")
if analyzed_count > 20:
recommendations.append(f"📦 已分析内容较多 ({analyzed_count} 项),建议归档")
if status['total_items'] == 0:
recommendations.append("📭 收件箱为空,暂无待处理内容")
return recommendations
# ==================== CLI Interface ====================
def run(self, trigger: str = "manual", scope: str = "all",
limit: int = None, report: bool = False):
"""
Main entry point for running the workflow
Args:
trigger: Trigger type (manual, scheduled)
scope: Processing scope
limit: Maximum items to process
report: Generate report after processing
"""
logger.info(f"KB Workflow started: trigger={trigger}, scope={scope}")
# Handle different triggers
if trigger == "status":
result = self.get_status()
elif trigger == "report":
result = self.generate_report()
elif trigger == "subscriptions":
result = self.check_subscriptions()
else:
# Process items
result = self.process_all(scope=scope, limit=limit)
# Generate report if requested
if report:
report_result = self.generate_report()
result['report'] = report_result
return result
def main():
"""CLI entry point"""
parser = argparse.ArgumentParser(
description='KB Workflow - Knowledge Base Workflow Orchestration',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Get workflow status
python kb_workflow.py --trigger status
# Process pending items manually
python kb_workflow.py --trigger manual --scope pending
# Process up to 5 items
python kb_workflow.py --trigger manual --limit 5
# Generate daily report
python kb_workflow.py --trigger scheduled --report
# Check subscriptions
python kb_workflow.py --trigger scheduled --scope subscriptions
"""
)
parser.add_argument(
'--trigger',
type=str,
default='manual',
choices=['manual', 'scheduled', 'webhook', 'status', 'report', 'subscriptions'],
help='Workflow trigger type (default: manual)'
)
parser.add_argument(
'--scope',
type=str,
default='all',
choices=['all', 'pending', 'metadata', 'content', 'analyzed', 'archive'],
help='Processing scope (default: all)'
)
parser.add_argument(
'--limit',
type=int,
default=None,
help='Maximum items to process'
)
parser.add_argument(
'--report',
action='store_true',
help='Generate report after processing'
)
parser.add_argument(
'--output',
type=str,
choices=['json', 'text'],
default='json',
help='Output format (default: json)'
)
parser.add_argument(
'--base-path',
type=str,
default=None,
help='Base path for knowledge base'
)
parser.add_argument(
'--verbose',
action='store_true',
help='Enable verbose logging'
)
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
# Initialize workflow
workflow = KBWorkflow(base_path=args.base_path)
# Run workflow
result = workflow.run(
trigger=args.trigger,
scope=args.scope,
limit=args.limit,
report=args.report
)
# Output result
if args.output == 'json':
print(json.dumps(result, ensure_ascii=False, indent=2, default=str))
else:
print(result)
return 0 if result else 1
if __name__ == '__main__':
sys.exit(main())
@@ -0,0 +1,36 @@
{
"generated_at": "2026-02-10T12:13:35.989689",
"period": "daily",
"start_date": "2026-02-09T12:13:35.989689",
"summary": {
"total_items_inbox": 0,
"total_archived": 0,
"items_by_stage": {
"pending": {
"count": 0,
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/pending"
},
"metadata": {
"count": 0,
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/metadata"
},
"content": {
"count": 0,
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/content"
},
"analyzed": {
"count": 0,
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/analyzed"
}
}
},
"processing_stats": {
"pending": 0,
"metadata": 0,
"content": 0,
"analyzed": 0
},
"recommendations": [
"📭 收件箱为空,暂无待处理内容"
]
}