feat: Add KB knowledge base series skills
- kb-link-parser: Multi-platform URL parser (Bilibili, YouTube, Zhihu) - kb-bilibili-metadata: Bilibili video/article/up主 metadata extractor - kb-zhihu-metadata: Zhihu question/answer/user metadata extractor - kb-bilibili-content: Bilibili video download and content extraction - kb-ocr-processor: OCR text extraction from video frames - kb-deep-analyzer: AI-powered deep content analysis - kb-workflow: Workflow orchestration (inbox, archive, subscriptions, report) Also add: - ARCHITECTURE_FLOWCHART.md: System architecture diagrams - DETAILED_DESIGN.md: Complete design documentation
This commit is contained in:
@@ -0,0 +1,285 @@
|
||||
---
|
||||
name: kb-workflow
|
||||
description: 知识库工作流编排技能,提供收件箱扫描、内容处理、归档、订阅检测和汇报生成功能。
|
||||
metadata:
|
||||
short-description: 知识库工作流编排
|
||||
---
|
||||
|
||||
# KB Workflow
|
||||
|
||||
## Description
|
||||
|
||||
KB Workflow 是一个知识库工作流编排技能,负责协调和管理知识库内容的完整生命周期:
|
||||
|
||||
**核心功能**:
|
||||
- 📥 **收件箱扫描** - 扫描四个子目录 (pending, metadata, content, analyzed)
|
||||
- 🔄 **内容处理** - 调用技能2-4处理内容
|
||||
- 📦 **智能归档** - 将处理完成的内容归档到 archive/
|
||||
- 🔔 **订阅检测** - 检测订阅源的更新
|
||||
- 📊 **汇报生成** - 生成处理统计和推荐
|
||||
|
||||
## Workflow Stages
|
||||
|
||||
```
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐
|
||||
│ pending │ → │ metadata│ → │ content │ → │ analyzed│ → │ archive/ │
|
||||
└─────────┘ └─────────┘ └─────────┘ └─────────┘ └──────────┘
|
||||
↓ ↓ ↓ ↓
|
||||
Skill 2 Skill 3 Skill 4 ↓
|
||||
(Link解析) (元数据提取) (内容分析) 最终归档
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
KB Workflow 作为 OpenClaw 技能套件的一部分,无需额外安装。
|
||||
|
||||
确保以下依赖已配置:
|
||||
- Python 3.8+
|
||||
- 知识库目录结构 (knowledge-base/)
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
knowledge-base/
|
||||
├── pending/ # 待处理的新内容
|
||||
├── metadata/ # 已提取元数据
|
||||
├── content/ # 已提取内容
|
||||
├── analyzed/ # 已完成分析
|
||||
├── archive/ # 归档内容
|
||||
│ └── YYYY-MM-DD/ # 按日期归档
|
||||
└── reports/ # 生成的汇报
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### 命令行用法
|
||||
|
||||
```bash
|
||||
# 查看工作流状态
|
||||
python kb_workflow.py --trigger status
|
||||
|
||||
# 手动处理待处理内容
|
||||
python kb_workflow.py --trigger manual --scope pending
|
||||
|
||||
# 处理所有新内容(限制5个)
|
||||
python kb_workflow.py --trigger manual --limit 5
|
||||
|
||||
# 生成日报
|
||||
python kb_workflow.py --trigger scheduled --report
|
||||
|
||||
# 检查订阅更新
|
||||
python kb_workflow.py --trigger subscriptions
|
||||
```
|
||||
|
||||
### 参数说明
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
|------|------|--------|
|
||||
| `--trigger` | 触发类型 | `manual` |
|
||||
| `--scope` | 处理范围 | `all` |
|
||||
| `--limit` | 处理数量限制 | 无限制 |
|
||||
| `--report` | 生成汇报 | False |
|
||||
| `--output` | 输出格式 | `json` |
|
||||
|
||||
### 触发类型
|
||||
|
||||
- `manual` - 手动触发,立即处理
|
||||
- `scheduled` - 定时任务触发
|
||||
- `webhook` - Webhook 触发
|
||||
- `status` - 仅查看状态
|
||||
- `report` - 生成汇报
|
||||
- `subscriptions` - 检查订阅
|
||||
|
||||
### 作用域
|
||||
|
||||
- `all` - 所有目录
|
||||
- `pending` - 仅待处理
|
||||
- `metadata` - 仅元数据阶段
|
||||
- `content` - 仅内容阶段
|
||||
- `analyzed` - 仅已完成分析
|
||||
- `archive` - 归档目录
|
||||
|
||||
## Integration
|
||||
|
||||
### 技能2-4 集成
|
||||
|
||||
KB Workflow 协调以下子技能:
|
||||
|
||||
1. **Skill 2: Link Parser** - 解析 URL/链接
|
||||
2. **Skill 3: Zhihu Metadata** - 提取元数据
|
||||
3. **Skill 4: Bilibili Analyzer** - 内容分析
|
||||
|
||||
### 调用示例
|
||||
|
||||
```python
|
||||
from kb_workflow import KBWorkflow
|
||||
|
||||
# 初始化
|
||||
workflow = KBWorkflow(base_path="/path/to/knowledge-base")
|
||||
|
||||
# 扫描收件箱
|
||||
inbox = workflow.scan_inbox(scope="pending")
|
||||
print(f"待处理: {len(inbox['pending'])} 项")
|
||||
|
||||
# 处理项目
|
||||
for item in inbox['pending'][:5]:
|
||||
result = workflow.process_item(item)
|
||||
print(f"处理结果: {result['success']}")
|
||||
|
||||
# 查看状态
|
||||
status = workflow.get_status()
|
||||
print(json.dumps(status, indent=2))
|
||||
|
||||
# 生成汇报
|
||||
report = workflow.generate_report(period="daily")
|
||||
```
|
||||
|
||||
## Input/Output Formats
|
||||
|
||||
### Input
|
||||
|
||||
#### 扫描收件箱输入
|
||||
```json
|
||||
{
|
||||
"scope": "all", // 或 "pending", "metadata", "content", "analyzed"
|
||||
"limit": 10 // 可选,限制返回数量
|
||||
}
|
||||
```
|
||||
|
||||
#### 处理项目输入
|
||||
```json
|
||||
{
|
||||
"name": "example.md",
|
||||
"path": "/path/to/knowledge-base/pending/example.md",
|
||||
"extension": ".md",
|
||||
"size": 1024,
|
||||
"modified": "2026-02-10T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### Output
|
||||
|
||||
#### 状态输出
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"timestamp": "2026-02-10T12:00:00",
|
||||
"inbox": {
|
||||
"pending": {"count": 5, "path": "/path/pending"},
|
||||
"metadata": {"count": 3, "path": "/path/metadata"},
|
||||
"content": {"count": 2, "path": "/path/content"},
|
||||
"analyzed": {"count": 8, "path": "/path/analyzed"}
|
||||
},
|
||||
"archive": {
|
||||
"count": 120,
|
||||
"path": "/path/archive"
|
||||
},
|
||||
"total_items": 18
|
||||
}
|
||||
```
|
||||
|
||||
#### 处理结果输出
|
||||
```json
|
||||
{
|
||||
"item": {"name": "example.md", ...},
|
||||
"success": true,
|
||||
"stages_completed": ["link_parsing", "metadata_extraction", "content_analysis"],
|
||||
"output_path": "/path/analyzed/example.md",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
#### 汇报输出
|
||||
```json
|
||||
{
|
||||
"generated_at": "2026-02-10T12:00:00",
|
||||
"period": "daily",
|
||||
"summary": {
|
||||
"total_items_inbox": 18,
|
||||
"total_archived": 120,
|
||||
"items_by_stage": {...}
|
||||
},
|
||||
"processing_stats": {...},
|
||||
"recommendations": [
|
||||
"⚠️ 待处理内容较多 (15 项),建议优先处理",
|
||||
"📦 已分析内容较多 (25 项),建议归档"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
KB Workflow 无需额外配置,自动使用以下路径:
|
||||
- 知识库根目录: `skills/kb-workflow/knowledge-base/`
|
||||
- 归档目录: `skills/kb-workflow/knowledge-base/archive/`
|
||||
- 汇报目录: `skills/kb-workflow/knowledge-base/reports/`
|
||||
|
||||
### 自定义路径
|
||||
|
||||
```python
|
||||
from kb_workflow import KBWorkflow
|
||||
|
||||
# 使用自定义路径
|
||||
workflow = KBWorkflow(
|
||||
base_path="/custom/knowledge-base",
|
||||
archive_path="/custom/archive",
|
||||
output_format="json"
|
||||
)
|
||||
```
|
||||
|
||||
## Workflow Steps
|
||||
|
||||
### 完整处理流程
|
||||
|
||||
1. **扫描收件箱**
|
||||
- 扫描 `pending/` 目录获取新内容
|
||||
- 扫描其他阶段目录获取待继续处理的内容
|
||||
|
||||
2. **分类处理**
|
||||
- 新内容 → 提取链接/URL (Skill 2)
|
||||
- 待提取元数据 → 提取元数据 (Skill 3)
|
||||
- 待分析内容 → 深度分析 (Skill 4)
|
||||
|
||||
3. **自动流转**
|
||||
- 处理完成后自动移动到下一阶段目录
|
||||
|
||||
4. **归档**
|
||||
- 分析完成后可归档到 `archive/YYYY-MM-DD/`
|
||||
|
||||
5. **汇报**
|
||||
- 生成处理统计
|
||||
- 提供处理建议
|
||||
|
||||
### 定时任务示例
|
||||
|
||||
```bash
|
||||
# 每天早上9点检查并处理新内容
|
||||
0 9 * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger scheduled --scope pending --report
|
||||
|
||||
# 每小时检查订阅更新
|
||||
0 * * * * cd /path/to/skills/kb-workflow && python kb_workflow.py --trigger subscriptions
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### 常见问题
|
||||
|
||||
**Q: 扫描不到内容?**
|
||||
A: 确保知识库目录存在且包含正确的子目录 (pending, metadata, content, analyzed)
|
||||
|
||||
**Q: 处理失败?**
|
||||
A: 检查日志输出,使用 `--verbose` 参数获取详细调试信息
|
||||
|
||||
**Q: 如何查看处理进度?**
|
||||
A: 使用 `--trigger status` 查看各阶段的项目数量
|
||||
|
||||
## Related Skills
|
||||
|
||||
- **obsidian-kb** - Obsidian 知识库管理
|
||||
- **link-parser** - 链接解析
|
||||
- **zhihu-metadata** - 知乎元数据提取
|
||||
- **bilibili-analyzer** - B站视频分析
|
||||
|
||||
---
|
||||
|
||||
*KB Workflow v1.0 - Knowledge Base Workflow Orchestration*
|
||||
Executable
+427
@@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Auto-trigger handler for obsidian-kb skill.
|
||||
|
||||
Detects video/article links in user messages and automatically processes them.
|
||||
Supports Bilibili, YouTube, and Zhihu.
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
import os
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Add path for imports
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), 'src'))
|
||||
|
||||
# Import short link resolver
|
||||
from shortlink_resolver import (
|
||||
ShortLinkResolver,
|
||||
detect_and_format,
|
||||
PlaywrightResolver
|
||||
)
|
||||
|
||||
# 导入主流程
|
||||
try:
|
||||
from main_pipeline import MainPipeline
|
||||
MAIN_PIPELINE_AVAILABLE = True
|
||||
except ImportError:
|
||||
MAIN_PIPELINE_AVAILABLE = False
|
||||
|
||||
# Pattern definitions for auto-detection
|
||||
PATTERNS = {
|
||||
'bilibili_full': re.compile(r'https?://(?:www\.)?bilibili\.com/video/(BV\w+)'),
|
||||
'bilibili_short': re.compile(r'https?://b23\.tv/(\w+)'),
|
||||
'bilibili_bv': re.compile(r'\b(BV[a-zA-Z0-9]{10})\b'),
|
||||
'youtube_full': re.compile(r'https?://(?:www\.)?youtube\.com/watch\?v=([a-zA-Z0-9_-]{11})'),
|
||||
'youtube_short': re.compile(r'https?://youtu\.be/([a-zA-Z0-9_-]{11})'),
|
||||
'zhihu_article': re.compile(r'https?://zhuanlan\.zhihu\.com/p/(\d+)'),
|
||||
'zhihu_answer': re.compile(r'https?://(?:www\.)?zhihu\.com/question/\d+/answer/(\d+)'),
|
||||
}
|
||||
|
||||
|
||||
def detect_links(text: str) -> List[Dict[str, str]]:
|
||||
"""
|
||||
Detect video/article links in text.
|
||||
|
||||
Args:
|
||||
text: Input text to search
|
||||
|
||||
Returns:
|
||||
List of detected links with metadata:
|
||||
[
|
||||
{
|
||||
'url': 'https://...',
|
||||
'platform': 'bilibili',
|
||||
'type': 'video',
|
||||
'id': 'BVxxx',
|
||||
'is_short': False
|
||||
},
|
||||
...
|
||||
]
|
||||
"""
|
||||
detected = []
|
||||
seen_ids = set() # Track IDs to avoid duplicates
|
||||
|
||||
for pattern_name, pattern in PATTERNS.items():
|
||||
matches = pattern.finditer(text)
|
||||
|
||||
for match in matches:
|
||||
# Determine platform and type
|
||||
if 'bilibili' in pattern_name:
|
||||
platform = 'bilibili'
|
||||
content_type = 'video'
|
||||
if 'short' in pattern_name:
|
||||
is_short = True
|
||||
url = match.group(0)
|
||||
content_id = None # Need to resolve short URL
|
||||
else:
|
||||
is_short = False
|
||||
if pattern_name == 'bilibili_bv':
|
||||
url = f"https://www.bilibili.com/video/{match.group(1)}"
|
||||
else:
|
||||
url = match.group(0)
|
||||
content_id = match.group(1)
|
||||
|
||||
elif 'youtube' in pattern_name:
|
||||
platform = 'youtube'
|
||||
content_type = 'video'
|
||||
is_short = 'short' in pattern_name
|
||||
url = match.group(0)
|
||||
content_id = match.group(1)
|
||||
|
||||
elif 'zhihu' in pattern_name:
|
||||
platform = 'zhihu'
|
||||
content_type = 'article'
|
||||
is_short = False
|
||||
url = match.group(0)
|
||||
content_id = match.group(1)
|
||||
|
||||
else:
|
||||
continue
|
||||
|
||||
# Skip duplicates by URL or ID
|
||||
unique_key = url if (not content_id or is_short) else content_id
|
||||
if unique_key in seen_ids:
|
||||
continue
|
||||
seen_ids.add(unique_key)
|
||||
|
||||
detected.append({
|
||||
'url': url,
|
||||
'platform': platform,
|
||||
'type': content_type,
|
||||
'id': content_id,
|
||||
'is_short': is_short,
|
||||
'pattern': pattern_name
|
||||
})
|
||||
|
||||
return detected
|
||||
|
||||
|
||||
def format_preview(detected: List[Dict[str, str]]) -> str:
|
||||
"""
|
||||
Format a preview message for detected links.
|
||||
|
||||
Args:
|
||||
detected: List of detected links
|
||||
|
||||
Returns:
|
||||
Formatted preview message
|
||||
"""
|
||||
if not detected:
|
||||
return ""
|
||||
|
||||
lines = ["🔍 检测到内容链接:\n"]
|
||||
|
||||
for i, item in enumerate(detected, 1):
|
||||
platform_emoji = {
|
||||
'bilibili': '📺',
|
||||
'youtube': '▶️',
|
||||
'zhihu': '📝'
|
||||
}.get(item['platform'], '🔗')
|
||||
|
||||
platform_name = {
|
||||
'bilibili': 'B站视频',
|
||||
'youtube': 'YouTube视频',
|
||||
'zhihu': '知乎文章'
|
||||
}.get(item['platform'], item['platform'])
|
||||
|
||||
lines.append(f"{platform_emoji} [{i}] {platform_name}")
|
||||
lines.append(f" URL: {item['url']}")
|
||||
|
||||
if item['is_short']:
|
||||
lines.append(f" ⚠️ 短链接,需要用户提供完整URL或ID")
|
||||
elif item['id']:
|
||||
lines.append(f" ID: {item['id']}")
|
||||
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_short_link_help(short_link: Optional[str] = None) -> str:
|
||||
"""
|
||||
Format help message for short links.
|
||||
|
||||
Args:
|
||||
short_link: Optional short link URL to mention
|
||||
|
||||
Returns:
|
||||
Help message
|
||||
"""
|
||||
return ShortLinkResolver.format_short_link_help(short_link)
|
||||
|
||||
|
||||
def get_action_prompt(detected: List[Dict[str, str]]) -> str:
|
||||
"""
|
||||
Generate prompt asking user what to do.
|
||||
|
||||
Args:
|
||||
detected: List of detected links
|
||||
|
||||
Returns:
|
||||
Action prompt
|
||||
"""
|
||||
has_short = any(item['is_short'] for item in detected)
|
||||
|
||||
if has_short:
|
||||
return f"{format_short_link_help()}\n\n其他链接是否立即处理?(y/n)"
|
||||
|
||||
count = len(detected)
|
||||
if count == 1:
|
||||
return "是否添加到收件箱并处理?(y/n)"
|
||||
else:
|
||||
return f"是否将上述 {count} 个内容添加到收件箱?(y/n)"
|
||||
|
||||
|
||||
def should_auto_process(text: str) -> bool:
|
||||
"""
|
||||
Check if user wants to auto-process based on their message.
|
||||
|
||||
Args:
|
||||
text: User's message
|
||||
|
||||
Returns:
|
||||
True if user indicates they want immediate processing
|
||||
"""
|
||||
auto_keywords = ['立即处理', '处理', 'process', '--process', '马上']
|
||||
return any(keyword in text.lower() for keyword in auto_keywords)
|
||||
|
||||
|
||||
def process_link(url: str, platform: str, auto_process: bool = False) -> Tuple[bool, str]:
|
||||
"""
|
||||
Process a single link (extract metadata, add to inbox).
|
||||
|
||||
Args:
|
||||
url: Content URL
|
||||
platform: Platform name (bilibili/youtube/zhihu)
|
||||
auto_process: Whether to immediately process after adding
|
||||
|
||||
Returns:
|
||||
(success, message) tuple
|
||||
"""
|
||||
try:
|
||||
# Import here to avoid issues if dependencies not available
|
||||
from sources.bilibili import BilibiliAdapter
|
||||
from models import Content, ContentStatus
|
||||
from storage import Storage
|
||||
|
||||
# Only support Bilibili for now
|
||||
if platform != 'bilibili':
|
||||
return False, f"暂不支持 {platform},请手动添加"
|
||||
|
||||
# Handle short links
|
||||
if 'b23.tv' in url:
|
||||
return False, "短链接无法自动处理,请提供完整URL或BV号"
|
||||
|
||||
# Create adapter
|
||||
adapter = BilibiliAdapter()
|
||||
|
||||
# Extract metadata
|
||||
content = adapter.extract_content(url)
|
||||
if not content:
|
||||
return False, "无法提取视频信息"
|
||||
|
||||
# Get transcript
|
||||
transcript = adapter.get_transcript(content)
|
||||
if transcript:
|
||||
content.transcript = transcript
|
||||
|
||||
# Save to inbox
|
||||
content.status = ContentStatus.CONFIRMED
|
||||
storage = Storage()
|
||||
storage.save_content(content)
|
||||
|
||||
# Build success message
|
||||
msg = f"✅ 已添加到收件箱\n"
|
||||
msg += f" 标题: {content.title}\n"
|
||||
msg += f" UP主: {content.author}\n"
|
||||
msg += f" 时长: {content.duration}秒\n"
|
||||
|
||||
if auto_process:
|
||||
# Process immediately
|
||||
try:
|
||||
from src.pipeline import ProcessingPipeline
|
||||
pipeline = ProcessingPipeline()
|
||||
content_text = content.transcript or content.summary or ""
|
||||
result = pipeline.process_full(content, content_text)
|
||||
storage.save_content(result)
|
||||
|
||||
msg += f"\n🔄 处理完成\n"
|
||||
msg += f" 📄 笔记: {result.output_path or 'N/A'}\n"
|
||||
msg += f" 🏷️ 标签: {', '.join(result.tags)}"
|
||||
except Exception as e:
|
||||
msg += f"\n⚠️ 处理失败: {e}"
|
||||
|
||||
return True, msg
|
||||
|
||||
except ImportError as e:
|
||||
return False, f"依赖模块未安装: {e}"
|
||||
except Exception as e:
|
||||
return False, f"处理失败: {e}"
|
||||
|
||||
|
||||
def process_video_full(source_id: str, enable_git_sync: bool = False) -> Tuple[bool, str]:
|
||||
"""
|
||||
使用主流程完整处理视频(新接口)
|
||||
|
||||
完整流程: 提取 → 处理 → 输出 → 同步
|
||||
|
||||
Args:
|
||||
source_id: 视频ID(支持BV号、URL)
|
||||
enable_git_sync: 是否启用Git同步
|
||||
|
||||
Returns:
|
||||
(success, message) tuple
|
||||
"""
|
||||
if not MAIN_PIPELINE_AVAILABLE:
|
||||
return False, "主流程模块未可用,请检查依赖"
|
||||
|
||||
try:
|
||||
# 导入配置
|
||||
from src.config import get_config
|
||||
|
||||
# 获取配置
|
||||
config = get_config()
|
||||
|
||||
# 创建主流程
|
||||
pipeline = MainPipeline(config=config, enable_git_sync=enable_git_sync)
|
||||
|
||||
# 处理视频
|
||||
result = pipeline.process_video(source_id)
|
||||
|
||||
# 返回结果
|
||||
if result['success']:
|
||||
return True, result['message']
|
||||
else:
|
||||
return False, result.get('error', 'Unknown error')
|
||||
|
||||
except Exception as e:
|
||||
return False, f"处理失败: {e}"
|
||||
|
||||
|
||||
def handle_message(text: str) -> Optional[str]:
|
||||
"""
|
||||
Main entry point: handle incoming message and return response.
|
||||
|
||||
Args:
|
||||
text: User's message text
|
||||
|
||||
Returns:
|
||||
Response message, or None if no action needed
|
||||
"""
|
||||
# First, check for short links and provide help
|
||||
has_short, short_msg = detect_and_format(text)
|
||||
|
||||
# Detect all links (including short ones)
|
||||
detected = detect_links(text)
|
||||
|
||||
if not detected:
|
||||
return None
|
||||
|
||||
# If we have short links, show help message first
|
||||
if has_short:
|
||||
return short_msg
|
||||
|
||||
# Check if this is a confirmation (y/yes)
|
||||
text_lower = text.lower().strip()
|
||||
if text_lower in ('y', 'yes', '是', '好的', 'ok'):
|
||||
# This would need state management - for now return a message
|
||||
# about using explicit command
|
||||
return "请使用 /kb-add <URL> 命令添加内容,或重新发送链接"
|
||||
|
||||
# Show preview
|
||||
preview = format_preview(detected)
|
||||
prompt = get_action_prompt(detected)
|
||||
|
||||
return f"{preview}\n{prompt}"
|
||||
|
||||
|
||||
# CLI interface for testing
|
||||
def main():
|
||||
"""CLI for testing the handler."""
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python3 handler.py <command> [options]")
|
||||
print("\nCommands:")
|
||||
print(" detect 'text' - 检测文本中的内容链接")
|
||||
print(" process <BV/URL> - 使用主流程处理视频")
|
||||
print(" process <BV/URL> --git - 处理视频并同步到Git")
|
||||
print("\nExamples:")
|
||||
print(' python3 handler.py detect "https://www.bilibili.com/video/BV1xx411c7mD"')
|
||||
print(' python3 handler.py process BV1xx411c7mD')
|
||||
print(' python3 handler.py process BV1xx411c7mD --git')
|
||||
sys.exit(1)
|
||||
|
||||
command = sys.argv[1].lower()
|
||||
|
||||
if command == 'detect':
|
||||
# 检测链接
|
||||
if len(sys.argv) < 3:
|
||||
print("Error: Please provide text to detect")
|
||||
sys.exit(1)
|
||||
text = ' '.join(sys.argv[2:])
|
||||
response = handle_message(text)
|
||||
if response:
|
||||
print(response)
|
||||
else:
|
||||
print("No content links detected")
|
||||
|
||||
elif command == 'process':
|
||||
# 使用主流程处理视频
|
||||
if len(sys.argv) < 3:
|
||||
print("Error: Please provide BV number or URL")
|
||||
sys.exit(1)
|
||||
|
||||
source_id = sys.argv[2]
|
||||
enable_git = '--git' in sys.argv
|
||||
|
||||
print(f"Processing video: {source_id}")
|
||||
if enable_git:
|
||||
print("Git sync: enabled")
|
||||
else:
|
||||
print("Git sync: disabled")
|
||||
print()
|
||||
|
||||
success, message = process_video_full(source_id, enable_git_sync=enable_git)
|
||||
|
||||
if success:
|
||||
print(message)
|
||||
sys.exit(0)
|
||||
else:
|
||||
print(f"\n❌ Failed: {message}")
|
||||
sys.exit(1)
|
||||
|
||||
else:
|
||||
# 兼容旧格式:直接传入文本进行检测
|
||||
text = sys.argv[1]
|
||||
response = handle_message(text)
|
||||
if response:
|
||||
print(response)
|
||||
else:
|
||||
print("No content links detected")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+489
@@ -0,0 +1,489 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Obsidian KB - 主流程集成
|
||||
|
||||
整合提取层、处理层、输出层,实现完整的数据流:
|
||||
提取(BilibiliExtractor) → 处理(DeepAnalyzer/ContentChunker) → 输出(ObsidianWriter) → 同步(GitSync)
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
from typing import Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# 添加项目路径
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from src.extractors.bilibili_extractor import BilibiliExtractor, ExtractionError
|
||||
from src.deep_analyzer import DeepAnalyzer
|
||||
from src.content_chunker import ContentChunker, Chunk
|
||||
from src.outputs.obsidian_writer import ObsidianWriter
|
||||
from src.outputs.git_sync import GitSync
|
||||
from src.models import Content, ContentType, ContentStatus, TimelineEntry
|
||||
from src.config import get_config
|
||||
|
||||
|
||||
# 配置日志
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
logging.StreamHandler(sys.stdout),
|
||||
logging.FileHandler('obsidian_kb.log', encoding='utf-8')
|
||||
]
|
||||
)
|
||||
logger = logging.getLogger('ObsidianKB')
|
||||
|
||||
|
||||
class MainPipeline:
|
||||
"""主流程集成器"""
|
||||
|
||||
def __init__(self, config=None, enable_git_sync: bool = False):
|
||||
"""
|
||||
初始化主流程
|
||||
|
||||
Args:
|
||||
config: 配置对象
|
||||
enable_git_sync: 是否启用Git同步
|
||||
"""
|
||||
self.config = config or get_config()
|
||||
self.enable_git_sync = enable_git_sync
|
||||
|
||||
logger.info("初始化主流程...")
|
||||
logger.info(f"Obsidian vault 路径: {self.config.vault_path}")
|
||||
logger.info(f"Git 同步: {'启用' if enable_git_sync else '禁用'}")
|
||||
|
||||
# 初始化各层组件
|
||||
self.extractor = BilibiliExtractor(config=self.config)
|
||||
self.analyzer = DeepAnalyzer(ai_client=None) # 暂时不传入ai_client
|
||||
self.chunker = ContentChunker()
|
||||
self.writer = ObsidianWriter(config=self.config)
|
||||
|
||||
# 初始化Git同步器(仅在需要时)
|
||||
self.git_sync = None
|
||||
if enable_git_sync and self._is_git_enabled():
|
||||
self.git_sync = GitSync(config=self.config)
|
||||
logger.info("Git 同步器已初始化")
|
||||
|
||||
def _is_git_enabled(self) -> bool:
|
||||
"""检查Git同步是否在配置中启用"""
|
||||
return self.config.get('git.enabled', False)
|
||||
|
||||
def process_video(self, source_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
处理单个视频的完整流程
|
||||
|
||||
Args:
|
||||
source_id: 视频ID(支持BV号、URL)
|
||||
|
||||
Returns:
|
||||
处理结果字典,包含:
|
||||
- success: bool - 是否成功
|
||||
- content: Content对象 - 处理的内容
|
||||
- message: str - 处理消息
|
||||
- error: str - 错误信息(如果失败)
|
||||
"""
|
||||
result = {
|
||||
'success': False,
|
||||
'content': None,
|
||||
'message': '',
|
||||
'error': ''
|
||||
}
|
||||
|
||||
try:
|
||||
logger.info("=" * 80)
|
||||
logger.info(f"开始处理视频: {source_id}")
|
||||
logger.info("=" * 80)
|
||||
|
||||
# Step 1: 提取层 - 提取视频信息
|
||||
logger.info("\n[Step 1/5] 提取视频信息...")
|
||||
extractor_data = self._extract_video(source_id)
|
||||
|
||||
# Step 2: 构建Content对象
|
||||
logger.info("\n[Step 2/5] 构建内容对象...")
|
||||
content = self._build_content(extractor_data)
|
||||
|
||||
# Step 3: 处理层 - 深度分析
|
||||
logger.info("\n[Step 3/5] 深度分析...")
|
||||
analysis_result = self._analyze_content(extractor_data)
|
||||
|
||||
# Step 4: 处理层 - 内容分片
|
||||
logger.info("\n[Step 4/5] 内容分片...")
|
||||
chunks = self._chunk_content(extractor_data)
|
||||
|
||||
# 合并分析结果和分片信息到content
|
||||
self._merge_analysis_to_content(content, analysis_result, chunks)
|
||||
|
||||
# Step 5: 输出层 - 写入Obsidian
|
||||
logger.info("\n[Step 5/5] 写入Obsidian...")
|
||||
output_path = self._write_obsidian_note(content)
|
||||
|
||||
# Step 6: 同步层 - Git同步(可选)
|
||||
if self.enable_git_sync and self.git_sync:
|
||||
logger.info("\n[Step 6/6] Git同步...")
|
||||
self._sync_to_git(content)
|
||||
|
||||
# 更新状态
|
||||
content.update_status(ContentStatus.GENERATED)
|
||||
content.output_path = output_path
|
||||
|
||||
# 构建成功结果
|
||||
result['success'] = True
|
||||
result['content'] = content
|
||||
result['message'] = self._format_success_message(content, chunks)
|
||||
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info(f"✅ 处理完成: {content.title}")
|
||||
logger.info(f" 📄 笔记路径: {output_path}")
|
||||
logger.info(f" 📊 字幕条数: {len(chunks) if chunks else 0}")
|
||||
logger.info("=" * 80 + "\n")
|
||||
|
||||
except ExtractionError as e:
|
||||
logger.error(f"❌ 提取失败: {e}")
|
||||
result['error'] = f"提取失败: {e}"
|
||||
result['message'] = f"提取视频信息失败: {e}"
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"❌ 处理失败: {e}", exc_info=True)
|
||||
result['error'] = str(e)
|
||||
result['message'] = f"处理过程中发生错误: {e}"
|
||||
|
||||
return result
|
||||
|
||||
def _extract_video(self, source_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
提取视频信息
|
||||
|
||||
Args:
|
||||
source_id: 视频ID或URL
|
||||
|
||||
Returns:
|
||||
提取器返回的数据结构
|
||||
|
||||
Raises:
|
||||
ExtractionError: 提取失败
|
||||
"""
|
||||
logger.info(f" 正在提取: {source_id}")
|
||||
|
||||
try:
|
||||
extractor_data = self.extractor.extract(source_id)
|
||||
logger.info(f" ✅ 提取成功: {extractor_data.get('metadata', {}).get('title', '未知标题')}")
|
||||
logger.info(f" 作者: {extractor_data.get('metadata', {}).get('author', '未知')}")
|
||||
logger.info(f" 时长: {extractor_data.get('metadata', {}).get('duration', '0')}")
|
||||
|
||||
# 检查字幕
|
||||
subtitle = extractor_data.get('subtitle', '')
|
||||
if subtitle:
|
||||
subtitle_lines = subtitle.split('\n')
|
||||
logger.info(f" 字幕: {len(subtitle_lines)} 条")
|
||||
else:
|
||||
logger.warning(f" ⚠️ 无字幕")
|
||||
|
||||
return extractor_data
|
||||
|
||||
except ExtractionError as e:
|
||||
logger.error(f" ❌ 提取失败: {e}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f" ❌ 提取异常: {e}", exc_info=True)
|
||||
raise ExtractionError(f"提取过程异常: {e}", source_id=source_id)
|
||||
|
||||
def _build_content(self, extractor_data: Dict[str, Any]) -> Content:
|
||||
"""
|
||||
从提取器数据构建Content对象
|
||||
|
||||
Args:
|
||||
extractor_data: 提取器返回的数据
|
||||
|
||||
Returns:
|
||||
Content对象
|
||||
"""
|
||||
metadata = extractor_data.get('metadata', {})
|
||||
|
||||
content = Content(
|
||||
source='bilibili',
|
||||
source_id=metadata.get('bvid', ''),
|
||||
url=metadata.get('url', ''),
|
||||
title=metadata.get('title', ''),
|
||||
author=metadata.get('author', ''),
|
||||
description=extractor_data.get('description', ''),
|
||||
duration=self._parse_duration(metadata.get('duration', '0')),
|
||||
content_type=ContentType.VIDEO,
|
||||
transcript=extractor_data.get('subtitle', ''),
|
||||
status=ContentStatus.SCANNED
|
||||
)
|
||||
|
||||
logger.info(f" ✅ Content对象已创建: {content.id}")
|
||||
return content
|
||||
|
||||
def _parse_duration(self, duration_str: str) -> Optional[int]:
|
||||
"""
|
||||
解析时长字符串为秒数
|
||||
|
||||
Args:
|
||||
duration_str: 时长字符串(如 "10:30")
|
||||
|
||||
Returns:
|
||||
秒数
|
||||
"""
|
||||
try:
|
||||
if ':' in duration_str:
|
||||
parts = duration_str.split(':')
|
||||
if len(parts) == 2:
|
||||
# MM:SS
|
||||
return int(parts[0]) * 60 + int(parts[1])
|
||||
elif len(parts) == 3:
|
||||
# HH:MM:SS
|
||||
return int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])
|
||||
except:
|
||||
pass
|
||||
return None
|
||||
|
||||
def _analyze_content(self, extractor_data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
深度分析内容
|
||||
|
||||
Args:
|
||||
extractor_data: 提取器返回的数据
|
||||
|
||||
Returns:
|
||||
分析结果字典
|
||||
"""
|
||||
try:
|
||||
logger.info(" 正在进行深度分析...")
|
||||
|
||||
# 使用DeepAnalyzer进行分析
|
||||
# DeepAnalyzer接收extractor_data作为输入
|
||||
analysis_result = self.analyzer.analyze(extractor_data)
|
||||
|
||||
# 将AnalysisResult转换为字典以便合并
|
||||
result_dict = {
|
||||
'main_topics': analysis_result.main_topics,
|
||||
'key_insights': analysis_result.key_insights,
|
||||
'action_items': analysis_result.action_items,
|
||||
'summary': analysis_result.summary,
|
||||
'timeline': analysis_result.timeline,
|
||||
'recommended_tags': analysis_result.recommended_tags,
|
||||
'full_analysis': analysis_result.full_analysis
|
||||
}
|
||||
|
||||
logger.info(f" ✅ 分析完成: {len(result_dict['main_topics'])} 个主题")
|
||||
|
||||
return result_dict
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f" ❌ 分析失败: {e}", exc_info=True)
|
||||
# 返回默认结果,不中断流程
|
||||
return {
|
||||
'main_topics': ['未分类'],
|
||||
'key_insights': [],
|
||||
'action_items': [],
|
||||
'summary': '分析失败,无法生成摘要',
|
||||
'timeline': [],
|
||||
'recommended_tags': ['未分类'],
|
||||
'full_analysis': ''
|
||||
}
|
||||
|
||||
def _chunk_content(self, extractor_data: Dict[str, Any]) -> list:
|
||||
"""
|
||||
内容分片
|
||||
|
||||
Args:
|
||||
extractor_data: 提取器返回的数据
|
||||
|
||||
Returns:
|
||||
Chunk对象列表
|
||||
"""
|
||||
try:
|
||||
logger.info(" 正在进行内容分片...")
|
||||
|
||||
# 使用ContentChunker进行分片
|
||||
chunks = self.chunker.chunk_from_extractor(extractor_data)
|
||||
|
||||
logger.info(f" ✅ 分片完成: {len(chunks)} 段")
|
||||
|
||||
return chunks
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f" ❌ 分片失败: {e}", exc_info=True)
|
||||
# 返回空列表,不中断流程
|
||||
return []
|
||||
|
||||
def _merge_analysis_to_content(self, content: Content, analysis_result: Dict[str, Any], chunks: list):
|
||||
"""
|
||||
将分析结果和分片信息合并到Content对象
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
analysis_result: 分析结果字典
|
||||
chunks: Chunk对象列表
|
||||
"""
|
||||
# 合并基本信息
|
||||
content.summary = analysis_result.get('summary', '')
|
||||
content.key_points = analysis_result.get('main_topics', [])
|
||||
content.tags = analysis_result.get('recommended_tags', [])
|
||||
|
||||
# 合并时间轴
|
||||
timeline_data = analysis_result.get('timeline', [])
|
||||
if timeline_data:
|
||||
content.timeline = [
|
||||
TimelineEntry(
|
||||
timestamp=item.get('start_time', ''),
|
||||
title=item.get('title', ''),
|
||||
description=item.get('summary', '')
|
||||
)
|
||||
for item in timeline_data
|
||||
]
|
||||
else:
|
||||
# 从chunks生成时间轴
|
||||
if chunks:
|
||||
content.timeline = [
|
||||
TimelineEntry(
|
||||
timestamp=chunk.time_range,
|
||||
title=f"片段 {chunk.index + 1}",
|
||||
description=f"时长 {int(chunk.duration)}秒"
|
||||
)
|
||||
for chunk in chunks
|
||||
]
|
||||
|
||||
logger.info(f" ✅ 数据合并完成: 时间轴 {len(content.timeline)} 条")
|
||||
|
||||
def _write_obsidian_note(self, content: Content) -> Optional[str]:
|
||||
"""
|
||||
写入Obsidian笔记
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
|
||||
Returns:
|
||||
文件路径,失败返回None
|
||||
"""
|
||||
try:
|
||||
logger.info(f" 正在写入笔记...")
|
||||
|
||||
# 确保输出目录存在
|
||||
self.config.vault_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 调用ObsidianWriter写入
|
||||
output_path = self.writer.write(content, self.config)
|
||||
|
||||
if output_path:
|
||||
logger.info(f" ✅ 笔记已写入: {output_path}")
|
||||
else:
|
||||
logger.warning(f" ⚠️ 笔记写入失败")
|
||||
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f" ❌ 写入失败: {e}", exc_info=True)
|
||||
return None
|
||||
|
||||
def _sync_to_git(self, content: Content):
|
||||
"""
|
||||
同步到Git仓库
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
"""
|
||||
if not self.git_sync:
|
||||
logger.warning(" ⚠️ Git同步器未初始化,跳过同步")
|
||||
return
|
||||
|
||||
try:
|
||||
logger.info(" 正在同步到Git...")
|
||||
|
||||
success = self.git_sync.sync(self.config)
|
||||
|
||||
if success:
|
||||
logger.info(" ✅ Git同步成功")
|
||||
else:
|
||||
logger.warning(" ⚠️ Git同步失败")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f" ❌ Git同步异常: {e}", exc_info=True)
|
||||
# Git同步失败不视为整体失败
|
||||
|
||||
def _format_success_message(self, content: Content, chunks: list) -> str:
|
||||
"""
|
||||
格式化成功消息
|
||||
|
||||
Args:
|
||||
content: Content对象
|
||||
chunks: Chunk列表
|
||||
|
||||
Returns:
|
||||
格式化的消息字符串
|
||||
"""
|
||||
lines = [
|
||||
"✅ 处理完成!",
|
||||
"",
|
||||
f"📺 标题: {content.title}",
|
||||
f"👤 作者: {content.author}",
|
||||
f"📄 笔记: {content.output_path or 'N/A'}",
|
||||
]
|
||||
|
||||
if content.duration:
|
||||
lines.append(f"⏱️ 时长: {self._format_duration(content.duration)}")
|
||||
|
||||
if chunks:
|
||||
lines.append(f"📊 分片: {len(chunks)} 段")
|
||||
|
||||
if content.tags:
|
||||
lines.append(f"🏷️ 标签: {', '.join(content.tags[:5])}")
|
||||
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def _format_duration(self, seconds: int) -> str:
|
||||
"""格式化时长"""
|
||||
if not seconds:
|
||||
return "N/A"
|
||||
|
||||
hours = seconds // 3600
|
||||
minutes = (seconds % 3600) // 60
|
||||
secs = seconds % 60
|
||||
|
||||
if hours > 0:
|
||||
return f"{hours}小时{minutes}分钟"
|
||||
elif minutes > 0:
|
||||
return f"{minutes}分钟{secs}秒"
|
||||
else:
|
||||
return f"{secs}秒"
|
||||
|
||||
|
||||
def main():
|
||||
"""命令行入口"""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description='Obsidian KB - 主流程集成')
|
||||
parser.add_argument('source_id', help='视频ID(BV号或URL)')
|
||||
parser.add_argument('--git', action='store_true', help='启用Git同步')
|
||||
parser.add_argument('--config', help='配置文件路径')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 加载配置
|
||||
config = get_config(args.config) if args.config else get_config()
|
||||
|
||||
# 创建主流程
|
||||
pipeline = MainPipeline(config=config, enable_git_sync=args.git)
|
||||
|
||||
# 处理视频
|
||||
result = pipeline.process_video(args.source_id)
|
||||
|
||||
# 输出结果
|
||||
print(result['message'])
|
||||
|
||||
if not result['success']:
|
||||
print(f"\n❌ 失败: {result['error']}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,674 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
KB Workflow - Knowledge Base Workflow Orchestration
|
||||
|
||||
工作流编排技能,整合以下功能:
|
||||
- 扫描收件箱目录 (pending, metadata, content, analyzed)
|
||||
- 调用技能2-4处理内容
|
||||
- 归档处理完成的条目
|
||||
- 订阅更新检测
|
||||
- 生成汇报
|
||||
|
||||
Usage:
|
||||
python kb_workflow.py --trigger manual
|
||||
python kb_workflow.py --trigger scheduled
|
||||
python kb_workflow.py --status
|
||||
python kb_workflow.py --report
|
||||
python kb_workflow.py --scope pending --limit 5
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Any
|
||||
from enum import Enum
|
||||
|
||||
# Setup logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger('KBWorkflow')
|
||||
|
||||
|
||||
class TriggerType(Enum):
|
||||
MANUAL = "manual"
|
||||
SCHEDULED = "scheduled"
|
||||
WEBHOOK = "webhook"
|
||||
|
||||
|
||||
class Scope(Enum):
|
||||
ALL = "all"
|
||||
PENDING = "pending"
|
||||
METADATA = "metadata"
|
||||
CONTENT = "content"
|
||||
ANALYZED = "analyzed"
|
||||
ARCHIVE = "archive"
|
||||
|
||||
|
||||
class KBWorkflow:
|
||||
"""Knowledge Base Workflow Orchestrator"""
|
||||
|
||||
# Base directories for the inbox structure
|
||||
INBOX_DIRS = ['pending', 'metadata', 'content', 'analyzed']
|
||||
|
||||
def __init__(self,
|
||||
base_path: str = None,
|
||||
archive_path: str = None,
|
||||
output_format: str = "json"):
|
||||
"""
|
||||
Initialize KB Workflow
|
||||
|
||||
Args:
|
||||
base_path: Base path for knowledge base (default: workspace/knowledge-base)
|
||||
archive_path: Path for archived items (default: archive/)
|
||||
output_format: Output format (json, text)
|
||||
"""
|
||||
self.base_path = Path(base_path) if base_path else Path(__file__).parent / "knowledge-base"
|
||||
self.archive_path = Path(archive_path) if archive_path else self.base_path / "archive"
|
||||
self.output_format = output_format
|
||||
|
||||
# Ensure directories exist
|
||||
self._ensure_directories()
|
||||
|
||||
# Initialize subsystems
|
||||
self._init_subsystems()
|
||||
|
||||
def _ensure_directories(self):
|
||||
"""Create necessary directories"""
|
||||
# Create main inbox directories
|
||||
for subdir in self.INBOX_DIRS:
|
||||
dir_path = self.base_path / subdir
|
||||
if not dir_path.exists():
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"Created directory: {dir_path}")
|
||||
|
||||
# Create archive directory
|
||||
if not self.archive_path.exists():
|
||||
self.archive_path.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"Created archive directory: {self.archive_path}")
|
||||
|
||||
def _init_subsystems(self):
|
||||
"""Initialize subsystem references"""
|
||||
# Placeholder for skill integrations
|
||||
self.subscriptions_file = self.base_path / "subscriptions.json"
|
||||
self.report_file = self.base_path / "reports" / f"report_{datetime.now().strftime('%Y%m%d')}.json"
|
||||
|
||||
# ==================== Inbox Scanning ====================
|
||||
|
||||
def scan_inbox(self, scope: str = "all", limit: int = None) -> Dict[str, List[Dict]]:
|
||||
"""
|
||||
Scan inbox directories for items
|
||||
|
||||
Args:
|
||||
scope: Which directories to scan (all, pending, metadata, content, analyzed)
|
||||
limit: Maximum number of items to return per directory
|
||||
|
||||
Returns:
|
||||
Dictionary with directory names as keys and item lists as values
|
||||
"""
|
||||
result = {}
|
||||
|
||||
if scope == "all":
|
||||
dirs_to_scan = self.INBOX_DIRS
|
||||
elif scope in self.INBOX_DIRS:
|
||||
dirs_to_scan = [scope]
|
||||
else:
|
||||
logger.warning(f"Unknown scope: {scope}, scanning all")
|
||||
dirs_to_scan = self.INBOX_DIRS
|
||||
|
||||
for dir_name in dirs_to_scan:
|
||||
dir_path = self.base_path / dir_name
|
||||
items = self._scan_directory(dir_path)
|
||||
|
||||
if limit:
|
||||
items = items[:limit]
|
||||
|
||||
result[dir_name] = items
|
||||
logger.info(f"Scanned {dir_name}: {len(items)} items")
|
||||
|
||||
return result
|
||||
|
||||
def _scan_directory(self, path: Path) -> List[Dict]:
|
||||
"""
|
||||
Scan a directory for items
|
||||
|
||||
Args:
|
||||
path: Directory path to scan
|
||||
|
||||
Returns:
|
||||
List of item dictionaries
|
||||
"""
|
||||
items = []
|
||||
|
||||
if not path.exists():
|
||||
return items
|
||||
|
||||
for item_path in path.iterdir():
|
||||
if item_path.is_file():
|
||||
item_info = {
|
||||
'name': item_path.name,
|
||||
'path': str(item_path),
|
||||
'extension': item_path.suffix,
|
||||
'size': item_path.stat().st_size,
|
||||
'modified': datetime.fromtimestamp(item_path.stat().st_mtime).isoformat(),
|
||||
'type': self._classify_file_type(item_path)
|
||||
}
|
||||
items.append(item_info)
|
||||
elif item_path.is_dir():
|
||||
# Treat subdirectories as items too
|
||||
item_info = {
|
||||
'name': item_path.name,
|
||||
'path': str(item_path),
|
||||
'type': 'directory',
|
||||
'modified': datetime.fromtimestamp(item_path.stat().st_mtime).isoformat()
|
||||
}
|
||||
items.append(item_info)
|
||||
|
||||
return items
|
||||
|
||||
def _classify_file_type(self, path: Path) -> str:
|
||||
"""Classify file type by extension"""
|
||||
suffix = path.suffix.lower()
|
||||
|
||||
type_mapping = {
|
||||
'.md': 'markdown',
|
||||
'.txt': 'text',
|
||||
'.json': 'json',
|
||||
'.yaml': 'yaml',
|
||||
'.yml': 'yaml',
|
||||
'.pdf': 'pdf',
|
||||
'.jpg': 'image',
|
||||
'.jpeg': 'image',
|
||||
'.png': 'image',
|
||||
'.mp4': 'video',
|
||||
'.mp3': 'audio',
|
||||
'.zip': 'archive',
|
||||
'.url': 'link'
|
||||
}
|
||||
|
||||
return type_mapping.get(suffix, 'unknown')
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current workflow status
|
||||
|
||||
Returns:
|
||||
Status dictionary
|
||||
"""
|
||||
inbox_status = {}
|
||||
total_items = 0
|
||||
|
||||
for dir_name in self.INBOX_DIRS:
|
||||
dir_path = self.base_path / dir_name
|
||||
items = self._scan_directory(dir_path)
|
||||
inbox_status[dir_name] = {
|
||||
'count': len(items),
|
||||
'path': str(dir_path)
|
||||
}
|
||||
total_items += len(items)
|
||||
|
||||
# Check archive
|
||||
archive_items = self._scan_directory(self.archive_path)
|
||||
|
||||
return {
|
||||
'status': 'ready',
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'inbox': inbox_status,
|
||||
'archive': {
|
||||
'count': len(archive_items),
|
||||
'path': str(self.archive_path)
|
||||
},
|
||||
'total_items': total_items
|
||||
}
|
||||
|
||||
# ==================== Item Processing ====================
|
||||
|
||||
def process_item(self, item: Dict, target_stage: str = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Process a single item through the workflow
|
||||
|
||||
This method calls skills 2-4 to process items:
|
||||
- Skill 2: Link parsing / URL extraction
|
||||
- Skill 3: Metadata extraction
|
||||
- Skill 4: Content analysis
|
||||
|
||||
Args:
|
||||
item: Item dictionary with path and metadata
|
||||
target_stage: Target stage to process to
|
||||
|
||||
Returns:
|
||||
Processing result dictionary
|
||||
"""
|
||||
result = {
|
||||
'item': item,
|
||||
'success': False,
|
||||
'stages_completed': [],
|
||||
'output_path': None,
|
||||
'error': None
|
||||
}
|
||||
|
||||
try:
|
||||
item_path = Path(item['path'])
|
||||
|
||||
# Determine processing stage based on current location
|
||||
current_stage = self._get_stage_from_path(item_path)
|
||||
|
||||
# Skill 2: Link/URL Parsing
|
||||
if current_stage in ['pending', None] or target_stage == 'metadata':
|
||||
logger.info(f"Stage 1 → 2: Processing link/URL for {item['name']}")
|
||||
# TODO: Integrate with link-parser skill
|
||||
result['stages_completed'].append('link_parsing')
|
||||
|
||||
# Skill 3: Metadata Extraction
|
||||
if current_stage in ['metadata', 'pending'] or target_stage == 'content':
|
||||
logger.info(f"Stage 2 → 3: Extracting metadata for {item['name']}")
|
||||
# TODO: Integrate with zhihu-metadata skill
|
||||
result['stages_completed'].append('metadata_extraction')
|
||||
|
||||
# Skill 4: Content Analysis
|
||||
if current_stage in ['content', 'metadata'] or target_stage == 'analyzed':
|
||||
logger.info(f"Stage 3 → 4: Analyzing content for {item['name']}")
|
||||
# TODO: Integrate with bilibili-analyzer skill
|
||||
result['stages_completed'].append('content_analysis')
|
||||
|
||||
# Move to next stage if successful
|
||||
if result['stages_completed']:
|
||||
self._move_to_next_stage(item_path, result['stages_completed'][-1])
|
||||
result['success'] = True
|
||||
|
||||
except Exception as e:
|
||||
result['error'] = str(e)
|
||||
logger.error(f"Processing failed: {e}")
|
||||
|
||||
return result
|
||||
|
||||
def _get_stage_from_path(self, path: Path) -> Optional[str]:
|
||||
"""Determine which stage a file is in based on its path"""
|
||||
for stage in self.INBOX_DIRS:
|
||||
if stage in str(path):
|
||||
return stage
|
||||
return None
|
||||
|
||||
def _move_to_next_stage(self, item_path: Path, completed_stage: str) -> None:
|
||||
"""Move item to the next stage directory"""
|
||||
stage_order = ['pending', 'metadata', 'content', 'analyzed']
|
||||
|
||||
try:
|
||||
current_idx = stage_order.index(completed_stage)
|
||||
if current_idx < len(stage_order) - 1:
|
||||
next_stage = stage_order[current_idx + 1]
|
||||
next_path = self.base_path / next_stage / item_path.name
|
||||
|
||||
# Move file
|
||||
if item_path.exists():
|
||||
item_path.rename(next_path)
|
||||
logger.info(f"Moved {item_path.name} → {next_stage}/")
|
||||
except ValueError:
|
||||
logger.warning(f"Unknown stage: {completed_stage}")
|
||||
|
||||
def process_all(self, scope: str = "pending", limit: int = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Process all items in specified scope
|
||||
|
||||
Args:
|
||||
scope: Which directories to process
|
||||
limit: Maximum items to process
|
||||
|
||||
Returns:
|
||||
Processing summary
|
||||
"""
|
||||
summary = {
|
||||
'scope': scope,
|
||||
'processed': 0,
|
||||
'failed': 0,
|
||||
'results': []
|
||||
}
|
||||
|
||||
inbox = self.scan_inbox(scope=scope)
|
||||
|
||||
for dir_name, items in inbox.items():
|
||||
if limit and summary['processed'] >= limit:
|
||||
break
|
||||
|
||||
for item in items:
|
||||
if limit and summary['processed'] >= limit:
|
||||
break
|
||||
|
||||
result = self.process_item(item)
|
||||
summary['results'].append(result)
|
||||
|
||||
if result['success']:
|
||||
summary['processed'] += 1
|
||||
else:
|
||||
summary['failed'] += 1
|
||||
|
||||
return summary
|
||||
|
||||
# ==================== Archiving ====================
|
||||
|
||||
def archive_item(self, item: Dict, analysis: Dict = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Archive a processed item to the archive directory
|
||||
|
||||
Args:
|
||||
item: Item to archive
|
||||
analysis: Optional analysis metadata
|
||||
|
||||
Returns:
|
||||
Archiving result
|
||||
"""
|
||||
result = {
|
||||
'success': False,
|
||||
'original_path': item['path'],
|
||||
'archive_path': None,
|
||||
'error': None
|
||||
}
|
||||
|
||||
try:
|
||||
item_path = Path(item['path'])
|
||||
|
||||
if not item_path.exists():
|
||||
result['error'] = f"Item not found: {item['path']}"
|
||||
return result
|
||||
|
||||
# Create archive subdirectory by date
|
||||
date_dir = self.archive_path / datetime.now().strftime('%Y-%m-%d')
|
||||
if not date_dir.exists():
|
||||
date_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Move item to archive
|
||||
archive_path = date_dir / item_path.name
|
||||
item_path.rename(archive_path)
|
||||
|
||||
# Save analysis metadata if provided
|
||||
if analysis:
|
||||
meta_path = date_dir / f"{item_path.stem}_meta.json"
|
||||
with open(meta_path, 'w', encoding='utf-8') as f:
|
||||
json.dump({
|
||||
'archived_at': datetime.now().isoformat(),
|
||||
'analysis': analysis
|
||||
}, f, ensure_ascii=False, indent=2)
|
||||
|
||||
result['success'] = True
|
||||
result['archive_path'] = str(archive_path)
|
||||
logger.info(f"Archived: {item['name']} → {archive_path}")
|
||||
|
||||
except Exception as e:
|
||||
result['error'] = str(e)
|
||||
logger.error(f"Archive failed: {e}")
|
||||
|
||||
return result
|
||||
|
||||
# ==================== Subscriptions ====================
|
||||
|
||||
def check_subscriptions(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check for updates from subscribed sources
|
||||
|
||||
Returns:
|
||||
Subscription check results
|
||||
"""
|
||||
result = {
|
||||
'checked_at': datetime.now().isoformat(),
|
||||
'sources': {},
|
||||
'new_items': 0,
|
||||
'errors': []
|
||||
}
|
||||
|
||||
# Load subscription list
|
||||
subscriptions = self._load_subscriptions()
|
||||
|
||||
for sub in subscriptions:
|
||||
try:
|
||||
source_name = sub.get('name', 'unknown')
|
||||
source_type = sub.get('type', 'rss')
|
||||
|
||||
# Placeholder: Check for new content
|
||||
logger.info(f"Checking subscription: {source_name}")
|
||||
|
||||
# TODO: Implement actual subscription checking
|
||||
# - RSS feeds
|
||||
# - YouTube channel updates
|
||||
# - Bilibili user updates
|
||||
# - Custom web hooks
|
||||
|
||||
result['sources'][source_name] = {
|
||||
'type': source_type,
|
||||
'status': 'checked',
|
||||
'new_items': 0
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
result['errors'].append({
|
||||
'source': sub.get('name', 'unknown'),
|
||||
'error': str(e)
|
||||
})
|
||||
|
||||
return result
|
||||
|
||||
def _load_subscriptions(self) -> List[Dict]:
|
||||
"""Load subscriptions from file"""
|
||||
if self.subscriptions_file.exists():
|
||||
try:
|
||||
with open(self.subscriptions_file, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load subscriptions: {e}")
|
||||
|
||||
return []
|
||||
|
||||
# ==================== Reporting ====================
|
||||
|
||||
def generate_report(self, period: str = "daily") -> Dict[str, Any]:
|
||||
"""
|
||||
Generate workflow report
|
||||
|
||||
Args:
|
||||
period: Report period (daily, weekly, monthly)
|
||||
|
||||
Returns:
|
||||
Report dictionary
|
||||
"""
|
||||
# Calculate date range
|
||||
now = datetime.now()
|
||||
|
||||
if period == "daily":
|
||||
start_date = now - timedelta(days=1)
|
||||
elif period == "weekly":
|
||||
start_date = now - timedelta(weeks=1)
|
||||
elif period == "monthly":
|
||||
start_date = now - timedelta(days=30)
|
||||
else:
|
||||
start_date = now - timedelta(days=1)
|
||||
|
||||
# Gather statistics
|
||||
status = self.get_status()
|
||||
inbox = self.scan_inbox(scope="all")
|
||||
|
||||
# Build report
|
||||
report = {
|
||||
'generated_at': now.isoformat(),
|
||||
'period': period,
|
||||
'start_date': start_date.isoformat(),
|
||||
'summary': {
|
||||
'total_items_inbox': status['total_items'],
|
||||
'total_archived': status['archive']['count'],
|
||||
'items_by_stage': status['inbox']
|
||||
},
|
||||
'processing_stats': {
|
||||
'pending': len(inbox.get('pending', [])),
|
||||
'metadata': len(inbox.get('metadata', [])),
|
||||
'content': len(inbox.get('content', [])),
|
||||
'analyzed': len(inbox.get('analyzed', []))
|
||||
},
|
||||
'recommendations': self._generate_recommendations(status, inbox)
|
||||
}
|
||||
|
||||
# Save report
|
||||
if self.report_file:
|
||||
self.report_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.report_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"Report saved: {self.report_file}")
|
||||
|
||||
return report
|
||||
|
||||
def _generate_recommendations(self, status: Dict, inbox: Dict) -> List[str]:
|
||||
"""Generate processing recommendations"""
|
||||
recommendations = []
|
||||
|
||||
pending_count = status['inbox'].get('pending', {}).get('count', 0)
|
||||
analyzed_count = status['inbox'].get('analyzed', {}).get('count', 0)
|
||||
|
||||
if pending_count > 10:
|
||||
recommendations.append(f"⚠️ 待处理内容较多 ({pending_count} 项),建议优先处理")
|
||||
|
||||
if analyzed_count > 20:
|
||||
recommendations.append(f"📦 已分析内容较多 ({analyzed_count} 项),建议归档")
|
||||
|
||||
if status['total_items'] == 0:
|
||||
recommendations.append("📭 收件箱为空,暂无待处理内容")
|
||||
|
||||
return recommendations
|
||||
|
||||
# ==================== CLI Interface ====================
|
||||
|
||||
def run(self, trigger: str = "manual", scope: str = "all",
|
||||
limit: int = None, report: bool = False):
|
||||
"""
|
||||
Main entry point for running the workflow
|
||||
|
||||
Args:
|
||||
trigger: Trigger type (manual, scheduled)
|
||||
scope: Processing scope
|
||||
limit: Maximum items to process
|
||||
report: Generate report after processing
|
||||
"""
|
||||
logger.info(f"KB Workflow started: trigger={trigger}, scope={scope}")
|
||||
|
||||
# Handle different triggers
|
||||
if trigger == "status":
|
||||
result = self.get_status()
|
||||
elif trigger == "report":
|
||||
result = self.generate_report()
|
||||
elif trigger == "subscriptions":
|
||||
result = self.check_subscriptions()
|
||||
else:
|
||||
# Process items
|
||||
result = self.process_all(scope=scope, limit=limit)
|
||||
|
||||
# Generate report if requested
|
||||
if report:
|
||||
report_result = self.generate_report()
|
||||
result['report'] = report_result
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='KB Workflow - Knowledge Base Workflow Orchestration',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Get workflow status
|
||||
python kb_workflow.py --trigger status
|
||||
|
||||
# Process pending items manually
|
||||
python kb_workflow.py --trigger manual --scope pending
|
||||
|
||||
# Process up to 5 items
|
||||
python kb_workflow.py --trigger manual --limit 5
|
||||
|
||||
# Generate daily report
|
||||
python kb_workflow.py --trigger scheduled --report
|
||||
|
||||
# Check subscriptions
|
||||
python kb_workflow.py --trigger scheduled --scope subscriptions
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--trigger',
|
||||
type=str,
|
||||
default='manual',
|
||||
choices=['manual', 'scheduled', 'webhook', 'status', 'report', 'subscriptions'],
|
||||
help='Workflow trigger type (default: manual)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--scope',
|
||||
type=str,
|
||||
default='all',
|
||||
choices=['all', 'pending', 'metadata', 'content', 'analyzed', 'archive'],
|
||||
help='Processing scope (default: all)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--limit',
|
||||
type=int,
|
||||
default=None,
|
||||
help='Maximum items to process'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--report',
|
||||
action='store_true',
|
||||
help='Generate report after processing'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
type=str,
|
||||
choices=['json', 'text'],
|
||||
default='json',
|
||||
help='Output format (default: json)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--base-path',
|
||||
type=str,
|
||||
default=None,
|
||||
help='Base path for knowledge base'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose',
|
||||
action='store_true',
|
||||
help='Enable verbose logging'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verbose:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
# Initialize workflow
|
||||
workflow = KBWorkflow(base_path=args.base_path)
|
||||
|
||||
# Run workflow
|
||||
result = workflow.run(
|
||||
trigger=args.trigger,
|
||||
scope=args.scope,
|
||||
limit=args.limit,
|
||||
report=args.report
|
||||
)
|
||||
|
||||
# Output result
|
||||
if args.output == 'json':
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2, default=str))
|
||||
else:
|
||||
print(result)
|
||||
|
||||
return 0 if result else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"generated_at": "2026-02-10T12:13:35.989689",
|
||||
"period": "daily",
|
||||
"start_date": "2026-02-09T12:13:35.989689",
|
||||
"summary": {
|
||||
"total_items_inbox": 0,
|
||||
"total_archived": 0,
|
||||
"items_by_stage": {
|
||||
"pending": {
|
||||
"count": 0,
|
||||
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/pending"
|
||||
},
|
||||
"metadata": {
|
||||
"count": 0,
|
||||
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/metadata"
|
||||
},
|
||||
"content": {
|
||||
"count": 0,
|
||||
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/content"
|
||||
},
|
||||
"analyzed": {
|
||||
"count": 0,
|
||||
"path": "/home/admin/.openclaw/workspace/skills/kb-workflow/knowledge-base/analyzed"
|
||||
}
|
||||
}
|
||||
},
|
||||
"processing_stats": {
|
||||
"pending": 0,
|
||||
"metadata": 0,
|
||||
"content": 0,
|
||||
"analyzed": 0
|
||||
},
|
||||
"recommendations": [
|
||||
"📭 收件箱为空,暂无待处理内容"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user