# 知识库维护系统 - 详细设计 > 版本: 1.0.0 > 状态: 草稿 --- ## 目录 1. [技能接口总览](#1-技能接口总览) 2. [技能1-link-parser](#2-技能1-link-parser) 3. [技能2-bilibili-metadata](#3-技能2-bilibili-metadata) 4. [技能2-zhihu-metadata](#4-技能2-zhihu-metadata) 5. [技能3-bilibili-content](#5-技能3-bilibili-content) 6. [技能3-ocr-processor](#6-技能3-ocr-processor) 7. [技能4-deep-analyzer](#7-技能4-deep-analyzer) 8. [技能5-kb-workflow](#8-技能5-kb-workflow) 9. [数据格式定义](#9-数据格式定义) 10. [配置文件规范](#10-配置文件规范) 11. [扩展指南](#11-扩展指南) --- ## 1. 技能接口总览 ### 1.1 调用约定 所有技能遵循以下调用约定: ```yaml 输入: - 命令行参数 - stdin JSON (可选) - 环境变量 输出: - stdout JSON - 0: 成功 - 1: 失败 (错误信息输出到 stderr) 状态码: - 0: 成功 - 1: 一般错误 - 2: 参数错误 - 3: 网络错误 - 4: 权限错误 ``` ### 1.2 通用错误响应 ```json { "success": false, "error": "错误描述", "error_code": "ERROR_CODE", "details": {} } ``` --- ## 2. 技能1: link-parser **功能**: 解析链接,提取平台+类型+ID,处理短链跳转 ### 2.1 命令行接口 ```bash # 基本用法 python link-parser.py --url "https://www.bilibili.com/video/BV1xx411c7mD" # 支持短链 python link-parser.py --url "https://b23.tv/xxxxx" # 批量处理 python link-parser.py --batch input.txt # 输出文件 python link-parser.py --url "..." --output result.json ``` ### 2.2 输入参数 ```yaml 参数: --url, -u # 必需,单个URL --batch, -b # 可选,批量文件(每行一个URL) --output, -o # 可选,输出文件路径 --format # 输出格式: json/yaml (默认: json) ``` ### 2.3 输出格式 **成功**: ```json { "success": true, "data": { "platform": "bilibili", "type": "video", "id": "BV1xx411c7mD", "source_url": "https://www.bilibili.com/video/BV1xx411c7mD", "canonical_url": "https://www.bilibili.com/video/BV1xx411c7mD", "short_link_resolved": false, "title": null, "metadata": {} }, "metadata": { "processed_at": "2024-01-15T10:00:00", "parser_version": "1.0.0" } } ``` **列表类型**: ```json { "success": true, "data": { "platform": "bilibili", "type": "up", "id": "7770724", "source_url": "https://space.bilibili.com/7770724", "is_list_type": true, "list_info": { "total_items": 156, "item_type": "video" } }, "metadata": { "requires_user_input": true, "input_options": { "scope": ["subscribe_only", "new_only", "all"], "subscription": ["daily", "weekly", "custom", "none"] } } } ``` ### 2.4 支持的链接格式 | 类型 | 平台 | 正则示例 | 输出 | |------|------|----------|------| | 短链 | B站 | `b23.tv/[a-zA-Z0-9]+` | resolved_url + 解析 | | 视频 | B站 | `bilibili.com/video/BV[a-zA-Z0-9]+` | `{type: video, id: BV...}` | | 专栏 | B站 | `bilibili.com/read/cv[0-9]+` | `{type: article, id: cv...}` | | 动态 | B站 | `bilibili.com/opus/[0-9]+` | `{type: opus, id: ...}` | | UP主 | B站 | `space.bilibili.com/[0-9]+` | `{type: up, id: ...}` | | 收藏夹 | B站 | `space.bilibili.com/[0-9]+/favlist?fid=[0-9]+` | `{type: favlist, id: ...}` | | 合集 | B站 | `space.bilibili.com/[0-9]+/lists/[0-9]+?type=season` | `{type: playlist, id: ...}` | | YouTube | YouTube | `(youtu.be/[a-zA-Z0-9]+|youtube.com/watch?v=[a-zA-Z0-9]+)` | `{type: video, id: ...}` | | 知乎 | 知乎 | `zhihu.com/question/[0-9]+` | `{type: question, id: ...}` | | 知乎 | 知乎 | `zhihu.com/answer/[0-9]+` | `{type: answer, id: ...}` | | 知乎 | 知乎 | `zhihu.com/people/[a-zA-Z0-9-]+` | `{type: user, id: ...}` | | 知乎 | 知乎 | `zhihu.com/collection/[0-9]+` | `{type: collection, id: ...}` | ### 2.5 复用代码 ```python # 复用现有代码 from shortlink_resolver import resolve_shortlink, extract_bilibili_id ``` --- ## 3. 技能2: bilibili-metadata **功能**: 反爬获取B站视频/专栏元数据 ### 3.1 命令行接口 ```bash # 基本用法 python bilibili-metadata.py --id "BV1xx411c7mD" # 指定输出目录 python bilibili-metadata.py --id "BV1xx" --output ./metadata/ # 批量处理 python bilibili-metadata.py --batch ids.txt --output ./ # 视频+专栏支持 python bilibili-metadata.py --id "cv43780186" --type "article" ``` ### 3.2 输入参数 ```yaml 参数: --id, -i # 必需,内容ID --type, -t # 内容类型: video/article/opus (默认: video) --output, -o # 输出目录 (默认: ./) --batch, -b # 批量文件 (每行: id [type]) --config, -c # 配置文件路径 ``` ### 3.3 输出格式 **视频元数据**: ```json { "success": true, "data": { "platform": "bilibili", "type": "video", "id": "BV1xx411c7mD", "source_url": "https://www.bilibili.com/video/BV1xx411c7mD", "title": "Python完整教程", "description": "这是一个Python教程...", "author": { "id": 7770724, "name": "UP主名字", "face": "https://i0.hdslb.com/xxx.jpg" }, "duration": 9000, "duration_text": "2:30:00", "publish_time": "2024-01-01T12:00:00", "tags": ["python", "tutorial", "编程"], "stats": { "view": 100000, "like": 5000, "coin": 2000, "favorite": 1000, "share": 500 }, "subtitle": { "has_subtitle": true, "languages": ["zh-CN"], "url": "https://api.bilibili.com/..." }, "pic": "https://i0.hdslb.com/xxx.jpg", "quality_score": null }, "metadata": { "fetched_at": "2024-01-15T10:00:00", "source": "bilibili_api" } } ``` **专栏文章元数据**: ```json { "success": true, "data": { "platform": "bilibili", "type": "article", "id": "cv43780186", "source_url": "https://www.bilibili.com/read/cv43780186", "title": "文章标题", "description": "文章摘要...", "author": { "id": 123456, "name": "作者名字", "face": "https://i0.hdslb.com/xxx.jpg" }, "publish_time": "2024-01-01T12:00:00", "stats": { "view": 50000, "like": 1000, "comment": 200 }, "word_count": 5000, "images_count": 10, "category": "技术" }, "metadata": { "fetched_at": "2024-01-15T10:00:00", "source": "bilibili_api" } } ``` **UP主/收藏夹列表**: ```json { "success": true, "data": { "platform": "bilibili", "type": "up", "id": "7770724", "source_url": "https://space.bilibili.com/7770724", "name": "UP主名字", "description": "UP主介绍...", "follower": 100000, "following": 100, "archive_count": 156, "content_stats": { "total_views": 10000000, "total_likes": 500000 }, "subscription": null }, "metadata": { "fetched_at": "2024-01-15T10:00:00" } } ``` ### 3.4 复用代码 ```python # 复用现有代码 from bilibili_extractor import BilibiliExtractor ``` --- ## 4. 技能2: zhihu-metadata **功能**: 获取知乎元数据 ### 4.1 命令行接口 ```bash python zhihu-metadata.py --id "266309957" --type "question" python zhihu-metadata.py --id "2003427891925509507" --type "answer" ``` ### 4.2 输入参数 ```yaml 参数: --id, -i # 必需,内容ID --type, -t # 内容类型: question/answer/article/user/collection (默认: question) --output, -o # 输出目录 --config, -c # 配置文件路径 ``` ### 4.3 输出格式 **问题元数据**: ```json { "success": true, "data": { "platform": "zhihu", "type": "question", "id": "266309957", "source_url": "https://www.zhihu.com/question/266309957", "title": "问题标题", "description": "问题描述...", "author": { "id": "user-xxx", "name": "作者名字", "url_token": "user-xxx" }, "answer_count": 50, "followers": 1000, "browse_count": 50000, "created_time": "2024-01-01T12:00:00", "updated_time": "2024-01-15T10:00:00", "tags": ["标签1", "标签2"] }, "metadata": { "fetched_at": "2024-01-15T10:00:00" } } ``` **回答元数据**: ```json { "success": true, "data": { "platform": "zhihu", "type": "answer", "id": "2003427891925509507", "source_url": "https://www.zhihu.com/question/266309957/answer/2003427891925509507", "question_id": "266309957", "question_title": "问题标题", "author": { "id": "user-xxx", "name": "作者名字", "headline": "作者简介" }, "vote_up": 1000, "comment_count": 50, "created_time": "2024-01-01T12:00:00", "updated_time": "2024-01-15T10:00:00", "content_summary": "回答摘要..." }, "metadata": { "fetched_at": "2024-01-15T10:00:00" } } ``` ### 4.4 复用代码 ```python # 复用现有代码 from zhihu_extractor import ZhihuDrissionExtractor ``` --- ## 5. 技能3: bilibili-content **功能**: 下载低分视频 + 提取字幕/专栏全文 ### 5.1 命令行接口 ```bash # 基本用法(视频) python bilibili-content.py --id "BV1xx411c7mD" --output ./content/ # 视频+字幕 python bilibili-content.py --id "BV1xx" --output ./ --with-subtitle # 专栏全文 python bilibili-content.py --id "cv43780186" --type article --output ./ # 低分辨率(用于OCR) python bilibili-content.py --id "BV1xx" --quality 480 --output ./ ``` ### 5.2 输入参数 ```yaml 参数: --id, -i # 必需,内容ID --type, -t # 内容类型: video/article/opus (默认: video) --output, -o # 输出目录 (默认: ./) --quality # 视频质量: 360/480/720/1080 (默认: 480) --with-subtitle # 是否提取字幕 (默认: true) --subtitle-lang # 字幕语言 (默认: zh-CN) --format # 输出格式: json/ndjson (默认: json) ``` ### 5.3 输出格式 **视频内容**: ```json { "success": true, "data": { "platform": "bilibili", "type": "video", "id": "BV1xx411c7mD", "video_file": { "path": "content/BV1xx_video.mp4", "size_bytes": 52428800, "duration": 9000, "quality": "480p" }, "subtitle": { "format": "srt", "path": "content/BV1xx_subtitle.srt", "language": "zh-CN", "text_length": 50000, "lines_count": 1500 }, "danmaku": { "path": "content/BV1xx_danmaku.xml", "count": 5000 }, "frames": { "output_dir": "content/BV1xx_frames", "count": 900, "fps": 0.1 } }, "metadata": { "fetched_at": "2024-01-15T10:00:00", "content_quality": "preview" } } ``` **专栏内容**: ```json { "success": true, "data": { "platform": "bilibili", "type": "article", "id": "cv43780186", "content": { "markdown": "# 文章标题\n\n内容...", "markdown_path": "content/cv43780186.md", "html": "...", "word_count": 5000, "images": [ { "url": "https://...", "path": "content/images/img_001.jpg", "description": "图片描述" } ] }, "author": { "name": "作者名字" }, "publish_time": "2024-01-01T12:00:00" }, "metadata": { "fetched_at": "2024-01-15T10:00:00" } } ``` ### 5.4 复用代码 ```python # 复用现有代码 from video_stream_extractor import VideoStreamExtractor from bilibili_extractor import BilibiliExtractor ``` --- ## 6. 技能3: ocr-processor **功能**: 通用字幕OCR(被技能3调用) ### 6.1 命令行接口 ```bash # 基本用法 python ocr-processor.py --input ./frames/ --output ./ocr.json # 指定引擎 python ocr-processor.py --input ./frames/ --engine paddleocr --output ./ # 只处理部分帧 python ocr-processor.py --input ./frames/ --start 0 --end 100 --output ./ ``` ### 6.2 输入参数 ```yaml 参数: --input, -i # 必需,帧图片目录或单个图片 --output, -o # 输出文件 (默认: ocr.json) --engine # OCR引擎: paddleocr/tesseract (默认: paddleocr) --region # 字幕区域: bottom/full/auto (默认: bottom) --fps # 抽帧频率 (默认: 0.1,即每10秒1帧) --start # 开始帧号 (默认: 0) --end # 结束帧号 (默认: -1,即全部) --confidence # 置信度阈值 (默认: 0.7) ``` ### 6.3 输出格式 ```json { "success": true, "data": { "frames": { "frame_0001.jpg": { "texts": [ { "text": "字幕文本", "confidence": 0.95, "bbox": [x1, y1, x2, y2] } ], "timestamp": 0.0 } }, "merged_text": "合并的字幕文本...", "statistics": { "total_frames": 100, "frames_with_text": 80, "total_texts": 500, "unique_texts": 200 } }, "metadata": { "engine": "paddleocr", "processed_at": "2024-01-15T10:00:00" } } ``` ### 6.4 复用代码 ```python # 复用现有代码 from ocr_processor import OCRProcessor ``` --- ## 7. 技能4: deep-analyzer **功能**: AI深度分析,按内容类型使用不同模板 ### 7.1 命令行接口 ```bash # 基本用法 python deep-analyzer.py --metadata meta.json --content content.json --output ./analysis.md # 指定类型 python deep-analyzer.py --metadata meta.json --type tutorial --output ./ # 使用自定义模板 python deep-analyzer.py --metadata meta.json --template my_template.md --output ./ # 带OCR结果 python deep-analyzer.py --metadata meta.json --content content.json --ocr ocr.json --output ./ ``` ### 7.2 输入参数 ```yaml 参数: --metadata, -m # 必需,元数据JSON文件 --content, -c # 必需,内容JSON文件 --output, -o # 必需,输出Markdown文件 --type, -t # 内容类型: auto/tutorial/documentation/news (默认: auto) --template # 自定义模板路径 --ocr # OCR结果JSON文件 (可选) --model # AI模型 (默认: 配置中指定) --tags # 预设标签 (可选) ``` ### 7.3 模板变量 **通用变量**: ```jinja2 {{ title }} # 标题 {{ source }} # 来源 (bilibili/zhihu) {{ url }} # 原始URL {{ author }} # 作者/UP主 {{ duration }} # 时长 {{ date }} # 发布日期 {{ tags }} # 标签列表 {{ summary }} # 详细摘要 {{ key_points }} # 核心要点 {{ quality_score }} # 质量评分 (1-100) ``` **教程视频额外变量**: ```jinja2 {{ prerequisites }} # 前置知识 {{ outline }} # 分段大纲 {{ pitfalls }} # 常见坑点 {{ practice_advice }} # 实践建议 ``` ### 7.4 输出格式 **教程类输出**: ```markdown --- title: "Python完整教程" source: "bilibili" url: "https://..." author: "CodeMaster" duration: "2:30:00" date: "2024-01-01T00:00:00" tags: - python - tutorial - 编程入门 quality_score: 95 --- ## 🎯 一句话总结 本教程系统讲解Python基础语法,适合零基础学习者。 ## 📚 前置知识 - 无需编程基础 - 电脑基本操作能力 ## 📝 步骤详解 ### 环境配置 [00:00-15:00] 首先下载Python... ![frame_0015: Python官网下载页面](./images/frame_0015.jpg) ```bash # 安装Python pip install python ``` ### 基础语法 [15:00-45:00] ... ## 💡 关键要点 1. Python使用缩进定义代码块 2. 变量不需要声明类型 3. ... ## ⚠️ 常见坑点 - Python 2 与 3 的区别 - 缩进问题导致报错 ## 🤔 实践建议 - 跟随视频动手编码 - 完成课后练习 ## 📚 相关资源 - [Python官网](https://www.python.org/) - [官方文档](https://docs.python.org/) ``` ### 7.5 复用代码 ```python # 复用现有代码 from deep_analyzer import DeepAnalyzer from ai_client import AIClient ``` --- ## 8. 技能5: kb-workflow **功能**: 工作流编排 + OpenClaw cron定时 + 手动触发 + 归档管理 ### 8.1 命令行接口 ```bash # 手动触发处理 python kb-workflow.py --trigger manual # 定时任务触发 python kb-workflow.py --trigger scheduled # 只处理待处理列表 python kb-workflow.py --trigger manual --scope pending # 强制刷新订阅 python kb-workflow.py --trigger manual --refresh-subscriptions # 生成汇报 python kb-workflow.py --report # 查看状态 python kb-workflow.py --status ``` ### 8.2 输入参数 ```yaml 参数: --trigger, -t # 触发方式: manual/scheduled (默认: manual) --scope # 处理范围: all/pending/metadata/content (默认: all) --limit # 最大处理数量 (默认: 5) --refresh-subs # 刷新订阅 --report # 生成汇报 --status # 显示状态 --config, -c # 配置文件路径 ``` ### 8.3 汇报文件格式 ```json { "run_id": "run_20240115_030000", "run_time": "2024-01-15T03:00:00", "trigger": "scheduled", "scope": "all", "processed": 5, "success": 4, "failed": 1, "details": { "total_inbox": 12, "pending": 2, "metadata_ready": 3, "content_ready": 5, "analyzed": 2, "archived": 10 }, "items": [ { "id": "BV1xx411c7mD", "type": "video", "status": "success", "duration_seconds": 45 }, { "id": "BV2yy522c8nE", "type": "video", "status": "failed", "error": "download timeout" } ], "subscriptions": { "checked": 3, "updates_found": 5, "new_items_added": 2 }, "storage": { "inbox_size_mb": 128, "archive_size_mb": 512 }, "generated_at": "2024-01-15T03:05:00" } ``` ### 8.4 工作流步骤 ```python def run_workflow(trigger, scope, limit): # 1. 扫描收件箱 pending_items = scan_pending() metadata_items = scan_metadata() content_items = scan_content() # 2. 检查订阅更新 if trigger == "scheduled": check_subscriptions() # 3. 按优先级处理 all_items = pending_items + metadata_items + content_items for item in all_items[:limit]: try: if item.status == "pending": metadata = run_skill2(item) update_item(item, metadata, status="metadata_ready") elif item.status == "metadata_ready": content = run_skill3(item) update_item(item, content, status="content_ready") elif item.status == "content_ready": analysis = run_skill4(item) archive_item(item, analysis) update_item(item, status="archived") except Exception as e: record_failure(item, str(e)) # 4. 生成汇报 generate_report() ``` ### 8.5 复用代码 ```python # 复用现有代码 from pipeline import Pipeline from git_sync import GitSync ``` --- ## 9. 数据格式定义 ### 9.1 收件箱目录结构 ``` inbox/ ├── pending/ │ └── bilibili_video_BV1xx_2024-01-15T100000.json ├── metadata/ │ └── bilibili_video_BV1xx_2024-01-15T100000.json ├── content/ │ └── bilibili_video_BV1xx_2024-01-15T100000.json └── analyzed/ └── bilibili_video_BV1xx_2024-01-15T100000.json ``` ### 9.2 元数据文件格式 ```json { "platform": "bilibili", "type": "video", "id": "BV1xx411c7mD", "source_url": "https://www.bilibili.com/video/BV1xx411c7mD", "status": "pending", "scope": "all", "title": "Python完整教程", "author": {"id": 7770724, "name": "UP主名字"}, "duration": 9000, "publish_time": "2024-01-01T12:00:00", "tags": ["python", "tutorial"], "stats": {"view": 100000, "like": 5000}, "subscription": { "enabled": true, "scope": "new_only", "frequency": "daily", "last_checked": "2024-01-15T00:00:00", "items_count": 156, "processed_count": 45 }, "created_at": "2024-01-15T00:00:00", "updated_at": "2024-01-15T10:00:00", "retry_count": 0 } ``` ### 9.3 归档目录结构 ``` archive/ ├── 2024/ │ ├── 01/ │ │ ├── video/ │ │ │ ├── bilibili_BV1xx_2024-01-15.md │ │ │ └── index.json │ │ ├── article/ │ │ │ └── index.json │ │ └── index.json │ └── index.json └── index.json ``` --- ## 10. 配置文件规范 ### 10.1 配置文件位置 优先级: 1. `./config.yaml` (当前目录) 2. `~/.config/obsidian-kb/config.yaml` 3. `~/.obsidian-kb/config.yaml` ### 10.2 完整配置示例 ```yaml # AI 配置 ai: provider: "openai" model: "gpt-4" api_key: "${OPENAI_API_KEY}" temperature: 0.3 max_tokens: 8192 # 存储配置 storage: inbox_dir: "~/work/inbox" archive_dir: "~/ObsidianVault/knowledge-base" temp_dir: "/tmp/obsidian-kb" max_concurrent: 3 cleanup_after_days: 7 # 输出配置 output: format: "markdown" encoding: "utf-8" include_timestamp: true include_source: true # Git 同步配置 git: enabled: true vault_path: "~/ObsidianVault" remote: "origin" branch: "main" auto_commit: true # Bilibili 提取器配置 bilibili: enabled: true cookie: "${BILIBILI_COOKIE}" use_playwright: true headless: true timeout: 30 subtitle: enabled: true language: "zh-CN" # 知乎提取器配置 zhihu: enabled: true use_drissionpage: true cookie_file: "~/.obsidian-kb/zhihu_cookies.json" # OCR 配置 ocr: engine: "paddleocr" confidence_threshold: 0.7 region: "bottom" fps: 0.1 max_frames: 500 # 工作流配置 workflow: schedule: "0 3 * * *" max_items_per_run: 5 subscription_check_interval: 24 # 汇报配置 reporting: enabled: true output_dir: "~/work/reports" format: "json" retention_days: 30 # 日志配置 logging: level: "INFO" format: "[%(asctime)s] %(levelname)s - %(message)s" file: "~/work/logs/obsidian-kb.log" ``` --- ## 11. 扩展指南 ### 11.1 新增平台步骤 以新增「小红书」为例: #### 步骤1: 创建链接解析器扩展 ```python # link-parser 的 platform_registry 中添加 PLATFORM_REGISTRY = { "xiaohongshu": { "name": "小红书", "patterns": [ r"xiaohongshu.com/notes/([a-zA-Z0-9]+)", ], "types": ["note", "user", "collection"] } } ``` #### 步骤2: 创建元数据获取器 ```bash # 创建文件: skills/xiaohongshu-metadata/ ``` ```python class XiaohongshuMetadata: def extract(self, url, config): # 实现获取逻辑 pass ``` #### 步骤3: 更新工作流配置 ```yaml xiaohongshu: enabled: true cookie: "${XIAOHONGSHU_COOKIE}" ``` ### 11.2 新增分析模板 ```jinja2 # templates/podcast.md --- title: "{{ title }}" source: "{{ source }}" url: "{{ url }}" author: "{{ author }}" duration: "{{ duration }}" date: "{{ date }}" tags: {{- for tag in tags }} - {{ tag }} {{- endfor }} --- ## 🎧 播客摘要 {{ summary }} ## 💡 核心要点 {{ key_points }} ``` --- ## 附录A: 技能清单 | # | 技能名 | 主要文件 | 复用代码 | |---|--------|----------|----------| | 1 | link-parser | `link-parser.py` | shortlink_resolver.py | | 2 | bilibili-metadata | `bilibili_metadata.py` | bilibili_extractor.py | | 2 | zhihu-metadata | `zhihu_metadata.py` | zhihu_extractor.py | | 3 | bilibili-content | `bilibili_content.py` | video_stream_extractor.py | | 3 | ocr-processor | `ocr_processor.py` | ocr_processor.py | | 4 | deep-analyzer | `deep_analyzer.py` | deep_analyzer.py | | 5 | kb-workflow | `kb_workflow.py` | pipeline.py, git_sync.py | --- ## 附录B: OpenClaw Cron 配置示例 ```yaml # 定时任务配置 schedule: "0 3 * * *" command: "python /path/to/skills/kb-workflow/kb_workflow.py --trigger scheduled" ``` --- 蒋先生,详细设计文档已完成! **包含内容:** - ✅ 7个技能的完整接口规范(命令行参数、输入输出格式) - ✅ 数据格式详细定义(收件箱、归档、汇报) - ✅ 配置文件规范 - ✅ 复用代码对照表 - ✅ 扩展指南(新增平台/模板步骤) **下一步:** 可以进入 **write-plan** 阶段,生成具体实现计划吗?