Commit Graph

52 Commits

Author SHA1 Message Date
OpenClaw Agent 3f74bcd377 docs: Add subtitle enhancement tasks to TODO
- Add visual model subtitle extraction (GPT-4V/Claude/GLM-4V)
- Add ASR subtitle extraction (Whisper/讯飞/阿里云)
2026-02-10 15:16:47 +08:00
OpenClaw Agent f4c6612849 docs: Add workspace configuration and memory files
- AGENTS.md: Update agent workspace guidelines (read TODO.md in workflow)
- TODO.md: Add security inspection and TTS configuration tasks
- memory/: Add knowledge base refactoring progress notes
- .gitignore: Add debug/ directory
2026-02-10 14:56:18 +08:00
OpenClaw Agent 47355b36d9 feat(kb-bilibili-content): Add article content extraction using Playwright
- Replace API-based article extraction with Playwright scraping
- Support Bilibili article (专栏) content extraction
- Extract full article text using .read-article-holder selector
- Use async Playwright API for proper async/await support
2026-02-10 14:53:05 +08:00
OpenClaw Agent 56907e31f5 refactor(kb-zhihu-metadata): Simplify to only support zvideo and zhuanlan
- Remove question, answer, user types (not needed for knowledge base)
- Keep only zvideo (知乎视频) and zhuanlan (知乎专栏文章)
- Update URL patterns and extraction logic
- Simplified CLI interface
2026-02-10 14:46:39 +08:00
OpenClaw Agent d216ce4ba7 feat(kb-bilibili-metadata): Add UP subscription tracking support
- Enhance _extract_up_async() with subscription tracking:
  - video_count: Number of videos
  - article_count: Number of articles
  - subscription_info: Last video/article for update detection
- Update CLI to support UP subscription queries
- All collection types now support basic_info + subscription status
2026-02-10 14:34:12 +08:00
OpenClaw Agent 26accdd726 fix(kb-bilibili-content): Improve browser headers and cookie support
- Add complete browser headers for better API compatibility
- Read BILIBILI_COOKIE from environment variable
- Add debug print statements for troubleshooting
2026-02-10 14:29:35 +08:00
OpenClaw Agent 4cdbc825ce feat(kb-bilibili-metadata): Add series/readlist/favlist metadata extraction
- Add extract_series_async() for series collection metadata
- Add extract_readlist_async() for article list metadata
- Add extract_favlist_async() for favorite list metadata
- Add CLI support for new types: series, readlist, favlist
- Update bilibili_metadata.py with new extraction functions
- Add --mid parameter for series type

Note: These types record basic info and subscription status only,
not the full content list (which may be large).
2026-02-10 14:27:14 +08:00
OpenClaw Agent 51347f671b feat: Improve kb-bilibili-content video download
- Simplify download logic using playurl API
- Support video download without Cookie (FLV format)
- Improve subtitle extraction
- Better error handling and progress display
- Remove unnecessary dependencies
2026-02-10 13:40:03 +08:00
OpenClaw Agent a7ab478dbc fix: Improve kb-bilibili-metadata output format
- Wrap result in standard format with success/data/metadata
- Change default headless to False (use API by default)
- Add error handling with structured output
- Improve CLI output readability
2026-02-10 13:18:58 +08:00
OpenClaw Agent 066395164a feat: Add Xiaohongshu (小红书) URL support
- Add xhslink.com short link resolution
- Add xiaohongshu.com note/explore/notes URL patterns
- Add discovery/item pattern (resolved from short links)
- Add user profile pattern
- Support multiple Xiaohongshu URL formats
2026-02-10 13:00:38 +08:00
OpenClaw Agent c14575ed1f feat: Add comprehensive Zhihu URL support
- Add zhuanlan article support (zhuanlan.zhihu.com/p/{id})
- Add organization page support (zhihu.com/org/{name})
- Add organization subpages: answers, zvideos, asks, posts, pins
- Add zvideo support (zhihu.com/zvideo/{id})
- Improve URL pattern matching for various Zhihu content types
2026-02-10 12:51:38 +08:00
OpenClaw Agent 3fb5af33ab feat: Add mobile opus URL support for Bilibili
- Support m.bilibili.com/opus/{id} for dynamic posts
- Handle mobile URLs when resolving short links
2026-02-10 12:38:24 +08:00
OpenClaw Agent a3623b5338 feat: Add mobile URL support for Bilibili
- Support m.bilibili.com/space/{id} for UP主 accounts
- Handle mobile video URLs when resolving short links
2026-02-10 12:35:06 +08:00
OpenClaw Agent d72090b957 feat: Add automatic short link resolution for b23.tv
- Use requests library to follow redirects and resolve b23.tv short links
- Automatically extract BV ID from resolved URL
- Support both short links and full URLs seamlessly
2026-02-10 12:33:34 +08:00
OpenClaw Agent 115d6e042e fix: Improve kb-link-parser URL patterns
- Add playlist pattern for bilibili series (lists/xxx?type=series)
- Add favlist pattern for bilibili favorites (favlist?fid=xxx)
- Add readlist pattern for bilibili article collections (read/readlist/rlxxx)
- Reorder pattern matching to check specific patterns before generic space pattern
2026-02-10 12:30:55 +08:00
OpenClaw Agent 5507b1d986 feat: Add KB knowledge base series skills
- kb-link-parser: Multi-platform URL parser (Bilibili, YouTube, Zhihu)
- kb-bilibili-metadata: Bilibili video/article/up主 metadata extractor
- kb-zhihu-metadata: Zhihu question/answer/user metadata extractor
- kb-bilibili-content: Bilibili video download and content extraction
- kb-ocr-processor: OCR text extraction from video frames
- kb-deep-analyzer: AI-powered deep content analysis
- kb-workflow: Workflow orchestration (inbox, archive, subscriptions, report)

Also add:
- ARCHITECTURE_FLOWCHART.md: System architecture diagrams
- DETAILED_DESIGN.md: Complete design documentation
2026-02-10 12:18:52 +08:00
OpenClaw Agent 2f5a269866 feat(bilibili-analyzer): 支持字幕区域裁剪
- 新增 --crop-ratio 参数,提取底部1/N区域(默认1/4)
- 新增 --no-crop 参数,禁用裁剪提取完整帧
- 优化字幕识别效率,只处理字幕区域
2026-02-09 23:51:42 +08:00
OpenClaw Agent 732dfb6549 清理GitHub相关技能 2026-02-09 23:37:58 +08:00
OpenClaw Agent 79381bf0ba 移除不需要的AI/开发技能 2026-02-09 23:25:01 +08:00
OpenClaw Agent 59d873e1ad 清理workspace:删除临时文件、测试脚本和项目文档 2026-02-09 23:19:15 +08:00
OpenClaw Agent 6051d05f0c 修改-analyzer技能 2026-02-09 22:21:51 +08:00
OpenClaw Agent e9872afd25 feat: 更新子代理监督规则和文件存放规则
- 新增子代理监督规则(子代理必须严格遵守技能要求,违规打回重做)
- 新增文件存放规则(默认输出到~/work/)
- 将MEMORY.md中的规则同步到AGENTS.md
2026-02-09 20:24:56 +08:00
OpenClaw Agent 00e55259a7 Update: workspace changes 2026-02-09 18:43:33 +08:00
OpenClaw Agent e0a2b323d9 Update obsidian-kb skill and add find-skills
- obsidian-kb: Add OCR processors (easyocr, tesseract)
- obsidian-kb: Fix subtitle detection with test files
- obsidian-kb: Add cookie verification script
- Add find-skills skill for discovering capabilities
- Add video processing scripts
- Update clawhub lock file
2026-02-09 11:53:42 +08:00
OpenClaw Agent d66eade77e docs: 更新实施计划 - 所有 Checkpoints 已完成
- 所有核心模块已实现
- 集成主流程完成
- 配置和文档已更新
- 代码已提交推送

项目状态:✅ 已完成所有 Checkpoints (1-6)
测试状态:⏳ 待实际视频验证

下一步:可在生产环境中直接使用硬字幕提取功能
2026-02-09 00:39:00 +08:00
OpenClaw Agent 1e96b8785c feat(hard-subtitle): 集成硬字幕提取功能
- 添加 VideoStreamExtractor - B站 API + OpenCV 流式读取
- 添加 SubtitleDetector - MSER 检测 + 颜色过滤 + LRU 缓存
- 添加 OCRProcessor - PaddleOCR 集成 + 批量识别 + 过滤去重
- 添加 HardSubtitleExtractor - 主协调器,整合所有模块
- 更新 BilibiliExtractor - 优先官方字幕,降级硬字幕提取
- 更新 __init__.py - 导出所有新模块
- 添加配置项支持
- 完成完整实施计划 Checkpoint 5

功能:
- 支持官方 CC 字幕(WBI API)
- 支持硬字幕提取(PaddleOCR)
- 自动检测字幕区域
- 字幕结果缓存
- 去重和噪声过滤
2026-02-09 00:37:16 +08:00
OpenClaw Agent cd80837c86 fix(bilibili): 改进字幕提取,支持 WBI API 和多语言优先级
- 使用 WBI API 获取字幕信息(支持 AI 字幕)
- 优先选择中文字幕(ai-zh > zh-CN > zh-Hans)
- 添加硬字幕提取实施计划
- 改进错误处理,静默失败不影响主流程
2026-02-09 00:24:19 +08:00
OpenClaw Agent f952c963d0 添加知乎 DrissionPage 计划和 Playwright 完成报告 2026-02-08 05:57:53 +08:00
OpenClaw Agent 1737800a3c 清理临时测试文件和报告 2026-02-08 05:57:40 +08:00
OpenClaw Agent 30eb8489a2 添加知乎 DrissionPage 完成报告和更新 .gitignore
- 添加最终完成报告到 memory/completion-reports/
- 更新 .gitignore 排除测试文件和临时报告
2026-02-08 05:57:22 +08:00
OpenClaw Agent a69225ff2b 知乎提取器:从 Playwright 迁移到 DrissionPage 方案
- 使用 DrissionPage 替代 Playwright
- 实现 Cookie 管理和反爬绕过
- 支持回答/文章/问题三种内容类型
- 灵活配置(文件/环境变量/配置文件)
- 更新文档和配置文件
2026-02-08 05:56:27 +08:00
OpenClaw Agent aa206a1679 Add knowledge base: B站视频《在2000多米高空 建座"会飞的"发电站》
- 新增 knowledge-base/ 目录
- 保存央视新闻视频:在2000多米高空 建座"会飞的"发电站
- 完善 obsidian-kb skill:添加知乎 Playwright 提取器
- 更新依赖和文档
2026-02-07 19:06:38 +08:00
OpenClaw Agent 027b82eb28 重构 B站提取器:移除技能配置,添加Python脚本实现 2026-02-07 17:27:23 +08:00
OpenClaw Agent e4c5dbf4b7 修改obsidian-kb技能描述:改为明确触发模式,不自动触发 2026-02-07 15:50:09 +08:00
OpenClaw Agent c79035d6ff 修复B站短链接解析问题和处理流程说明
1. 修复短链接解析bug:
   - 新增 _resolve_shortlink_bvid() 方法
   - 使用 Playwright + stealth 模式解析 b23.tv 短链接
   - 确保反爬机制正常工作

2. 更新 SKILL.md 使用说明:
   - 添加明确的处理流程说明
   - 说明收到链接后直接处理,不要问用户
   - 添加示例触发词

修复问题:短链接无法自动解析,反爬机制未生效
2026-02-07 15:31:47 +08:00
OpenClaw Agent 8163c37e83 refactor: 移除bilibili_extractor插件,改用obsidian-kb
- 删除 bilibili_extractor.py 及相关文档和配置
- 删除 bilibili_venv 虚拟环境
- 删除测试脚本和生成的B站笔记
- 以后使用 skills/obsidian-kb 提供的B站提取功能
2026-02-07 14:44:58 +08:00
OpenClaw Agent f15c494d37 feat: 添加B站登录配置支持和视频笔记
- bilibili_extractor.py: 添加SESSDATA登录凭证支持
- 新增 bilibili_config.example 配置模板
- 更新 .gitignore: 排除虚拟环境和敏感配置文件
- 添加今日记忆 (2026-02-07.md)
- 新增B站视频笔记: Qwen3-TTS上新、时代真变了、用了几天Clawdbot
2026-02-07 14:38:19 +08:00
OpenClaw Agent 08f7c9f26c feat: integrate bilibili-extractor into obsidian-kb
- Add extractors layer with BilibiliExtractor (Playwright stealth mode)
- Add outputs layer with ObsidianWriter and GitSync
- Implement subtitle/no-subtitle graceful degradation
- Update config with output and bilibili extraction settings
- Complete integration pipeline: extract → process → output → sync
- Add comprehensive tests (100% pass rate)
- Update documentation (SKILL.md, README.md, requirements.txt)
2026-02-07 13:07:43 +08:00
OpenClaw Agent ceeeb0f84c 添加bilibili-extractor技能
创建了独立的skill目录和SKILL.md文档:
- 完整的功能说明和使用方法
- 新会话可通过技能系统使用B站视频提取功能
- 包含故障排除、技术栈、依赖说明等

技能位置: skills/bilibili-extractor/
2026-02-07 09:54:19 +08:00
OpenClaw Agent 87ba038d39 添加B站视频内容提取工具(反爬虫方案)
功能特性:
- 使用Playwright + undetected-playwright绕过B站极验验证码
- 支持提取视频基本信息、统计数据、字幕
- 生成结构化Markdown笔记
- 多策略数据提取(JS变量→HTML解析→API备用)
- 完善的错误处理和超时机制

文件说明:
- bilibili_extractor.py: 主程序脚本(21KB)
- BILIBILI_EXTRACTOR_README.md: 完整技术文档(6.0KB)
- USAGE.md: 快速使用指南(1.5KB)
- PROJECT_SUMMARY.md: 项目总结(2.9KB)
- notes/: 生成的视频笔记目录
- .gitignore: 忽略Python虚拟环境等

测试验证:
- 成功提取BV11t64BJEDZ视频内容
- 绕过B站极验验证码
- 生成完整的Markdown笔记
2026-02-07 09:51:22 +08:00
OpenClaw Agent f41f945fe0 添加B站链接转换和浏览器安装脚本
- bilibili-short-to-long.sh: B站短链接转长链接工具
  - 支持短链接、BV号、AV号多种格式
  - 自动清理URL中的tracking参数
  - 友好的输出格式

- install-browser-for-openclaw.sh: OpenClaw浏览器环境一键安装脚本
  - 支持Ubuntu/Debian/CentOS
  - 自动安装Chromium浏览器
  - 配置OpenClaw Gateway
2026-02-07 09:07:42 +08:00
OpenClaw Agent 7032ddb1a3 feat(ai-client): 实现多 AI 提供商支持与 fallback 机制
- 支持 MiniMax、智谱、通义千问等多家 AI 提供商
- 实现自动 fallback 机制(主失败则自动切换备用)
- 修复 MiniMax Anthropic 格式响应解析
- 修复智谱 API 端点(自动添加 /chat/completions)
- 新增 _call_api_with_fallback 方法统一处理 fallback 逻辑
- 更新 _load_ai_config 加载主配置和 fallbacks 配置

配置文件位置: ~/.config/obsidian-kb/config.yaml
2026-02-07 07:08:46 +08:00
OpenClaw Agent fda4b7680b fix: add YAML frontmatter to obsidian-kb SKILL.md for skill loading 2026-02-07 06:45:17 +08:00
OpenClaw Agent 1f76a40d1f fix(obsidian-kb): fix SKILL.md YAML format and restore files 2026-02-07 05:50:00 +08:00
OpenClaw Agent a0e43a5136 fix(obsidian-kb): restore files and shorten description 2026-02-07 05:32:26 +08:00
OpenClaw Agent 8bce4ac23b feat: 添加本地字幕生成功能
- 新增 subtitle_generator.py 核心模块
  - AudioDownloader: 使用 yt-dlp 下载视频音频
  - WhisperTranscriber: Whisper 语音识别器
  - SubtitleFormatter: SRT 格式化器
  - SubtitleCache: 缓存管理器
  - SubtitleGenerator: 统一接口

- 集成到 pipeline.py
  - 添加 _ensure_subtitle() 方法
  - 自动检测无字幕视频并生成字幕
  - 支持缓存机制避免重复处理

- 更新配置文件
  - config.yaml.example 添加字幕配置项
  - requirements.txt 添加依赖包

- 文档更新
  - SKILL.md 添加字幕生成说明
  - SUBTITLE_FEATURE.md 详细功能文档

- 测试脚本
  - test_subtitle_generator.py 完整测试套件

特性:
- 本地处理,数据不上传,隐私安全
- 支持中英文识别及自动检测
- 多种模型可选(tiny/base/small/medium/large)
- 智能缓存,提升处理速度
- 完善的错误处理和日志输出
2026-02-07 01:57:13 +08:00
OpenClaw Agent 42a8dd9642 docs(obsidian-kb): 添加短链接功能实现总结文档 2026-02-06 21:23:17 +08:00
OpenClaw Agent fbcc3df2cf feat(obsidian-kb): 实现短链接解析功能
- 新增 shortlink_resolver.py 模块
  * 检测 B站短链接(b23.tv)和其他短链接服务
  * 提供友好的用户引导信息
  * 支持 Playwright 浏览器自动化解析(可选)

- 集成到 handler.py
  * 短链接自动检测和提示
  * 智能区分短链接和完整 URL

- 更新 add_video.py
  * 支持短链接解析
  * 添加 --playwright 参数启用浏览器解析

- 新增 test_shortlink.py 测试脚本
  * 测试单个短链接检测
  * 测试多个短链接检测
  * 测试混合内容场景
  * 所有测试通过

- 更新文档
  * SKILL.md: 添加短链接说明
  * AUTO_TRIGGER.md: 添加短链接章节
  * obsidian-kb-improvements.md: 标记任务2为已完成

功能特点:
- 方案A(用户引导):已实现并作为默认行为
- 方案B(浏览器自动化):提供 PlaywrightResolver 类,可选使用
- 方案C(代理池):未实现(复杂度高,收益低)

验收标准达成:
✅ 短链接能够友好处理(显示引导信息)
✅ 用户不会感到困惑
✅ 支持 b23.tv 和其他短链接服务(检测)
✅ 集成到 handler.py
✅ 测试各种短链接场景
2026-02-06 21:22:45 +08:00
OpenClaw Agent 7a1ad6088a feat(obsidian-kb): 实现自动触发机制
- 更新 SKILL.md:添加 frontmatter 元数据和自动触发规则说明
- 创建 handler.py:实现链接检测和预览生成
- 支持多种 URL 模式:B站(完整/BV号/短链接)、YouTube、知乎
- 智能去重避免重复显示
- 短链接显示友好提示
- 支持批量检测多个链接
- 添加 AUTO_TRIGGER.md 实现文档
- 更新任务文件:标记任务1已完成

测试通过:
✅ B站完整 URL、BV号、短链接
✅ YouTube 视频
✅ 知乎文章
✅ 批量检测和去重
2026-02-06 21:05:20 +08:00
OpenClaw Agent ee18cde745 feat: Add /kb-add command for quick video addition
- Add cmd_add() function to support adding single videos to inbox
- Support BV IDs, full URLs, and short links (with friendly warning)
- Add --process flag to immediately process after adding
- Add --yes flag to skip confirmation prompts
- Update SKILL.md with /kb-add command documentation
2026-02-06 20:42:11 +08:00