18 Commits

Author SHA1 Message Date
starmen1265 5ccabec3e4 Merge pull request 'fix(ci): correct skill directory names in deploy.yml' (#27) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/27
2026-03-17 23:41:51 +08:00
starmen1265 289be86fc6 Merge pull request 'fix/ci-deploy-reset' (#26) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/26
2026-03-17 23:28:17 +08:00
starmen1265 79cf9635c6 Merge pull request 'fix/ci-deploy-reset' (#25) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/25
2026-03-17 15:35:59 +08:00
starmen1265 bc5e02249d Merge pull request 'feat(kb_workflow): add archive_file to process_command return value' (#24) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/24
2026-03-17 13:54:58 +08:00
starmen1265 36de6cf6dd Merge pull request 'fix(download): return author name instead of mid in _extract_author' (#23) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/23
2026-03-17 13:51:49 +08:00
starmen1265 23254afc49 Merge pull request 'fix(debug): resolve kb-pipeline import issue in skill directory' (#22) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/22
2026-03-17 11:18:24 +08:00
starmen1265 00b8e103c0 Merge pull request 'fix/ci-deploy-reset' (#21) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/21
2026-03-17 00:41:46 +08:00
starmen1265 7a1b2fc244 Merge pull request 'fix/ci-deploy-reset' (#20) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/20
2026-03-17 00:26:41 +08:00
starmen1265 61c4fd401f Merge pull request 'fix/ci-deploy-reset' (#19) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/19
2026-03-16 18:21:01 +08:00
starmen1265 71176c9a12 Merge pull request 'restore: 恢复误删的技能目录并修复导入方式' (#18) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/18
2026-03-16 17:28:19 +08:00
starmen1265 b5cd81d6e7 Merge pull request '删除就技能' (#17) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/17
2026-03-16 16:56:03 +08:00
starmen1265 46aab1d9f4 Merge pull request 'fix/ci-deploy-reset' (#16) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 1s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/16
2026-03-16 15:38:48 +08:00
starmen1265 ed1de1a5b1 Merge pull request 'fix(zhihu): remove noscript tags to prevent duplicate images' (#15) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/15
2026-03-16 15:09:57 +08:00
starmen1265 b89627b8ce Merge pull request 'fix(bilibili): correct metadata and filename in archive' (#14) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/14
2026-03-16 13:49:42 +08:00
starmen1265 33c2fd4bcd Merge pull request 'refactor: remove deprecated kb-xxx sub-skills' (#13) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/13
2026-03-16 12:09:09 +08:00
starmen1265 b3e40b3344 Merge pull request 'fix/ci-deploy-reset' (#12) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/12
2026-03-16 00:25:42 +08:00
starmen1265 9f12b35e6d Merge pull request 'fix/ci-deploy-reset' (#11) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/11
2026-03-15 23:01:13 +08:00
starmen1265 4fde411be3 Merge pull request 'fix/ci-deploy-reset' (#10) from fix/ci-deploy-reset into main
Deploy kb-skills / deploy (push) Successful in 2s
Reviewed-on: http://192.168.1.22:8418/starmen1265/kb-skills/pulls/10
2026-03-15 18:18:24 +08:00
2 changed files with 58 additions and 79 deletions
+57 -74
View File
@@ -5,8 +5,6 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
# KB Workflow # KB Workflow
知识库内容处理管道,支持 B站、知乎等平台的内容抓取、语音识别、深度分析和归档。
## 什么时候使用 ## 什么时候使用
当用户发送包含以下模式的消息时,此技能自动触发: 当用户发送包含以下模式的消息时,此技能自动触发:
@@ -14,7 +12,7 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
| 触发词 | 行为 | 示例 | | 触发词 | 行为 | 示例 |
|--------|------|------| |--------|------|------|
| `添加到知识库` + 链接 | 添加到知识库并处理 | "添加到知识库 https://www.bilibili.com/video/BVxxx" | | `添加到知识库` + 链接 | 添加到知识库并处理 | "添加到知识库 https://www.bilibili.com/video/BVxxx" |
| `保存到知识库` + 链接 | 同上 | "保存到知识库 https://zhuanlan.zhihu.com/p/xxx" | | `保存到知识库` + 链接 | 同上 | "保存到知识库 https://www.bilibili.com/video/BVxxx" |
## 环境变量 ## 环境变量
@@ -24,22 +22,23 @@ description: 当收到"添加到知识库 + 链接" 或"保存到知识库 + 链
**必须的环境变量**(缺失则报错退出): **必须的环境变量**(缺失则报错退出):
配置方式:将变量写入 `~/.kb_env` 文件:
```bash ```bash
# ~/.kb_env 文件
export AI_API_KEY="your-api-key" export AI_API_KEY="your-api-key"
export AI_PROVIDER="zhipu" export AI_PROVIDER="zhipu"
export AI_MODEL="glm-5" export AI_MODEL="glm-5"
export AI_BASE_URL="https://open.bigmodel.cn/api/coding/paas/v4" export AI_BASE_URL="https://open.bigmodel.cn/api/coding/paas/v4"
export KB_INTERMEDIATE_PATH="~/work/intermediate/analysis"
export KB_ARCHIVE_PATH="~/work/zq/30.资源/AI分析"
``` ```
**可选的环境变量**: **可选的环境变量**:
| 变量名 | 用途 | | 变量名 | 用途 | 示例 |
|--------|------| |--------|------|------|
| `BILIBILI_COOKIE` | 下载会员视频 | | `BILIBILI_COOKIE` | 下载会员视频 | `SESSDATA=xxx; bili_jct=xxx` |
| `KB_GIT_USER` / `KB_GIT_EMAIL` | Git 提交信息 | | `KB_GIT_USER` | Git提交用户名 | `Your Name` |
| `HF_TOKEN` | HuggingFace Token(说话者分离) | | `KB_GIT_EMAIL` | Git提交邮箱 | `your@email.com` |
| `KB_ARCHIVE_PATH` | 归档目录路径 | `~/work/zq/30.资源/AI分析` |
| `KB_GIT_ROOT` | Git仓库根目录 | `~/work/zq` |
| `HF_TOKEN` | HuggingFace Token(说话者分离) | `hf_xxx` |
### 检查命令 ### 检查命令
@@ -50,7 +49,7 @@ if [ -f ~/.kb_env ]; then
fi fi
# 检查必须的环境变量 # 检查必须的环境变量
if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ]; then if [ -z "$AI_API_KEY" ] || [ -z "$AI_PROVIDER" ] || [ -z "$AI_MODEL" ] || [ -z "$AI_BASE_URL" ]; then
echo "❌ 错误:必须的环境变量未设置" echo "❌ 错误:必须的环境变量未设置"
echo "请配置 ~/.kb_env 文件" echo "请配置 ~/.kb_env 文件"
exit 1 exit 1
@@ -61,78 +60,72 @@ echo "✅ 环境变量检查通过"
## Workflow 执行方式 ## Workflow 执行方式
### 处理链接 ### 一键处理(推荐)
使用管道 CLI 一键完成所有步骤:
```bash ```bash
# 处理 URL(自动执行完整管道) python3 kb-workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx"
python kb_workflow/kb_workflow.py process "https://www.bilibili.com/video/BVxxx" ```
**该命令自动执行:**
1. URL 解析 → 识别平台和内容 ID
2. 元数据提取 → 获取标题、作者等信息
3. 内容下载 → 字幕/视频/文章内容
4. 语音识别 → Whisper/FunASR(如需要)
5. 深度分析 → AI 生成分析报告
6. 归档 → 移动到归档目录
**命令参数:**
```bash
# 指定输出目录 # 指定输出目录
python kb_workflow/kb_workflow.py process "https://..." -o /path/to/output python3 kb-workflow/kb_workflow.py process "https://..." -o /path/to/output
# 强制重新执行(忽略已完成状态) # 强制重新执行(忽略已完成状态)
python kb_workflow/kb_workflow.py process "https://..." --no-skip python3 kb-workflow/kb_workflow.py process "https://..." --no-skip
``` ```
### 知乎登录
```bash
# 二维码扫码登录(Cookie 自动保存)
python kb_workflow/kb_workflow.py login
# 检查 Cookie 状态
python kb_workflow/kb_workflow.py login --status
# 从 JSON 文件导入 Cookie
python kb_workflow/kb_workflow.py login --import-cookie cookies.json
```
**注意:** 知乎链接处理时会自动检测 Cookie 是否过期,过期则自动触发扫码登录。
## 支持的平台和内容类型 ## 支持的平台和内容类型
| 平台 | 内容类型 | URL 示例 | | 平台 | 内容类型 | URL 示例 |
|------|----------|----------| |------|----------|----------|
| B站 | 视频 | `bilibili.com/video/BVxxx` | | B站 | 视频 | `bilibili.com/video/BVxxx` |
| B站 | 专栏 | `bilibili.com/read/cvxxx` | | B站 | 专栏 | `bilibili.com/read/cvxxx` |
| 知乎 | 问答 | `zhihu.com/question/xxx/answer/xxx` | | 知乎 | 问答 | `zhihu.com/question/xxx` |
| 知乎 | 专栏 | `zhuanlan.zhihu.com/p/xxx` | | 知乎 | 专栏 | `zhuanlan.zhihu.com/p/xxx` |
| 知乎 | 视频 | `zhihu.com/zvideo/xxx` |
## 处理流程 ## 处理流程
``` ```
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────┐
│ Pipeline 处理流程 │ │ kb-workflow process │
├─────────────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────────────┤
│ 1. Parse → LinkParser 解析 URL,识别平台和 ID │ │ 1. Parse → LinkParser 解析 URL │
│ 2. Download → 平台适配器获取元数据和内容 │ │ 2. Download → 平台适配器获取元数据和内容 │
│ 3. ASR → 语音识别(视频无字幕时) │ │ 3. ASR → 语音识别(视频无字幕时) │
│ 4. Analyze → DeepAnalyzer 生成深度分析报告 │ │ 4. Analyze → DeepAnalyzer 生成分析报告 │
│ 5. Archive → 归档到 KB_ARCHIVE_PATH │ │ 5. Archive → 归档到 KB_ARCHIVE_PATH │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
``` ```
**知乎特殊处理:**
- 评论默认爬取并追加到正文末尾
- Cookie 过期时自动弹出二维码,扫码后继续执行
- 无需手动设置 Cookie
## 目录结构 ## 目录结构
**工作目录(KB_INTERMEDIATE_PATH):** **工作目录(默认 ~/work/knowledge-base/):**
``` ```
~/work/intermediate/analysis/ ~/work/knowledge-base/
└── {platform}_{content_id}/ ├── pending/ # 待处理链接
├── metadata.json # 元数据 ├── metadata/ # 元数据 JSON
├── content.md # 内容(含评论) ├── content/ # 内容文件
└── downloads/ # 下载的媒体文件 │ └── downloads/ # 下载的视频/字幕
└── analyzed/ # 分析报告
``` ```
**归档目录(KB_ARCHIVE_PATH):** **归档目录(默认 ~/work/zq/30.资源/AI分析/):**
``` ```
~/work/zq/30.资源/AI分析/ ~/work/zq/30.资源/AI分析/
└── YYYY-MM-DD/ └── YYYY-MM-DD/
└── {platform}_{id}_analysis.md # 深度分析报告 └── {platform}_{id}.md # 深度分析报告
``` ```
## ⚠️ 严格失败处理原则 ## ⚠️ 严格失败处理原则
@@ -147,44 +140,34 @@ python kb_workflow/kb_workflow.py login --import-cookie cookies.json
## 示例 ## 示例
``` ```
用户: 添加到知识库 https://zhuanlan.zhihu.com/p/123456789 用户: 添加到知识库 https://www.bilibili.com/video/BV1xx
Claude: Claude:
✅ 环境变量检查通过 ✅ 环境变量检查通过
◆ 处理中... ◆ 处理中...
→ 解析 URL: zhihu 专栏 123456789 → 解析 URL: bilibili 视频 BV1xx
→ 获取元数据... → 获取元数据...
→ 爬取内容和评论... → 下载字幕...
→ 深度分析... → 深度分析...
✅ 完成! ✅ 完成!
标题: 深度学习在推荐系统中的应用 标题: 【教程】如何使用 Python 进行数据分析
内容目录: ~/work/intermediate/analysis/zhihu_123456789/ 处理阶段: 全部完成
归档文件: ~/work/zq/30.资源/AI分析/2026-03-17/zhihu_123456789_analysis.md 最终文件: ~/work/zq/30.资源/AI分析/2026-03-15/bilibili_BV1xx.md
``` ```
## 技术架构 ## 技术架构
``` ```
kb_workflow/ kb-workflow/kb_workflow.py (CLI)
├── kb_workflow.py — CLI 入口 └── kb_pipeline/ (核心管道包)
├── auth/ ├── pipeline.py — Pipeline 编排引擎
│ └── zhihu_login.py — 知乎登录和 Cookie 管理 ├── processors/ — 处理器实现
├── crawler/ │ ├── parse.py — URL 解析
│ ├── zhihu_crawler.py — 知乎爬虫(问答/专栏/评论) │ ├── download.py — 内容下载
│ └── models.py — 数据模型 │ ├── asr.py — 语音识别
├── pipeline/ │ └── analyze.py — 深度分析
│ ├── pipeline.py — Pipeline 编排引擎 └── schemas.py — 数据模型
│ ├── processors/ — 处理器实现
│ │ ├── parse.py — URL 解析
│ │ ├── download.py — 内容下载
│ │ ├── asr.py — 语音识别
│ │ ├── analyze.py — 深度分析
│ │ └── archive.py — 归档
│ ├── platforms/ — 平台适配器
│ │ └── zhihu.py — 知乎适配器(使用 ZhihuCrawler)
│ └── schemas.py — 数据模型
└── deep_analyzer/ — AI 分析器
``` ```
+1 -5
View File
@@ -19,8 +19,7 @@ from playwright.sync_api import Route, Response, sync_playwright
from tenacity import Retrying, stop_after_attempt, wait_exponential, RetryError from tenacity import Retrying, stop_after_attempt, wait_exponential, RetryError
from kb_workflow.metadata.zhihu_extractor import PlaywrightBrowser from kb_workflow.metadata.zhihu_extractor import PlaywrightBrowser
# NOTE: get_settings is imported lazily in _get_cookie() to avoid circular import from kb_workflow.pipeline.config import get_settings
# Circular: zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler
from .models import ( from .models import (
AnswerData, AnswerData,
ArticleData, ArticleData,
@@ -130,9 +129,6 @@ class ZhihuCrawler:
Returns: Returns:
Cookie string from settings Cookie string from settings
""" """
# Lazy import to avoid circular dependency:
# zhihu_crawler -> pipeline.config -> pipeline.platforms -> zhihu_crawler
from kb_workflow.pipeline.config import get_settings
settings = get_settings() settings = get_settings()
return settings.zhihu_cookie.get_secret_value() return settings.zhihu_cookie.get_secret_value()