docs: map existing codebase
This commit is contained in:
@@ -0,0 +1,154 @@
|
||||
# Architecture
|
||||
|
||||
**Analysis Date:** 2026-03-13
|
||||
|
||||
## Pattern Overview
|
||||
|
||||
**Overall:** Multi-skill workflow orchestration with skill-based modular architecture
|
||||
|
||||
**Key Characteristics:**
|
||||
- Skill-based modular design: Each skill is an independent module with its own `SKILL.md` specification
|
||||
- Pipeline workflow pattern: Sequential processing stages orchestrated by `kb-workflow`
|
||||
- Stateful session management: tmux-based persistent sessions for GSD-OpenClaw
|
||||
- Platform abstraction: Unified interfaces for multiple content platforms (Bilibili, Zhihu, YouTube, etc.)
|
||||
|
||||
## Layers
|
||||
|
||||
**Skill Definition Layer:**
|
||||
- Purpose: Define skill capabilities, triggers, and execution steps
|
||||
- Location: `kb-*/SKILL.md`, `gsd-openclaw/skill/SKILL.md`
|
||||
- Contains: Markdown documents with YAML frontmatter and structured workflows
|
||||
- Depends on: Nothing (declarative)
|
||||
- Used by: AI agents that execute skills
|
||||
|
||||
**Core Implementation Layer:**
|
||||
- Purpose: Implement actual functionality for each skill
|
||||
- Location: `kb-*/*.py`, `gsd-openclaw/src/gsd_openclaw/*.py`
|
||||
- Contains: Python modules with CLI entry points and core logic
|
||||
- Depends on: External APIs, platform SDKs, AI providers
|
||||
- Used by: Skill orchestration layer
|
||||
|
||||
**Orchestration Layer:**
|
||||
- Purpose: Coordinate multiple skills in workflows
|
||||
- Location: `kb-workflow/kb_workflow.py`
|
||||
- Contains: Workflow coordination utilities, directory management, Git integration
|
||||
- Depends on: All skill modules
|
||||
- Used by: Main agent executing workflows
|
||||
|
||||
**Session Management Layer:**
|
||||
- Purpose: Manage persistent tmux sessions for long-running operations
|
||||
- Location: `gsd-openclaw/src/gsd_openclaw/tmux_session.py`
|
||||
- Contains: TmuxSessionManager, session lifecycle management
|
||||
- Depends on: libtmux, StateManager, WorkspaceManager
|
||||
- Used by: GSD command orchestration
|
||||
|
||||
**State Persistence Layer:**
|
||||
- Purpose: Provide crash-safe state checkpointing
|
||||
- Location: `gsd-openclaw/src/gsd_openclaw/state_manager.py`
|
||||
- Contains: StateManager, atomic save/load with backup rotation
|
||||
- Depends on: File system only
|
||||
- Used by: TmuxSessionManager, workspace management
|
||||
|
||||
## Data Flow
|
||||
|
||||
**Content Processing Pipeline:**
|
||||
|
||||
1. URL Input → `kb-link-parser` extracts platform, type, and ID
|
||||
2. Metadata Extraction → Platform-specific skill (`kb-*-metadata`) fetches metadata
|
||||
3. Content Download → Platform-specific skill (`kb-*-content`) downloads content
|
||||
4. Speech Recognition → `kb-whisper`/`kb-funasr` transcribes audio (if needed)
|
||||
5. Deep Analysis → `kb-deep-analyzer` generates AI-powered analysis report
|
||||
6. Archive → `kb-workflow` archives to Git-managed directory
|
||||
|
||||
**GSD Session Flow:**
|
||||
|
||||
1. `/gsd:*` command received
|
||||
2. `TmuxSessionManager.create_session()` or `resume_session()`
|
||||
3. State persisted via `StateManager.save()`
|
||||
4. Command executed in tmux session
|
||||
5. Output captured and formatted via `FeishuResponseFormatter`
|
||||
|
||||
**State Management:**
|
||||
- State stored in `~/gsd-workspaces/{project}/.gsd/state.json`
|
||||
- Atomic writes via temp file + rename
|
||||
- Automatic backup rotation (state.json → state.json.bak)
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
**Skill:**
|
||||
- Purpose: Represents an independent capability unit
|
||||
- Examples: `kb-link-parser/link-parser.py`, `kb-deep-analyzer/deep_analyzer.py`
|
||||
- Pattern: CLI-based with JSON output, defined by `SKILL.md` specification
|
||||
|
||||
**TmuxSession:**
|
||||
- Purpose: Represents a managed tmux session for project isolation
|
||||
- Examples: `gsd-openclaw/src/gsd_openclaw/tmux_session.py`
|
||||
- Pattern: Dataclass with session_id, project_name, tmux_session_name
|
||||
|
||||
**Workspace:**
|
||||
- Purpose: Isolated directory structure per project
|
||||
- Examples: `gsd-openclaw/src/gsd_openclaw/workspace.py`
|
||||
- Pattern: Defined subdirectories (.gsd/, code/, io-logs/)
|
||||
|
||||
**AnalysisResult:**
|
||||
- Purpose: Structured output from deep analysis
|
||||
- Examples: `kb-deep-analyzer/deep_analyzer.py`
|
||||
- Pattern: Dataclass with summary, main_topics, key_insights, tags
|
||||
|
||||
## Entry Points
|
||||
|
||||
**CLI Entry Points:**
|
||||
- Location: `kb-*/*.py` with `if __name__ == '__main__': main()`
|
||||
- Triggers: Command-line invocation with argparse
|
||||
- Responsibilities: Parse arguments, invoke core logic, output JSON
|
||||
|
||||
**gsd-ocl CLI:**
|
||||
- Location: `gsd-openclaw/src/gsd_openclaw/cli.py`
|
||||
- Triggers: `gsd-ocl create|list|destroy|resume|cleanup`
|
||||
- Responsibilities: Session lifecycle management
|
||||
|
||||
**Skill Trigger:**
|
||||
- Location: `SKILL.md` frontmatter with `name` and `description`
|
||||
- Triggers: AI agent pattern matching on user messages
|
||||
- Responsibilities: Define when and how to invoke the skill
|
||||
|
||||
## Error Handling
|
||||
|
||||
**Strategy:** Fail-fast with structured JSON error output
|
||||
|
||||
**Patterns:**
|
||||
- All CLI tools output JSON with `success: true/false` field
|
||||
- Error responses include `error` and `detail` fields
|
||||
- Deep analysis failure terminates entire workflow (per `kb-workflow/SKILL.md`)
|
||||
- State manager uses backup fallback on corruption
|
||||
|
||||
**Error Response Format:**
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "Error description",
|
||||
"detail": "Additional context"
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-Cutting Concerns
|
||||
|
||||
**Logging:** Console output with `[ModuleName]` prefixes (e.g., `[DeepAnalyzer]`, `[AI Client]`)
|
||||
|
||||
**Validation:**
|
||||
- Environment variable checks before workflow execution (`kb-workflow/SKILL.md`)
|
||||
- Required field validation in `StateManager.load()`
|
||||
- Platform/type validation in link parser
|
||||
|
||||
**Authentication:**
|
||||
- Platform cookies via environment variables (`BILIBILI_COOKIE`, `ZHIHU_COOKIE`)
|
||||
- AI API keys via environment variables (`AI_API_KEY`, `HF_TOKEN`)
|
||||
- Git credentials via environment variables (`KB_GIT_USER`, `KB_GIT_EMAIL`)
|
||||
|
||||
**Configuration:**
|
||||
- Environment variables loaded from `~/.kb_env`
|
||||
- Path configuration via environment variables (`KB_BASE_PATH`, `KB_ARCHIVE_PATH`, `KB_GIT_ROOT`)
|
||||
|
||||
---
|
||||
|
||||
*Architecture analysis: 2026-03-13*
|
||||
@@ -0,0 +1,180 @@
|
||||
# Codebase Concerns
|
||||
|
||||
**Analysis Date:** 2026-03-13
|
||||
|
||||
## Tech Debt
|
||||
|
||||
### 异常处理不完善
|
||||
- Issue: 多处使用 bare `except:` 或 `except Exception` 吞掉所有异常,没有记录错误信息
|
||||
- Files:
|
||||
- `kb-bilibili-metadata/bilibili_extractor.py` (第 310、340 行)
|
||||
- `kb-deep-analyzer/ai_client.py` (第 228、489 行)
|
||||
- `kb-deep-analyzer/deep_analyzer.py` (第 394、410、506、516 行)
|
||||
- `kb-whisper-diarization/whisper_diarization.py` (第 70、326 行)
|
||||
- `kb-funasr/funasr_tool.py` (第 81、186 行)
|
||||
- Impact: 错误被静默忽略,调试困难,生产问题难以追踪
|
||||
- Fix approach: 使用具体异常类型,添加日志记录
|
||||
|
||||
### 过多的 return None 模式
|
||||
- Issue: 多个函数在异常或边界情况返回 `None`,调用方需要额外检查
|
||||
- Files:
|
||||
- `kb-bilibili-metadata/bilibili_extractor.py` (第 140、161、176 行)
|
||||
- `kb-bilibili-content/video_downloader.py` (第 36、48、68、75、90、106、148 行)
|
||||
- `kb-link-parser/link-parser.py` (第 197、245、379、436 行)
|
||||
- `kb-link-parser/link_parser_shortlink.py` (第 223、237、249 行)
|
||||
- `kb-zhihu-metadata/zhihu_extractor.py` (第 137、152、410、437、450 行)
|
||||
- Impact: 调用方需要频繁检查 None,代码冗长,容易遗漏
|
||||
- Fix approach: 使用 Result 模式或抛出具体异常
|
||||
|
||||
### 硬编码的默认路径
|
||||
- Issue: 默认知识库路径硬编码在代码中
|
||||
- Files: `kb-workflow/kb_workflow.py` (第 104、113、122 行)
|
||||
- Impact: 不同用户或环境需要修改代码或设置环境变量
|
||||
- Fix approach: 使用配置文件或更灵活的路径发现机制
|
||||
|
||||
### AI 客户端重复实现
|
||||
- Issue: 存在两套 AI 客户端实现 (`ai_client.py` 和 `deep_analyzer.py` 中的 `SimpleAIClient`)
|
||||
- Files:
|
||||
- `kb-deep-analyzer/ai_client.py` (514 行,功能完整)
|
||||
- `kb-deep-analyzer/deep_analyzer.py` (37-123 行,简化版本)
|
||||
- Impact: 代码重复,维护困难,行为可能不一致
|
||||
- Fix approach: 统一使用一个 AI 客户端实现
|
||||
|
||||
## Known Bugs
|
||||
|
||||
### CI/CD 部署列表与实际目录不匹配
|
||||
- Symptoms: `.gitea/workflows/deploy.yml` 中的技能列表包含 `dev-workflow`,但该目录已被删除(根据 git log)
|
||||
- Files: `.gitea/workflows/deploy.yml` (第 23 行)
|
||||
- Trigger: 推送到 main 分支时,CI 会尝试同步不存在的目录
|
||||
- Workaround: 目录不存在时 rsync 会跳过,但会产生混乱的日志
|
||||
|
||||
### Playwright 依赖检测不完整
|
||||
- Symptoms: 多个模块检测 Playwright 可用性但不处理所有边界情况
|
||||
- Files: `kb-bilibili-metadata/bilibili_extractor.py` (第 28-44 行)
|
||||
- Trigger: 当 Playwright 不可用时回退到 API,但 httpx 也可能不可用
|
||||
- Workaround: 检查 HTTPX_AVAILABLE 并返回错误信息
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### 环境变量中传递敏感信息
|
||||
- Risk: API Keys 和 Cookies 通过环境变量传递,可能在日志或错误信息中泄露
|
||||
- Files:
|
||||
- `kb-deep-analyzer/deep_analyzer.py` (第 49-52 行直接读取环境变量)
|
||||
- `kb-deep-analyzer/ai_client.py` (第 32、44 行)
|
||||
- `kb-zhihu-metadata/zhihu_extractor.py` (Cookie 处理)
|
||||
- Current mitigation: 使用 `.env.example` 文档说明,但无运行时保护
|
||||
- Recommendations:
|
||||
- 敏感信息不应出现在日志输出中
|
||||
- 考虑使用 secrets 管理工具
|
||||
|
||||
### Git 凭据在代码中配置
|
||||
- Risk: `kb_workflow.py` 中 Git 配置通过环境变量传递用户名和邮箱
|
||||
- Files: `kb-workflow/kb_workflow.py` (第 196-200 行)
|
||||
- Current mitigation: 环境变量方式
|
||||
- Recommendations: 考虑使用 Git credential helper
|
||||
|
||||
## Performance Bottlenecks
|
||||
|
||||
### 固定的 HTTP 超时
|
||||
- Problem: 多处 HTTP 请求使用固定的 30 秒超时
|
||||
- Files:
|
||||
- `kb-deep-analyzer/ai_client.py` (第 135、169、191、213 行)
|
||||
- `kb-bilibili-metadata/bilibili_extractor.py` (第 262 行)
|
||||
- Cause: 大文件下载或慢速 API 可能超时
|
||||
- Improvement path: 根据请求类型使用可配置的超时
|
||||
|
||||
### AI 调用长超时
|
||||
- Problem: AI API 调用使用 600 秒(10 分钟)超时
|
||||
- Files: `kb-deep-analyzer/deep_analyzer.py` (第 87、97、116 行)
|
||||
- Cause: 长文本分析需要时间,但 10 分钟过长
|
||||
- Improvement path: 根据内容长度动态调整超时
|
||||
|
||||
### Playwright 浏览器启动开销
|
||||
- Problem: 每次提取都启动新的浏览器实例
|
||||
- Files:
|
||||
- `kb-bilibili-metadata/bilibili_extractor.py`
|
||||
- `kb-zhihu-metadata/zhihu_extractor.py`
|
||||
- Cause: 无浏览器实例复用
|
||||
- Improvement path: 考虑浏览器实例池或持久化
|
||||
|
||||
## Fragile Areas
|
||||
|
||||
### 网页选择器依赖
|
||||
- Files:
|
||||
- `kb-zhihu-metadata/zhihu_extractor.py` (第 183-327 行)
|
||||
- `kb-bilibili-metadata/bilibili_extractor.py`
|
||||
- Why fragile: 网站更新会改变 CSS 类名和 DOM 结构
|
||||
- Safe modification: 添加选择器配置化,使用多个备用选择器
|
||||
- Test coverage: 缺少针对选择器变更的测试
|
||||
|
||||
### 元数据格式多样性
|
||||
- Files: `kb-deep-analyzer/deep_analyzer.py` (第 382-414 行 `_normalize_metadata`)
|
||||
- Why fragile: 不同来源的元数据格式不一致(扁平、嵌套、混合)
|
||||
- Safe modification: 使用数据验证库(如 Pydantic)标准化格式
|
||||
- Test coverage: 部分覆盖
|
||||
|
||||
### Tmux 会话状态管理
|
||||
- Files: `gsd-openclaw/src/gsd_openclaw/tmux_session.py`
|
||||
- Why fragile: tmux 会话可能被外部操作破坏,状态文件可能不同步
|
||||
- Safe modification: 添加状态校验和恢复机制
|
||||
- Test coverage: 有测试 (`test_tmux_session.py`),但边缘情况可能不足
|
||||
|
||||
## Scaling Limits
|
||||
|
||||
### 单实例设计
|
||||
- Current capacity: 单用户、单进程
|
||||
- Limit: 无法并行处理多个分析任务
|
||||
- Scaling path: 引入任务队列和 worker 进程
|
||||
|
||||
### 内存中的用户上下文缓存
|
||||
- Current capacity: 无限(但无清理机制)
|
||||
- Limit: 长时间运行后内存泄漏
|
||||
- Files: `gsd-openclaw/src/gsd_openclaw/orchestrator.py` (第 70 行 `user_context_managers` 字典)
|
||||
- Scaling path: 添加 LRU 缓存或定期清理
|
||||
|
||||
## Dependencies at Risk
|
||||
|
||||
### 外部 API 依赖
|
||||
- Risk: 依赖 Bilibili、知乎等第三方 API,可能随时变更
|
||||
- Impact: 抓取功能失效
|
||||
- Migration plan: 使用官方 SDK(如果有),或添加 API 版本检测
|
||||
|
||||
### Playwright 和 undetected-playwright
|
||||
- Risk: `undetected_playwright` 是小众库,维护可能不活跃
|
||||
- Impact: 反爬虫规避功能失效
|
||||
- Files: `kb-bilibili-metadata/bilibili_extractor.py` (第 35-38 行)
|
||||
- Migration plan: 监控库更新,准备降级方案
|
||||
|
||||
## Missing Critical Features
|
||||
|
||||
### 缺少统一日志
|
||||
- Problem: 各模块使用 `print()` 输出日志,无统一日志级别控制
|
||||
- Blocks: 生产环境日志管理、问题排查
|
||||
|
||||
### 缺少健康检查端点
|
||||
- Problem: 无服务健康状态监控
|
||||
- Blocks: 运维监控、自动故障恢复
|
||||
|
||||
## Test Coverage Gaps
|
||||
|
||||
### kb-* 技能缺少单元测试
|
||||
- What's not tested: 所有 kb-* 技能目录(kb-bilibili-content, kb-zhihu-metadata 等)都没有测试文件
|
||||
- Files: 所有 `kb-*/**/*.py` 文件
|
||||
- Risk: 重构或修改可能引入回归 bug
|
||||
- Priority: High - 核心功能缺少测试保护
|
||||
|
||||
### 集成测试缺失
|
||||
- What's not tested: 技能之间的集成(如 kb-workflow 调用其他技能)
|
||||
- Files: 跨技能调用路径
|
||||
- Risk: 接口变更导致集成失败
|
||||
- Priority: Medium
|
||||
|
||||
### 边缘情况测试不足
|
||||
- What's not tested: 网络错误、超时、无效输入等
|
||||
- Files: `gsd-openclaw/tests/` 中的测试主要覆盖正常路径
|
||||
- Risk: 生产环境异常处理不当
|
||||
- Priority: Medium
|
||||
|
||||
---
|
||||
|
||||
*Concerns audit: 2026-03-13*
|
||||
@@ -0,0 +1,162 @@
|
||||
# External Integrations
|
||||
|
||||
**Analysis Date:** 2026-03-13
|
||||
|
||||
## APIs & External Services
|
||||
|
||||
**AI Providers:**
|
||||
- **智谱 AI (Zhipu)** - 主要 AI 分析服务
|
||||
- SDK: `requests` HTTP 客户端
|
||||
- Auth: `AI_API_KEY` 环境变量
|
||||
- Base URL: `https://open.bigmodel.cn/api/paas/v4`
|
||||
- 模型: `glm-5`, `glm-4`
|
||||
|
||||
- **MiniMax** - AI 备选服务
|
||||
- SDK: `requests` HTTP 客户端
|
||||
- Auth: `AI_API_KEY` 环境变量
|
||||
- Base URL: `https://api.minimaxi.com/anthropic/v1/messages`
|
||||
|
||||
- **OpenAI** - AI 备选服务
|
||||
- SDK: `requests` HTTP 客户端
|
||||
- Auth: `AI_API_KEY` 环境变量
|
||||
|
||||
- **阿里云通义千问 (Alibaba/Qwen)** - AI 备选服务
|
||||
- SDK: `requests` HTTP 客户端
|
||||
- Auth: `AI_API_KEY` 环境变量
|
||||
|
||||
**视频平台 API:**
|
||||
- **Bilibili** - 视频/专栏元数据提取
|
||||
- SDK: `httpx`, `playwright`
|
||||
- Auth: `BILIBILI_COOKIE` 环境变量 (SESSDATA, bili_jct)
|
||||
- 文件: `kb-bilibili-metadata/bilibili_extractor.py`
|
||||
|
||||
- **知乎** - 问答/专栏/视频内容
|
||||
- SDK: `yt-dlp`, `playwright`
|
||||
- Auth: `ZHIHU_COOKIE` 环境变量
|
||||
- 文件: `kb-zhihu-metadata/zhihu_metadata.py`, `kb-zhihu-content/zhihu_content.py`
|
||||
|
||||
- **YouTube** - 视频内容(通过链接解析)
|
||||
- SDK: `yt-dlp`
|
||||
- 文件: `kb-link-parser/link-parser.py`
|
||||
|
||||
- **小红书** - 笔记内容
|
||||
- 短链接: `xhslink.com`
|
||||
- 文件: `kb-link-parser/link-parser.py`
|
||||
|
||||
## Data Storage
|
||||
|
||||
**Databases:**
|
||||
- 无数据库 - 所有数据以文件形式存储
|
||||
|
||||
**File Storage:**
|
||||
- 本地文件系统
|
||||
- 工作目录: `~/work/knowledge-base/`
|
||||
- 结构:
|
||||
- `pending/` - 待处理链接
|
||||
- `metadata/` - 元数据 JSON
|
||||
- `content/downloads/` - 下载的媒体文件
|
||||
- `analyzed/` - AI 分析报告
|
||||
|
||||
**归档存储:**
|
||||
- 路径: `~/work/zq/30.资源/AI分析/`
|
||||
- 按日期组织: `YYYY-MM-DD/`
|
||||
|
||||
**Caching:**
|
||||
- 模型缓存: `~/.cache/modelscope/hub/` - FunASR 模型
|
||||
|
||||
## Authentication & Identity
|
||||
|
||||
**Auth Provider:**
|
||||
- Cookie-based 认证
|
||||
- Bilibili: `BILIBILI_COOKIE`
|
||||
- 知乎: `ZHIHU_COOKIE`
|
||||
|
||||
**AI API 认证:**
|
||||
- API Key 认证
|
||||
- `AI_API_KEY` - AI 服务密钥
|
||||
- `HF_TOKEN` - HuggingFace Token (说话者分离)
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
**Error Tracking:**
|
||||
- 无外部监控服务
|
||||
|
||||
**Logs:**
|
||||
- 控制台输出 - 所有技能通过 `print()` 输出日志
|
||||
- 建议使用 `python3 -u` 禁用缓冲以实时查看输出
|
||||
|
||||
## CI/CD & Deployment
|
||||
|
||||
**Hosting:**
|
||||
- Self-hosted (本地服务器)
|
||||
- 技能部署位置: `/home/starmen1265/.openclaw/skills/`
|
||||
|
||||
**CI Pipeline:**
|
||||
- Gitea Actions (self-hosted runner)
|
||||
- 配置文件: `.gitea/workflows/deploy.yml`
|
||||
- 触发: push to main
|
||||
- 使用 `rsync` 同步技能目录
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
**必须的环境变量:**
|
||||
```bash
|
||||
AI_API_KEY="your-api-key" # AI 服务密钥
|
||||
AI_PROVIDER="zhipu" # 或 openai, minimax, alibaba
|
||||
AI_MODEL="glm-5" # 模型名称
|
||||
AI_BASE_URL="https://..." # API 地址
|
||||
```
|
||||
|
||||
**可选的环境变量:**
|
||||
```bash
|
||||
BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx" # B站登录态
|
||||
ZHIHU_COOKIE="..." # 知乎登录态
|
||||
HF_TOKEN="hf_xxx" # HuggingFace Token
|
||||
KB_GIT_USER="Your Name" # Git 提交用户名
|
||||
KB_GIT_EMAIL="your@email.com" # Git 提交邮箱
|
||||
KB_ARCHIVE_PATH="~/work/zq/30.资源/AI分析" # 归档目录
|
||||
KB_GIT_ROOT="~/work/zq" # Git 仓库根目录
|
||||
KB_BASE_PATH="~/work/knowledge-base" # 知识库基础路径
|
||||
WHISPER_CPP_PATH="~/whisper.cpp" # Whisper.cpp 路径
|
||||
```
|
||||
|
||||
**配置文件位置:**
|
||||
- `~/.kb_env` - 知识库专用配置(推荐)
|
||||
- `kb-workflow/.env.example` - 示例配置
|
||||
|
||||
**Secrets location:**
|
||||
- 环境变量(明文存储在 `~/.kb_env`)
|
||||
- 无加密/secrets 管理
|
||||
|
||||
## Webhooks & Callbacks
|
||||
|
||||
**Incoming:**
|
||||
- 无 Webhook 端点
|
||||
|
||||
**Outgoing:**
|
||||
- 无主动 Webhook 调用
|
||||
|
||||
## External Tools
|
||||
|
||||
**音视频处理:**
|
||||
- `ffmpeg` / `ffprobe` - 音视频格式转换和元数据提取
|
||||
- `yt-dlp` - 视频下载(知乎视频)
|
||||
|
||||
**语音识别:**
|
||||
- `whisper.cpp` - 本地 Whisper 推理
|
||||
- `funasr` / `modelscope` - 阿里 FunASR
|
||||
|
||||
**浏览器自动化:**
|
||||
- `playwright` - 无头浏览器
|
||||
- `undetected_playwright` - 反检测浏览器
|
||||
|
||||
**OCR:**
|
||||
- `tesseract` - OCR 引擎
|
||||
- `easyocr` - 深度学习 OCR
|
||||
|
||||
**会话管理:**
|
||||
- `tmux` - 终端会话管理(gsd-openclaw)
|
||||
|
||||
---
|
||||
|
||||
*Integration audit: 2026-03-13*
|
||||
@@ -0,0 +1,108 @@
|
||||
# Technology Stack
|
||||
|
||||
**Analysis Date:** 2026-03-13
|
||||
|
||||
## Languages
|
||||
|
||||
**Primary:**
|
||||
- Python 3.12 - 主要用于 kb-* 技能模块(内容采集、语音识别、AI分析等)
|
||||
- JavaScript/Node.js >=16.7.0 - 用于 gsd (Get-Shit-Done) 元提示系统
|
||||
|
||||
**Secondary:**
|
||||
- Shell/Bash - 用于部署脚本和工具链(ffmpeg, yt-dlp)
|
||||
|
||||
## Runtime
|
||||
|
||||
**Environment:**
|
||||
- Python 3.12.3
|
||||
- Node.js v22.22.0
|
||||
|
||||
**Package Manager:**
|
||||
- pip/uv (Python) - gsd-openclaw 使用 uv.lock
|
||||
- npm (Node.js) - gsd 使用 npm
|
||||
|
||||
**Lockfile:**
|
||||
- `uv.lock` (gsd-openclaw)
|
||||
- `package-lock.json` (gsd)
|
||||
|
||||
## Frameworks
|
||||
|
||||
**Core:**
|
||||
- 无传统 Web 框架 - 这是一个 CLI 技能集合,非 Web 应用
|
||||
|
||||
**Testing:**
|
||||
- pytest >=7.0.0 - Python 单元测试 (gsd-openclaw)
|
||||
- Node.js 内置测试 - gsd 使用自定义 `scripts/run-tests.cjs`
|
||||
|
||||
**Build/Dev:**
|
||||
- esbuild ^0.24.0 - gsd hooks 构建
|
||||
- c8 ^11.0.0 - 测试覆盖率工具
|
||||
|
||||
## Key Dependencies
|
||||
|
||||
**Python 核心依赖:**
|
||||
- `requests` - HTTP 客户端,用于 AI API 调用
|
||||
- `httpx` - 异步 HTTP 客户端(Bilibili 元数据提取)
|
||||
- `playwright` / `undetected_playwright` - 浏览器自动化(反爬虫)
|
||||
- `easyocr>=1.7.0` - OCR 文字识别
|
||||
- `pytesseract>=0.3.10` - Tesseract OCR 接口
|
||||
- `Pillow>=9.0.0` - 图像处理
|
||||
- `torch>=2.0.0` - PyTorch 深度学习框架
|
||||
- `pyannote.audio>=1.0.0` - 说话者分离
|
||||
- `funasr` / `modelscope` - 阿里 FunASR 语音识别
|
||||
- `libtmux>=0.55.0` - tmux 会话管理 (gsd-openclaw)
|
||||
|
||||
**Node.js 核心依赖:**
|
||||
- 无外部运行时依赖 - gsd 是纯 CommonJS 工具集
|
||||
|
||||
## Configuration
|
||||
|
||||
**环境变量配置:**
|
||||
- 配置文件: `~/.kb_env` - 知识库专用环境变量
|
||||
- `.env.example` - 各技能目录下提供示例配置
|
||||
|
||||
**关键配置文件:**
|
||||
- `pyproject.toml` - gsd-openclaw 项目配置
|
||||
- `requirements.txt` - Python 技能依赖声明
|
||||
- `package.json` - gsd Node.js 项目配置
|
||||
|
||||
**构建配置:**
|
||||
- `scripts/build-hooks.js` - gsd hooks 构建
|
||||
- `tool.setuptools` - gsd-openclaw 打包配置
|
||||
|
||||
## Platform Requirements
|
||||
|
||||
**Development:**
|
||||
- ffmpeg - 音视频格式转换
|
||||
- yt-dlp - 视频下载工具
|
||||
- whisper.cpp - 本地语音识别(或 GPU 版 PyTorch)
|
||||
- tesseract - OCR 引擎(可选)
|
||||
- tmux - 会话管理
|
||||
|
||||
**Production:**
|
||||
- 自托管 Gitea Actions runner
|
||||
- 部署目标: `~/.openclaw/skills/` 目录
|
||||
- rsync 同步部署
|
||||
|
||||
## Project Structure
|
||||
|
||||
**Monorepo 结构:**
|
||||
```
|
||||
kb-skills/
|
||||
├── gsd/ # Get-Shit-Done 元提示系统 (Node.js)
|
||||
├── gsd-openclaw/ # GSD-OpenClaw 桥接技能 (Python)
|
||||
├── kb-workflow/ # 知识库工作流编排入口
|
||||
├── kb-link-parser/ # 多平台 URL 解析
|
||||
├── kb-bilibili-*/ # B站内容采集技能
|
||||
├── kb-zhihu-*/ # 知乎内容采集技能
|
||||
├── kb-whisper/ # Whisper 语音识别
|
||||
├── kb-whisper-diarization/ # 说话者分离
|
||||
├── kb-funasr/ # 阿里 FunASR 语音识别
|
||||
├── kb-ocr-processor/ # OCR 文字提取
|
||||
├── kb-deep-analyzer/ # AI 深度内容分析
|
||||
└── todo-manager/ # 待办管理技能
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Stack analysis: 2026-03-13*
|
||||
@@ -0,0 +1,186 @@
|
||||
# Codebase Structure
|
||||
|
||||
**Analysis Date:** 2026-03-13
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
/home/starmen1265/work/kb-skills/
|
||||
├── .claude/ # Claude Code configuration
|
||||
├── .gitea/ # Gitea CI/CD workflows
|
||||
├── .planning/ # Planning documents
|
||||
│ ├── codebase/ # Codebase analysis docs
|
||||
│ └── research/ # Research documents
|
||||
├── gsd/ # GSD (Get Shit Done) framework - Node.js
|
||||
│ ├── agents/ # Agent definitions
|
||||
│ ├── bin/ # Installation scripts
|
||||
│ ├── commands/ # Slash command definitions
|
||||
│ ├── docs/ # Documentation
|
||||
│ ├── get-shit-done/ # Core GSD implementation
|
||||
│ │ ├── bin/ # GSD tools
|
||||
│ │ ├── templates/ # Document templates
|
||||
│ │ ├── workflows/ # Workflow definitions
|
||||
│ │ └── references/ # Reference documentation
|
||||
│ ├── hooks/ # Git hooks
|
||||
│ └── scripts/ # Build/test scripts
|
||||
├── gsd-openclaw/ # GSD-OpenClaw bridge - Python
|
||||
│ ├── src/gsd_openclaw/ # Source package
|
||||
│ │ └── prompts/ # Prompt templates
|
||||
│ ├── tests/ # Test files
|
||||
│ ├── skill/ # Skill definition
|
||||
│ └── .planning/ # Phase planning docs
|
||||
├── kb-bilibili-content/ # Bilibili content download skill
|
||||
├── kb-bilibili-metadata/ # Bilibili metadata extraction skill
|
||||
├── kb-deep-analyzer/ # AI deep analysis skill
|
||||
├── kb-funasr/ # FunASR speech recognition skill
|
||||
├── kb-link-parser/ # Multi-platform URL parser skill
|
||||
├── kb-ocr-processor/ # OCR text extraction skill
|
||||
├── kb-whisper/ # Whisper speech recognition skill
|
||||
├── kb-whisper-diarization/ # Whisper + speaker diarization skill
|
||||
├── kb-workflow/ # Knowledge base workflow orchestrator
|
||||
├── kb-zhihu-content/ # Zhihu content download skill
|
||||
├── kb-zhihu-metadata/ # Zhihu metadata extraction skill
|
||||
├── todo-manager/ # Todo management skill
|
||||
└── README.md # Project overview
|
||||
```
|
||||
|
||||
## Directory Purposes
|
||||
|
||||
**kb-* Skill Directories:**
|
||||
- Purpose: Independent skill modules for content processing
|
||||
- Contains: Python implementation, SKILL.md specification, optional .env.example
|
||||
- Key files: `*.py` (implementation), `SKILL.md` (specification)
|
||||
|
||||
**gsd/ Directory:**
|
||||
- Purpose: Node.js-based GSD framework for project management
|
||||
- Contains: Commands, agents, templates, workflows
|
||||
- Key files: `package.json`, `commands/gsd/*.md`, `get-shit-done/bin/gsd-tools.cjs`
|
||||
|
||||
**gsd-openclaw/ Directory:**
|
||||
- Purpose: Python bridge between GSD and OpenClaw chat interface
|
||||
- Contains: Session management, state persistence, response formatting
|
||||
- Key files: `src/gsd_openclaw/cli.py`, `src/gsd_openclaw/tmux_session.py`
|
||||
|
||||
**gsd-openclaw/src/gsd_openclaw/ Directory:**
|
||||
- Purpose: Core Python package for GSD-OpenClaw
|
||||
- Contains: CLI, session management, workspace management, state persistence
|
||||
- Key files: `cli.py`, `tmux_session.py`, `workspace.py`, `state_manager.py`
|
||||
|
||||
**gsd-openclaw/skill/ Directory:**
|
||||
- Purpose: Skill definition for GSD-OpenClaw
|
||||
- Contains: SKILL.md that defines how the skill triggers and works
|
||||
|
||||
## Key File Locations
|
||||
|
||||
**Entry Points:**
|
||||
- `kb-workflow/kb_workflow.py`: Knowledge base workflow orchestrator CLI
|
||||
- `kb-deep-analyzer/deep_analyzer.py`: AI deep analysis CLI
|
||||
- `kb-link-parser/link-parser.py`: URL parsing CLI
|
||||
- `gsd-openclaw/src/gsd_openclaw/cli.py`: GSD-OpenClaw session management CLI
|
||||
|
||||
**Configuration:**
|
||||
- `~/.kb_env`: Environment variables for knowledge base (loaded by kb-workflow)
|
||||
- `gsd-openclaw/pyproject.toml`: Python package configuration
|
||||
- `gsd/package.json`: Node.js package configuration
|
||||
|
||||
**Core Logic:**
|
||||
- `gsd-openclaw/src/gsd_openclaw/tmux_session.py`: Tmux session lifecycle
|
||||
- `gsd-openclaw/src/gsd_openclaw/state_manager.py`: State persistence with backup
|
||||
- `gsd-openclaw/src/gsd_openclaw/workspace.py`: Project workspace management
|
||||
- `kb-deep-analyzer/ai_client.py`: Multi-provider AI client
|
||||
|
||||
**Testing:**
|
||||
- `gsd-openclaw/tests/test_*.py`: pytest test files
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
**Skill Directories:**
|
||||
- Pattern: `kb-{platform}-{function}` or `kb-{function}`
|
||||
- Examples: `kb-bilibili-metadata`, `kb-deep-analyzer`, `kb-whisper`
|
||||
|
||||
**Python Files:**
|
||||
- Pattern: `{platform}_{function}.py` or `{function}.py`
|
||||
- Examples: `bilibili_metadata.py`, `deep_analyzer.py`, `link_parser.py`
|
||||
- Note: Some use hyphens in filename (e.g., `link-parser.py`)
|
||||
|
||||
**Output Files:**
|
||||
- Pattern: `{platform}_{id}.{type}.json`
|
||||
- Examples: `bilibili_BVxxx.meta.json`, `bilibili_BVxxx.link.json`
|
||||
|
||||
**Skill Specifications:**
|
||||
- Pattern: `SKILL.md` in each skill directory
|
||||
- Format: YAML frontmatter + Markdown body
|
||||
|
||||
## Where to Add New Code
|
||||
|
||||
**New Content Platform Skill:**
|
||||
- Create directory: `kb-{platform}-{function}/`
|
||||
- Add implementation: `kb-{platform}-{function}/{platform}_{function}.py`
|
||||
- Add specification: `kb-{platform}-{function}/SKILL.md`
|
||||
- Update `kb-link-parser/link-parser.py` to recognize platform URLs
|
||||
- Update `kb-workflow/SKILL.md` to include new skill in workflow
|
||||
|
||||
**New Processing Tool Skill:**
|
||||
- Create directory: `kb-{function}/`
|
||||
- Add implementation: `kb-{function}/{function}.py`
|
||||
- Add specification: `kb-{function}/SKILL.md`
|
||||
- Integrate with `kb-workflow/SKILL.md` if part of main pipeline
|
||||
|
||||
**New GSD Command:**
|
||||
- Create command definition: `gsd/commands/gsd/{command-name}.md`
|
||||
- Add agent if needed: `gsd/agents/gsd-{agent-name}.md`
|
||||
- Update CLI in `gsd-openclaw/src/gsd_openclaw/cli.py` if needed
|
||||
|
||||
**New Session Management Feature:**
|
||||
- Primary code: `gsd-openclaw/src/gsd_openclaw/tmux_session.py`
|
||||
- Tests: `gsd-openclaw/tests/test_tmux_session.py`
|
||||
- State handling: `gsd-openclaw/src/gsd_openclaw/state_manager.py`
|
||||
|
||||
**New AI Provider:**
|
||||
- Add method to `kb-deep-analyzer/ai_client.py` `SimpleAIClient` class
|
||||
- Pattern: `_call_{provider}(self, prompt, max_tokens, api_key, model, base_url)`
|
||||
- Update `kb-deep-analyzer/deep_analyzer.py` `generate()` method
|
||||
|
||||
## Special Directories
|
||||
|
||||
**gsd-openclaw/.venv:**
|
||||
- Purpose: Python virtual environment
|
||||
- Generated: Yes (uv/pip)
|
||||
- Committed: No
|
||||
|
||||
**gsd-openclaw/.pytest_cache:**
|
||||
- Purpose: pytest cache
|
||||
- Generated: Yes
|
||||
- Committed: No
|
||||
|
||||
**gsd/.git:**
|
||||
- Purpose: Separate git repository for GSD submodule
|
||||
- Contains: Full GSD framework history
|
||||
- Committed: As submodule reference
|
||||
|
||||
**__pycache__/:**
|
||||
- Purpose: Python bytecode cache
|
||||
- Generated: Yes
|
||||
- Committed: No
|
||||
|
||||
**Workspace Directory (~/gsd-workspaces/):**
|
||||
- Purpose: Project workspaces for GSD sessions
|
||||
- Structure: `{project}/.gsd/state.json`, `{project}/code/`, `{project}/io-logs/`
|
||||
- Generated: At runtime
|
||||
- Committed: No (runtime data)
|
||||
|
||||
**Knowledge Base Directory (~/work/knowledge-base/):**
|
||||
- Purpose: Working directory for content processing
|
||||
- Structure: `pending/`, `metadata/`, `content/downloads/`, `analyzed/`
|
||||
- Generated: At runtime via `kb_workflow.py ensure_directories()`
|
||||
- Committed: No (runtime data)
|
||||
|
||||
**Archive Directory (~/work/zq/30.资源/AI分析/):**
|
||||
- Purpose: Final archive location for analysis reports
|
||||
- Structure: `YYYY-MM-DD/{platform}_{id}.md`
|
||||
- Generated: At runtime
|
||||
- Committed: Yes (via Git in archive workflow)
|
||||
|
||||
---
|
||||
|
||||
*Structure analysis: 2026-03-13*
|
||||
Reference in New Issue
Block a user