diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 0000000..5f57187 --- /dev/null +++ b/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,154 @@ +# Architecture + +**Analysis Date:** 2026-03-13 + +## Pattern Overview + +**Overall:** Multi-skill workflow orchestration with skill-based modular architecture + +**Key Characteristics:** +- Skill-based modular design: Each skill is an independent module with its own `SKILL.md` specification +- Pipeline workflow pattern: Sequential processing stages orchestrated by `kb-workflow` +- Stateful session management: tmux-based persistent sessions for GSD-OpenClaw +- Platform abstraction: Unified interfaces for multiple content platforms (Bilibili, Zhihu, YouTube, etc.) + +## Layers + +**Skill Definition Layer:** +- Purpose: Define skill capabilities, triggers, and execution steps +- Location: `kb-*/SKILL.md`, `gsd-openclaw/skill/SKILL.md` +- Contains: Markdown documents with YAML frontmatter and structured workflows +- Depends on: Nothing (declarative) +- Used by: AI agents that execute skills + +**Core Implementation Layer:** +- Purpose: Implement actual functionality for each skill +- Location: `kb-*/*.py`, `gsd-openclaw/src/gsd_openclaw/*.py` +- Contains: Python modules with CLI entry points and core logic +- Depends on: External APIs, platform SDKs, AI providers +- Used by: Skill orchestration layer + +**Orchestration Layer:** +- Purpose: Coordinate multiple skills in workflows +- Location: `kb-workflow/kb_workflow.py` +- Contains: Workflow coordination utilities, directory management, Git integration +- Depends on: All skill modules +- Used by: Main agent executing workflows + +**Session Management Layer:** +- Purpose: Manage persistent tmux sessions for long-running operations +- Location: `gsd-openclaw/src/gsd_openclaw/tmux_session.py` +- Contains: TmuxSessionManager, session lifecycle management +- Depends on: libtmux, StateManager, WorkspaceManager +- Used by: GSD command orchestration + +**State Persistence Layer:** +- Purpose: Provide crash-safe state checkpointing +- Location: `gsd-openclaw/src/gsd_openclaw/state_manager.py` +- Contains: StateManager, atomic save/load with backup rotation +- Depends on: File system only +- Used by: TmuxSessionManager, workspace management + +## Data Flow + +**Content Processing Pipeline:** + +1. URL Input → `kb-link-parser` extracts platform, type, and ID +2. Metadata Extraction → Platform-specific skill (`kb-*-metadata`) fetches metadata +3. Content Download → Platform-specific skill (`kb-*-content`) downloads content +4. Speech Recognition → `kb-whisper`/`kb-funasr` transcribes audio (if needed) +5. Deep Analysis → `kb-deep-analyzer` generates AI-powered analysis report +6. Archive → `kb-workflow` archives to Git-managed directory + +**GSD Session Flow:** + +1. `/gsd:*` command received +2. `TmuxSessionManager.create_session()` or `resume_session()` +3. State persisted via `StateManager.save()` +4. Command executed in tmux session +5. Output captured and formatted via `FeishuResponseFormatter` + +**State Management:** +- State stored in `~/gsd-workspaces/{project}/.gsd/state.json` +- Atomic writes via temp file + rename +- Automatic backup rotation (state.json → state.json.bak) + +## Key Abstractions + +**Skill:** +- Purpose: Represents an independent capability unit +- Examples: `kb-link-parser/link-parser.py`, `kb-deep-analyzer/deep_analyzer.py` +- Pattern: CLI-based with JSON output, defined by `SKILL.md` specification + +**TmuxSession:** +- Purpose: Represents a managed tmux session for project isolation +- Examples: `gsd-openclaw/src/gsd_openclaw/tmux_session.py` +- Pattern: Dataclass with session_id, project_name, tmux_session_name + +**Workspace:** +- Purpose: Isolated directory structure per project +- Examples: `gsd-openclaw/src/gsd_openclaw/workspace.py` +- Pattern: Defined subdirectories (.gsd/, code/, io-logs/) + +**AnalysisResult:** +- Purpose: Structured output from deep analysis +- Examples: `kb-deep-analyzer/deep_analyzer.py` +- Pattern: Dataclass with summary, main_topics, key_insights, tags + +## Entry Points + +**CLI Entry Points:** +- Location: `kb-*/*.py` with `if __name__ == '__main__': main()` +- Triggers: Command-line invocation with argparse +- Responsibilities: Parse arguments, invoke core logic, output JSON + +**gsd-ocl CLI:** +- Location: `gsd-openclaw/src/gsd_openclaw/cli.py` +- Triggers: `gsd-ocl create|list|destroy|resume|cleanup` +- Responsibilities: Session lifecycle management + +**Skill Trigger:** +- Location: `SKILL.md` frontmatter with `name` and `description` +- Triggers: AI agent pattern matching on user messages +- Responsibilities: Define when and how to invoke the skill + +## Error Handling + +**Strategy:** Fail-fast with structured JSON error output + +**Patterns:** +- All CLI tools output JSON with `success: true/false` field +- Error responses include `error` and `detail` fields +- Deep analysis failure terminates entire workflow (per `kb-workflow/SKILL.md`) +- State manager uses backup fallback on corruption + +**Error Response Format:** +```json +{ + "success": false, + "error": "Error description", + "detail": "Additional context" +} +``` + +## Cross-Cutting Concerns + +**Logging:** Console output with `[ModuleName]` prefixes (e.g., `[DeepAnalyzer]`, `[AI Client]`) + +**Validation:** +- Environment variable checks before workflow execution (`kb-workflow/SKILL.md`) +- Required field validation in `StateManager.load()` +- Platform/type validation in link parser + +**Authentication:** +- Platform cookies via environment variables (`BILIBILI_COOKIE`, `ZHIHU_COOKIE`) +- AI API keys via environment variables (`AI_API_KEY`, `HF_TOKEN`) +- Git credentials via environment variables (`KB_GIT_USER`, `KB_GIT_EMAIL`) + +**Configuration:** +- Environment variables loaded from `~/.kb_env` +- Path configuration via environment variables (`KB_BASE_PATH`, `KB_ARCHIVE_PATH`, `KB_GIT_ROOT`) + +--- + +*Architecture analysis: 2026-03-13* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md new file mode 100644 index 0000000..82bc8ab --- /dev/null +++ b/.planning/codebase/CONCERNS.md @@ -0,0 +1,180 @@ +# Codebase Concerns + +**Analysis Date:** 2026-03-13 + +## Tech Debt + +### 异常处理不完善 +- Issue: 多处使用 bare `except:` 或 `except Exception` 吞掉所有异常,没有记录错误信息 +- Files: + - `kb-bilibili-metadata/bilibili_extractor.py` (第 310、340 行) + - `kb-deep-analyzer/ai_client.py` (第 228、489 行) + - `kb-deep-analyzer/deep_analyzer.py` (第 394、410、506、516 行) + - `kb-whisper-diarization/whisper_diarization.py` (第 70、326 行) + - `kb-funasr/funasr_tool.py` (第 81、186 行) +- Impact: 错误被静默忽略,调试困难,生产问题难以追踪 +- Fix approach: 使用具体异常类型,添加日志记录 + +### 过多的 return None 模式 +- Issue: 多个函数在异常或边界情况返回 `None`,调用方需要额外检查 +- Files: + - `kb-bilibili-metadata/bilibili_extractor.py` (第 140、161、176 行) + - `kb-bilibili-content/video_downloader.py` (第 36、48、68、75、90、106、148 行) + - `kb-link-parser/link-parser.py` (第 197、245、379、436 行) + - `kb-link-parser/link_parser_shortlink.py` (第 223、237、249 行) + - `kb-zhihu-metadata/zhihu_extractor.py` (第 137、152、410、437、450 行) +- Impact: 调用方需要频繁检查 None,代码冗长,容易遗漏 +- Fix approach: 使用 Result 模式或抛出具体异常 + +### 硬编码的默认路径 +- Issue: 默认知识库路径硬编码在代码中 +- Files: `kb-workflow/kb_workflow.py` (第 104、113、122 行) +- Impact: 不同用户或环境需要修改代码或设置环境变量 +- Fix approach: 使用配置文件或更灵活的路径发现机制 + +### AI 客户端重复实现 +- Issue: 存在两套 AI 客户端实现 (`ai_client.py` 和 `deep_analyzer.py` 中的 `SimpleAIClient`) +- Files: + - `kb-deep-analyzer/ai_client.py` (514 行,功能完整) + - `kb-deep-analyzer/deep_analyzer.py` (37-123 行,简化版本) +- Impact: 代码重复,维护困难,行为可能不一致 +- Fix approach: 统一使用一个 AI 客户端实现 + +## Known Bugs + +### CI/CD 部署列表与实际目录不匹配 +- Symptoms: `.gitea/workflows/deploy.yml` 中的技能列表包含 `dev-workflow`,但该目录已被删除(根据 git log) +- Files: `.gitea/workflows/deploy.yml` (第 23 行) +- Trigger: 推送到 main 分支时,CI 会尝试同步不存在的目录 +- Workaround: 目录不存在时 rsync 会跳过,但会产生混乱的日志 + +### Playwright 依赖检测不完整 +- Symptoms: 多个模块检测 Playwright 可用性但不处理所有边界情况 +- Files: `kb-bilibili-metadata/bilibili_extractor.py` (第 28-44 行) +- Trigger: 当 Playwright 不可用时回退到 API,但 httpx 也可能不可用 +- Workaround: 检查 HTTPX_AVAILABLE 并返回错误信息 + +## Security Considerations + +### 环境变量中传递敏感信息 +- Risk: API Keys 和 Cookies 通过环境变量传递,可能在日志或错误信息中泄露 +- Files: + - `kb-deep-analyzer/deep_analyzer.py` (第 49-52 行直接读取环境变量) + - `kb-deep-analyzer/ai_client.py` (第 32、44 行) + - `kb-zhihu-metadata/zhihu_extractor.py` (Cookie 处理) +- Current mitigation: 使用 `.env.example` 文档说明,但无运行时保护 +- Recommendations: + - 敏感信息不应出现在日志输出中 + - 考虑使用 secrets 管理工具 + +### Git 凭据在代码中配置 +- Risk: `kb_workflow.py` 中 Git 配置通过环境变量传递用户名和邮箱 +- Files: `kb-workflow/kb_workflow.py` (第 196-200 行) +- Current mitigation: 环境变量方式 +- Recommendations: 考虑使用 Git credential helper + +## Performance Bottlenecks + +### 固定的 HTTP 超时 +- Problem: 多处 HTTP 请求使用固定的 30 秒超时 +- Files: + - `kb-deep-analyzer/ai_client.py` (第 135、169、191、213 行) + - `kb-bilibili-metadata/bilibili_extractor.py` (第 262 行) +- Cause: 大文件下载或慢速 API 可能超时 +- Improvement path: 根据请求类型使用可配置的超时 + +### AI 调用长超时 +- Problem: AI API 调用使用 600 秒(10 分钟)超时 +- Files: `kb-deep-analyzer/deep_analyzer.py` (第 87、97、116 行) +- Cause: 长文本分析需要时间,但 10 分钟过长 +- Improvement path: 根据内容长度动态调整超时 + +### Playwright 浏览器启动开销 +- Problem: 每次提取都启动新的浏览器实例 +- Files: + - `kb-bilibili-metadata/bilibili_extractor.py` + - `kb-zhihu-metadata/zhihu_extractor.py` +- Cause: 无浏览器实例复用 +- Improvement path: 考虑浏览器实例池或持久化 + +## Fragile Areas + +### 网页选择器依赖 +- Files: + - `kb-zhihu-metadata/zhihu_extractor.py` (第 183-327 行) + - `kb-bilibili-metadata/bilibili_extractor.py` +- Why fragile: 网站更新会改变 CSS 类名和 DOM 结构 +- Safe modification: 添加选择器配置化,使用多个备用选择器 +- Test coverage: 缺少针对选择器变更的测试 + +### 元数据格式多样性 +- Files: `kb-deep-analyzer/deep_analyzer.py` (第 382-414 行 `_normalize_metadata`) +- Why fragile: 不同来源的元数据格式不一致(扁平、嵌套、混合) +- Safe modification: 使用数据验证库(如 Pydantic)标准化格式 +- Test coverage: 部分覆盖 + +### Tmux 会话状态管理 +- Files: `gsd-openclaw/src/gsd_openclaw/tmux_session.py` +- Why fragile: tmux 会话可能被外部操作破坏,状态文件可能不同步 +- Safe modification: 添加状态校验和恢复机制 +- Test coverage: 有测试 (`test_tmux_session.py`),但边缘情况可能不足 + +## Scaling Limits + +### 单实例设计 +- Current capacity: 单用户、单进程 +- Limit: 无法并行处理多个分析任务 +- Scaling path: 引入任务队列和 worker 进程 + +### 内存中的用户上下文缓存 +- Current capacity: 无限(但无清理机制) +- Limit: 长时间运行后内存泄漏 +- Files: `gsd-openclaw/src/gsd_openclaw/orchestrator.py` (第 70 行 `user_context_managers` 字典) +- Scaling path: 添加 LRU 缓存或定期清理 + +## Dependencies at Risk + +### 外部 API 依赖 +- Risk: 依赖 Bilibili、知乎等第三方 API,可能随时变更 +- Impact: 抓取功能失效 +- Migration plan: 使用官方 SDK(如果有),或添加 API 版本检测 + +### Playwright 和 undetected-playwright +- Risk: `undetected_playwright` 是小众库,维护可能不活跃 +- Impact: 反爬虫规避功能失效 +- Files: `kb-bilibili-metadata/bilibili_extractor.py` (第 35-38 行) +- Migration plan: 监控库更新,准备降级方案 + +## Missing Critical Features + +### 缺少统一日志 +- Problem: 各模块使用 `print()` 输出日志,无统一日志级别控制 +- Blocks: 生产环境日志管理、问题排查 + +### 缺少健康检查端点 +- Problem: 无服务健康状态监控 +- Blocks: 运维监控、自动故障恢复 + +## Test Coverage Gaps + +### kb-* 技能缺少单元测试 +- What's not tested: 所有 kb-* 技能目录(kb-bilibili-content, kb-zhihu-metadata 等)都没有测试文件 +- Files: 所有 `kb-*/**/*.py` 文件 +- Risk: 重构或修改可能引入回归 bug +- Priority: High - 核心功能缺少测试保护 + +### 集成测试缺失 +- What's not tested: 技能之间的集成(如 kb-workflow 调用其他技能) +- Files: 跨技能调用路径 +- Risk: 接口变更导致集成失败 +- Priority: Medium + +### 边缘情况测试不足 +- What's not tested: 网络错误、超时、无效输入等 +- Files: `gsd-openclaw/tests/` 中的测试主要覆盖正常路径 +- Risk: 生产环境异常处理不当 +- Priority: Medium + +--- + +*Concerns audit: 2026-03-13* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md new file mode 100644 index 0000000..be515f8 --- /dev/null +++ b/.planning/codebase/INTEGRATIONS.md @@ -0,0 +1,162 @@ +# External Integrations + +**Analysis Date:** 2026-03-13 + +## APIs & External Services + +**AI Providers:** +- **智谱 AI (Zhipu)** - 主要 AI 分析服务 + - SDK: `requests` HTTP 客户端 + - Auth: `AI_API_KEY` 环境变量 + - Base URL: `https://open.bigmodel.cn/api/paas/v4` + - 模型: `glm-5`, `glm-4` + +- **MiniMax** - AI 备选服务 + - SDK: `requests` HTTP 客户端 + - Auth: `AI_API_KEY` 环境变量 + - Base URL: `https://api.minimaxi.com/anthropic/v1/messages` + +- **OpenAI** - AI 备选服务 + - SDK: `requests` HTTP 客户端 + - Auth: `AI_API_KEY` 环境变量 + +- **阿里云通义千问 (Alibaba/Qwen)** - AI 备选服务 + - SDK: `requests` HTTP 客户端 + - Auth: `AI_API_KEY` 环境变量 + +**视频平台 API:** +- **Bilibili** - 视频/专栏元数据提取 + - SDK: `httpx`, `playwright` + - Auth: `BILIBILI_COOKIE` 环境变量 (SESSDATA, bili_jct) + - 文件: `kb-bilibili-metadata/bilibili_extractor.py` + +- **知乎** - 问答/专栏/视频内容 + - SDK: `yt-dlp`, `playwright` + - Auth: `ZHIHU_COOKIE` 环境变量 + - 文件: `kb-zhihu-metadata/zhihu_metadata.py`, `kb-zhihu-content/zhihu_content.py` + +- **YouTube** - 视频内容(通过链接解析) + - SDK: `yt-dlp` + - 文件: `kb-link-parser/link-parser.py` + +- **小红书** - 笔记内容 + - 短链接: `xhslink.com` + - 文件: `kb-link-parser/link-parser.py` + +## Data Storage + +**Databases:** +- 无数据库 - 所有数据以文件形式存储 + +**File Storage:** +- 本地文件系统 + - 工作目录: `~/work/knowledge-base/` + - 结构: + - `pending/` - 待处理链接 + - `metadata/` - 元数据 JSON + - `content/downloads/` - 下载的媒体文件 + - `analyzed/` - AI 分析报告 + +**归档存储:** +- 路径: `~/work/zq/30.资源/AI分析/` +- 按日期组织: `YYYY-MM-DD/` + +**Caching:** +- 模型缓存: `~/.cache/modelscope/hub/` - FunASR 模型 + +## Authentication & Identity + +**Auth Provider:** +- Cookie-based 认证 + - Bilibili: `BILIBILI_COOKIE` + - 知乎: `ZHIHU_COOKIE` + +**AI API 认证:** +- API Key 认证 + - `AI_API_KEY` - AI 服务密钥 + - `HF_TOKEN` - HuggingFace Token (说话者分离) + +## Monitoring & Observability + +**Error Tracking:** +- 无外部监控服务 + +**Logs:** +- 控制台输出 - 所有技能通过 `print()` 输出日志 +- 建议使用 `python3 -u` 禁用缓冲以实时查看输出 + +## CI/CD & Deployment + +**Hosting:** +- Self-hosted (本地服务器) +- 技能部署位置: `/home/starmen1265/.openclaw/skills/` + +**CI Pipeline:** +- Gitea Actions (self-hosted runner) + - 配置文件: `.gitea/workflows/deploy.yml` + - 触发: push to main + - 使用 `rsync` 同步技能目录 + +## Environment Configuration + +**必须的环境变量:** +```bash +AI_API_KEY="your-api-key" # AI 服务密钥 +AI_PROVIDER="zhipu" # 或 openai, minimax, alibaba +AI_MODEL="glm-5" # 模型名称 +AI_BASE_URL="https://..." # API 地址 +``` + +**可选的环境变量:** +```bash +BILIBILI_COOKIE="SESSDATA=xxx; bili_jct=xxx" # B站登录态 +ZHIHU_COOKIE="..." # 知乎登录态 +HF_TOKEN="hf_xxx" # HuggingFace Token +KB_GIT_USER="Your Name" # Git 提交用户名 +KB_GIT_EMAIL="your@email.com" # Git 提交邮箱 +KB_ARCHIVE_PATH="~/work/zq/30.资源/AI分析" # 归档目录 +KB_GIT_ROOT="~/work/zq" # Git 仓库根目录 +KB_BASE_PATH="~/work/knowledge-base" # 知识库基础路径 +WHISPER_CPP_PATH="~/whisper.cpp" # Whisper.cpp 路径 +``` + +**配置文件位置:** +- `~/.kb_env` - 知识库专用配置(推荐) +- `kb-workflow/.env.example` - 示例配置 + +**Secrets location:** +- 环境变量(明文存储在 `~/.kb_env`) +- 无加密/secrets 管理 + +## Webhooks & Callbacks + +**Incoming:** +- 无 Webhook 端点 + +**Outgoing:** +- 无主动 Webhook 调用 + +## External Tools + +**音视频处理:** +- `ffmpeg` / `ffprobe` - 音视频格式转换和元数据提取 +- `yt-dlp` - 视频下载(知乎视频) + +**语音识别:** +- `whisper.cpp` - 本地 Whisper 推理 +- `funasr` / `modelscope` - 阿里 FunASR + +**浏览器自动化:** +- `playwright` - 无头浏览器 +- `undetected_playwright` - 反检测浏览器 + +**OCR:** +- `tesseract` - OCR 引擎 +- `easyocr` - 深度学习 OCR + +**会话管理:** +- `tmux` - 终端会话管理(gsd-openclaw) + +--- + +*Integration audit: 2026-03-13* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md new file mode 100644 index 0000000..71895e4 --- /dev/null +++ b/.planning/codebase/STACK.md @@ -0,0 +1,108 @@ +# Technology Stack + +**Analysis Date:** 2026-03-13 + +## Languages + +**Primary:** +- Python 3.12 - 主要用于 kb-* 技能模块(内容采集、语音识别、AI分析等) +- JavaScript/Node.js >=16.7.0 - 用于 gsd (Get-Shit-Done) 元提示系统 + +**Secondary:** +- Shell/Bash - 用于部署脚本和工具链(ffmpeg, yt-dlp) + +## Runtime + +**Environment:** +- Python 3.12.3 +- Node.js v22.22.0 + +**Package Manager:** +- pip/uv (Python) - gsd-openclaw 使用 uv.lock +- npm (Node.js) - gsd 使用 npm + +**Lockfile:** +- `uv.lock` (gsd-openclaw) +- `package-lock.json` (gsd) + +## Frameworks + +**Core:** +- 无传统 Web 框架 - 这是一个 CLI 技能集合,非 Web 应用 + +**Testing:** +- pytest >=7.0.0 - Python 单元测试 (gsd-openclaw) +- Node.js 内置测试 - gsd 使用自定义 `scripts/run-tests.cjs` + +**Build/Dev:** +- esbuild ^0.24.0 - gsd hooks 构建 +- c8 ^11.0.0 - 测试覆盖率工具 + +## Key Dependencies + +**Python 核心依赖:** +- `requests` - HTTP 客户端,用于 AI API 调用 +- `httpx` - 异步 HTTP 客户端(Bilibili 元数据提取) +- `playwright` / `undetected_playwright` - 浏览器自动化(反爬虫) +- `easyocr>=1.7.0` - OCR 文字识别 +- `pytesseract>=0.3.10` - Tesseract OCR 接口 +- `Pillow>=9.0.0` - 图像处理 +- `torch>=2.0.0` - PyTorch 深度学习框架 +- `pyannote.audio>=1.0.0` - 说话者分离 +- `funasr` / `modelscope` - 阿里 FunASR 语音识别 +- `libtmux>=0.55.0` - tmux 会话管理 (gsd-openclaw) + +**Node.js 核心依赖:** +- 无外部运行时依赖 - gsd 是纯 CommonJS 工具集 + +## Configuration + +**环境变量配置:** +- 配置文件: `~/.kb_env` - 知识库专用环境变量 +- `.env.example` - 各技能目录下提供示例配置 + +**关键配置文件:** +- `pyproject.toml` - gsd-openclaw 项目配置 +- `requirements.txt` - Python 技能依赖声明 +- `package.json` - gsd Node.js 项目配置 + +**构建配置:** +- `scripts/build-hooks.js` - gsd hooks 构建 +- `tool.setuptools` - gsd-openclaw 打包配置 + +## Platform Requirements + +**Development:** +- ffmpeg - 音视频格式转换 +- yt-dlp - 视频下载工具 +- whisper.cpp - 本地语音识别(或 GPU 版 PyTorch) +- tesseract - OCR 引擎(可选) +- tmux - 会话管理 + +**Production:** +- 自托管 Gitea Actions runner +- 部署目标: `~/.openclaw/skills/` 目录 +- rsync 同步部署 + +## Project Structure + +**Monorepo 结构:** +``` +kb-skills/ +├── gsd/ # Get-Shit-Done 元提示系统 (Node.js) +├── gsd-openclaw/ # GSD-OpenClaw 桥接技能 (Python) +├── kb-workflow/ # 知识库工作流编排入口 +├── kb-link-parser/ # 多平台 URL 解析 +├── kb-bilibili-*/ # B站内容采集技能 +├── kb-zhihu-*/ # 知乎内容采集技能 +├── kb-whisper/ # Whisper 语音识别 +├── kb-whisper-diarization/ # 说话者分离 +├── kb-funasr/ # 阿里 FunASR 语音识别 +├── kb-ocr-processor/ # OCR 文字提取 +├── kb-deep-analyzer/ # AI 深度内容分析 +└── todo-manager/ # 待办管理技能 +``` + +--- + +*Stack analysis: 2026-03-13* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md new file mode 100644 index 0000000..34a68a7 --- /dev/null +++ b/.planning/codebase/STRUCTURE.md @@ -0,0 +1,186 @@ +# Codebase Structure + +**Analysis Date:** 2026-03-13 + +## Directory Layout + +``` +/home/starmen1265/work/kb-skills/ +├── .claude/ # Claude Code configuration +├── .gitea/ # Gitea CI/CD workflows +├── .planning/ # Planning documents +│ ├── codebase/ # Codebase analysis docs +│ └── research/ # Research documents +├── gsd/ # GSD (Get Shit Done) framework - Node.js +│ ├── agents/ # Agent definitions +│ ├── bin/ # Installation scripts +│ ├── commands/ # Slash command definitions +│ ├── docs/ # Documentation +│ ├── get-shit-done/ # Core GSD implementation +│ │ ├── bin/ # GSD tools +│ │ ├── templates/ # Document templates +│ │ ├── workflows/ # Workflow definitions +│ │ └── references/ # Reference documentation +│ ├── hooks/ # Git hooks +│ └── scripts/ # Build/test scripts +├── gsd-openclaw/ # GSD-OpenClaw bridge - Python +│ ├── src/gsd_openclaw/ # Source package +│ │ └── prompts/ # Prompt templates +│ ├── tests/ # Test files +│ ├── skill/ # Skill definition +│ └── .planning/ # Phase planning docs +├── kb-bilibili-content/ # Bilibili content download skill +├── kb-bilibili-metadata/ # Bilibili metadata extraction skill +├── kb-deep-analyzer/ # AI deep analysis skill +├── kb-funasr/ # FunASR speech recognition skill +├── kb-link-parser/ # Multi-platform URL parser skill +├── kb-ocr-processor/ # OCR text extraction skill +├── kb-whisper/ # Whisper speech recognition skill +├── kb-whisper-diarization/ # Whisper + speaker diarization skill +├── kb-workflow/ # Knowledge base workflow orchestrator +├── kb-zhihu-content/ # Zhihu content download skill +├── kb-zhihu-metadata/ # Zhihu metadata extraction skill +├── todo-manager/ # Todo management skill +└── README.md # Project overview +``` + +## Directory Purposes + +**kb-* Skill Directories:** +- Purpose: Independent skill modules for content processing +- Contains: Python implementation, SKILL.md specification, optional .env.example +- Key files: `*.py` (implementation), `SKILL.md` (specification) + +**gsd/ Directory:** +- Purpose: Node.js-based GSD framework for project management +- Contains: Commands, agents, templates, workflows +- Key files: `package.json`, `commands/gsd/*.md`, `get-shit-done/bin/gsd-tools.cjs` + +**gsd-openclaw/ Directory:** +- Purpose: Python bridge between GSD and OpenClaw chat interface +- Contains: Session management, state persistence, response formatting +- Key files: `src/gsd_openclaw/cli.py`, `src/gsd_openclaw/tmux_session.py` + +**gsd-openclaw/src/gsd_openclaw/ Directory:** +- Purpose: Core Python package for GSD-OpenClaw +- Contains: CLI, session management, workspace management, state persistence +- Key files: `cli.py`, `tmux_session.py`, `workspace.py`, `state_manager.py` + +**gsd-openclaw/skill/ Directory:** +- Purpose: Skill definition for GSD-OpenClaw +- Contains: SKILL.md that defines how the skill triggers and works + +## Key File Locations + +**Entry Points:** +- `kb-workflow/kb_workflow.py`: Knowledge base workflow orchestrator CLI +- `kb-deep-analyzer/deep_analyzer.py`: AI deep analysis CLI +- `kb-link-parser/link-parser.py`: URL parsing CLI +- `gsd-openclaw/src/gsd_openclaw/cli.py`: GSD-OpenClaw session management CLI + +**Configuration:** +- `~/.kb_env`: Environment variables for knowledge base (loaded by kb-workflow) +- `gsd-openclaw/pyproject.toml`: Python package configuration +- `gsd/package.json`: Node.js package configuration + +**Core Logic:** +- `gsd-openclaw/src/gsd_openclaw/tmux_session.py`: Tmux session lifecycle +- `gsd-openclaw/src/gsd_openclaw/state_manager.py`: State persistence with backup +- `gsd-openclaw/src/gsd_openclaw/workspace.py`: Project workspace management +- `kb-deep-analyzer/ai_client.py`: Multi-provider AI client + +**Testing:** +- `gsd-openclaw/tests/test_*.py`: pytest test files + +## Naming Conventions + +**Skill Directories:** +- Pattern: `kb-{platform}-{function}` or `kb-{function}` +- Examples: `kb-bilibili-metadata`, `kb-deep-analyzer`, `kb-whisper` + +**Python Files:** +- Pattern: `{platform}_{function}.py` or `{function}.py` +- Examples: `bilibili_metadata.py`, `deep_analyzer.py`, `link_parser.py` +- Note: Some use hyphens in filename (e.g., `link-parser.py`) + +**Output Files:** +- Pattern: `{platform}_{id}.{type}.json` +- Examples: `bilibili_BVxxx.meta.json`, `bilibili_BVxxx.link.json` + +**Skill Specifications:** +- Pattern: `SKILL.md` in each skill directory +- Format: YAML frontmatter + Markdown body + +## Where to Add New Code + +**New Content Platform Skill:** +- Create directory: `kb-{platform}-{function}/` +- Add implementation: `kb-{platform}-{function}/{platform}_{function}.py` +- Add specification: `kb-{platform}-{function}/SKILL.md` +- Update `kb-link-parser/link-parser.py` to recognize platform URLs +- Update `kb-workflow/SKILL.md` to include new skill in workflow + +**New Processing Tool Skill:** +- Create directory: `kb-{function}/` +- Add implementation: `kb-{function}/{function}.py` +- Add specification: `kb-{function}/SKILL.md` +- Integrate with `kb-workflow/SKILL.md` if part of main pipeline + +**New GSD Command:** +- Create command definition: `gsd/commands/gsd/{command-name}.md` +- Add agent if needed: `gsd/agents/gsd-{agent-name}.md` +- Update CLI in `gsd-openclaw/src/gsd_openclaw/cli.py` if needed + +**New Session Management Feature:** +- Primary code: `gsd-openclaw/src/gsd_openclaw/tmux_session.py` +- Tests: `gsd-openclaw/tests/test_tmux_session.py` +- State handling: `gsd-openclaw/src/gsd_openclaw/state_manager.py` + +**New AI Provider:** +- Add method to `kb-deep-analyzer/ai_client.py` `SimpleAIClient` class +- Pattern: `_call_{provider}(self, prompt, max_tokens, api_key, model, base_url)` +- Update `kb-deep-analyzer/deep_analyzer.py` `generate()` method + +## Special Directories + +**gsd-openclaw/.venv:** +- Purpose: Python virtual environment +- Generated: Yes (uv/pip) +- Committed: No + +**gsd-openclaw/.pytest_cache:** +- Purpose: pytest cache +- Generated: Yes +- Committed: No + +**gsd/.git:** +- Purpose: Separate git repository for GSD submodule +- Contains: Full GSD framework history +- Committed: As submodule reference + +**__pycache__/:** +- Purpose: Python bytecode cache +- Generated: Yes +- Committed: No + +**Workspace Directory (~/gsd-workspaces/):** +- Purpose: Project workspaces for GSD sessions +- Structure: `{project}/.gsd/state.json`, `{project}/code/`, `{project}/io-logs/` +- Generated: At runtime +- Committed: No (runtime data) + +**Knowledge Base Directory (~/work/knowledge-base/):** +- Purpose: Working directory for content processing +- Structure: `pending/`, `metadata/`, `content/downloads/`, `analyzed/` +- Generated: At runtime via `kb_workflow.py ensure_directories()` +- Committed: No (runtime data) + +**Archive Directory (~/work/zq/30.资源/AI分析/):** +- Purpose: Final archive location for analysis reports +- Structure: `YYYY-MM-DD/{platform}_{id}.md` +- Generated: At runtime +- Committed: Yes (via Git in archive workflow) + +--- + +*Structure analysis: 2026-03-13*