240 lines
8.7 KiB
Markdown
240 lines
8.7 KiB
Markdown
---
|
|
phase: 02-platform-skills
|
|
plan: 01
|
|
type: execute
|
|
wave: 1
|
|
depends_on: []
|
|
files_modified:
|
|
- kb_pipeline/link_parser.py
|
|
- kb_pipeline/idempotency.py
|
|
- kb_pipeline/__init__.py
|
|
- tests/test_pipeline/test_link_parser.py
|
|
- tests/test_pipeline/test_idempotency.py
|
|
autonomous: true
|
|
requirements: [PLAT-01, PLAT-02, PLAT-03]
|
|
user_setup: []
|
|
|
|
must_haves:
|
|
truths:
|
|
- "输入B站视频URL，解析出 platform=bilibili, type=video, id=BVxxx"
|
|
- "输入知乎专栏URL，解析出 platform=zhihu, type=zhuanlan, id=xxx"
|
|
- "输入b23.tv短链，自动解析到真实URL并提取ID"
|
|
- "输入无法识别的URL，返回None和错误信息（不抛异常）"
|
|
- "幂等检查能正确判断文件是否已存在"
|
|
artifacts:
|
|
- path: "kb_pipeline/link_parser.py"
|
|
provides: "统一链接解析器"
|
|
exports: ["LinkParser", "ParsedLink"]
|
|
- path: "kb_pipeline/idempotency.py"
|
|
provides: "幂等检查机制"
|
|
exports: ["IdempotencyChecker"]
|
|
- path: "tests/test_pipeline/test_link_parser.py"
|
|
provides: "LinkParser 单元测试"
|
|
min_lines: 50
|
|
- path: "tests/test_pipeline/test_idempotency.py"
|
|
provides: "幂等检查单元测试"
|
|
min_lines: 30
|
|
key_links:
|
|
- from: "kb_pipeline/link_parser.py"
|
|
to: "kb_pipeline/config.py"
|
|
via: "get_settings for base_path"
|
|
pattern: "from.*config.*import"
|
|
- from: "kb_pipeline/idempotency.py"
|
|
to: "kb_pipeline/link_parser.py"
|
|
via: "ParsedLink for content_id"
|
|
pattern: "from.*link_parser.*import"
|
|
---
|
|
|
|
<objective>
|
|
创建统一的 LinkParser 类和幂等检查机制，为后续平台适配器提供链接解析和重复处理防护能力。
|
|
|
|
Purpose: 为所有平台提供统一的 URL 解析入口，支持 B站/知乎的内容类型识别和 ID 提取，同时提供幂等检查避免重复处理。
|
|
Output: LinkParser 类、ParsedLink 数据类、IdempotencyChecker 类
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
|
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/STATE.md
|
|
@.planning/phases/02-platform-skills/02-CONTEXT.md
|
|
@.planning/phases/02-platform-skills/02-RESEARCH.md
|
|
@.planning/phases/02-platform-skills/02-VALIDATION.md
|
|
|
|
<interfaces>
|
|
<!-- Phase 1 基础设施 - LinkParser 将复用这些 -->
|
|
|
|
From kb_pipeline/schemas.py:
|
|
```python
|
|
class StepInput(BaseModel):
|
|
content_id: str = Field(..., min_length=1, description="内容唯一标识")
|
|
platform: Literal["bilibili", "zhihu"] = Field(..., description="平台名称")
|
|
|
|
class StepStatus(str, Enum):
|
|
PENDING = "pending"
|
|
COMPLETED = "completed"
|
|
FAILED = "failed"
|
|
```
|
|
|
|
From kb_pipeline/config.py:
|
|
```python
|
|
class KBSettings(BaseSettings):
|
|
base_path: str = Field(..., description="知识库根目录")
|
|
archive_path: str = Field(..., description="归档目录")
|
|
|
|
@lru_cache
|
|
def get_settings() -> KBSettings:
|
|
"""获取配置单例"""
|
|
```
|
|
|
|
From kb_pipeline/errors.py:
|
|
```python
|
|
class KBError(Exception):
|
|
def __init__(self, message, error_type, step_name, input_summary, suggestion, cause=None)
|
|
|
|
class ValidationError(KBError):
|
|
def __init__(self, step_name, input_summary, validation_errors=None, cause=None)
|
|
```
|
|
|
|
From kb_pipeline/retry.py:
|
|
```python
|
|
retry_on_network_error = retry(
|
|
stop=stop_after_attempt(3),
|
|
wait=wait_exponential(multiplier=1, min=2, max=10),
|
|
retry=retry_if_exception_type(NetworkError),
|
|
)
|
|
```
|
|
</interfaces>
|
|
|
|
<existing_code>
|
|
<!-- 现有 link-parser.py 的核心模式，需要整合到 kb_pipeline/link_parser.py -->
|
|
|
|
From kb-link-parser/link-parser.py:
|
|
```python
|
|
class LinkParser:
|
|
BILIBILI_VIDEO_PATTERN = re.compile(r'https?://(?:www\.)?bilibili\.com/video/(BV[a-zA-Z0-9]+)')
|
|
BILIBILI_ARTICLE_PATTERN = re.compile(r'https?://(?:www\.)?bilibili\.com/(?:article|read)/([a-zA-Z0-9]+)')
|
|
ZHIHAU_ANSWER_PATTERN = re.compile(r'https?://(?:www\.)?zhihu\.com/question/\d+/answer/(\d+)')
|
|
ZHIHAU_ZHUANLAN_PATTERN = re.compile(r'https?://zhuanlan\.zhihu\.com/p/(\d+)')
|
|
|
|
def resolve_shortlink(self, url: str) -> Optional[str]:
|
|
# 使用 requests.head(url, allow_redirects=True, timeout=10)
|
|
|
|
def parse(self, url: str) -> Dict[str, Any]:
|
|
# 返回 {platform, type, id, original_url, resolved_url, success, error}
|
|
```
|
|
</existing_code>
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 1: Write interface contracts and tests for LinkParser</name>
|
|
<files>kb_pipeline/link_parser.py, tests/test_pipeline/test_link_parser.py</files>
|
|
<behavior>
|
|
- Test 1: 解析B站视频URL返回正确的 ParsedLink (platform=bilibili, type=video, id=BVxxx)
|
|
- Test 2: 解析B站专栏URL返回正确的 ParsedLink (platform=bilibili, type=article, id=cvxxx)
|
|
- Test 3: 解析知乎问答URL返回正确的 ParsedLink (platform=zhihu, type=answer, id=xxx)
|
|
- Test 4: 解析知乎专栏URL返回正确的 ParsedLink (platform=zhihu, type=zhuanlan, id=xxx)
|
|
- Test 5: 解析b23.tv短链返回解析后的真实URL
|
|
- Test 6: 无法识别的URL返回None（不抛异常）
|
|
</behavior>
|
|
<action>
|
|
1. 创建 kb_pipeline/link_parser.py:
|
|
- 定义 ParsedLink dataclass，包含: platform, content_type, content_id, original_url, resolved_url
|
|
- 定义 LinkParser 类，实现 parse(url) -> Optional[ParsedLink]
|
|
- 支持 URL 类型:
|
|
* B站视频: bilibili.com/video/BVxxx
|
|
* B站专栏: bilibili.com/read/cvxxx
|
|
* 知乎问答: zhihu.com/question/xxx/answer/xxx
|
|
* 知乎专栏: zhuanlan.zhihu.com/p/xxx
|
|
- 短链支持: b23.tv 使用 requests.head(allow_redirects=True) 解析
|
|
- 错误处理: 无法识别返回 None，不抛异常
|
|
|
|
2. 创建 tests/test_pipeline/test_link_parser.py:
|
|
- 使用 pytest 和 parametrize 测试各种 URL 格式
|
|
- Mock 短链解析以避免真实网络请求
|
|
- 覆盖所有 CONTEXT.md 定义的支持 URL 类型
|
|
</action>
|
|
<verify>
|
|
<automated>pytest tests/test_pipeline/test_link_parser.py -x -v</automated>
|
|
</verify>
|
|
<done>LinkParser 能正确解析所有支持的 URL 类型，无法识别的 URL 返回 None</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: Write interface contracts and tests for IdempotencyChecker</name>
|
|
<files>kb_pipeline/idempotency.py, tests/test_pipeline/test_idempotency.py</files>
|
|
<behavior>
|
|
- Test 1: 文件存在时 check_exists 返回 (True, [file_list])
|
|
- Test 2: 文件不存在时 check_exists 返回 (False, [])
|
|
- Test 3: 部分文件存在时返回正确的存在文件列表
|
|
</behavior>
|
|
<action>
|
|
1. 创建 kb_pipeline/idempotency.py:
|
|
- 定义 IdempotencyChecker 类
|
|
- 实现 check_exists(output_dir: Path, content_id: str, platform: str) -> Tuple[bool, list[Path]]
|
|
- 检查文件:
|
|
* 元数据: {platform}_{content_id}_metadata.json
|
|
* 内容: {platform}_{content_id}_content.md
|
|
- 使用 pathlib.Path 进行路径操作
|
|
|
|
2. 创建 tests/test_pipeline/test_idempotency.py:
|
|
- 使用 tmp_path fixture 创建临时文件
|
|
- 测试文件存在/不存在的各种场景
|
|
- 测试部分文件存在的情况
|
|
</action>
|
|
<verify>
|
|
<automated>pytest tests/test_pipeline/test_idempotency.py -x -v</automated>
|
|
</verify>
|
|
<done>IdempotencyChecker 能正确检查文件是否存在，返回准确的文件列表</done>
|
|
</task>
|
|
|
|
<task type="auto">
|
|
<name>Task 3: Update package exports</name>
|
|
<files>kb_pipeline/__init__.py</files>
|
|
<action>
|
|
更新 kb_pipeline/__init__.py，导出新模块:
|
|
```python
|
|
from .link_parser import LinkParser, ParsedLink
|
|
from .idempotency import IdempotencyChecker
|
|
|
|
__all__ = [
|
|
# ... existing exports ...
|
|
"LinkParser",
|
|
"ParsedLink",
|
|
"IdempotencyChecker",
|
|
]
|
|
```
|
|
</action>
|
|
<verify>
|
|
<automated>python -c "from kb_pipeline import LinkParser, ParsedLink, IdempotencyChecker; print('OK')"</automated>
|
|
</verify>
|
|
<done>LinkParser, ParsedLink, IdempotencyChecker 可从 kb_pipeline 包根导入</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<verification>
|
|
- 所有测试通过: `pytest tests/test_pipeline/test_link_parser.py tests/test_pipeline/test_idempotency.py -v`
|
|
- 导入测试: `python -c "from kb_pipeline import LinkParser, IdempotencyChecker"`
|
|
- 覆盖率: `pytest tests/test_pipeline/ --cov=kb_pipeline --cov-report=term-missing`
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- [ ] LinkParser 正确解析 B站视频/专栏、知乎问答/专栏 URL
|
|
- [ ] b23.tv 短链自动解析到真实 URL
|
|
- [ ] 无法识别的 URL 返回 None（不抛异常）
|
|
- [ ] IdempotencyChecker 正确检查文件存在性
|
|
- [ ] 所有测试通过（至少 10 个测试用例）
|
|
- [ ] 公共 API 可从 kb_pipeline 包根导入
|
|
</success_criteria>
|
|
|
|
<output>
|
|
After completion, create `.planning/phases/02-platform-skills/02-01-SUMMARY.md`
|
|
</output>
|