195 lines
6.9 KiB
Markdown
195 lines
6.9 KiB
Markdown
---
|
||
phase: 10-proxy-pool
|
||
plan: 01
|
||
type: execute
|
||
wave: 1
|
||
depends_on: []
|
||
files_modified:
|
||
- kb_workflow/crawler/proxy_pool.py
|
||
- tests/test_crawler/test_proxy_pool.py
|
||
- tests/test_crawler/conftest.py
|
||
autonomous: true
|
||
requirements:
|
||
- PROXY-01
|
||
- PROXY-02
|
||
user_setup: []
|
||
|
||
must_haves:
|
||
truths:
|
||
- "用户可以从 YAML 文件加载多个代理地址"
|
||
- "用户可以从环境变量加载单个代理（向后兼容）"
|
||
- "每次请求自动选择下一个可用代理（Round-Robin）"
|
||
artifacts:
|
||
- path: "kb_workflow/crawler/proxy_pool.py"
|
||
provides: "ProxyPool, ProxyEntry, ProxyPoolConfig 类"
|
||
min_lines: 80
|
||
- path: "tests/test_crawler/test_proxy_pool.py"
|
||
provides: "代理池单元测试"
|
||
contains: "test_load_from_yaml, test_load_from_env, test_round_robin"
|
||
key_links:
|
||
- from: "ProxyPool.from_yaml()"
|
||
to: "~/.kb_proxies.yaml"
|
||
via: "PyYAML safe_load"
|
||
pattern: "yaml\\.safe_load"
|
||
- from: "ProxyPool.get_next_proxy()"
|
||
to: "ProxyEntry 列表"
|
||
via: "索引指针递增"
|
||
pattern: "_current_index.*\\+.*%.*len"
|
||
---
|
||
|
||
<objective>
|
||
实现代理池核心功能：配置加载和代理轮换。
|
||
|
||
Purpose: 建立多代理配置基础设施，支持从 YAML 文件或环境变量加载代理列表，实现 Round-Robin 轮换策略。
|
||
Output: proxy_pool.py 模块，包含 ProxyPool、ProxyEntry、ProxyPoolConfig 类。
|
||
</objective>
|
||
|
||
<execution_context>
|
||
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
|
||
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
|
||
</execution_context>
|
||
|
||
<context>
|
||
@.planning/PROJECT.md
|
||
@.planning/ROADMAP.md
|
||
@.planning/STATE.md
|
||
@.planning/phases/10-proxy-pool/10-CONTEXT.md
|
||
@.planning/phases/10-proxy-pool/10-RESEARCH.md
|
||
|
||
<interfaces>
|
||
<!-- Key interfaces from existing codebase that executor needs -->
|
||
|
||
From kb_workflow/crawler/config.py:
|
||
```python
|
||
class ProxyConfig(BaseModel):
|
||
server: str
|
||
username: Optional[str] = None
|
||
password: Optional[str] = None
|
||
|
||
def to_playwright_format(self) -> Dict[str, Any]:
|
||
"""Convert to Playwright proxy format."""
|
||
```
|
||
|
||
From kb_workflow/crawler/config.py:
|
||
```python
|
||
def get_crawler_config() -> CrawlerConfig:
|
||
"""Get crawler configuration singleton."""
|
||
```
|
||
|
||
From tests/test_crawler/conftest.py:
|
||
```python
|
||
@pytest.fixture
|
||
def mock_browser():
|
||
"""Mock PlaywrightBrowser instance."""
|
||
```
|
||
</interfaces>
|
||
</context>
|
||
|
||
<tasks>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 0: 创建测试脚手架</name>
|
||
<files>tests/test_crawler/test_proxy_pool.py, tests/test_crawler/conftest.py</files>
|
||
<behavior>
|
||
- Test load from YAML: 加载包含多个代理的 YAML 文件，返回 ProxyPool 实例
|
||
- Test load from env: 从 HTTPS_PROXY 环境变量加载单代理，返回 ProxyPool 实例
|
||
- Test round-robin: 连续调用 get_next_proxy() 返回按顺序轮换的代理
|
||
- Test empty pool: 无代理时 get_next_proxy() 返回 None
|
||
- Test single proxy: 单代理时总是返回该代理
|
||
</behavior>
|
||
<action>
|
||
创建 tests/test_crawler/test_proxy_pool.py 测试文件。
|
||
|
||
测试用例：
|
||
1. `test_load_from_yaml`: 使用 tmp_path 创建临时 YAML 文件，验证 ProxyPool.from_yaml() 正确加载代理列表
|
||
2. `test_load_from_yaml_not_found`: YAML 文件不存在时返回空 ProxyPool
|
||
3. `test_load_from_env`: mock HTTPS_PROXY 环境变量，验证 ProxyPool.from_env_fallback() 正确加载
|
||
4. `test_round_robin`: 创建包含 3 个代理的 ProxyPool，连续调用 get_next_proxy() 6 次，验证返回顺序为 A-B-C-A-B-C
|
||
5. `test_empty_pool`: 空 ProxyPool 的 get_next_proxy() 返回 None
|
||
6. `test_single_proxy`: 单代理 ProxyPool 总是返回该代理
|
||
|
||
在 conftest.py 中添加 proxy_pool_config fixture（如果需要）。
|
||
|
||
注意：这些测试应该先失败（RED），因为 ProxyPool 类尚未实现。
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_crawler/test_proxy_pool.py -x -v 2>&1 | grep -E "(FAILED|ERROR|test_)" || echo "Tests not failing as expected for Wave 0"</automated>
|
||
</verify>
|
||
<done>测试文件存在，包含 6 个测试用例，运行后显示失败（RED 阶段）</done>
|
||
</task>
|
||
|
||
<task type="auto" tdd="true">
|
||
<name>Task 1: 实现 ProxyEntry + ProxyPool 核心类</name>
|
||
<files>kb_workflow/crawler/proxy_pool.py, tests/test_crawler/test_proxy_pool.py</files>
|
||
<behavior>
|
||
- ProxyEntry(url) 创建代理条目，默认 is_available=True, fail_count=0
|
||
- ProxyPool(config) 初始化代理池，从 config.proxies 创建 ProxyEntry 列表
|
||
- ProxyPool.from_yaml(path) 从 YAML 文件加载代理池配置
|
||
- ProxyPool.from_env_fallback() 从环境变量加载单代理
|
||
- ProxyPool.get_next_proxy() 返回下一个代理（Round-Robin）
|
||
- ProxyPool.has_proxies() 返回是否有配置的代理
|
||
</behavior>
|
||
<action>
|
||
创建 kb_workflow/crawler/proxy_pool.py 模块。
|
||
|
||
实现以下类（基于 RESEARCH.md Pattern 1 设计）：
|
||
|
||
1. **ProxyEntry (Pydantic BaseModel)**:
|
||
- 字段: url: str, fail_count: int = 0, last_fail_time: Optional[datetime] = None, is_available: bool = True
|
||
- 方法: record_success(), record_failure()（此 task 仅 stub，Task 2 实现完整逻辑）
|
||
|
||
2. **ProxyPoolConfig (Pydantic BaseModel)**:
|
||
- 字段: proxies: list[str] = [], cooldown_seconds: int = 300, max_fail_count: int = 3
|
||
|
||
3. **ProxyPool**:
|
||
- `__init__(self, config: ProxyPoolConfig)`: 初始化代理池
|
||
- `from_yaml(cls, path: Path) -> ProxyPool`: 从 YAML 文件加载
|
||
- `from_env_fallback(cls) -> ProxyPool`: 从环境变量加载（向后兼容）
|
||
- `get_next_proxy(self) -> Optional[ProxyEntry]`: Round-Robin 轮换
|
||
- `has_proxies(self) -> bool`: 检查是否有代理
|
||
|
||
配置文件格式（~/.kb_proxies.yaml）:
|
||
```yaml
|
||
proxies:
|
||
- "http://127.0.0.1:7890"
|
||
- "http://user:pass@proxy.example.com:8080"
|
||
cooldown_seconds: 300
|
||
max_fail_count: 3
|
||
```
|
||
|
||
轮换逻辑：维护 _current_index 指针，每次调用 get_next_proxy() 递增并循环。
|
||
|
||
运行测试直到全部通过（GREEN 阶段）。
|
||
</action>
|
||
<verify>
|
||
<automated>pytest tests/test_crawler/test_proxy_pool.py -x</automated>
|
||
</verify>
|
||
<done>
|
||
- proxy_pool.py 存在，包含 ProxyEntry, ProxyPoolConfig, ProxyPool 类
|
||
- 测试 test_load_from_yaml, test_load_from_env, test_round_robin 通过
|
||
- 代码通过 ruff check（如果项目配置了）
|
||
</done>
|
||
</task>
|
||
|
||
</tasks>
|
||
|
||
<verification>
|
||
**Wave 1 验证检查：**
|
||
1. `pytest tests/test_crawler/test_proxy_pool.py -x` 全部通过
|
||
2. ProxyPool 可从 YAML 文件加载多代理
|
||
3. ProxyPool 可从环境变量加载单代理（向后兼容）
|
||
4. Round-Robin 轮换正常工作
|
||
</verification>
|
||
|
||
<success_criteria>
|
||
- [x] 用户可以从 YAML 文件加载多个代理地址（PROXY-01）
|
||
- [x] 用户可以从环境变量加载单代理（PROXY-01 向后兼容）
|
||
- [x] 每次请求自动选择下一个可用代理（PROXY-02）
|
||
- [ ] 代理健康检查（PROXY-03 - Plan 02）
|
||
- [ ] 代理失败回退（PROXY-04 - Plan 02）
|
||
</success_criteria>
|
||
|
||
<output>
|
||
After completion, create `.planning/phases/10-proxy-pool/10-01-SUMMARY.md`
|
||
</output>
|