---
phase: 08-crawler
plan: 03
type: execute
wave: 2
depends_on: [08-01]
files_modified:
  - kb_workflow/crawler/config.py
  - kb_workflow/crawler/zhihu_crawler.py
  - tests/test_crawler/test_proxy_config.py
  - tests/test_crawler/test_rate_limiter.py
  - tests/test_crawler/test_api_intercept.py
autonomous: true
requirements: [NET-01, NET-02, NET-03]
user_setup:
  - service: proxy
    why: "防止 IP 被知乎封禁"
    env_vars:
      - name: HTTP_PROXY
        source: "代理服务地址，如 http://127.0.0.1:7890"
      - name: HTTPS_PROXY
        source: "代理服务地址，如 http://127.0.0.1:7890"
    dashboard_config: []

must_haves:
  truths:
    - "用户可以通过环境变量配置代理 IP"
    - "不配置代理时，爬虫拒绝运行并提示错误"
    - "每次请求间有随机延迟（1-3秒）"
    - "请求失败时有限重试（3次），间隔递增"
    - "x-zse-96 签名由浏览器自动处理，无需手动逆向"
  artifacts:
    - path: "kb_workflow/crawler/config.py"
      provides: "CrawlerConfig 配置类，代理和延迟设置"
      contains: "class CrawlerConfig"
    - path: "kb_workflow/crawler/zhihu_crawler.py"
      provides: "代理应用和重试逻辑"
      contains: "def _get_proxy_config"
    - path: "tests/test_crawler/test_proxy_config.py"
      provides: "代理配置测试"
      exports: ["test_proxy_required"]
    - path: "tests/test_crawler/test_rate_limiter.py"
      provides: "频率控制测试"
      exports: ["test_random_delay"]
  key_links:
    - from: "CrawlerConfig"
      to: "HTTP_PROXY/HTTPS_PROXY"
      via: "环境变量读取"
      pattern: "os\\.environ\\.get.*PROXY"
    - from: "ZhihuCrawler"
      to: "browser.new_context(proxy=...)"
      via: "Playwright 代理配置"
      pattern: "new_context.*proxy"
---

<objective>
实现网络反爬措施：强制代理配置、请求频率控制、有限重试机制，以及利用浏览器自动处理 x-zse-96 签名。

Purpose: 降低被封风险，提高爬取稳定性
Output: CrawlerConfig + 代理/延迟/重试逻辑 + 测试
</objective>

<execution_context>
@/home/starmen1265/.claude/get-shit-done/workflows/execute-plan.md
@/home/starmen1265/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/08-crawler/08-CONTEXT.md
@.planning/phases/08-crawler/08-RESEARCH.md
@.planning/phases/08-crawler/08-01-PLAN.md

<interfaces>
<!-- 从 08-01 创建的接口 -->

From kb_workflow/crawler/zhihu_crawler.py (Plan 01):
```python
class ZhihuCrawler:
    def __init__(self, headless: bool = True, timeout: int = 30)
    def _get_cookie(self) -> str
    def crawl_answer(self, url: str) -> CrawlResult
    def crawl_article(self, url: str) -> CrawlResult
```

代理配置模式 (from RESEARCH.md):
```python
import os
from urllib.parse import urlparse

def get_proxy_config():
    proxy_url = os.environ.get("HTTPS_PROXY") or os.environ.get("HTTP_PROXY")
    if not proxy_url:
        return None

    parsed = urlparse(proxy_url)
    config = {"server": f"{parsed.scheme}://{parsed.hostname}:{parsed.port}"}

    if parsed.username and parsed.password:
        config["username"] = parsed.username
        config["password"] = parsed.password

    return config
```

重试模式 (from RESEARCH.md):
```python
from tenacity import Retrying, stop_after_attempt, wait_exponential

for attempt in Retrying(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    reraise=True
):
    with attempt:
        # 爬取逻辑
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: 创建 CrawlerConfig 配置类</name>
  <files>kb_workflow/crawler/config.py, tests/test_crawler/test_proxy_config.py</files>
  <action>
创建爬虫配置类，处理代理和延迟设置。

**1. 创建 kb_workflow/crawler/config.py:**

```python
"""Crawler configuration

Handles proxy settings, rate limiting, and retry configuration.
"""

import os
from urllib.parse import urlparse
from typing import Optional, Dict, Any

from pydantic import BaseModel, Field


class ProxyConfig(BaseModel):
    """代理配置"""
    server: str = Field(..., description="代理服务器地址")
    username: Optional[str] = Field(None, description="代理用户名")
    password: Optional[str] = Field(None, description="代理密码")

    def to_playwright_format(self) -> Dict[str, Any]:
        """转换为 Playwright 代理格式"""
        config = {"server": self.server}
        if self.username:
            config["username"] = self.username
        if self.password:
            config["password"] = self.password
        return config


class CrawlerConfig(BaseModel):
    """爬虫配置"""

    # 代理设置
    proxy: Optional[ProxyConfig] = Field(None, description="代理配置")
    proxy_required: bool = Field(default=True, description="是否强制要求代理")

    # 请求频率
    min_delay: float = Field(default=1.0, description="最小延迟（秒）")
    max_delay: float = Field(default=3.0, description="最大延迟（秒）")

    # 重试设置
    max_retries: int = Field(default=3, description="最大重试次数")
    retry_min_wait: float = Field(default=2.0, description="重试最小等待（秒）")
    retry_max_wait: float = Field(default=10.0, description="重试最大等待（秒）")

    @classmethod
    def from_env(cls) -> "CrawlerConfig":
        """从环境变量加载配置"""
        proxy = cls._parse_proxy_env()
        return cls(proxy=proxy)

    @staticmethod
    def _parse_proxy_env() -> Optional[ProxyConfig]:
        """解析代理环境变量"""
        proxy_url = os.environ.get("HTTPS_PROXY") or os.environ.get("HTTP_PROXY")

        if not proxy_url:
            return None

        parsed = urlparse(proxy_url)
        server = f"{parsed.scheme}://{parsed.hostname}"

        if parsed.port:
            server += f":{parsed.port}"

        return ProxyConfig(
            server=server,
            username=parsed.username,
            password=parsed.password
        )

    def validate_proxy(self) -> None:
        """验证代理配置

        Raises:
            ValueError: 如果 proxy_required=True 但未配置代理
        """
        if self.proxy_required and not self.proxy:
            raise ValueError(
                "代理配置缺失！请设置环境变量 HTTP_PROXY 或 HTTPS_PROXY。\n"
                "示例: export HTTPS_PROXY=http://127.0.0.1:7890"
            )


def get_crawler_config() -> CrawlerConfig:
    """获取爬虫配置单例"""
    return CrawlerConfig.from_env()
```

**2. 创建 tests/test_crawler/test_proxy_config.py:**

```python
import os
import pytest
from unittest.mock import patch

from kb_workflow.crawler.config import (
    CrawlerConfig,
    ProxyConfig,
    get_crawler_config
)


class TestProxyConfig:

    def test_parse_proxy_from_https_env(self):
        """从 HTTPS_PROXY 解析代理"""
        with patch.dict(os.environ, {"HTTPS_PROXY": "http://127.0.0.1:7890"}):
            config = CrawlerConfig.from_env()
            assert config.proxy is not None
            assert config.proxy.server == "http://127.0.0.1:7890"

    def test_parse_proxy_from_http_env(self):
        """从 HTTP_PROXY 解析代理（HTTPS_PROXY 优先）"""
        with patch.dict(os.environ, {
            "HTTP_PROXY": "http://127.0.0.1:8080",
            "HTTPS_PROXY": "http://127.0.0.1:7890"
        }, clear=False):
            config = CrawlerConfig.from_env()
            assert config.proxy.server == "http://127.0.0.1:7890"

    def test_parse_proxy_with_auth(self):
        """解析带认证的代理"""
        with patch.dict(os.environ, {
            "HTTPS_PROXY": "http://user:pass@proxy.example.com:8080"
        }):
            config = CrawlerConfig.from_env()
            assert config.proxy.username == "user"
            assert config.proxy.password == "pass"

    def test_no_proxy_configured(self):
        """未配置代理"""
        with patch.dict(os.environ, {}, clear=True):
            # 清除所有代理相关环境变量
            os.environ.pop("HTTP_PROXY", None)
            os.environ.pop("HTTPS_PROXY", None)
            config = CrawlerConfig.from_env()
            assert config.proxy is None

    def test_proxy_required_validation_fails(self):
        """强制代理但未配置时抛出异常"""
        config = CrawlerConfig(proxy=None, proxy_required=True)
        with pytest.raises(ValueError, match="代理配置缺失"):
            config.validate_proxy()

    def test_proxy_required_validation_passes(self):
        """配置代理后验证通过"""
        config = CrawlerConfig(
            proxy=ProxyConfig(server="http://127.0.0.1:7890"),
            proxy_required=True
        )
        config.validate_proxy()  # 不应抛出异常

    def test_proxy_not_required(self):
        """不强制代理时验证通过"""
        config = CrawlerConfig(proxy=None, proxy_required=False)
        config.validate_proxy()  # 不应抛出异常

    def test_to_playwright_format(self):
        """转换为 Playwright 格式"""
        proxy = ProxyConfig(
            server="http://127.0.0.1:7890",
            username="user",
            password="pass"
        )
        result = proxy.to_playwright_format()
        assert result == {
            "server": "http://127.0.0.1:7890",
            "username": "user",
            "password": "pass"
        }
```
</action>
  <verify>
    <automated>pytest tests/test_crawler/test_proxy_config.py -x --tb=short</automated>
  </verify>
  <done>
    - CrawlerConfig 类定义完成
    - 从 HTTP_PROXY/HTTPS_PROXY 读取代理
    - proxy_required=True 时未配置代理抛出异常
    - 所有测试通过
  </done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: 整合代理和延迟到 ZhihuCrawler</name>
  <files>kb_workflow/crawler/zhihu_crawler.py, tests/test_crawler/test_rate_limiter.py, tests/test_crawler/test_api_intercept.py</files>
  <behavior>
    - Test 1: 未配置代理且 proxy_required=True 时抛出异常
    - Test 2: 请求间有随机延迟 (1-3 秒)
    - Test 3: 失败请求重试 3 次，间隔递增
    - Test 4: API 拦截正确处理 x-zse-96 签名（由浏览器自动处理）
  </behavior>
  <action>
更新 ZhihuCrawler 类，整合代理配置、随机延迟和重试机制。

**1. 更新 kb_workflow/crawler/zhihu_crawler.py:**

```python
import random
import time
from typing import Optional, Dict, Any
from playwright.sync_api import Route, Response
from tenacity import Retrying, stop_after_attempt, wait_exponential, RetryError

from kb_workflow.metadata.zhihu_extractor import PlaywrightBrowser
from kb_workflow.pipeline.config import get_settings
from .models import AnswerData, ArticleData, ContentMetadata, AuthorInfo, CrawlResult
from .config import CrawlerConfig, get_crawler_config


class ZhihuCrawler:
    """知乎内容爬虫

    支持:
    - 强制代理配置
    - 随机请求延迟
    - 有限重试
    - 自动处理 x-zse-96 签名
    """

    # ... URL patterns 等现有代码 ...

    def __init__(
        self,
        headless: bool = True,
        timeout: int = 30,
        config: Optional[CrawlerConfig] = None
    ):
        self.headless = headless
        self.timeout = timeout
        self.config = config or get_crawler_config()
        self.browser: Optional[PlaywrightBrowser] = None
        self.interceptor: Optional[ZhihuApiInterceptor] = None

    def _get_cookie(self) -> str:
        """获取 Cookie"""
        settings = get_settings()
        return settings.zhihu_cookie.get_secret_value()

    def _apply_random_delay(self) -> None:
        """应用随机延迟"""
        delay = random.uniform(self.config.min_delay, self.config.max_delay)
        time.sleep(delay)

    def _start_browser_with_proxy(self, cookie: str) -> None:
        """启动浏览器并应用代理配置"""
        # 验证代理
        self.config.validate_proxy()

        from playwright.sync_api import sync_playwright

        self.playwright = sync_playwright().start()
        browser_args = {"headless": self.headless}

        # 应用代理
        context_args = {
            'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        }

        if self.config.proxy:
            context_args["proxy"] = self.config.proxy.to_playwright_format()

        # 阻止 Service Worker 干扰 API 拦截
        context_args["service_workers"] = "block"

        self.browser = self.playwright.chromium.launch(**browser_args)
        self.context = self.browser.new_context(**context_args)

        # 添加 Cookie
        if cookie:
            self._add_cookies(cookie)

        self.page = self.context.new_page()
        self.page.set_default_timeout(self.timeout * 1000)

    def _add_cookies(self, cookie_str: str) -> None:
        """添加 Cookie 到浏览器上下文"""
        cookies = []
        for item in cookie_str.split(';'):
            item = item.strip()
            if '=' in item:
                name, value = item.split('=', 1)
                cookies.append({
                    'name': name.strip(),
                    'value': value.strip(),
                    'domain': '.zhihu.com',
                    'path': '/',
                    'secure': True,
                    'httpOnly': False,
                    'sameSite': 'Lax'
                })
        self.context.add_cookies(cookies)

    def _close_browser(self) -> None:
        """关闭浏览器"""
        if hasattr(self, 'page') and self.page:
            self.page.close()
        if hasattr(self, 'context') and self.context:
            self.context.close()
        if hasattr(self, 'browser') and self.browser:
            self.browser.close()
        if hasattr(self, 'playwright'):
            self.playwright.stop()

    def _crawl_with_retry(self, crawl_func, *args, **kwargs):
        """带重试的爬取"""
        try:
            for attempt in Retrying(
                stop=stop_after_attempt(self.config.max_retries),
                wait=wait_exponential(
                    multiplier=1,
                    min=self.config.retry_min_wait,
                    max=self.config.retry_max_wait
                ),
                reraise=True
            ):
                with attempt:
                    # 随机延迟
                    self._apply_random_delay()
                    return crawl_func(*args, **kwargs)
        except RetryError as e:
            return CrawlResult(
                success=False,
                error=f"重试 {self.config.max_retries} 次后仍失败: {str(e)}"
            )

    def crawl_answer(self, url: str, include_comments: bool = False) -> CrawlResult:
        """爬取知乎回答（带重试）"""
        return self._crawl_with_retry(self._crawl_answer_impl, url, include_comments)

    def _crawl_answer_impl(self, url: str, include_comments: bool = False) -> CrawlResult:
        """爬取知乎回答实现"""
        import re

        match = re.search(self.ANSWER_PATTERN, url)
        if not match:
            return CrawlResult(success=False, error="Invalid answer URL")

        question_id, answer_id = match.groups()

        try:
            self._start_browser_with_proxy(self._get_cookie())
            self.interceptor = ZhihuApiInterceptor()

            # 设置路由拦截 - x-zse-96 签名由浏览器自动处理
            self.page.route("**/api/v4/**", self.interceptor.handle_route)

            # 访问页面触发 API 请求
            self.page.goto(url, wait_until="networkidle")

            # 等待数据捕获
            time.sleep(2)

            if "answer" not in self.interceptor.captured_data:
                return CrawlResult(success=False, error="Failed to capture answer data")

            answer_data = self.interceptor.captured_data["answer"]

            result_data = AnswerData(
                id=int(answer_id),
                question_id=int(question_id),
                question_title=answer_data.get("question", {}).get("title", ""),
                content=answer_data.get("content", ""),
                excerpt=answer_data.get("excerpt", ""),
                metadata=self._parse_metadata(answer_data)
            )

            result = CrawlResult(success=True, content_type="answer", data=result_data)

            # 可选：爬取评论
            if include_comments:
                comments_result = self.crawl_comments(
                    content_type="answers",
                    content_id=int(answer_id)
                )
                result.comments = comments_result.comments

            return result

        except Exception as e:
            return CrawlResult(success=False, error=str(e))
        finally:
            self._close_browser()

    def crawl_article(self, url: str, include_comments: bool = False) -> CrawlResult:
        """爬取知乎专栏文章（带重试）"""
        return self._crawl_with_retry(self._crawl_article_impl, url, include_comments)

    def _crawl_article_impl(self, url: str, include_comments: bool = False) -> CrawlResult:
        """爬取知乎专栏文章实现"""
        # ... 类似 crawl_answer 的实现 ...
        pass

    # crawl_comments 方法保持不变
    # ...
```

**2. 创建 tests/test_crawler/test_rate_limiter.py:**

```python
import pytest
import time
from unittest.mock import Mock, patch, MagicMock

from kb_workflow.crawler.zhihu_crawler import ZhihuCrawler
from kb_workflow.crawler.config import CrawlerConfig


class TestRateLimiter:

    def test_random_delay_applied(self):
        """请求间有随机延迟"""
        config = CrawlerConfig(
            min_delay=0.1,
            max_delay=0.2,
            proxy_required=False
        )
        crawler = ZhihuCrawler(config=config)

        start = time.time()
        crawler._apply_random_delay()
        elapsed = time.time() - start

        assert 0.1 <= elapsed <= 0.25  # 允许少量误差

    def test_delay_is_random(self):
        """延迟是随机的"""
        config = CrawlerConfig(
            min_delay=0.05,
            max_delay=0.15,
            proxy_required=False
        )
        crawler = ZhihuCrawler(config=config)

        delays = []
        for _ in range(10):
            start = time.time()
            crawler._apply_random_delay()
            delays.append(time.time() - start)

        # 延迟应该有变化
        assert max(delays) - min(delays) > 0.01

    def test_exponential_backoff_retry(self):
        """重试间隔递增"""
        config = CrawlerConfig(
            max_retries=3,
            retry_min_wait=0.1,
            retry_max_wait=1.0,
            proxy_required=False
        )
        crawler = ZhihuCrawler(config=config)

        call_times = []

        def failing_func():
            call_times.append(time.time())
            raise Exception("Simulated failure")

        result = crawler._crawl_with_retry(failing_func)

        assert result.success is False
        assert "重试 3 次后仍失败" in result.error
        assert len(call_times) == 3  # 初始 + 2 次重试
```

**3. 创建 tests/test_crawler/test_api_intercept.py:**

```python
import pytest
from unittest.mock import Mock, MagicMock

from kb_workflow.crawler.zhihu_crawler import ZhihuApiInterceptor


class TestApiIntercept:

    def test_intercept_answer_api(self):
        """拦截答案 API"""
        interceptor = ZhihuApiInterceptor()

        # 模拟 route 对象
        route = Mock()
        route.request.url = "https://www.zhihu.com/api/v4/answers/12345"

        mock_response = Mock()
        mock_response.json.return_value = {"id": 12345, "content": "test"}
        route.fetch.return_value = mock_response

        interceptor.handle_route(route)

        assert "answer" in interceptor.captured_data
        assert interceptor.captured_data["answer"]["id"] == 12345

    def test_intercept_article_api(self):
        """拦截文章 API"""
        interceptor = ZhihuApiInterceptor()

        route = Mock()
        route.request.url = "https://www.zhihu.com/api/v4/articles/67890"

        mock_response = Mock()
        mock_response.json.return_value = {"id": 67890, "title": "test article"}
        route.fetch.return_value = mock_response

        interceptor.handle_route(route)

        assert "article" in interceptor.captured_data

    def test_non_api_requests_continue(self):
        """非 API 请求继续正常加载"""
        interceptor = ZhihuApiInterceptor()

        route = Mock()
        route.request.url = "https://www.zhihu.com/static/some.js"

        interceptor.handle_route(route)

        route.continue_.assert_called_once()
        route.fulfill.assert_not_called()

    def test_x_zse_96_auto_handled(self):
        """x-zse-96 签名由浏览器自动处理（通过 route.fetch）"""
        interceptor = ZhihuApiInterceptor()

        route = Mock()
        route.request.url = "https://www.zhihu.com/api/v4/answers/123"

        interceptor.handle_route(route)

        # fetch() 会自动带上浏览器生成的签名
        route.fetch.assert_called_once()
        # fulfill() 返回响应给页面
        route.fulfill.assert_called_once()
```
</action>
  <verify>
    <automated>pytest tests/test_crawler/test_rate_limiter.py tests/test_crawler/test_api_intercept.py -x --tb=short</automated>
  </verify>
  <done>
    - 代理配置整合到 ZhihuCrawler
    - 随机延迟 1-3 秒生效
    - 失败重试 3 次，间隔递增
    - x-zse-96 签名由浏览器自动处理
    - 所有测试通过
  </done>
</task>

</tasks>

<verification>
**自动化验证:**
```bash
pytest tests/test_crawler/ -x -v
```

**代理配置验证:**
```bash
# 不配置代理时应报错
unset HTTP_PROXY HTTPS_PROXY
python -c "from kb_workflow.crawler import ZhihuCrawler; ZhihuCrawler().crawl_answer('https://www.zhihu.com/question/1/answer/1')"
# 应输出: ValueError: 代理配置缺失！
```
</verification>

<success_criteria>
1. 代理配置从 HTTP_PROXY/HTTPS_PROXY 读取
2. proxy_required=True 时未配置代理抛出异常
3. 请求间有随机延迟 (1-3 秒)
4. 失败请求重试 3 次，间隔递增
5. x-zse-96 签名由浏览器自动处理
6. 所有测试通过
</success_criteria>

<output>
After completion, create `.planning/phases/08-crawler/08-03-SUMMARY.md`
</output>