kb-skills/.planning/phases/08-crawler/08-03-SUMMARY.md at dfbbabceea25a87306bc89c403a4e9d39c493dec

Archived

This repository has been archived on 2026-08-07. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

starmen1265 123e307522 docs(08-03): complete anti-crawl measures plan

- Proxy configuration from HTTP_PROXY/HTTPS_PROXY
- Random delay (1-3s) between requests
- Exponential backoff retry (3 attempts)
- x-zse-96 signature handled by browser

All 44 crawler tests passing

Requirements: NET-01, NET-02, NET-03 completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-17 22:20:38 +08:00

3.0 KiB

Raw Blame History

phase, plan, subsystem, tags, requires, provides, tech-stack, key-files, key-decisions, requirements-completed, duration, completed

phase

plan

subsystem

tags

requires

provides

tech-stack

key-files

key-decisions

requirements-completed

duration

completed

08-crawler

crawler

proxy

rate-limiter

retry

anti-crawl

playwright

phase	provides
08-01	ZhihuCrawler base class

CrawlerConfig for proxy/rate limiting configuration

Random delay (1-3s) between requests

Exponential backoff retry (3 attempts)

Automatic x-zse-96 signature handling via browser

added

patterns

tenacity (retry library)

Environment variable proxy configuration

Exponential backoff retry pattern

Random delay for rate limiting

created

modified

kb_workflow/crawler/config.py

tests/test_crawler/test_proxy_config.py

tests/test_crawler/test_rate_limiter.py

tests/test_crawler/test_api_intercept.py

kb_workflow/crawler/zhihu_crawler.py

kb_workflow/crawler/models.py

tests/test_crawler/test_comment_crawler.py

Mandatory proxy configuration (configurable via proxy_required flag)

Use tenacity library for retry with exponential backoff

x-zse-96 signature handled automatically by browser via route.fetch()

Random delay range
1-3 seconds (configurable)

NET-01

NET-02

NET-03

15min

2026-03-17

Summary

实现了网络反爬措施：强制代理配置、请求频率控制、有限重试机制，以及利用浏览器自动处理 x-zse-96 签名。

What Was Built

CrawlerConfig 类 (kb_workflow/crawler/config.py)

ProxyConfig - 代理配置模型
从环境变量 HTTP_PROXY/HTTPS_PROXY 读取代理设置
proxy_required 标志控制是否强制代理（默认 True）
随机延迟配置（min_delay, max_delay）
重试配置（max_retries, retry_min_wait, retry_max_wait）

ZhihuCrawler 增强

_start_browser_with_proxy() - 启动浏览器并应用代理配置
_apply_random_delay() - 应用随机延迟（1-3 秒）
_crawl_with_retry() - 带指数退避的重试逻辑
浏览器自动处理 x-zse-96 签名（通过 route.fetch()）

Test Coverage

44 个测试全部通过

test_proxy_config.py - 11 个代理配置测试
test_rate_limiter.py - 8 个频率控制测试
test_api_intercept.py - 4 个 API 拦截测试
其他测试文件保持通过

Commits

626fb73 feat(08-03): create CrawlerConfig for proxy and rate limiting
9e1452d feat(08-03): integrate proxy, delay, and retry into ZhihuCrawler

Key Decisions

强制代理: 默认 proxy_required=True，未配置代理时抛出 ValueError
重试机制: 使用 tenacity 库，最多 3 次重试，间隔指数递增（2s → 10s）
签名处理: 利用浏览器 route.fetch() 自动携带 x-zse-96 签名，无需手动逆向

Requirements Completed

NET-01: 用户可以通过环境变量配置代理 IP ✓
NET-02: 不配置代理时，爬虫拒绝运行并提示错误 ✓
NET-03: 每次请求间有随机延迟（1-3秒），失败重试 3 次 ✓

3.0 KiB Raw Blame History Unescape Escape