feat(assets): 白模资产包+参考资料+图片下载工具
- BlockingStarterPack 白模资产包（墙体/地板/楼梯/门窗等） - 参考图：放学后少年游戏截图 - tools/image-downloader: 图片批量下载工具
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# 参考图素材批量下载工具
|
||||
|
||||
从 markdown 文件中提取 URL，按域名自动分类，调用合适的开源工具批量下载图片。
|
||||
|
||||
## 支持的站点类型
|
||||
|
||||
| 引擎 | 支持站点 | 说明 |
|
||||
|------|---------|------|
|
||||
| **gallery-dl** | Pinterest、Instagram、Reddit | 需 Chrome 登录态 cookies |
|
||||
| **yt-dlp** | YouTube | 仅下载视频缩略图 |
|
||||
| **BeautifulSoup4** | GAME Watch、博客、Wiki 等 | 静态页面通用抓取 |
|
||||
| **Playwright** | LaunchBox、MobyGames、Getty 等 | 动态页面 headless 浏览器 |
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
# 安装 Python 依赖
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 安装 Playwright 浏览器引擎（BS4 不需要此步骤）
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
```bash
|
||||
# 1. 先 dry-run 查看分类结果
|
||||
python download.py "../../ai-art-research/参考图/参考图素材链接清单.md" --dry-run
|
||||
|
||||
# 2. 快速模式（仅 gallery-dl + yt-dlp）
|
||||
python download.py "../../ai-art-research/参考图/参考图素材链接清单.md" --quick
|
||||
|
||||
# 3. 完整下载
|
||||
python download.py "../../ai-art-research/参考图/参考图素材链接清单.md"
|
||||
|
||||
# 4. 仅下载某个章节
|
||||
python download.py "../../ai-art-research/参考图/参考图素材链接清单.md" --section 我的暑假
|
||||
|
||||
# 5. 指定输出目录
|
||||
python download.py "../../ai-art-research/参考图/参考图素材链接清单.md" -o "../../ai-art-research/参考图"
|
||||
```
|
||||
|
||||
## 命令行参数
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `markdown` | 包含 URL 的 markdown 文件路径（必填） |
|
||||
| `-o, --output` | 输出根目录（默认：markdown 文件所在目录） |
|
||||
| `--dry-run` | 仅打印 URL 分类结果，不下载 |
|
||||
| `--section 关键词` | 仅处理包含指定关键词的章节 |
|
||||
| `--quick` | 快速模式：仅使用 gallery-dl 和 yt-dlp |
|
||||
|
||||
## 输出目录结构
|
||||
|
||||
```
|
||||
参考图/
|
||||
├── 游戏截图/
|
||||
│ ├── 我的暑假/ ← 章节一
|
||||
│ └── 放学后少年/ ← 章节二
|
||||
├── 美术风格/ ← 章节四（Lowpoly/怀旧美术）
|
||||
├── 真实场景/ ← 章节三（千禧年中国城郊）
|
||||
└── 其他/ ← 未匹配章节
|
||||
```
|
||||
|
||||
## 站点规则配置
|
||||
|
||||
`site_rules.json` 可自定义各站点的抓取行为：
|
||||
|
||||
```json
|
||||
{
|
||||
"example.com": {
|
||||
"engine": "bs4",
|
||||
"img_selector": "img.article-image",
|
||||
"min_size_kb": 20,
|
||||
"note": "说明文字"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| `engine` | string | 下载引擎：`bs4` / `playwright` / `gallery-dl` / `yt-dlp` |
|
||||
| `img_selector` | string | CSS 选择器（仅 bs4 引擎） |
|
||||
| `min_size_kb` | int | 最小图片体积，小于此值的图标被跳过 |
|
||||
| `wait_seconds` | int | Playwright 页面等待时间（秒） |
|
||||
| `scroll_to_load` | bool | 是否滚动页面触发懒加载 |
|
||||
| `cookies_required` | bool | 是否需要浏览器 cookies（gallery-dl） |
|
||||
| `watermark_warning` | bool | 水印警告标记 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
- **Pinterest / Instagram**：需要先在 Chrome 浏览器中登录，gallery-dl 会自动读取 cookies
|
||||
- **Getty / Alamy / iStock**：下载的图片带水印，仅供美术参考
|
||||
- **反爬延迟**：每次请求间自动加入 1.5~4 秒随机延迟
|
||||
- **版权声明**：所有抓取图片仅供个人美术参考/灵感素材，不应直接用于商业发布
|
||||
@@ -0,0 +1,668 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
参考图素材批量下载工具
|
||||
|
||||
从 markdown 文件中提取 URL，按域名自动分类，调用合适的工具下载图片。
|
||||
支持 gallery-dl / yt-dlp / BeautifulSoup4 / Playwright 四种下载引擎。
|
||||
|
||||
用法:
|
||||
python download.py 参考图素材链接清单.md
|
||||
python download.py 参考图素材链接清单.md --dry-run
|
||||
python download.py 参考图素材链接清单.md --section 我的暑假
|
||||
python download.py 参考图素材链接清单.md --quick
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
# Windows 终端 UTF-8 输出
|
||||
if sys.platform == "win32":
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 常量
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
SITE_RULES_PATH = SCRIPT_DIR / "site_rules.json"
|
||||
|
||||
# 章节标题关键词 → 输出子目录
|
||||
SECTION_MAP = [
|
||||
("我的暑假", "游戏截图/我的暑假"),
|
||||
("放学后少年", "游戏截图/放学后少年"),
|
||||
("城郊", "真实场景"),
|
||||
("真实场景", "真实场景"),
|
||||
("Lowpoly", "美术风格"),
|
||||
("低多边形", "美术风格"),
|
||||
("怀旧美术", "美术风格"),
|
||||
("美术风格", "美术风格"),
|
||||
]
|
||||
|
||||
# 默认 fallback
|
||||
DEFAULT_SECTION_DIR = "其他"
|
||||
|
||||
# 请求头
|
||||
HEADERS = {
|
||||
"User-Agent": (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/125.0.0.0 Safari/537.36"
|
||||
)
|
||||
}
|
||||
|
||||
# 下载延迟范围（秒），防止被封
|
||||
DELAY_RANGE = (1.5, 4.0)
|
||||
|
||||
# 默认最小图片体积 (KB)，过滤掉图标/像素点
|
||||
DEFAULT_MIN_SIZE_KB = 10
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown 解析器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_markdown(md_path: str) -> list[dict]:
|
||||
"""解析 markdown 文件，返回 [{url, section, subsection, line}]"""
|
||||
content = Path(md_path).read_text(encoding="utf-8")
|
||||
lines = content.splitlines()
|
||||
|
||||
results = []
|
||||
current_section = ""
|
||||
current_subsection = ""
|
||||
|
||||
for i, line in enumerate(lines, 1):
|
||||
stripped = line.strip()
|
||||
|
||||
# 追踪章节标题
|
||||
if stripped.startswith("## "):
|
||||
current_section = stripped.lstrip("#").strip()
|
||||
current_subsection = ""
|
||||
elif stripped.startswith("### "):
|
||||
current_subsection = stripped.lstrip("#").strip()
|
||||
|
||||
# 跳过非 URL 行（表格分隔行等）
|
||||
if "---" in stripped and not "http" in stripped:
|
||||
continue
|
||||
|
||||
# 提取 URL
|
||||
for m in re.finditer(r'https?://[^\s\)|\]]+', stripped):
|
||||
url = m.group(0).rstrip(".,;:)") # 清理尾部标点
|
||||
results.append({
|
||||
"url": url,
|
||||
"section": current_section,
|
||||
"subsection": current_subsection,
|
||||
"line": i,
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def section_to_dir(section: str, subsection: str) -> str:
|
||||
"""将章节标题映射为输出目录名"""
|
||||
# 先检查 section
|
||||
for keyword, dir_name in SECTION_MAP:
|
||||
if keyword in section:
|
||||
return dir_name
|
||||
# 再检查 subsection
|
||||
for keyword, dir_name in SECTION_MAP:
|
||||
if keyword in subsection:
|
||||
return dir_name
|
||||
return DEFAULT_SECTION_DIR
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 域名路由器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_site_rules() -> dict:
|
||||
"""加载 site_rules.json"""
|
||||
if SITE_RULES_PATH.exists():
|
||||
with open(SITE_RULES_PATH, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
return {}
|
||||
|
||||
|
||||
def route_url(url: str, site_rules: dict) -> dict:
|
||||
"""根据 URL 域名和 site_rules 确定下载引擎"""
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc.lower()
|
||||
|
||||
# 去掉 www. 前缀尝试匹配
|
||||
domain_no_www = re.sub(r"^www\.", "", domain)
|
||||
|
||||
# 精确匹配
|
||||
if domain in site_rules:
|
||||
return {"tool": site_rules[domain].get("engine", "bs4"), "rules": site_rules[domain]}
|
||||
if domain_no_www in site_rules:
|
||||
return {"tool": site_rules[domain_no_www].get("engine", "bs4"), "rules": site_rules[domain_no_www]}
|
||||
|
||||
# 后缀匹配（处理子域名）
|
||||
for rule_domain, rules in site_rules.items():
|
||||
if domain.endswith(rule_domain) or rule_domain.endswith(domain):
|
||||
return {"tool": rules.get("engine", "bs4"), "rules": rules}
|
||||
|
||||
# 默认使用 BS4
|
||||
return {"tool": "bs4", "rules": {}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 下载引擎：gallery-dl
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def download_gallery_dl(url: str, output_dir: str, rules: dict) -> bool:
|
||||
"""调用 gallery-dl 下载图片"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd = ["gallery-dl", "-D", str(output_path)]
|
||||
|
||||
# Pinterest / Instagram 需要 cookies
|
||||
if rules.get("cookies_required"):
|
||||
cmd.extend(["--cookies-from-browser", "chrome"])
|
||||
|
||||
cmd.append(url)
|
||||
|
||||
print(f" [gallery-dl] {url}")
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
|
||||
if result.returncode == 0:
|
||||
print(f" ✓ 完成")
|
||||
return True
|
||||
else:
|
||||
print(f" ✗ 失败: {result.stderr[:200]}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f" ✗ 超时")
|
||||
return False
|
||||
except FileNotFoundError:
|
||||
print(f" ✗ gallery-dl 未安装 (pip install gallery-dl)")
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 下载引擎：yt-dlp
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def download_yt_dlp(url: str, output_dir: str, rules: dict) -> bool:
|
||||
"""调用 yt-dlp 下载 YouTube 视频缩略图"""
|
||||
try:
|
||||
import yt_dlp
|
||||
except ImportError:
|
||||
print(f" ✗ yt-dlp 未安装 (pip install yt-dlp)")
|
||||
return False
|
||||
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print(f" [yt-dlp] {url}")
|
||||
|
||||
opts = {
|
||||
"writethumbnail": True,
|
||||
"skip_download": True,
|
||||
"convertthumbnails": {"default": "png"},
|
||||
"outtmpl": str(output_path / "%(title)s.%(ext)s"),
|
||||
"quiet": True,
|
||||
"no_warnings": True,
|
||||
}
|
||||
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(opts) as ydl:
|
||||
ydl.download([url])
|
||||
print(f" ✓ 缩略图已下载")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" ✗ 失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 下载引擎：BeautifulSoup4 静态抓取
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def download_bs4(url: str, output_dir: str, rules: dict) -> bool:
|
||||
"""用 requests + BeautifulSoup 抓取静态页面中的图片"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
min_size_kb = rules.get("min_size_kb", DEFAULT_MIN_SIZE_KB)
|
||||
img_selector = rules.get("img_selector")
|
||||
|
||||
print(f" [bs4] {url}")
|
||||
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=30)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException as e:
|
||||
print(f" ✗ 请求失败: {e}")
|
||||
return False
|
||||
|
||||
soup = BeautifulSoup(resp.text, "lxml")
|
||||
|
||||
# 提取图片 URL
|
||||
img_urls = []
|
||||
|
||||
if img_selector:
|
||||
# 先尝试自定义 CSS 选择器
|
||||
for img in soup.select(img_selector):
|
||||
src = img.get("src") or img.get("data-src") or img.get("data-original")
|
||||
if src:
|
||||
img_urls.append(urljoin(url, src))
|
||||
|
||||
# 选择器无结果时 fallback 到通用提取
|
||||
if not img_urls:
|
||||
for img in soup.find_all("img"):
|
||||
for attr in ("src", "data-src", "data-original", "data-lazy-src"):
|
||||
src = img.get(attr)
|
||||
if src:
|
||||
img_urls.append(urljoin(url, src))
|
||||
break
|
||||
|
||||
# 额外策略：扫描 <a> 标签中直接链接的图片
|
||||
for a in soup.find_all("a", href=True):
|
||||
href = a["href"]
|
||||
if any(href.lower().endswith(ext) for ext in (".jpg", ".jpeg", ".png", ".gif", ".webp")):
|
||||
img_urls.append(urljoin(url, href))
|
||||
|
||||
if not img_urls:
|
||||
print(f" ⚠ 未找到图片")
|
||||
return False
|
||||
|
||||
# 去重
|
||||
img_urls = list(dict.fromkeys(img_urls))
|
||||
|
||||
# 下载图片
|
||||
downloaded = 0
|
||||
skipped = 0
|
||||
for img_url in img_urls:
|
||||
# 跳过 base64 内嵌图片
|
||||
if img_url.startswith("data:"):
|
||||
continue
|
||||
|
||||
# 跳过 SVG 图标、追踪像素等
|
||||
parsed_path = urlparse(img_url).path.lower()
|
||||
if parsed_path.endswith((".svg",)):
|
||||
continue
|
||||
|
||||
try:
|
||||
# 先 HEAD 请求检查大小
|
||||
head_resp = requests.head(img_url, headers=HEADERS, timeout=10, allow_redirects=True)
|
||||
content_length = int(head_resp.headers.get("content-length", 0))
|
||||
if content_length < min_size_kb * 1024:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
# 下载图片
|
||||
img_resp = requests.get(img_url, headers=HEADERS, timeout=30, stream=True)
|
||||
img_resp.raise_for_status()
|
||||
|
||||
# 生成文件名
|
||||
filename = Path(urlparse(img_url).path).name
|
||||
if not filename or "." not in filename:
|
||||
ext = ".jpg"
|
||||
content_type = img_resp.headers.get("content-type", "")
|
||||
if "png" in content_type:
|
||||
ext = ".png"
|
||||
elif "webp" in content_type:
|
||||
ext = ".webp"
|
||||
filename = f"img_{downloaded + 1}{ext}"
|
||||
|
||||
# 避免文件名冲突
|
||||
filepath = output_path / filename
|
||||
counter = 1
|
||||
while filepath.exists():
|
||||
stem = filepath.stem
|
||||
suffix = filepath.suffix
|
||||
filepath = output_path / f"{stem}_{counter}{suffix}"
|
||||
counter += 1
|
||||
|
||||
with open(filepath, "wb") as f:
|
||||
for chunk in img_resp.iter_content(8192):
|
||||
f.write(chunk)
|
||||
|
||||
downloaded += 1
|
||||
|
||||
# 延迟
|
||||
time.sleep(random.uniform(*DELAY_RANGE))
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠ 跳过 {img_url[:80]}: {e}")
|
||||
|
||||
print(f" ✓ 下载 {downloaded} 张，跳过 {skipped} 张小图")
|
||||
return downloaded > 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 下载引擎：Playwright 动态抓取
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def download_playwright(url: str, output_dir: str, rules: dict) -> bool:
|
||||
"""用 Playwright headless 浏览器抓取动态页面中的图片"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
min_size_kb = rules.get("min_size_kb", DEFAULT_MIN_SIZE_KB)
|
||||
wait_seconds = rules.get("wait_seconds", 3)
|
||||
scroll_to_load = rules.get("scroll_to_load", False)
|
||||
|
||||
print(f" [playwright] {url}")
|
||||
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
except ImportError:
|
||||
print(f" ✗ playwright 未安装 (pip install playwright && playwright install chromium)")
|
||||
return False
|
||||
|
||||
try:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
page = browser.new_page()
|
||||
page.goto(url, wait_until="networkidle", timeout=30000)
|
||||
|
||||
# 等待额外渲染时间
|
||||
page.wait_for_timeout(wait_seconds * 1000)
|
||||
|
||||
# 滚动页面触发懒加载
|
||||
if scroll_to_load:
|
||||
_scroll_page(page)
|
||||
|
||||
# 提取图片 URL
|
||||
img_elements = page.query_selector_all("img")
|
||||
img_urls = []
|
||||
for img in img_elements:
|
||||
src = img.get_attribute("src") or img.get_attribute("data-src") or img.get_attribute("data-original")
|
||||
if src and not src.startswith("data:"):
|
||||
img_urls.append(urljoin(url, src))
|
||||
|
||||
browser.close()
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Playwright 失败: {e}")
|
||||
return False
|
||||
|
||||
if not img_urls:
|
||||
print(f" ⚠ 未找到图片")
|
||||
return False
|
||||
|
||||
# 去重
|
||||
img_urls = list(dict.fromkeys(img_urls))
|
||||
|
||||
# 下载图片
|
||||
downloaded = 0
|
||||
skipped = 0
|
||||
for img_url in img_urls:
|
||||
parsed_path = urlparse(img_url).path.lower()
|
||||
if parsed_path.endswith((".svg",)):
|
||||
continue
|
||||
|
||||
try:
|
||||
head_resp = requests.head(img_url, headers=HEADERS, timeout=10, allow_redirects=True)
|
||||
content_length = int(head_resp.headers.get("content-length", 0))
|
||||
if content_length < min_size_kb * 1024:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
img_resp = requests.get(img_url, headers=HEADERS, timeout=30, stream=True)
|
||||
img_resp.raise_for_status()
|
||||
|
||||
filename = Path(urlparse(img_url).path).name
|
||||
if not filename or "." not in filename:
|
||||
ext = ".jpg"
|
||||
content_type = img_resp.headers.get("content-type", "")
|
||||
if "png" in content_type:
|
||||
ext = ".png"
|
||||
elif "webp" in content_type:
|
||||
ext = ".webp"
|
||||
filename = f"img_{downloaded + 1}{ext}"
|
||||
|
||||
filepath = output_path / filename
|
||||
counter = 1
|
||||
while filepath.exists():
|
||||
stem = filepath.stem
|
||||
suffix = filepath.suffix
|
||||
filepath = output_path / f"{stem}_{counter}{suffix}"
|
||||
counter += 1
|
||||
|
||||
with open(filepath, "wb") as f:
|
||||
for chunk in img_resp.iter_content(8192):
|
||||
f.write(chunk)
|
||||
|
||||
downloaded += 1
|
||||
time.sleep(random.uniform(*DELAY_RANGE))
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠ 跳过 {img_url[:80]}: {e}")
|
||||
|
||||
print(f" ✓ 下载 {downloaded} 张，跳过 {skipped} 张小图")
|
||||
return downloaded > 0
|
||||
|
||||
|
||||
def _scroll_page(page):
|
||||
"""滚动页面到底部，触发懒加载"""
|
||||
try:
|
||||
last_height = page.evaluate("document.body.scrollHeight")
|
||||
for _ in range(10): # 最多滚动 10 次
|
||||
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
page.wait_for_timeout(1000)
|
||||
new_height = page.evaluate("document.body.scrollHeight")
|
||||
if new_height == last_height:
|
||||
break
|
||||
last_height = new_height
|
||||
# 滚回顶部
|
||||
page.evaluate("window.scrollTo(0, 0)")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 调度器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TOOL_DISPATCHER = {
|
||||
"gallery-dl": download_gallery_dl,
|
||||
"yt-dlp": download_yt_dlp,
|
||||
"bs4": download_bs4,
|
||||
"playwright": download_playwright,
|
||||
}
|
||||
|
||||
|
||||
def process_urls(
|
||||
entries: list[dict],
|
||||
output_root: str,
|
||||
quick: bool = False,
|
||||
section_filter: str = "",
|
||||
) -> None:
|
||||
"""处理所有 URL 条目"""
|
||||
site_rules = load_site_rules()
|
||||
|
||||
# 按 section 过滤
|
||||
if section_filter:
|
||||
entries = [e for e in entries if section_filter in e["section"] or section_filter in e["subsection"]]
|
||||
if not entries:
|
||||
print(f"⚠ 没有匹配章节 '{section_filter}' 的 URL")
|
||||
return
|
||||
|
||||
# 统计
|
||||
stats = {"total": 0, "success": 0, "fail": 0, "skip": 0}
|
||||
by_tool = {}
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 参考图素材批量下载")
|
||||
print(f" 共 {len(entries)} 个 URL")
|
||||
if quick:
|
||||
print(f" 模式：快速（仅 gallery-dl + yt-dlp）")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
for i, entry in enumerate(entries, 1):
|
||||
url = entry["url"]
|
||||
section = entry["section"]
|
||||
dir_name = section_to_dir(section, entry["subsection"])
|
||||
output_dir = os.path.join(output_root, dir_name)
|
||||
|
||||
route = route_url(url, site_rules)
|
||||
tool = route["tool"]
|
||||
rules = route["rules"]
|
||||
|
||||
# quick 模式跳过 BS4/Playwright
|
||||
if quick and tool in ("bs4", "playwright"):
|
||||
print(f" [{i}/{len(entries)}] 跳过 (quick模式): {url[:80]}")
|
||||
stats["skip"] += 1
|
||||
continue
|
||||
|
||||
# 水印警告
|
||||
if rules.get("watermark_warning"):
|
||||
print(f" ⚠ 注意：该站点图片带水印，仅供参考")
|
||||
|
||||
print(f"\n[{i}/{len(entries)}] {section} → {dir_name}")
|
||||
|
||||
stats["total"] += 1
|
||||
by_tool[tool] = by_tool.get(tool, 0) + 1
|
||||
|
||||
downloader = TOOL_DISPATCHER.get(tool, download_bs4)
|
||||
success = downloader(url, output_dir, rules)
|
||||
|
||||
if success:
|
||||
stats["success"] += 1
|
||||
else:
|
||||
stats["fail"] += 1
|
||||
|
||||
# 请求间延迟
|
||||
if i < len(entries):
|
||||
delay = random.uniform(*DELAY_RANGE)
|
||||
time.sleep(delay)
|
||||
|
||||
# 打印统计
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 下载完成！")
|
||||
print(f" 总计: {stats['total']} 个 URL")
|
||||
print(f" 成功: {stats['success']}")
|
||||
print(f" 失败: {stats['fail']}")
|
||||
print(f" 跳过: {stats['skip']}")
|
||||
print(f" 引擎分布: {by_tool}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
|
||||
def dry_run(entries: list[dict]) -> None:
|
||||
"""仅打印 URL 分类结果，不下载"""
|
||||
site_rules = load_site_rules()
|
||||
|
||||
print(f"\n{'='*70}")
|
||||
print(f" Dry Run — URL 分类预览 ({len(entries)} 个 URL)")
|
||||
print(f"{'='*70}\n")
|
||||
|
||||
current_section = ""
|
||||
by_tool = {}
|
||||
by_dir = {}
|
||||
|
||||
for entry in entries:
|
||||
url = entry["url"]
|
||||
section = entry["section"]
|
||||
dir_name = section_to_dir(section, entry["subsection"])
|
||||
route = route_url(url, site_rules)
|
||||
tool = route["tool"]
|
||||
|
||||
by_tool[tool] = by_tool.get(tool, 0) + 1
|
||||
by_dir[dir_name] = by_dir.get(dir_name, 0) + 1
|
||||
|
||||
if section != current_section:
|
||||
current_section = section
|
||||
print(f"\n >> {section}")
|
||||
print(f" {'~'*60}")
|
||||
|
||||
watermark = " [水印]" if route.get("rules", {}).get("watermark_warning") else ""
|
||||
cookies = " [需Cookie]" if route.get("rules", {}).get("cookies_required") else ""
|
||||
print(f" [{tool:>11}] {dir_name:<20} {url[:65]}{watermark}{cookies}")
|
||||
|
||||
print(f"\n{'~'*70}")
|
||||
print(f" 引擎分布: {by_tool}")
|
||||
print(f" 目录分布: {by_dir}")
|
||||
print(f"{'='*70}\n")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI 入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="参考图素材批量下载工具 — 从 markdown 提取 URL 并自动下载图片",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
示例:
|
||||
python download.py 参考图素材链接清单.md
|
||||
python download.py 参考图素材链接清单.md --dry-run
|
||||
python download.py 参考图素材链接清单.md --section 我的暑假
|
||||
python download.py 参考图素材链接清单.md --quick
|
||||
python download.py 参考图素材链接清单.md -o ./output
|
||||
""",
|
||||
)
|
||||
parser.add_argument(
|
||||
"markdown",
|
||||
help="包含 URL 的 markdown 文件路径",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
default=None,
|
||||
help="输出根目录（默认：markdown 文件所在目录）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="仅打印 URL 分类结果，不下载",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--section",
|
||||
default="",
|
||||
help="仅处理包含指定关键词的章节",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--quick",
|
||||
action="store_true",
|
||||
help="快速模式：仅使用 gallery-dl 和 yt-dlp（跳过自定义抓取）",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 检查 markdown 文件
|
||||
md_path = Path(args.markdown)
|
||||
if not md_path.exists():
|
||||
print(f"✗ 文件不存在: {md_path}")
|
||||
sys.exit(1)
|
||||
|
||||
# 解析 URL
|
||||
entries = parse_markdown(str(md_path))
|
||||
if not entries:
|
||||
print("⚠ 未在文件中找到任何 URL")
|
||||
sys.exit(0)
|
||||
|
||||
# 确定输出目录
|
||||
if args.output:
|
||||
output_root = args.output
|
||||
else:
|
||||
output_root = str(md_path.parent)
|
||||
|
||||
# 执行
|
||||
if args.dry_run:
|
||||
dry_run(entries)
|
||||
else:
|
||||
process_urls(
|
||||
entries,
|
||||
output_root=output_root,
|
||||
quick=args.quick,
|
||||
section_filter=args.section,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,6 @@
|
||||
gallery-dl>=1.27
|
||||
yt-dlp>=2025.1
|
||||
requests>=2.31
|
||||
beautifulsoup4>=4.12
|
||||
lxml>=5.0
|
||||
playwright>=1.40
|
||||
@@ -0,0 +1,189 @@
|
||||
{
|
||||
"game.watch.impress.co.jp": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "日系静态页面，2007年老格式，一图一页"
|
||||
},
|
||||
"gamesdb.launchbox-app.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 5,
|
||||
"scroll_to_load": true,
|
||||
"note": "需要等待 JS 渲染，可能有分页"
|
||||
},
|
||||
"www.mobygames.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 4,
|
||||
"scroll_to_load": true,
|
||||
"note": "动态加载截图画廊"
|
||||
},
|
||||
"www.famitsu.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 5,
|
||||
"scroll_to_load": true,
|
||||
"note": "动态加载图片画廊，有懒加载"
|
||||
},
|
||||
"www.gettyimages.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 3,
|
||||
"min_size_kb": 15,
|
||||
"watermark_warning": true,
|
||||
"note": "⚠️ 下载的图片带水印，仅供参考"
|
||||
},
|
||||
"www.alamy.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 3,
|
||||
"watermark_warning": true,
|
||||
"note": "⚠️ 下载的图片带水印，仅供参考"
|
||||
},
|
||||
"www.istockphoto.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 3,
|
||||
"watermark_warning": true,
|
||||
"note": "⚠️ 下载的图片带水印，仅供参考"
|
||||
},
|
||||
"www.awwwards.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 5,
|
||||
"note": "SPA 架构，截图类图片"
|
||||
},
|
||||
"summer-afternoon.vlucendo.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 8,
|
||||
"scroll_to_load": true,
|
||||
"note": "WebGL 项目，需等待渲染完成，可能需要手动截图"
|
||||
},
|
||||
"zhuanlan.zhihu.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 3,
|
||||
"img_selector": "img[data-original]",
|
||||
"use_data_attr": "data-original",
|
||||
"note": "知乎专栏使用懒加载，图片在 data-original 属性中"
|
||||
},
|
||||
"www.zhihu.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 3,
|
||||
"note": "知乎问答页面，需登录才能看完整内容"
|
||||
},
|
||||
"store.steampowered.com": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 4,
|
||||
"scroll_to_load": true,
|
||||
"note": "Steam 策展页面，有动态加载"
|
||||
},
|
||||
"www.lp.zone": {
|
||||
"engine": "playwright",
|
||||
"wait_seconds": 5,
|
||||
"scroll_to_load": true,
|
||||
"note": "论坛帖文，图片较多需翻页/滚动"
|
||||
},
|
||||
"www.pinterest.com": {
|
||||
"engine": "gallery-dl",
|
||||
"cookies_required": true,
|
||||
"note": "需要 Chrome 登录态 cookies"
|
||||
},
|
||||
"jp.pinterest.com": {
|
||||
"engine": "gallery-dl",
|
||||
"cookies_required": true,
|
||||
"note": "需要 Chrome 登录态 cookies"
|
||||
},
|
||||
"www.youtube.com": {
|
||||
"engine": "yt-dlp",
|
||||
"download_thumbnails": true,
|
||||
"skip_video": true,
|
||||
"note": "仅下载缩略图"
|
||||
},
|
||||
"www.instagram.com": {
|
||||
"engine": "gallery-dl",
|
||||
"cookies_required": true,
|
||||
"note": "需要 Chrome 登录态 cookies"
|
||||
},
|
||||
"www.reddit.com": {
|
||||
"engine": "gallery-dl",
|
||||
"note": "Reddit 帖子图片下载"
|
||||
},
|
||||
"www.video-games-museum.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "小型游戏截图数据库，静态页面"
|
||||
},
|
||||
"tinycartridge.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "博客类，简单的 img 标签"
|
||||
},
|
||||
"radii.co": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "文章类页面"
|
||||
},
|
||||
"www.macfilos.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "摄影文章，图片较多"
|
||||
},
|
||||
"david-colson.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "技术博客文章"
|
||||
},
|
||||
"aesthetics.fandom.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "Fandom 维基页面"
|
||||
},
|
||||
"indie-hive.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "独立游戏介绍页面"
|
||||
},
|
||||
"www.gamedeveloper.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "游戏开发文章"
|
||||
},
|
||||
"gamerswithglasses.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "游戏推荐文章"
|
||||
},
|
||||
"bordersdown.net": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "论坛回顾帖"
|
||||
},
|
||||
"edmoise.sites.clemson.edu": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 15,
|
||||
"note": "学术照片集页面"
|
||||
},
|
||||
"pioneerproject.net": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "博客攻略页面"
|
||||
},
|
||||
"xneo.jp": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "日站攻略页面"
|
||||
},
|
||||
"www.mexigame.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "游戏介绍页面"
|
||||
},
|
||||
"sghexport.shobserver.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "新闻文章页面"
|
||||
},
|
||||
"www.ourjiangsu.com": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "新闻文章页面"
|
||||
},
|
||||
"zjnews.zjol.com.cn": {
|
||||
"engine": "bs4",
|
||||
"min_size_kb": 10,
|
||||
"note": "新闻文章页面"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user