0.2.6-dev: /data/assets.json and /data/local-assets.json are served with content-hashed URLs (/assets/~<hash>/…, one hash per Spine model, the committed files unchanged); the ~hash routes of /assets/ /fonts/ /media/ answer immutable when the hash names the file and the current file with no-cache otherwise; content ETags — host bandwidth layer 1.2; golden: unchanged
This commit is contained in:
sganggs committed
1 parent
521dd984e7
commit
893d17e725
12 files changed
+571
-34
No files matched your search
@@ -124,6 +124,7 @@ npm start # 启动服务器：http://localhost:3000
|
||||
| `DEBUG` | 空 | 设为任意值输出详细日志 |
|
||||
| `SP_NO_BROWSER` | 空 | 设为 `1` 时启动脚本不自动打开浏览器 |
|
||||
| `SP_BOT_EMOTES` | 空 | AI 队友的对局表情回应（战斗、整备、合成、联防、收礼时即刻的 `m.emote` 广播）；设为 `0` 时整体关闭 |
|
||||
| `SP_ASSET_INDEX` | 空 | 素材地址带内容哈希、浏览器长期缓存（0.2.6 起默认开启，哈希存在 `.cache/asset-hashes.json`）；设为 `0` 时关闭，素材清单原样下发 |
|
||||
|
||||
设置方式：macOS / Linux `PORT=8080 npm start`；PowerShell `$env:PORT=8080; npm start`；cmd `set "PORT=8080" && npm start`。健康检查：`GET /healthz`。
|
||||
|
||||
|
||||
@@ -77,7 +77,7 @@ so old imports keep working: `public/js/ui/gameLogic.js` (`public/js/ui/gameLogi
|
||||
| path | what |
|
||||
|---|---|
|
||||
| `server/index.js` | the process entry (`npm start`); `startServer()` wires `server/http/` |
|
||||
| `server/http/` | `config.js` (environment), `websocket.js` (sessions, `/ws`, permessage-deflate), `static.js` (the mounts), `media.js`, `files.js` (MIME, gzip, ETag, ranges), `buildTag.js`, `routes.js` (`/healthz`), `traffic.js` (bytes sent by category and on the WebSocket, `/healthz?traffic=1`), `assetIndex.js` (the content hashes of `public/assets/` and `public/fonts/`, cached in `.cache/asset-hashes.json`), `common.js`, `boot.js` (a pending update package first, banner, shutdown) |
|
||||
| `server/http/` | `config.js` (environment), `websocket.js` (sessions, `/ws`, permessage-deflate), `static.js` (the mounts, the `~<hash>` asset routes), `media.js`, `files.js` (MIME, gzip, ETag, ranges), `buildTag.js`, `routes.js` (`/healthz`), `traffic.js` (bytes sent by category and on the WebSocket, `/healthz?traffic=1`), `assetIndex.js` (the content hashes of `public/assets/` and `public/fonts/`, cached in `.cache/asset-hashes.json`), `assetViews.js` (the served `/data/assets.json` and `/data/local-assets.json`: every art URL with its hash segment; ASSETS.md), `common.js`, `boot.js` (a pending update package first, banner, shutdown) |
|
||||
| `server/net.js` | sessions and reconnect tokens, rate limits, message validation |
|
||||
| `server/lobby.js` | rooms, seats, AI seats, spectators; starts a `Match` |
|
||||
| `server/data.js` | loads `data/*.json` once (frozen) |
|
||||
@@ -131,7 +131,8 @@ data/*.json ──────────┬─▶ server/data.js ────
|
||||
├─▶ public/js/data.js ──────────▶ the UI
|
||||
└─▶ public/js/battle/runner.js ─▶ the browser's battles (setSimData)
|
||||
data/i18n/en.json ──────▶ public/js/data.js ──────────▶ the English game texts
|
||||
public/assets/ ─────────▶ public/js/render/, public/js/audio.js (URLs listed in data/assets.json)
|
||||
public/assets/ ─────────▶ public/js/render/, public/js/audio.js (URLs listed in data/assets.json,
|
||||
served with content-hash segments: server/http/assetViews.js)
|
||||
```
|
||||
|
||||
- **Game data.** `tools/build-data.mjs` downloads the official zh_CN tables into `.cache/gamedata/` (or reads them
|
||||
|
||||
+34
-1
@@ -215,7 +215,7 @@ The `stem` of a Spine model is the upstream file name. Two examples: `char_107_l
|
||||
|
||||
## Manifest schema (`data/assets.json`)
|
||||
|
||||
All paths are URL paths relative to the site root, for example `/assets/char/avatar/char_002_amiya.png`. **Only entries whose files exist on disk are emitted.** When a key is missing, the client should use its fallback (research 07 §5.6).
|
||||
All paths are URL paths relative to the site root, for example `/assets/char/avatar/char_002_amiya.png` — in the committed file; a server hands them out with a content-hash segment ("The committed manifest and the served manifests" below). **Only entries whose files exist on disk are emitted.** When a key is missing, the client should use its fallback (research 07 §5.6).
|
||||
|
||||
```js
|
||||
{
|
||||
@@ -470,6 +470,39 @@ Other renderer rules from research 07 §5.4–5.5:
|
||||
- **Missing unit SFX:** use `audio.sfx.battle.enemyHit` or a WebAudio blip (research 07 §6.4).
|
||||
- **Bond icons:** if the real glyph ever fails, the stored file is the nation camp logo.
|
||||
|
||||
## The committed manifest and the served manifests
|
||||
|
||||
`data/assets.json` in git (and as `tools/fetch-assets.mjs` writes it) and `data/local-assets.json` (as
|
||||
`tools/local-extract` writes it) hold plain URLs. Since 0.2.6 (host bandwidth, layer 1) the server hands out a view of
|
||||
each (`server/http/assetIndex.js`, `server/http/assetViews.js`):
|
||||
|
||||
- Shortly after the start (1 s) the server hashes every file under `public/assets/` and `public/fonts/` in the
|
||||
background: SHA-256, the first 16 hex digits. The hashes are kept in `.cache/asset-hashes.json` by path + size +
|
||||
mtime, so a restart only stats the files and reads none of the unchanged ones (12,691 files / 724 MB on the
|
||||
maintainer's machine: 1.3 s for the first build, 0.25 s after that). A file whose size or mtime changed is hashed
|
||||
again; a re-extraction, `setup --force` or a Docker rebuild then costs the players only the files whose bytes changed.
|
||||
- `GET /data/assets.json` and `GET /data/local-assets.json` answer with every art URL carrying its file's hash as the
|
||||
first segment: `/assets/char/avatar/char_003_kalts.png` → `/assets/~398f32e2ff78d9e6/char/avatar/char_003_kalts.png`,
|
||||
`/fonts/~<hash>/…`. Audio keeps its `/assets/~<hash>/audio/….mp3` form; `public/js/media.js` asks for it as
|
||||
`/media/~<hash>/…` (the download-manager route, `shared/media.js`).
|
||||
- A Spine model's skeleton, atlas and pages carry one hash, the model's (SHA-256 of the files' hashes): pixi-spine finds
|
||||
the atlas by the skeleton's name and the pages next to the atlas, so the URLs share a folder. In `data/assets.json`
|
||||
every `{ skel, atlas, textures }` object is a model; in `data/local-assets.json` every `spine/…` group (the files a
|
||||
`spineLocal` record names).
|
||||
- The server strips the segment and serves the file at the plain path. When the segment names the file's current bytes
|
||||
(its own hash, or its model's) the answer is `Cache-Control: public, max-age=31536000, immutable`; any other segment —
|
||||
a page loaded before an update, a URL derived from another file's — gets the current file with `no-cache`, so an old
|
||||
page keeps working. The ETag of every indexed file is its content hash (`"398f32e2ff78d9e6"`, plus `-gz` when
|
||||
compressed), on the plain URL too.
|
||||
- A path segment, never `?v=`: `assets.js validSpine` matches `/\.skel$/`, `audio.js SKILL_MODE_FILE` the `.mp3` ending,
|
||||
pixi-spine takes the atlas name from the skeleton's basename and resolves the pages against the atlas's folder — a
|
||||
query string survives none of that.
|
||||
- A URL the index has no hash for (the file is missing, or not hashed yet) stays plain; plain URLs work as before
|
||||
(`public, max-age=86400`, revalidated after a day).
|
||||
- Until the index is ready, with `SP_ASSET_INDEX=0`, or on a static host without this server, the manifests go out as
|
||||
committed. The views are rebuilt in memory when a manifest or the hashes change; neither file is ever written by the
|
||||
server, and the `hash` field is the committed one. The index lives in `.cache/` only and is never packaged.
|
||||
|
||||
## Verification
|
||||
|
||||
`node --test test/assets.test.js` covers the pure helpers: the resolver, the atlas normalizer, the format sniffers, WOFF2, audio banks, the plan id sets, the downloader against a fake network, and the self-heal of corrupt skeletons.
|
||||
|
||||
+2
-1
@@ -9,7 +9,7 @@
|
||||
|---|---|
|
||||
| 服务器 CPU | 战斗在各玩家浏览器里模拟（DESIGN §14），服务器只负责回合、经济和校验：**每个房间每个作战回合约 1 ms CPU**。AI 队友 / 掉线玩家的战场由服务器模拟：作战开始时 3 个 AI 战场在开发机上约 0.2–0.5 s CPU，小主机上可能要几秒（分成 8 ms 小片执行，不会卡住其他房间）。`SP_VERIFY=all` 会复算每个真人战场，CPU 明显增加，小主机建议保持 `off` 或 `sample`。 |
|
||||
| 服务器内存 | 空闲约 100 MB，每个进行中的对局再增加几 MB。 |
|
||||
| 网络 | 4 人对局中服务器每回合下行约 0.25 MB（DESIGN §14 实测；0.2.6 起 1 KB 以上的 WebSocket 消息用 permessage-deflate 压缩，实测一局单人对局的服务器消息从 169 KB 降到 63 KB，约为原来的四成）。首次进入游戏时浏览器要从主机下载所需的图片 / Spine 模型 / 音频（按需加载，之后走浏览器缓存），公网隧道带宽小时第一次会慢一些。 |
|
||||
| 网络 | 4 人对局中服务器每回合下行约 0.25 MB（DESIGN §14 实测；0.2.6 起 1 KB 以上的 WebSocket 消息用 permessage-deflate 压缩，实测一局单人对局的服务器消息从 169 KB 降到 63 KB，约为原来的四成）。首次进入游戏时浏览器要从主机下载所需的图片 / Spine 模型 / 音频（按需加载，之后走浏览器缓存），公网隧道带宽小时第一次会慢一些。0.2.6 起服务器启动后在后台给素材算内容哈希（第一次约 1–20 秒，看机器和硬盘；之后只检查大小和修改时间），下发的素材清单里每个地址带上哈希：浏览器把图片、模型和音频缓存一年，服务器更新、重新解压或 Docker 重建之后，玩家只重新下载内容真的变了的文件。哈希存在 `.cache/asset-hashes.json`，删掉会重新计算；目录不可写（例如只读的容器）时只在内存里算。 |
|
||||
| 磁盘 | 素材约 550 MB（`public/assets`，含中文约 65 MB、日文约 85 MB 两套干员语音）+ 依赖约 125 MB（`node_modules`；整合包只带运行依赖，约 65 MB）；可选的本地提取约 40 MB（`.venv-extract`）+ 70 MB 贴图（见第 6 节）。完整包解压后约 710 MB。 |
|
||||
| 玩家设备 | 支持 WebGL 的现代浏览器（Chrome / Edge / Firefox / Safari 最新版），电脑或手机平板（横屏）。老旧设备可在设置里调低画质或访问 `/?board=2d`。 |
|
||||
|
||||
@@ -295,6 +295,7 @@ services:
|
||||
| 表情显示成默认图标、「玩法说明」只有文字要点 | 素材没下载完整：重新运行 `node tools/setup.mjs`（表情和教程图随其他素材一起从公开镜像下载，不需要客户端）；缺失明细在 `.cache/assets-report.json` |
|
||||
| 本地提取失败 | 游戏照常运行，只是第 6 节表格里的几样换成替代样式。确认客户端已下载全部资源；Python 版本太新导致依赖安装失败时，安装 Python 3.12 后删除 `.venv-extract` 再运行 `node tools/setup.mjs --local` |
|
||||
| 3D 棋盘没出现 | 需要本地提取的棋盘贴图（`node tools/doctor.mjs` 会显示「3D 棋盘可用」），以及支持 WebGL2 的浏览器。没有客户端的服务器可以从同一版本的整合包复制本地素材（第 6 节） |
|
||||
| 素材地址里的 `~` 加一串字母数字 | `/assets/~<哈希>/…` 是 0.2.6 起的带版本地址（见第 0 节「网络」），浏览器可以长期缓存，属于正常现象。怀疑它导致问题时，可以用 `SP_ASSET_INDEX=0` 启动服务器临时关闭（清单改回普通地址），并请在 issue 里说明 |
|
||||
| 想知道开服电脑上传了多少流量 | 浏览器打开 `http://localhost:3000/healthz?traffic=1`（换成你的端口）：`traffic.http` 是服务器启动以来各类内容（`assets` 图片和模型、`audio` 音频、`code` 页面和脚本、`data` 游戏数据和素材清单）发出的字节数和请求数（压缩后的实际字节，`notModified` 是只回了「没变」的 304），`traffic.ws` 是 WebSocket 的线上字节（`wireOut` / `wireIn`，压缩后）和压缩前的消息字节（`payloadOut` / `payloadIn`）。数字只在内存里，重启后从 0 开始。玩家那边：设置 →「问题反馈」复制的诊断里有一行 `traffic:`，是这个页面从打开起下载的字节数（`cached` 是浏览器缓存直接给的，不占流量；`external` 是其他网站，例如 Google 字体） |
|
||||
| 断线 | 同盟模拟 10 分钟内、独立模拟 24 小时内（`config.constants.singleReconnectTime`）用同一浏览器重新打开页面，自动回到原座位。同盟掉线期间按原阵容自动作战、到时自动准备（不会代为购买；想让 AI 代打请用「离开模拟 → 暂离（AI 托管）」）；独立模拟不计时，等你回来 |
|
||||
|
||||
|
||||
@@ -44,13 +44,16 @@ server/
|
||||
config.js ROOT, served dirs, env (PORT, HOST, TRUST_PROXY, DEBUG), options handed to net.js / lobby.js
|
||||
websocket.js session wiring (SessionRegistry → Lobby → Network), WebSocket upgrade at /ws (64 KB frames,
|
||||
permessage-deflate from 1 KB)
|
||||
static.js mounts / → public, /data/, /shared/, /sim/ (.js only), the /data.js browser stand-in, path guards
|
||||
static.js mounts / → public, /data/, /shared/, /sim/ (.js only), the /data.js browser stand-in, path guards,
|
||||
the ~<hash> segment of /assets/ /fonts/ /media/ URLs stripped (immutable when it names the file)
|
||||
media.js /media/… extension-less audio → public/assets/audio
|
||||
files.js one file → response: MIME, gzip (.skel/.atlas/.json/.js/.css …), ETag / 304, cache policy, ranges
|
||||
files.js one file → response: MIME, gzip (.skel/.atlas/.json/.js/.css …), ETag (content hash when known) / 304,
|
||||
cache policy, ranges; serveBuffer for the served manifests
|
||||
buildTag.js build tag of the served browser runtime (/healthz build, public/js/ui/buildGuard.js)
|
||||
routes.js request listener: 414 / 400 / 405 guards, GET /healthz (?traffic=1: the counters), then static files
|
||||
traffic.js traffic counters: HTTP body bytes by category (assets / audio / code / data), WebSocket wire / payload
|
||||
assetIndex.js content hashes (SHA-256, 16 hex) of public/assets + public/fonts, background build, .cache/asset-hashes.json
|
||||
assetViews.js the served /data/assets.json + /data/local-assets.json: art URLs as /assets/~<hash>/… (immutable when it matches)
|
||||
common.js security headers, URL split, error page, JSON replies, bare 400 for unparseable requests
|
||||
boot.js boot banner (Local / LAN / tunnel URLs), port-in-use hint, graceful SIGINT / SIGTERM
|
||||
net.js session registry, send helpers, per-connection rate limit, message validation (uses shared/protocol.js)
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
// server/http/assetViews.js — the served manifests (host bandwidth, layer 1; docs/ASSETS.md "The served manifests"):
|
||||
// data/assets.json and data/local-assets.json as this server hands them out, every art URL carrying the content hash
|
||||
// of its file as its first segment (shared/assetPaths.js) — `/assets/char/avatar/x.png` →
|
||||
// `/assets/~<hash>/char/avatar/x.png`, `/fonts/~<hash>/…` — so the browser keeps the file for a year and asks again
|
||||
// only when the bytes change. Audio keeps the `/assets/~<hash>/audio/….mp3` form: public/js/media.js turns it into
|
||||
// `/media/~<hash>/…` when it fetches it.
|
||||
//
|
||||
// * A Spine model's skeleton, atlas and pages carry ONE hash, that of the model (assetIndex.js models): pixi-spine
|
||||
// finds the atlas by the skeleton's name and the pages next to the atlas, so their URLs share a folder. A model is
|
||||
// every `{ skel, atlas, textures }` object of data/assets.json (chars' front / back, tokens, enemies) and every
|
||||
// `spine/…` group of data/local-assets.json (the files a `spineLocal` record names).
|
||||
// * A URL the index has no hash for (not on disk, not hashed yet) stays as it is.
|
||||
// * The committed data/assets.json is never written: a view is built in memory from the file and the index, again
|
||||
// when either changes (the file's size / mtime, the index's generation), and once after every index build, so the
|
||||
// index knows the model lists before any page asks.
|
||||
// * No index, or not ready yet: no view (`get` → null) and the files go out as they are.
|
||||
|
||||
import fsp from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { hashedPath } from '../../shared/assetPaths.js';
|
||||
import { hash16 } from './assetIndex.js';
|
||||
import { noopLog } from './config.js';
|
||||
|
||||
/** The manifests that have a view, by their name under data/. */
|
||||
export const VIEW_MANIFESTS = Object.freeze(['assets.json', 'local-assets.json']);
|
||||
|
||||
const isObj = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
|
||||
const ART_URL = /^\/(?:assets|fonts)\/./;
|
||||
|
||||
/** URL path of a manifest → the index's rel path ('/assets/a%20b.png' → 'assets/a b.png'), or null. */
|
||||
export function relOf(url) {
|
||||
if (typeof url !== 'string' || !ART_URL.test(url) || /[?#]/.test(url)) return null;
|
||||
let p;
|
||||
try { p = decodeURIComponent(url); } catch { return null; }
|
||||
if (p.includes('\0') || p.includes('\\')) return null;
|
||||
const segs = p.slice(1).split('/');
|
||||
if (segs.some((s) => s === '' || s === '.' || s === '..')) return null;
|
||||
return segs.join('/');
|
||||
}
|
||||
|
||||
/** Is `o` a Spine object of data/assets.json (`{ skel: '/….skel', atlas, textures? }`)? */
|
||||
const isSpine = (o) => isObj(o) && typeof o.skel === 'string' && o.skel.startsWith('/') && /\.skel$/.test(o.skel);
|
||||
|
||||
/** The URLs of a Spine object: skeleton, atlas, pages (the `<skel name>.png` when it lists none). */
|
||||
function spineUrls(o) {
|
||||
const out = [o.skel];
|
||||
if (typeof o.atlas === 'string' && o.atlas) out.push(o.atlas);
|
||||
const pages = Array.isArray(o.textures) ? o.textures.filter((u) => typeof u === 'string' && u) : [];
|
||||
out.push(...(pages.length ? pages : [o.skel.replace(/\.skel$/, '.png')]));
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The served copy of a manifest and its model lists. `index`: hashOf(rel) / modelHash(rels) of assetIndex.js.
|
||||
* @param {any} doc the parsed manifest (not changed)
|
||||
* @param {'assets.json'|'local-assets.json'} name
|
||||
* @param {{ hashOf: (rel: string) => string | null, modelHash: (rels: string[]) => string | null }} index
|
||||
* @returns {{ doc: any, models: string[][], hashed: number, plain: number }}
|
||||
*/
|
||||
export function rewriteManifest(doc, name, index) {
|
||||
const models = [];
|
||||
let hashed = 0;
|
||||
let plain = 0;
|
||||
const own = (url) => {
|
||||
const rel = relOf(url);
|
||||
const h = rel ? index.hashOf(rel) : null;
|
||||
if (h) { hashed++; return hashedPath(url, h); }
|
||||
if (rel) plain++;
|
||||
return url;
|
||||
};
|
||||
/** All URLs of one model with the model's hash (or all left plain when a member is unknown). */
|
||||
const model = (urls) => {
|
||||
const rels = urls.map(relOf);
|
||||
if (rels.some((r) => !r)) return null;
|
||||
models.push(rels);
|
||||
const h = index.modelHash(rels);
|
||||
if (!h) { plain += urls.length; return null; }
|
||||
hashed += urls.length;
|
||||
return h;
|
||||
};
|
||||
const walk = (v) => {
|
||||
if (typeof v === 'string') return ART_URL.test(v) ? own(v) : v;
|
||||
if (Array.isArray(v)) return v.map(walk);
|
||||
if (!isObj(v)) return v;
|
||||
if (name === 'assets.json' && isSpine(v)) {
|
||||
const h = model(spineUrls(v));
|
||||
const out = {};
|
||||
for (const [k, x] of Object.entries(v)) {
|
||||
if (h && k === 'skel') out[k] = hashedPath(x, h);
|
||||
else if (h && k === 'atlas' && typeof x === 'string') out[k] = hashedPath(x, h);
|
||||
else if (h && k === 'textures' && Array.isArray(x)) out[k] = x.map((u) => (typeof u === 'string' ? hashedPath(u, h) : u));
|
||||
else out[k] = walk(x);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
const out = {};
|
||||
for (const [k, x] of Object.entries(v)) out[k] = walk(x);
|
||||
return out;
|
||||
};
|
||||
if (name === 'local-assets.json' && isObj(doc) && isObj(doc.groups)) {
|
||||
const groups = {};
|
||||
for (const [g, entries] of Object.entries(doc.groups)) {
|
||||
if (g.startsWith('spine/') && isObj(entries)) {
|
||||
const names = Object.keys(entries).filter((n) => isObj(entries[n]) && typeof entries[n].path === 'string').sort();
|
||||
const h = names.length ? model(names.map((n) => entries[n].path)) : null;
|
||||
groups[g] = {};
|
||||
for (const [n, e] of Object.entries(entries)) {
|
||||
groups[g][n] = h && isObj(e) && typeof e.path === 'string' ? { ...e, path: hashedPath(e.path, h) } : walk(e);
|
||||
}
|
||||
} else groups[g] = walk(entries);
|
||||
}
|
||||
return { doc: { ...walk({ ...doc, groups: null }), groups }, models, hashed, plain };
|
||||
}
|
||||
return { doc: walk(doc), models, hashed, plain };
|
||||
}
|
||||
|
||||
/**
|
||||
* The views of one server.
|
||||
* @param {{ dataDir: string, index: ReturnType<typeof import('./assetIndex.js').createAssetIndex>, log?: object }} opts
|
||||
*/
|
||||
export function createManifestViews({ dataDir, index, log = noopLog }) {
|
||||
const dir = path.resolve(dataDir);
|
||||
/** @type {Map<string, { key: string, view: { body: Buffer, hash: string, hashed: number, plain: number } | null, models: string[][] }>} */
|
||||
const cache = new Map();
|
||||
/** @type {Map<string, Promise<any>>} */
|
||||
const pending = new Map();
|
||||
const warned = new Set();
|
||||
|
||||
const pushModels = () => index.setModels([...cache.values()].flatMap((c) => c.models));
|
||||
|
||||
/** Build (or reuse) the view of `name`; null without the file, while the index is not ready or empty, or for a broken file. */
|
||||
async function build(name) {
|
||||
if (!index.ready || !index.size) return null;
|
||||
const file = path.join(dir, name);
|
||||
const st = await fsp.stat(file).catch(() => null);
|
||||
if (!st || !st.isFile()) {
|
||||
if (cache.delete(name)) pushModels();
|
||||
return null;
|
||||
}
|
||||
const key = `${index.generation}|${st.size}|${st.mtimeMs}`;
|
||||
const hit = cache.get(name);
|
||||
if (hit && hit.key === key) return hit.view;
|
||||
let doc;
|
||||
try {
|
||||
doc = JSON.parse(await fsp.readFile(file, 'utf8'));
|
||||
} catch (e) {
|
||||
if (!warned.has(name)) { warned.add(name); log.warn?.(`[assets] data/${name} unreadable (${e?.message || e}); served as it is`); }
|
||||
cache.set(name, { key, view: null, models: [] });
|
||||
pushModels();
|
||||
return null;
|
||||
}
|
||||
// one synchronous step: the view and the model lists the index checks a model's URLs against
|
||||
const { doc: out, models, hashed, plain } = rewriteManifest(doc, name, index);
|
||||
const body = Buffer.from(JSON.stringify(out));
|
||||
const view = { body, hash: hash16(body), hashed, plain };
|
||||
cache.set(name, { key, view, models });
|
||||
pushModels();
|
||||
return view;
|
||||
}
|
||||
|
||||
/** The view of `name` (one build at a time per manifest). */
|
||||
function get(name) {
|
||||
if (!VIEW_MANIFESTS.includes(name)) return Promise.resolve(null);
|
||||
const running = pending.get(name);
|
||||
if (running) return running;
|
||||
const p = build(name).catch((e) => { log.error?.('[assets] manifest view failed', e); return null; }).finally(() => pending.delete(name));
|
||||
pending.set(name, p);
|
||||
return p;
|
||||
}
|
||||
|
||||
const rebuildAll = () => Promise.all(VIEW_MANIFESTS.map(get));
|
||||
// after every build that changed the hashes (and the first one): the views, and with them the index's model lists
|
||||
index.onChange(() => { rebuildAll(); });
|
||||
index.whenReady().then(() => rebuildAll(), () => {});
|
||||
|
||||
return {
|
||||
get,
|
||||
rebuildAll,
|
||||
stats() {
|
||||
const out = {};
|
||||
for (const [name, c] of cache) out[name] = c.view ? { bytes: c.view.body.length, hashed: c.view.hashed, plain: c.view.plain } : null;
|
||||
return out;
|
||||
},
|
||||
};
|
||||
}
|
||||
+45
-9
@@ -1,9 +1,11 @@
|
||||
// server/http/files.js — one file on disk → an HTTP response (static.js and media.js decide which file):
|
||||
// server/http/files.js — one file on disk (serveFile) or one body in memory (serveBuffer, the served manifests of
|
||||
// assetViews.js) → an HTTP response (static.js and media.js decide which file):
|
||||
// MIME types incl. .mjs/.js text/javascript, .skel application/octet-stream, .atlas text/plain;
|
||||
// gzip for text-like types, .skel and uncompressed fonts when the client accepts it (small files are
|
||||
// compressed once and cached in memory); strong ETag + Last-Modified with 304s; Cache-Control
|
||||
// (html & code/data: no-cache + revalidate; public/assets|fonts|vendor: 1 day; any `?v=` URL: immutable);
|
||||
// single byte-range requests (206/416, used by <audio>).
|
||||
// compressed once and cached in memory); strong ETag + Last-Modified with 304s — the ETag is the content hash
|
||||
// when the caller has one (assetIndex.js), else size + mtime; Cache-Control (html & code/data: no-cache +
|
||||
// revalidate; public/assets|fonts|vendor: 1 day; a URL whose content-hash segment matches the file (static.js,
|
||||
// media.js): immutable for a year; any `?v=` URL: immutable); single byte-range requests (206/416, used by <audio>).
|
||||
|
||||
import fs from 'node:fs';
|
||||
import fsp from 'node:fs/promises';
|
||||
@@ -84,13 +86,17 @@ export class GzipCache {
|
||||
|
||||
/** @returns {Promise<Buffer>} gzip of the file identified by (path, size, mtime) */
|
||||
get(absPath, stat) {
|
||||
const key = `${absPath}\0${stat.size}\0${stat.mtimeMs}`;
|
||||
return this.compressed(`${absPath}\0${stat.size}\0${stat.mtimeMs}`, () => fsp.readFile(absPath));
|
||||
}
|
||||
|
||||
/** @returns {Promise<Buffer>} gzip of the body `read()` returns, cached under `key` (one compression per key at a time) */
|
||||
compressed(key, read) {
|
||||
const hit = this.map.get(key);
|
||||
if (hit) { this.map.delete(key); this.map.set(key, hit); return Promise.resolve(hit); }
|
||||
const pending = this.inflight.get(key);
|
||||
if (pending) return pending;
|
||||
const p = (async () => {
|
||||
const raw = await fsp.readFile(absPath);
|
||||
const raw = await read();
|
||||
const gz = await gzipAsync(raw, { level: 6 });
|
||||
this.store(key, gz);
|
||||
return gz;
|
||||
@@ -190,20 +196,25 @@ function cacheControlFor(ext, mountName, segments, query) {
|
||||
return 'no-cache';
|
||||
}
|
||||
|
||||
export async function serveFile(req, res, absPath, stat, mountName, segments, query, gzipCache, log) {
|
||||
/**
|
||||
* One file → the response.
|
||||
* @param {{ contentHash?: string | null, cacheControl?: string | null }} [opts] contentHash: the file's content hash
|
||||
* (assetIndex.js) — the ETag then; cacheControl: overrides the policy of its path (a content-hashed URL)
|
||||
*/
|
||||
export async function serveFile(req, res, absPath, stat, mountName, segments, query, gzipCache, log, opts = {}) {
|
||||
const ext = path.extname(absPath).toLowerCase();
|
||||
const type = MIME[ext] || 'application/octet-stream';
|
||||
const compressible = COMPRESSIBLE.has(ext);
|
||||
const rangeHeader = req.headers.range;
|
||||
const useGzip = compressible && stat.size >= GZIP_MIN_BYTES && !rangeHeader && acceptsGzip(req.headers['accept-encoding']);
|
||||
const baseTag = `${stat.size.toString(16)}-${Math.floor(stat.mtimeMs).toString(16)}`;
|
||||
const baseTag = opts.contentHash || `${stat.size.toString(16)}-${Math.floor(stat.mtimeMs).toString(16)}`;
|
||||
const etag = `"${baseTag}${useGzip ? '-gz' : ''}"`;
|
||||
const lastModified = stat.mtime.toUTCString();
|
||||
const isHead = req.method === 'HEAD';
|
||||
|
||||
const headers = {
|
||||
'Content-Type': type,
|
||||
'Cache-Control': cacheControlFor(ext, mountName, segments, query),
|
||||
'Cache-Control': opts.cacheControl || cacheControlFor(ext, mountName, segments, query),
|
||||
ETag: etag,
|
||||
'Last-Modified': lastModified,
|
||||
};
|
||||
@@ -253,6 +264,31 @@ export async function serveFile(req, res, absPath, stat, mountName, segments, qu
|
||||
await streamTo(fs.createReadStream(absPath, { start, end }), res, log);
|
||||
}
|
||||
|
||||
/**
|
||||
* One body in memory → the response (the served manifests): its ETag from `hash`, gzip when accepted (cached under
|
||||
* `key`), 304 on a matching If-None-Match, no ranges.
|
||||
* @param {{ body: Buffer, hash: string, type: string, cacheControl: string, key: string, cache: GzipCache }} o
|
||||
*/
|
||||
export async function serveBuffer(req, res, { body, hash, type, cacheControl, key, cache }) {
|
||||
const useGzip = body.length >= GZIP_MIN_BYTES && acceptsGzip(req.headers['accept-encoding']);
|
||||
const etag = `"${hash}${useGzip ? '-gz' : ''}"`;
|
||||
const headers = { 'Content-Type': type, 'Cache-Control': cacheControl, ETag: etag, Vary: 'Accept-Encoding' };
|
||||
const inm = req.headers['if-none-match'];
|
||||
if (typeof inm === 'string' && (inm.trim() === '*' || inm.split(',').some((t) => stripWeak(t) === etag))) {
|
||||
res.writeHead(304, headers);
|
||||
res.end();
|
||||
return;
|
||||
}
|
||||
let out = body;
|
||||
if (useGzip) {
|
||||
out = await cache.compressed(`${key}\0${hash}`, async () => body);
|
||||
headers['Content-Encoding'] = 'gzip';
|
||||
}
|
||||
headers['Content-Length'] = out.length;
|
||||
res.writeHead(200, headers);
|
||||
res.end(req.method === 'HEAD' ? undefined : out);
|
||||
}
|
||||
|
||||
async function streamTo(src, res, log, transform) {
|
||||
try {
|
||||
if (transform) await pipeline(src, transform, res);
|
||||
|
||||
+10
-3
@@ -6,6 +6,7 @@ import path from 'node:path';
|
||||
import { AUDIO_EXTS } from '../../shared/media.js';
|
||||
import { sendError } from './common.js';
|
||||
import { serveFile } from './files.js';
|
||||
import { hashedFileOpts } from './assetIndex.js';
|
||||
|
||||
/**
|
||||
* Extension-less audio route: `/media/bgm/act1` → `public/assets/audio/bgm/act1.mp3`.
|
||||
@@ -15,8 +16,12 @@ import { serveFile } from './files.js';
|
||||
* Requests for the direct `/assets/audio/…` URLs keep working (they are the fallback for plain static hosts).
|
||||
* `MEDIA_PREFIX` / `AUDIO_EXTS` live in `shared/media.js`: the browser decides which URLs to rewrite with the
|
||||
* same two values, and they must not drift apart.
|
||||
* `/media/~<hash>/bgm/act1` (a served manifest's `/assets/~<hash>/audio/bgm/act1.mp3`, public/js/media.js): static.js
|
||||
* strips the segment and hands it over as `urlHash`; the answer is `immutable` when it names the resolved file's bytes
|
||||
* (assetIndex.js), else `no-cache`, and the ETag is the file's content hash when the index has it.
|
||||
* @param {{ urlHash?: string | null, assetIndex?: ReturnType<typeof import('./assetIndex.js').createAssetIndex> | null }} [hashing]
|
||||
*/
|
||||
export async function serveMedia(req, res, rest, query, publicDir, gzipCache, log) {
|
||||
export async function serveMedia(req, res, rest, query, publicDir, gzipCache, log, { urlHash = null, assetIndex = null } = {}) {
|
||||
const root = path.join(path.resolve(publicDir), 'assets', 'audio');
|
||||
const segments = String(rest || '').split('/').filter((s) => s.length > 0);
|
||||
if (!segments.length || rest.endsWith('/')) { sendError(req, res, 404, '页面不存在 · Not found'); return; }
|
||||
@@ -44,9 +49,11 @@ export async function serveMedia(req, res, rest, query, publicDir, gzipCache, lo
|
||||
} catch { continue; }
|
||||
if (!stat.isFile()) continue;
|
||||
// serveFile decides Content-Type from the resolved name (`.mp3` → audio/mpeg) — Range/ETag handling is shared.
|
||||
// Cache policy is that of the public path the client would otherwise have asked for (`/assets/audio/…`, 1 day).
|
||||
// Cache policy is that of the public path the client would otherwise have asked for (`/assets/audio/…`, 1 day),
|
||||
// or that of its content-hashed form.
|
||||
const rel = ['assets', 'audio', ...segments.slice(0, -1), stem + ext].join('/');
|
||||
// eslint-disable-next-line no-await-in-loop
|
||||
await serveFile(req, res, absPath, stat, 'public', ['assets', 'audio', ...segments], query, gzipCache, log);
|
||||
await serveFile(req, res, absPath, stat, 'public', ['assets', 'audio', ...segments], query, gzipCache, log, hashedFileOpts(assetIndex, rel, stat, urlHash));
|
||||
return;
|
||||
}
|
||||
sendError(req, res, 404, '页面不存在 · Not found');
|
||||
|
||||
+37
-8
@@ -15,16 +15,27 @@
|
||||
// /packs/index.json → the content packs of this server (server/packs.js: the language packs of
|
||||
// public/i18n/ and the pack folders of packs/, re-read when they change; never cached)
|
||||
// /packs/<id>/<file> → a file of a pack folder — only one its manifest names (server/packs.js servable)
|
||||
// /assets/~<hash>/…, /fonts/~<hash>/…, /media/~<hash>/… → the same files: the content-hash segment
|
||||
// (shared/assetPaths.js) is stripped. A hash that names the file's current bytes (assetIndex.js
|
||||
// urlHashOk: its own hash, or its Spine model's) answers `immutable` for a year; any other hash —
|
||||
// an old page, a URL derived from another file's — the current file with `no-cache`
|
||||
// /data/assets.json, /data/local-assets.json → the served manifests (assetViews.js: every art URL with its
|
||||
// content-hash segment; no-cache, the ETag of the view) once the asset index is ready, the files as
|
||||
// they are before that or without an index
|
||||
//
|
||||
// Traversal & dotfile protection, a directory without its trailing slash → 301, 404 page; an absent
|
||||
// data/local-assets.json is answered with an empty manifest. Files go out through files.js serveFile.
|
||||
// Files under public/assets/ and public/fonts/ get their content hash as ETag when the index has it (a file whose
|
||||
// size or mtime changed since asks for a rescan), others size + mtime. Traversal & dotfile protection, a directory
|
||||
// without its trailing slash → 301, 404 page; an absent data/local-assets.json is answered with an empty manifest.
|
||||
// Files go out through files.js serveFile.
|
||||
|
||||
import fsp from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { MEDIA_PREFIX } from '../../shared/media.js';
|
||||
import { splitHashedPath } from '../../shared/assetPaths.js';
|
||||
import { ROOT, noopLog } from './config.js';
|
||||
import { sendError, sendJson } from './common.js';
|
||||
import { MIME, GzipCache, isNotModified, serveFile } from './files.js';
|
||||
import { MIME, GzipCache, isNotModified, serveFile, serveBuffer } from './files.js';
|
||||
import { ASSET_INDEX_ROOTS, hashedFileOpts } from './assetIndex.js';
|
||||
import { serveMedia } from './media.js';
|
||||
import { createPackRegistry } from '../packs.js';
|
||||
import { PACKS_URL, PACK_INDEX_FILE } from '../../shared/packs.js';
|
||||
@@ -51,11 +62,15 @@ const EMPTY_LOCAL_ART = Buffer.from(JSON.stringify({ version: 1, source: 'none',
|
||||
/**
|
||||
* Create the static request handler.
|
||||
* @param {{ publicDir: string, dataDir: string, sharedDir: string, simDir?: string, packsDir?: string,
|
||||
* packs?: ReturnType<typeof createPackRegistry>, log?: object }} dirs
|
||||
* packs: the server's pack registry (default: one over publicDir, dataDir and packsDir — ROOT/packs)
|
||||
* packs?: ReturnType<typeof createPackRegistry>, log?: object,
|
||||
* assetIndex?: ReturnType<typeof import('./assetIndex.js').createAssetIndex> | null,
|
||||
* views?: ReturnType<typeof import('./assetViews.js').createManifestViews> | null }} dirs
|
||||
* packs: the server's pack registry (default: one over publicDir, dataDir and packsDir — ROOT/packs);
|
||||
* assetIndex / views: the content hashes and the served manifests (none: plain URLs, size-mtime ETags)
|
||||
* @returns {(req: import('node:http').IncomingMessage, res: import('node:http').ServerResponse, rawPath: string, query: string) => Promise<void>}
|
||||
*/
|
||||
export function createStaticHandler({ publicDir, dataDir, sharedDir, simDir = path.join(ROOT, 'server', 'sim'), packsDir = path.join(ROOT, 'packs'), packs = null, log = noopLog }) {
|
||||
export function createStaticHandler({ publicDir, dataDir, sharedDir, simDir = path.join(ROOT, 'server', 'sim'), packsDir = path.join(ROOT, 'packs'), packs = null, log = noopLog,
|
||||
assetIndex = null, views = null }) {
|
||||
const registry = packs || createPackRegistry({ publicDir, dataDir, packsDir }, { log: /** @type {any} */ (log) });
|
||||
const mounts = [
|
||||
{ prefix: '/data/', name: 'data', dir: path.resolve(dataDir) },
|
||||
@@ -75,6 +90,10 @@ export function createStaticHandler({ publicDir, dataDir, sharedDir, simDir = pa
|
||||
sendError(req, res, 400, '请求地址无效 · Bad request');
|
||||
return;
|
||||
}
|
||||
// /assets/~<hash>/x → /assets/x (the hash decides the cache policy below)
|
||||
const hashed = splitHashedPath(decoded);
|
||||
const urlHash = hashed.hash;
|
||||
decoded = hashed.path;
|
||||
if (decoded === '/data.js') {
|
||||
const headers = { 'Content-Type': MIME['.js'], 'Cache-Control': 'no-cache', ETag: shimTag, 'Content-Length': shimBody.length };
|
||||
if (isNotModified(req, shimTag, new Date(0))) { delete headers['Content-Length']; res.writeHead(304, headers); res.end(); return; }
|
||||
@@ -94,9 +113,17 @@ export function createStaticHandler({ publicDir, dataDir, sharedDir, simDir = pa
|
||||
}
|
||||
// Extension-less audio (download-manager avoidance): /media/bgm/act1 → /assets/audio/bgm/act1.mp3
|
||||
if (decoded.startsWith(MEDIA_PREFIX)) {
|
||||
await serveMedia(req, res, decoded.slice(MEDIA_PREFIX.length), query, publicDir, gzipCache, log);
|
||||
await serveMedia(req, res, decoded.slice(MEDIA_PREFIX.length), query, publicDir, gzipCache, log, { urlHash, assetIndex });
|
||||
return;
|
||||
}
|
||||
// The served manifests: every art URL with its content hash (assetViews.js), once the index is ready
|
||||
if (views && (decoded === '/data/assets.json' || decoded === '/data/local-assets.json')) {
|
||||
const view = await views.get(decoded.slice('/data/'.length));
|
||||
if (view) {
|
||||
await serveBuffer(req, res, { body: view.body, hash: view.hash, type: MIME['.json'], cacheControl: 'no-cache', key: `view:${decoded}`, cache: gzipCache });
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Bare mount paths (e.g. "/data") → treat as the mount directory.
|
||||
const mount = mounts.find((m) => decoded.startsWith(m.prefix) || decoded === m.prefix.slice(0, -1)) || mounts[mounts.length - 1];
|
||||
const rest = decoded.length > mount.prefix.length ? decoded.slice(mount.prefix.length) : '';
|
||||
@@ -154,6 +181,8 @@ export function createStaticHandler({ publicDir, dataDir, sharedDir, simDir = pa
|
||||
}
|
||||
return;
|
||||
}
|
||||
await serveFile(req, res, absPath, stat, mount.name, segments, query, gzipCache, log);
|
||||
const indexed = mount.name === 'public' && segments.length > 1 && ASSET_INDEX_ROOTS.includes(segments[0]);
|
||||
const opts = indexed ? hashedFileOpts(assetIndex, segments.join('/'), stat, urlHash) : (urlHash ? { cacheControl: 'no-cache' } : {});
|
||||
await serveFile(req, res, absPath, stat, mount.name, segments, query, gzipCache, log, opts);
|
||||
};
|
||||
}
|
||||
+9
-5
@@ -11,6 +11,8 @@
|
||||
// http/files.js one file → response: MIME, gzip + memory cache, ETag / Last-Modified / 304, Cache-Control, ranges
|
||||
// http/assetIndex.js the content hashes of public/assets and public/fonts (built in the background, cached in
|
||||
// .cache/asset-hashes.json by path + size + mtime; Spine models share one hash)
|
||||
// http/assetViews.js the served /data/assets.json and /data/local-assets.json: every art URL with its hash segment
|
||||
// (/assets/~<hash>/…, immutable when the hash matches); the committed files stay unchanged
|
||||
// http/buildTag.js the build tag of the served browser runtime (/healthz `build`, public/js/ui/buildGuard.js)
|
||||
// http/routes.js the request listener: security headers, 414 / 400 / 405, GET /healthz → JSON status, else static
|
||||
// http/traffic.js the traffic counters: bytes sent per category (assets / audio / code / data) and on the
|
||||
@@ -37,6 +39,7 @@ import { BUILD_INPUTS, computeBuildTag, buildTag, resetBuildTag } from './http/b
|
||||
import { createRequestHandler } from './http/routes.js';
|
||||
import { createTrafficCounter } from './http/traffic.js';
|
||||
import { createAssetIndex } from './http/assetIndex.js';
|
||||
import { createManifestViews } from './http/assetViews.js';
|
||||
import { answerClientError } from './http/common.js';
|
||||
import { lanUrls, displayHost, isProcessEntry, runMain } from './http/boot.js';
|
||||
|
||||
@@ -63,7 +66,7 @@ export {
|
||||
* lobby: import('./lobby.js').Lobby, network: import('./net.js').Network,
|
||||
* registry: import('./net.js').SessionRegistry, packs: ReturnType<typeof createPackRegistry>,
|
||||
* traffic: ReturnType<typeof createTrafficCounter>, assetIndex: ReturnType<typeof createAssetIndex> | null,
|
||||
* close: () => Promise<void> }>}
|
||||
* views: ReturnType<typeof createManifestViews> | null, close: () => Promise<void> }>}
|
||||
*/
|
||||
export async function startServer(opts = {}) {
|
||||
const { port, host } = listenAddress(opts);
|
||||
@@ -76,17 +79,18 @@ export async function startServer(opts = {}) {
|
||||
// content packs (docs/PACKS.md): scanned now — the start log names them — and again whenever their folders change
|
||||
const packs = createPackRegistry({ publicDir, dataDir, packsDir }, { log });
|
||||
packs.refresh(true);
|
||||
// the content hashes of the art (built in the background once listening)
|
||||
// the content hashes of the art (built in the background once listening) and the manifests served with them
|
||||
const indexOptions = assetIndexOptionsFrom(opts, publicDir);
|
||||
const assetIndex = indexOptions ? createAssetIndex({ publicDir, log, ...indexOptions }) : null;
|
||||
const serveStatic = createStaticHandler({ publicDir, dataDir, sharedDir, packsDir, packs, log });
|
||||
const views = assetIndex ? createManifestViews({ dataDir, index: assetIndex, log }) : null;
|
||||
const serveStatic = createStaticHandler({ publicDir, dataDir, sharedDir, packsDir, packs, log, assetIndex, views });
|
||||
const startedAt = Date.now();
|
||||
// The tag is per process (see buildTag): read the browser runtime once, here, not on every /healthz.
|
||||
resetBuildTag();
|
||||
buildTag();
|
||||
|
||||
const traffic = createTrafficCounter();
|
||||
const server = http.createServer(createRequestHandler({ serveStatic, health: { startedAt, network, registry, lobby, traffic, assetIndex }, log }));
|
||||
const server = http.createServer(createRequestHandler({ serveStatic, health: { startedAt, network, registry, lobby, traffic, assetIndex, views }, log }));
|
||||
server.on('clientError', answerClientError);
|
||||
const wss = attachWebSocket(server, { network, log, traffic });
|
||||
|
||||
@@ -148,7 +152,7 @@ export async function startServer(opts = {}) {
|
||||
return closing;
|
||||
}
|
||||
|
||||
return { port: actualPort, host: boundHost, url, server, wss, lobby, network, registry, packs, traffic, assetIndex, close };
|
||||
return { port: actualPort, host: boundHost, url, server, wss, lobby, network, registry, packs, traffic, assetIndex, views, close };
|
||||
}
|
||||
|
||||
// `node server/index.js` / npm start: listen, print the banner, stop on SIGINT / SIGTERM (http/boot.js).
|
||||
|
||||
+234
-2
@@ -1,15 +1,23 @@
|
||||
// Content-hashed asset URLs (host bandwidth, layer 1; docs/ASSETS.md "The served manifests"):
|
||||
// * server/http/assetIndex.js — SHA-256 (16 hex) of every file under public/assets and public/fonts, cached by path +
|
||||
// size + mtime in a JSON file, never trusted for a file whose size or mtime changed; Spine models share one hash.
|
||||
import { test, describe } from 'node:test';
|
||||
// size + mtime in a JSON file, never trusted for a file whose size or mtime changed; Spine models share one hash;
|
||||
// * server/http/assetViews.js — the served data/assets.json / data/local-assets.json with every art URL carrying its
|
||||
// hash segment, the committed files untouched;
|
||||
// * the ~hash routes of static.js / media.js — the segment stripped, `immutable` only when it names the file's bytes,
|
||||
// the current file with `no-cache` otherwise; content ETags; the plain URLs as before.
|
||||
import { test, describe, before, after } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { createHash, randomBytes } from 'node:crypto';
|
||||
import { createAssetIndex, hash16 } from '../server/http/assetIndex.js';
|
||||
import { rewriteManifest, relOf } from '../server/http/assetViews.js';
|
||||
import { startServer } from '../server/index.js';
|
||||
|
||||
const sha16 = (buf) => createHash('sha256').update(buf).digest('hex').slice(0, 16);
|
||||
const IMMUTABLE = 'public, max-age=31536000, immutable';
|
||||
|
||||
function tree(files) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sp-hash-'));
|
||||
@@ -21,6 +29,18 @@ function tree(files) {
|
||||
return dir;
|
||||
}
|
||||
|
||||
function get(port, p, headers = {}, method = 'GET') {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = http.request({ host: '127.0.0.1', port, path: p, headers, method, agent: false }, (res) => {
|
||||
const chunks = [];
|
||||
res.on('data', (c) => chunks.push(c));
|
||||
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: Buffer.concat(chunks) }));
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
const until = async (fn, ms = 3000) => { for (const t0 = Date.now(); Date.now() - t0 < ms;) { if (await fn()) return true; await new Promise((r) => setTimeout(r, 20)); } return false; };
|
||||
|
||||
describe('the asset index', () => {
|
||||
@@ -122,3 +142,215 @@ describe('the asset index', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('the served manifests (rewriteManifest)', () => {
|
||||
const H = { '/assets/char/a.png': 'aaaaaaaaaaaaaaaa', '/assets/audio/bgm/x.mp3': 'bbbbbbbbbbbbbbbb', '/fonts/f.woff2': 'cccccccccccccccc',
|
||||
'/assets/spine/x/x.skel': '1111111111111111', '/assets/spine/x/x.atlas': '2222222222222222', '/assets/spine/x/x.png': '3333333333333333',
|
||||
'/assets/local/spine/token/t/t.skel': '4444444444444444', '/assets/local/spine/token/t/t.atlas': '5555555555555555',
|
||||
'/assets/local/spine/token/t/t.png': '6666666666666666', '/assets/local/ui/battle/b.png': '7777777777777777' };
|
||||
const index = {
|
||||
hashOf: (rel) => H[`/${rel}`] || null,
|
||||
modelHash: (rels) => (rels.every((r) => H[`/${r}`]) ? hash16(rels.map((r) => H[`/${r}`]).join('\n')) : null),
|
||||
};
|
||||
|
||||
test('data/assets.json: own hashes, one model hash per Spine object, unknown files and names left alone, the input untouched', () => {
|
||||
const doc = {
|
||||
version: 1, hash: 'a1b2c3d4e5f6', generator: 'tools/fetch-assets.mjs',
|
||||
chars: { c: { avatar: '/assets/char/a.png', portrait: '/assets/char/missing.png',
|
||||
spine: { front: { skel: '/assets/spine/x/x.skel', atlas: '/assets/spine/x/x.atlas', textures: ['/assets/spine/x/x.png'], pma: false, anims: { idle: 'Idle' } } } } },
|
||||
enemies: { e: { icon: '/assets/char/a.png', spineLocal: { group: 'spine/enemy/e', skel: 'e.skel', atlas: 'e.atlas', textures: ['e.png'] } } },
|
||||
audio: { bgm: { lobby: { loop: '/assets/audio/bgm/x.mp3' } } },
|
||||
fonts: { css: '/fonts/fonts.css', faces: { f: { woff2: '/fonts/f.woff2', family: 'F' } } },
|
||||
};
|
||||
const copy = JSON.parse(JSON.stringify(doc));
|
||||
const r = rewriteManifest(doc, 'assets.json', index);
|
||||
assert.deepEqual(doc, copy, 'the committed manifest object is not changed');
|
||||
const m = index.modelHash(['assets/spine/x/x.skel', 'assets/spine/x/x.atlas', 'assets/spine/x/x.png']);
|
||||
assert.equal(r.doc.chars.c.avatar, '/assets/~aaaaaaaaaaaaaaaa/char/a.png');
|
||||
assert.equal(r.doc.chars.c.portrait, '/assets/char/missing.png', 'no hash: the plain URL');
|
||||
assert.deepEqual(r.doc.chars.c.spine.front, { skel: `/assets/~${m}/spine/x/x.skel`, atlas: `/assets/~${m}/spine/x/x.atlas`, textures: [`/assets/~${m}/spine/x/x.png`], pma: false, anims: { idle: 'Idle' } });
|
||||
assert.deepEqual(r.doc.enemies.e.spineLocal, copy.enemies.e.spineLocal, 'spineLocal holds names, not URLs');
|
||||
assert.equal(r.doc.audio.bgm.lobby.loop, '/assets/~bbbbbbbbbbbbbbbb/audio/bgm/x.mp3', 'audio keeps /assets/…/audio/….mp3 (media.js makes it /media/~hash/…)');
|
||||
assert.equal(r.doc.fonts.faces.f.woff2, '/fonts/~cccccccccccccccc/f.woff2');
|
||||
assert.equal(r.doc.fonts.css, '/fonts/fonts.css');
|
||||
assert.equal(r.doc.hash, 'a1b2c3d4e5f6');
|
||||
assert.deepEqual(r.models, [['assets/spine/x/x.skel', 'assets/spine/x/x.atlas', 'assets/spine/x/x.png']]);
|
||||
assert.equal(r.hashed, 7);
|
||||
assert.equal(r.plain, 2);
|
||||
});
|
||||
|
||||
test('data/local-assets.json: a spine/ group is one model, other groups own hashes', () => {
|
||||
const doc = { version: 1, source: 'local-client', count: 4, groups: {
|
||||
'spine/token/t': { 't.atlas': { path: '/assets/local/spine/token/t/t.atlas', kind: 'TextAsset' }, 't.png': { path: '/assets/local/spine/token/t/t.png', w: 4, h: 4 }, 't.skel': { path: '/assets/local/spine/token/t/t.skel' } },
|
||||
'ui/battle': { b: { path: '/assets/local/ui/battle/b.png', w: 2, h: 2, kind: 'Sprite' } },
|
||||
} };
|
||||
const r = rewriteManifest(doc, 'local-assets.json', index);
|
||||
const m = index.modelHash(['assets/local/spine/token/t/t.atlas', 'assets/local/spine/token/t/t.png', 'assets/local/spine/token/t/t.skel']);
|
||||
assert.ok(m);
|
||||
for (const [n, e] of Object.entries(r.doc.groups['spine/token/t'])) assert.equal(e.path, doc.groups['spine/token/t'][n].path.replace('/assets/', `/assets/~${m}/`), n);
|
||||
assert.equal(r.doc.groups['spine/token/t']['t.png'].w, 4);
|
||||
assert.deepEqual(r.doc.groups['ui/battle'].b, { path: '/assets/~7777777777777777/local/ui/battle/b.png', w: 2, h: 2, kind: 'Sprite' });
|
||||
assert.deepEqual([r.doc.version, r.doc.source, r.doc.count], [1, 'local-client', 4]);
|
||||
assert.equal(r.models.length, 1);
|
||||
});
|
||||
|
||||
test('relOf: URL path → index path, refusing what the static server refuses', () => {
|
||||
assert.equal(relOf('/assets/a%20b.png'), 'assets/a b.png');
|
||||
assert.equal(relOf('/fonts/f.woff2'), 'fonts/f.woff2');
|
||||
for (const u of ['/data/x.json', '/assets/../x', '/assets//x', '/assets/x?v=1', '/assets/%E0%A4%A', 'assets/x', null]) assert.equal(relOf(u), null, String(u));
|
||||
});
|
||||
});
|
||||
|
||||
describe('content-hashed URLs through a real server', () => {
|
||||
let srv;
|
||||
let dir;
|
||||
const png = randomBytes(3000);
|
||||
const mp3 = randomBytes(6000);
|
||||
const skel = Buffer.from('skel-binary'.repeat(80));
|
||||
const atlas = Buffer.from('x.png\nsize: 64,64\n'.repeat(30));
|
||||
const page = randomBytes(1500);
|
||||
const manifest = {
|
||||
version: 1, hash: 'abcdefabcdef',
|
||||
chars: { c: { avatar: '/assets/char/a.png', spine: { front: { skel: '/assets/spine/x/x.skel', atlas: '/assets/spine/x/x.atlas', textures: ['/assets/spine/x/x.png'], anims: { idle: 'Idle' } } } } },
|
||||
audio: { bgm: { lobby: { loop: '/assets/audio/bgm/lobby.mp3' } } },
|
||||
fonts: { css: '/fonts/fonts.css', faces: { f: { woff2: '/fonts/f.woff2' } } },
|
||||
stats: { note: 'a manifest large enough to be compressed '.repeat(40) },
|
||||
};
|
||||
const manifestText = JSON.stringify(manifest, null, 2);
|
||||
|
||||
before(async () => {
|
||||
dir = tree({
|
||||
'secret.txt': 'TOP-SECRET-CONTENT',
|
||||
'public/index.html': '<p>hi</p>', 'public/assets/char/a.png': png, 'public/assets/audio/bgm/lobby.mp3': mp3,
|
||||
'public/assets/spine/x/x.skel': skel, 'public/assets/spine/x/x.atlas': atlas, 'public/assets/spine/x/x.png': page,
|
||||
'public/assets/spine/x/tiles.json': '{"t":1}', 'public/fonts/f.woff2': randomBytes(500), 'public/fonts/fonts.css': '@font-face{}',
|
||||
'data/assets.json': manifestText, 'data/config.json': '{}', 'shared/x.js': 'export const y = 2;',
|
||||
});
|
||||
srv = await startServer({ port: 0, host: '127.0.0.1', quiet: true, publicDir: path.join(dir, 'public'), dataDir: path.join(dir, 'data'),
|
||||
sharedDir: path.join(dir, 'shared'), assetIndex: { delayMs: 0 } });
|
||||
await srv.assetIndex.whenReady();
|
||||
});
|
||||
after(async () => {
|
||||
await srv?.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const view = async () => JSON.parse((await get(srv.port, '/data/assets.json')).body);
|
||||
|
||||
test('/data/assets.json is served with hashed URLs (no-cache, its own ETag, 304, gzip, HEAD); the committed file stays as it was', async () => {
|
||||
const r = await get(srv.port, '/data/assets.json');
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.headers['content-type'], 'application/json; charset=utf-8');
|
||||
assert.equal(r.headers['cache-control'], 'no-cache');
|
||||
assert.match(r.headers.etag, /^"[0-9a-f]{16}"$/);
|
||||
const v = JSON.parse(r.body);
|
||||
assert.equal(v.chars.c.avatar, `/assets/~${sha16(png)}/char/a.png`);
|
||||
assert.equal(v.audio.bgm.lobby.loop, `/assets/~${sha16(mp3)}/audio/bgm/lobby.mp3`);
|
||||
assert.match(v.fonts.faces.f.woff2, /^\/fonts\/~[0-9a-f]{16}\/f\.woff2$/);
|
||||
const m = hash16([skel, atlas, page].map(sha16).join('\n'));
|
||||
assert.deepEqual([v.chars.c.spine.front.skel, v.chars.c.spine.front.atlas, v.chars.c.spine.front.textures[0]],
|
||||
[`/assets/~${m}/spine/x/x.skel`, `/assets/~${m}/spine/x/x.atlas`, `/assets/~${m}/spine/x/x.png`]);
|
||||
assert.equal((await get(srv.port, '/data/assets.json', { 'if-none-match': r.headers.etag })).status, 304);
|
||||
const gz = await get(srv.port, '/data/assets.json', { 'accept-encoding': 'gzip' });
|
||||
assert.equal(gz.headers['content-encoding'], 'gzip');
|
||||
assert.notEqual(gz.headers.etag, r.headers.etag);
|
||||
const head = await get(srv.port, '/data/assets.json', {}, 'HEAD');
|
||||
assert.equal(head.status, 200);
|
||||
assert.equal(head.body.length, 0);
|
||||
assert.equal(fs.readFileSync(path.join(dir, 'data/assets.json'), 'utf8'), manifestText, 'never written');
|
||||
assert.equal((await get(srv.port, '/data/local-assets.json')).status, 200, 'an absent local manifest is still the empty stand-in');
|
||||
assert.deepEqual(JSON.parse((await get(srv.port, '/data/local-assets.json')).body).groups, {});
|
||||
});
|
||||
|
||||
test('a hashed URL: immutable with a content ETag when the hash names the file; the current file with no-cache otherwise; plain URLs as before', async () => {
|
||||
const v = await view();
|
||||
const ok = await get(srv.port, v.chars.c.avatar);
|
||||
assert.equal(ok.status, 200);
|
||||
assert.deepEqual(ok.body, png);
|
||||
assert.equal(ok.headers['cache-control'], IMMUTABLE);
|
||||
assert.equal(ok.headers.etag, `"${sha16(png)}"`);
|
||||
const old = await get(srv.port, '/assets/~0000000000000000/char/a.png');
|
||||
assert.deepEqual(old.body, png, 'an old page still gets the file');
|
||||
assert.equal(old.headers['cache-control'], 'no-cache');
|
||||
const plain = await get(srv.port, '/assets/char/a.png');
|
||||
assert.equal(plain.headers['cache-control'], 'public, max-age=86400');
|
||||
assert.equal(plain.headers.etag, `"${sha16(png)}"`, 'a content ETag on the plain URL too (an mtime change alone is a 304)');
|
||||
assert.equal((await get(srv.port, '/assets/char/a.png', { 'if-none-match': `"${sha16(png)}"` })).status, 304);
|
||||
const font = await get(srv.port, v.fonts.faces.f.woff2);
|
||||
assert.equal(font.headers['cache-control'], IMMUTABLE);
|
||||
const query = await get(srv.port, '/assets/char/a.png?v=1');
|
||||
assert.match(query.headers['cache-control'], /immutable/, 'the old ?v= rule is kept');
|
||||
});
|
||||
|
||||
test('a Spine model: skeleton, atlas, page and the atlas derived from the skeleton URL are immutable under the model hash; a file derived beside them is no-cache', async () => {
|
||||
const sp = (await view()).chars.c.spine.front;
|
||||
for (const u of [sp.skel, sp.atlas, sp.textures[0], sp.skel.replace(/\.skel$/, '.atlas')]) {
|
||||
const r = await get(srv.port, u);
|
||||
assert.equal(r.status, 200, u);
|
||||
assert.equal(r.headers['cache-control'], IMMUTABLE, u);
|
||||
}
|
||||
const sk = await get(srv.port, sp.skel);
|
||||
assert.deepEqual(sk.body, skel);
|
||||
assert.equal(sk.headers.etag, `"${sha16(skel)}"`, 'the ETag is the file\'s own content hash');
|
||||
const tiles = await get(srv.port, sp.skel.replace(/[^/]+$/, 'tiles.json'));
|
||||
assert.equal(tiles.status, 200);
|
||||
assert.equal(tiles.headers['cache-control'], 'no-cache', 'not a member of the model');
|
||||
});
|
||||
|
||||
test('audio: /media/~hash/… resolves like /media/…, immutable when the hash matches, ranges and ETags as before', async () => {
|
||||
const loop = (await view()).audio.bgm.lobby.loop;
|
||||
const media = loop.replace(/^\/assets\/(~[0-9a-f]{16})\/audio\/(.+)\.mp3$/, '/media/$1/$2');
|
||||
assert.match(media, /^\/media\/~[0-9a-f]{16}\/bgm\/lobby$/);
|
||||
const r = await get(srv.port, media);
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.headers['content-type'], 'audio/mpeg');
|
||||
assert.equal(r.headers['cache-control'], IMMUTABLE);
|
||||
assert.equal(r.headers.etag, `"${sha16(mp3)}"`);
|
||||
assert.deepEqual(r.body, mp3);
|
||||
const range = await get(srv.port, media, { range: 'bytes=0-9' });
|
||||
assert.equal(range.status, 206);
|
||||
assert.deepEqual(range.body, mp3.subarray(0, 10));
|
||||
assert.equal((await get(srv.port, loop)).headers['cache-control'], IMMUTABLE, 'the .mp3 fallback URL too');
|
||||
assert.equal((await get(srv.port, '/media/~0000000000000000/bgm/lobby')).headers['cache-control'], 'no-cache');
|
||||
assert.equal((await get(srv.port, '/media/bgm/lobby')).headers['cache-control'], 'public, max-age=86400');
|
||||
});
|
||||
|
||||
test('the segment never opens a way out of the served folders', async () => {
|
||||
const h = sha16(png);
|
||||
for (const p of [`/assets/~${h}/../../secret.txt`, `/assets/~${h}/%2e%2e/%2e%2e/secret.txt`, `/assets/~${h}/..%2f..%2fsecret.txt`,
|
||||
`/media/~${h}/..%2f..%2f..%2fsecret.txt`, `/fonts/~${h}/../../secret.txt`, `/assets/~${h}/.hidden`, `/assets/~${h}`, '/assets/~/char/a.png']) {
|
||||
const r = await get(srv.port, p);
|
||||
assert.ok([301, 400, 403, 404].includes(r.status), `${p} → ${r.status}`);
|
||||
assert.ok(!r.body.toString().includes('TOP-SECRET'), `${p} leaked`);
|
||||
}
|
||||
});
|
||||
|
||||
test('a file replaced on disk: its old hashed URL turns no-cache at once, and after the rescan the view names the new bytes', async () => {
|
||||
const before = (await view()).chars.c.avatar;
|
||||
const next = randomBytes(3100);
|
||||
fs.writeFileSync(path.join(dir, 'public/assets/char/a.png'), next);
|
||||
const stale = await get(srv.port, before);
|
||||
assert.deepEqual(stale.body, next, 'the current file');
|
||||
assert.equal(stale.headers['cache-control'], 'no-cache', 'the index no longer vouches for these bytes');
|
||||
// the miss asked for a rescan (RESCAN_GAP_MS after the last build) — trigger it now instead of waiting
|
||||
await srv.assetIndex.refresh();
|
||||
assert.ok(await until(async () => (await view()).chars.c.avatar === `/assets/~${sha16(next)}/char/a.png`), 'the view follows');
|
||||
assert.equal((await get(srv.port, `/assets/~${sha16(next)}/char/a.png`)).headers['cache-control'], IMMUTABLE);
|
||||
});
|
||||
});
|
||||
|
||||
test('without the index (assetIndex: false) the manifest goes out as committed and a hashed URL is served no-cache', async () => {
|
||||
const dir = tree({ 'public/assets/a.png': 'png', 'data/assets.json': JSON.stringify({ chars: { c: { avatar: '/assets/a.png' } } }) });
|
||||
const srv = await startServer({ port: 0, host: '127.0.0.1', quiet: true, publicDir: path.join(dir, 'public'), dataDir: path.join(dir, 'data'), assetIndex: false });
|
||||
try {
|
||||
assert.equal(srv.assetIndex, null);
|
||||
assert.equal(JSON.parse((await get(srv.port, '/data/assets.json')).body).chars.c.avatar, '/assets/a.png');
|
||||
const r = await get(srv.port, `/assets/~${sha16(Buffer.from('png'))}/a.png`);
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.headers['cache-control'], 'no-cache');
|
||||
assert.match(r.headers.etag, /^"[0-9a-f]+-[0-9a-f]+"$/, 'size-mtime ETag');
|
||||
} finally {
|
||||
await srv.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
+6
-1
@@ -138,7 +138,10 @@ describe('static http server', () => {
|
||||
srv = await startServer({
|
||||
port: 0, host: '127.0.0.1', quiet: true,
|
||||
publicDir: path.join(tmp, 'public'), dataDir: path.join(tmp, 'data'), sharedDir: path.join(tmp, 'shared'),
|
||||
assetIndex: { delayMs: 0 },
|
||||
});
|
||||
// the content hashes of public/assets + public/fonts (the ETags of those files): ready before the first request
|
||||
await srv.assetIndex.whenReady();
|
||||
});
|
||||
|
||||
after(async () => {
|
||||
@@ -226,9 +229,11 @@ describe('static http server', () => {
|
||||
const gzEtag = (await httpReq(srv.port, '/js/app.js', { headers: { 'accept-encoding': 'gzip' } })).headers.etag;
|
||||
assert.notEqual(gzEtag, etag, 'gzip variant has its own ETag');
|
||||
|
||||
// unversioned art/fonts/vendor: cached for a day (files can be rewritten in place by the asset tools)
|
||||
// unversioned art/fonts/vendor: cached for a day (files can be rewritten in place by the asset tools); the art and
|
||||
// fonts carry their content hash as ETag (server/http/assetIndex.js), so a rewrite with the same bytes is a 304
|
||||
const asset = await httpReq(srv.port, '/assets/img/a.png');
|
||||
assert.equal(asset.headers['cache-control'], 'public, max-age=86400');
|
||||
assert.match(asset.headers.etag, /^"[0-9a-f]{16}"$/);
|
||||
const font = await httpReq(srv.port, '/fonts/f.woff2');
|
||||
assert.equal(font.headers['cache-control'], 'public, max-age=86400');
|
||||
const assetV = await httpReq(srv.port, '/assets/img/a.png?v=abc');
|
||||
|
||||
Reference in new issue
Block a user