Report missing asset leaves and add opt-in strict mode (#421)

This commit is contained in:
sganggs committed 2026-10-09 17:13:13 +08:00
1 parent 174ccdf94d
commit 3e3919c917
6 files changed
+211 -14

No files matched your search

+23 -3
View File
@@ -24,6 +24,7 @@ npm run assets # = node tools/vendor.mjs && node tools/fetch-assets.mjs
| `--voice-all` | Plan every official voice slot, including the prep-only lines no battle plays (干员报到 / 编入队伍 / 任命队长 — 360 files, one per operator and slot). Off by default: nothing requests them, so planning them only makes every run download more. | | `--voice-all` | Plan every official voice slot, including the prep-only lines no battle plays (干员报到 / 编入队伍 / 任命队长 — 360 files, one per operator and slot). Off by default: nothing requests them, so planning them only makes every run download more. |
| `--prune` | Delete files under `public/assets/` that the manifest no longer references, for example after a mapping change. Without this flag they are only listed in the report. `public/assets/local/` (written by `tools/local-extract`) is never pruned. Implies `--allow-shrink`. | | `--prune` | Delete files under `public/assets/` that the manifest no longer references, for example after a mapping change. Without this flag they are only listed in the report. `public/assets/local/` (written by `tools/local-extract`) is never pruned. Implies `--allow-shrink`. |
| `--allow-shrink` | Write `data/assets.json` even when it loses entries the current one has (see "The manifest never shrinks by accident" below). | | `--allow-shrink` | Write `data/assets.json` even when it loses entries the current one has (see "The manifest never shrinks by accident" below). |
| `--strict` | Exit 1 when a leaf was dropped for having no file on disk, instead of only reporting it (see "A dropped leaf is reported" below). The same as the environment variable `SP_ASSETS_STRICT=1` — for CI and packaging builds. |
| `--add-only` | For a checkout whose `public/assets/` and `public/fonts/` are shared with another one (a git worktree with symlinked asset folders): download only the files missing on disk and never re-download, rewrite or delete an existing file — atlases already on disk are left as they are, the fonts are not rebuilt (the manifest keeps its current `fonts`). Not with `--prune` / `--force`. | | `--add-only` | For a checkout whose `public/assets/` and `public/fonts/` are shared with another one (a git worktree with symlinked asset folders): download only the files missing on disk and never re-download, rewrite or delete an existing file — atlases already on disk are left as they are, the fonts are not rebuilt (the manifest keeps its current `fonts`). Not with `--prune` / `--force`. |
| `--local-spines` | Rewrite `tools/assets/local-enemy-spines.json` and `tools/assets/local-token-spines.json` (the metadata of the enemy and token models only the local client has, see "Enemy aliases" and "Token models from the local client") from the models `tools/local-extract/extract.py` extracted to `public/assets/local/spine/enemy/` and `public/assets/local/spine/token/`. Run it after a game update changed them; without it the committed files are used and a differing extraction only gets a warning. | | `--local-spines` | Rewrite `tools/assets/local-enemy-spines.json` and `tools/assets/local-token-spines.json` (the metadata of the enemy and token models only the local client has, see "Enemy aliases" and "Token models from the local client") from the models `tools/local-extract/extract.py` extracted to `public/assets/local/spine/enemy/` and `public/assets/local/spine/token/`. Run it after a game update changed them; without it the committed files are used and a differing extraction only gets a warning. |
@@ -38,6 +39,23 @@ smaller manifest is intended, for example after a mapping change. Build fields (
plan legitimately narrows — like the 干员战斗语音 default, which no longer plans the three prep-only slots (360 entries, plan legitimately narrows — like the 干员战斗语音 default, which no longer plans the three prep-only slots (360 entries,
DESIGN §21.30) — reports exactly those entries and needs `--allow-shrink` once; the list it prints is the check. DESIGN §21.30) — reports exactly those entries and needs `--allow-shrink` once; the list it prints is the check.
**A dropped leaf is reported, never silent.** `resolveTemplate` leaves out every entry none of whose alternative files
is on disk — for a leaf that is an entry the client loses outright: a unit whose `attack` sound was never downloaded has
no `audio.sfx.units[charId].attack` at all, so the sound is simply missing (nothing falls back; there is no URL to
retry). That is a different thing from an entry whose *fallback* was used. The run therefore prints a summary naming at most eight such leaves
and records the complete list as `droppedLeaves` (a subset of `misses`, which also lists the
unresolved Spine models):
```
[assets] 1 leaf dropped: no alternative on disk (audio.sfx.units.some_char.attack)
```
A run with nothing dropped prints `[assets] no leaf dropped: every planned leaf has a file on disk`. This is only
observability: which files are planned, downloaded and written does not change. `--strict` (or `SP_ASSETS_STRICT=1`)
turns those drops into a failure (exit 1, the leaves named) so a CI or packaging run cannot ship a manifest with such
holes — the committed manifest's shrink guard cannot see them once the entry is already gone. The manifest may
already have been written before this check returns failure; strict mode does not roll it back.
The script is **idempotent**. A file on disk is kept, not re-downloaded, when any one of these holds: The script is **idempotent**. A file on disk is kept, not re-downloaded, when any one of these holds:
- its size matches the ledger entry from a previous download (`.cache/assets-ledger.json`); - its size matches the ledger entry from a previous download (`.cache/assets-ledger.json`);
- its size matches the byte count recorded in research; - its size matches the byte count recorded in research;
@@ -59,10 +77,12 @@ Outputs:
- `data/assets.json`: the manifest (committed). - `data/assets.json`: the manifest (committed).
- `public/assets/**`: art and audio (git-ignored). - `public/assets/**`: art and audio (git-ignored).
- `public/fonts/*`: fonts and `fonts.css`. - `public/fonts/*`: fonts and `fonts.css`.
- `.cache/assets-report.json`: misses, fallbacks and notes from the last run. - `.cache/assets-report.json`: misses, fallbacks, `droppedLeaves` (the leaves left out of the manifest for having no file
on disk) and notes from the last run.
- `.cache/spine-info.json`: skeleton parse cache. - `.cache/spine-info.json`: skeleton parse cache.
The run exits with code 1 if any pool operator is missing its avatar, its portrait or its Front Spine. The run exits with code 1 if any pool operator is missing its avatar, its portrait or its Front Spine, and — with
`--strict` / `SP_ASSETS_STRICT=1` — if any leaf was dropped for having no file on disk.
Upstream indexes are cached under `.cache/`. They are downloaded when missing: Upstream indexes are cached under `.cache/`. They are downloaded when missing:
- `.cache/gamedata/excel/audio_data.json`, from `Kengxxiao/ArknightsGameData` (zh_CN). - `.cache/gamedata/excel/audio_data.json`, from `Kengxxiao/ArknightsGameData` (zh_CN).
@@ -100,7 +120,7 @@ The `stem` of a Spine model is the upstream file name. Two examples: `char_107_l
### Id scope ### Id scope
- **Operators:** all 138 pool charIds from `activity_table` (`charShopChessDatas[*].charId ∪ backupCharId`), including hidden chess and backup operators; plus every unit of `data/backups.json` research 07 does not list — the 71 自选 owned-6★ picks (`diy.ownedPool`, DATA.md §18; the collab operators are not in the data, and 焰狐龙梓兰's entries left the manifest when she left the pool in 0.2.0) — planned from 07's URL patterns (`tools/assets/plan.mjs patternOperator`, `tools/fetch-assets.mjs dataExtras`): avatar and portrait (E0–E1 and E2), the default-skin battle Spine Front / Back, the icon and skill sound of each skill, the sub-profession icon. 209 operators in all (206 with a Back model). - **Operators:** all 138 pool charIds from `activity_table` (`charShopChessDatas[*].charId ∪ backupCharId`), including hidden chess and backup operators; plus every unit of `data/backups.json` research 07 does not list — the 72 自选 owned-6★ picks (`diy.ownedPool`, DATA.md §18; the collab operators are not in the data, and 焰狐龙梓兰's entries left the manifest when she left the pool in 0.2.0) — planned from 07's URL patterns (`tools/assets/plan.mjs patternOperator`, `tools/fetch-assets.mjs dataExtras`): avatar and portrait (E0–E1 and E2), the default-skin battle Spine Front / Back, the icon and skill sound of each skill, the sub-profession icon. 209 operators in all (206 with a Back model).
- **Tokens:** the 20 pool tokens, and the 38 summons of the 自选 picks (`data/backups.json tokens`) as extra tokens (below): their avatars; no dump carries the battle Spine of the 自选 summons (upstream has at most skin variants, which the default locations miss), so the web manifest has no model for them — 35 of them, and 4 pool summons, have the official model as an optional local-client overlay ("Token models from the local client"); 3 have no model in the game at all. - **Tokens:** the 20 pool tokens, and the 38 summons of the 自选 picks (`data/backups.json tokens`) as extra tokens (below): their avatars; no dump carries the battle Spine of the 自选 summons (upstream has at most skin variants, which the default locations miss), so the web manifest has no model for them — 35 of them, and 4 pool summons, have the official model as an optional local-client overlay ("Token models from the local client"); 3 have no model in the game at all.
- **Enemies:** 253 ids planned, 252 in the manifest (心烛 has no assets). The set is the union of: - **Enemies:** 253 ids planned, 252 in the manifest (心烛 has no assets). The set is the union of:
- the 07 enemy list; - the 07 enemy list;
+9
View File
@@ -87,3 +87,12 @@ Only that granted trait additionally listens to `dollSwap`, in both directions,
Deployment and switches share the original cap, active-bond checks and grant deduplication. Other deployment Deployment and switches share the original cap, active-bond checks and grant deduplication. Other deployment
traits keep their existing triggers. The 7 regression cases cover both real dollkeepers, natural and S2 returns, traits keep their existing triggers. The 7 regression cases cover both real dollkeepers, natural and S2 returns,
shared caps, duplicate grant holders, inactive bonds and unrelated timed windows. shared caps, duplicate grant holders, inactive bonds and unrelated timed windows.
### 28.6 Asset download visibility — PR #421
Ported from [PR #421](https://github.com/sganggs/Stronghold-Protocol/pull/421) by @LimitlessHPPK.
The asset resolver records every planned leaf with no alternative on disk as `droppedLeaves` in its report;
the terminal prints a bounded summary. `--strict` / `SP_ASSETS_STRICT=1` returns exit 1 for those omissions.
The default remains a warning, and the existing manifest shrink guard remains in force. Strict mode changes
the outcome code; it is not a transaction and does not roll back a manifest already written. Temporary test
assets are cleaned up after each case. This is an opt-in tool check, not an added packaging requirement.
+40
View File
@@ -443,10 +443,12 @@ describe('audio banks and plan id sets', () => {
const r = resolveTemplate(tpl, { root: ROOT, spine: new Map([['k', { skel: '/assets/s.skel', atlas: '/assets/s.atlas', textures: [] }]]) }); const r = resolveTemplate(tpl, { root: ROOT, spine: new Map([['k', { skel: '/assets/s.skel', atlas: '/assets/s.atlas', textures: [] }]]) });
assert.deepEqual(Object.keys(r.value).sort(), ['keep', 'm']); assert.deepEqual(Object.keys(r.value).sort(), ['keep', 'm']);
assert.deepEqual(r.misses.sort(), ['a', 'b.c']); assert.deepEqual(r.misses.sort(), ['a', 'b.c']);
assert.deepEqual(r.droppedLeaves, ['a', 'b.c'], 'a leaf no alternative of which is on disk is a dropped leaf');
assert.equal(collectLeaves(tpl).length, 2); assert.equal(collectLeaves(tpl).length, 2);
const ok = resolveTemplate({ p: { alts: [{ rel: 'nope.png', urls: ['x'] }, { rel: 'package.json', urls: ['y'] }] } }, { root: ROOT, spine: new Map() }); const ok = resolveTemplate({ p: { alts: [{ rel: 'nope.png', urls: ['x'] }, { rel: 'package.json', urls: ['y'] }] } }, { root: ROOT, spine: new Map() });
assert.equal(ok.value.p, '/assets/package.json'); assert.equal(ok.value.p, '/assets/package.json');
assert.equal(ok.fallbacks.length, 1); assert.equal(ok.fallbacks.length, 1);
assert.deepEqual(ok.droppedLeaves, [], 'a fallback on disk is not a drop');
}); });
test('an array whose lines are all missing is dropped, with the entry it empties (a voice slot never downloaded)', () => { test('an array whose lines are all missing is dropped, with the entry it empties (a voice slot never downloaded)', () => {
@@ -462,6 +464,44 @@ describe('audio banks and plan id sets', () => {
assert.deepEqual(r.value.voice, { b: { select: ['/assets/package.json'] } }, 'operator a has no line on disk: no entry at all'); assert.deepEqual(r.value.voice, { b: { select: ['/assets/package.json'] } }, 'operator a has no line on disk: no entry at all');
assert.deepEqual(r.value.empty, [], 'an array that was empty in the template stays'); assert.deepEqual(r.value.empty, [], 'an array that was empty in the template stays');
assert.deepEqual(r.misses.sort(), ['voice.a.place[0]', 'voice.a.select[0]', 'voice.a.select[1]', 'voice.b.select[0]']); assert.deepEqual(r.misses.sort(), ['voice.a.place[0]', 'voice.a.select[0]', 'voice.a.select[1]', 'voice.b.select[0]']);
assert.deepEqual(r.droppedLeaves.sort(), r.misses.slice().sort(), 'every dropped line is a dropped leaf too');
});
// A unit's attack sound is a leaf like any other: when its file was never downloaded (or a checkout lost it) the whole
// key vanishes from data/assets.json and the client plays nothing — there is no fallback and no URL to retry. The
// resolution therefore reports it as a dropped leaf, which the run prints and --strict fails on
// (tools/fetch-assets.mjs; docs/ASSETS.md "A dropped leaf is reported, never silent").
test('move a resolved sound aside: the leaf leaves the manifest and is reported as dropped, never silently', async (t) => {
const { mkdtemp, mkdir, writeFile, rm } = await import('node:fs/promises');
const { tmpdir } = await import('node:os');
const root = await mkdtemp(join(tmpdir(), 'sp-leaf-drop-'));
t.after(() => rm(root, { recursive: true, force: true }));
const rel = 'audio/sfx/player/p_atk/p_atk_somechar_n.mp3';
const imp = 'audio/sfx/player/p_imp/p_imp_somechar_n.mp3';
const ui = 'audio/sfx/general/g_ui/g_ui_btn_h.mp3';
const leaf = (r) => ({ alts: [{ rel: r, urls: [`https://raw.githubusercontent.com/o/r/main/${r}`], kind: 'mp3' }] });
const tpl = { audio: { sfx: { units: { char_x: { attack: leaf(rel), hit: leaf(imp) } }, ui: { click: leaf(ui) } } } };
for (const r of [rel, imp, ui]) {
await mkdir(join(root, dirname(r)), { recursive: true });
await writeFile(join(root, r), 'ID3');
}
const before = resolveTemplate(tpl, { root, spine: new Map() });
assert.equal(JSON.stringify(before.value), JSON.stringify({
audio: {
sfx: {
units: { char_x: { attack: `/assets/${rel}`, hit: `/assets/${imp}` } },
ui: { click: `/assets/${ui}` },
},
},
}), 'every file on disk: the manifest is exactly the plan, byte for byte');
assert.deepEqual([before.misses, before.droppedLeaves, before.fallbacks], [[], [], []]);
await rm(join(root, rel)); // the download never happened / the file went missing
const after = resolveTemplate(tpl, { root, spine: new Map() });
assert.equal(Object.hasOwn(after.value.audio.sfx.units.char_x, 'attack'), false, 'the key is gone from the manifest');
assert.equal(after.value.audio.sfx.units.char_x.hit, `/assets/${imp}`, 'the roles whose file is on disk stay');
assert.deepEqual(after.droppedLeaves, ['audio.sfx.units.char_x.attack'], 'and the drop is reported, not silent');
assert.ok(after.misses.includes('audio.sfx.units.char_x.attack'), 'the wider miss list names it too');
}); });
}); });
+38
View File
@@ -96,4 +96,42 @@ test('fetch-assets still runs as a script: --help lists --allow-shrink', () => {
assert.equal(r.status, 0, r.stderr); assert.equal(r.status, 0, r.stderr);
assert.match(r.stdout, /--allow-shrink/); assert.match(r.stdout, /--allow-shrink/);
assert.match(r.stdout, /--prune .*\n.*implies --allow-shrink/); assert.match(r.stdout, /--prune .*\n.*implies --allow-shrink/);
assert.match(r.stdout, /--strict\s+exit 1 when a leaf was dropped/);
assert.match(r.stdout, /SP_ASSETS_STRICT=1 is the same as --strict/);
});
// A leaf whose every alternative file is missing on disk disappears from data/assets.json: the client has no entry at all
// (a unit whose attack sound was never downloaded plays nothing), and the shrink guard cannot see it once it is gone from
// the committed manifest. The run reports it as `droppedLeaves`; --strict / SP_ASSETS_STRICT=1 makes it a failure for CI
// and packaging runs, which is the only part that changes a run's outcome (docs/ASSETS.md).
test('--strict / SP_ASSETS_STRICT=1 fails a run that dropped a leaf for having no file on disk, instead of warning', async () => {
const { parseArgs, runExitCode, leafDropSummary, strictLeafDropError } = await import('../tools/fetch-assets.mjs');
const leaf = 'audio.sfx.units.char_x.attack';
assert.equal(parseArgs([]).strict, false, 'a plain run only warns');
assert.equal(parseArgs(['--strict']).strict, true);
assert.throws(() => parseArgs(['--stricts']), /unknown option/);
// the environment variable is the CI / packaging switch, like SP_ASSET_SOURCE
const saved = process.env.SP_ASSETS_STRICT;
try {
for (const v of ['1', 'true', 'YES', 'on']) { process.env.SP_ASSETS_STRICT = v; assert.equal(parseArgs([]).strict, true, `SP_ASSETS_STRICT=${v}`); }
for (const v of ['', '0', 'false', 'no', 'off']) { process.env.SP_ASSETS_STRICT = v; assert.equal(parseArgs([]).strict, false, `SP_ASSETS_STRICT=${v}`); }
delete process.env.SP_ASSETS_STRICT;
assert.equal(parseArgs([]).strict, false, 'unset');
} finally {
if (saved === undefined) delete process.env.SP_ASSETS_STRICT; else process.env.SP_ASSETS_STRICT = saved;
}
// the exit code: a dropped leaf fails a strict run and only warns otherwise
assert.equal(runExitCode({ droppedLeaves: 1 }), 0, 'default: reported, the run still succeeds');
assert.equal(runExitCode({ droppedLeaves: 48, strict: true }), 1, '--strict: the drop is a failure');
assert.equal(runExitCode({ droppedLeaves: 0, strict: true }), 0, 'nothing dropped: a strict run succeeds');
assert.equal(runExitCode({}), 0);
assert.equal(runExitCode({ requiredMisses: 1 }), 1, 'unchanged: a missing avatar / portrait / Front Spine always fails');
assert.equal(runExitCode({ manifestWritten: false }), 1, 'unchanged: the shrink guard keeps the run red');
// and the messages name the leaf
assert.match(strictLeafDropError([leaf]), /^ERROR: 1 leaf dropped for having no alternative on disk \(--strict \/ SP_ASSETS_STRICT=1\): audio\.sfx\.units\.char_x\.attack$/);
assert.match(strictLeafDropError([leaf].concat(Array.from({ length: 12 }, (_, i) => `ui.g${i}`))), /all 13 in \.cache/);
assert.equal(leafDropSummary([leaf]), '[assets] 1 leaf dropped: no alternative on disk (audio.sfx.units.char_x.attack)');
assert.match(leafDropSummary(['a', 'b']), /^\[assets\] 2 leaves dropped: no alternative on disk \(a, b\)$/);
assert.match(leafDropSummary(Array.from({ length: 9 }, (_, i) => `f${i}`)), /\(f0, f1, f2, f3, f4, f5, f6, f7, …\)$/, 'long lists are truncated on the line');
assert.match(leafDropSummary([]), /^\[assets\] no leaf dropped: every planned leaf has a file on disk$/);
}); });
+16 -2
View File
@@ -3,6 +3,13 @@
// files that actually exist on disk into data/assets.json. Entries whose files // files that actually exist on disk into data/assets.json. Entries whose files
// are missing are dropped (never emitted as broken URLs); a `literal(value)` node // are missing are dropped (never emitted as broken URLs); a `literal(value)` node
// is emitted as it is (no files behind it). // is emitted as it is (no files behind it).
//
// A dropped entry is never silent: resolveTemplate reports every leaf none of whose
// alternatives is on disk in `droppedLeaves` (a leaf whose file is missing is an
// entry the client loses — a sound that never plays, not a broken URL it recovers
// from), besides the wider `misses` list (leaves and unresolved Spine models). The
// caller prints that list, keeps it in .cache/assets-report.json and can turn it
// into a failure (fetch-assets.mjs --strict / SP_ASSETS_STRICT).
import { existsSync, statSync } from 'node:fs'; import { existsSync, statSync } from 'node:fs';
import { createHash } from 'node:crypto'; import { createHash } from 'node:crypto';
@@ -80,10 +87,14 @@ export async function downloadLeaves(leaves, dl, root, label = 'files') {
* @param {Map<string, any>} o.spine resolved Spine entries by model key * @param {Map<string, any>} o.spine resolved Spine entries by model key
* @param {(rel:string)=>string|undefined} [o.sourceOf] URL a file was downloaded from (ledger), to report * @param {(rel:string)=>string|undefined} [o.sourceOf] URL a file was downloaded from (ledger), to report
* fallbacks that share the primary's path (e.g. an enemy icon taken from its base id) * fallbacks that share the primary's path (e.g. an enemy icon taken from its base id)
* @returns {{ value: any, misses: string[], fallbacks: string[], files: Set<string> }} * @returns {{ value: any, misses: string[], droppedLeaves: string[], fallbacks: string[], files: Set<string> }}
* `droppedLeaves`: the dotted paths of the leaves no alternative of which is on disk, i.e. exactly the entries the
* manifest loses here (a subset of `misses`, which also names the unresolved Spine models) — what a caller reports
* as dropped, and what --strict fails on.
*/ */
export function resolveTemplate(template, { root, spine, sourceOf = () => undefined }) { export function resolveTemplate(template, { root, spine, sourceOf = () => undefined }) {
const misses = []; const misses = [];
const droppedLeaves = [];
const fallbacks = []; const fallbacks = [];
const files = new Set(); const files = new Set();
const walk = (node, path) => { const walk = (node, path) => {
@@ -103,6 +114,9 @@ export function resolveTemplate(template, { root, spine, sourceOf = () => undefi
return assetUrl(a.rel); return assetUrl(a.rel);
} }
} }
// No alternative on disk: the entry is left out of the manifest, which is a real loss (never a broken URL) — so
// it is named as a dropped leaf, not only among the misses.
droppedLeaves.push(path);
misses.push(path); misses.push(path);
return undefined; return undefined;
} }
@@ -130,7 +144,7 @@ export function resolveTemplate(template, { root, spine, sourceOf = () => undefi
}; };
const isContainer = (v) => v && typeof v === 'object' && !isLeaf(v) && !isModelRef(v) && !isLiteral(v); const isContainer = (v) => v && typeof v === 'object' && !isLeaf(v) && !isModelRef(v) && !isLiteral(v);
const value = walk(template, ''); const value = walk(template, '');
return { value, misses, fallbacks, files }; return { value, misses, droppedLeaves, fallbacks, files };
} }
/** /**
+85 -9
View File
@@ -30,6 +30,16 @@
// the entries it would drop and exits 1; --allow-shrink (or --prune) writes the // the entries it would drop and exits 1; --allow-shrink (or --prune) writes the
// smaller manifest. // smaller manifest.
// //
// Every leaf the plan had resolved but no alternative of which is on disk is also
// reported on its own line — `[assets] N leaves dropped: no alternative on disk
// (…, audio.sfx.units.some_char.attack, …)` — and kept as `droppedLeaves` in the
// report, whether or not it shrinks the committed manifest: such a leaf is an
// entry the client loses (a sound that never plays), not a URL it falls back
// from. --strict (or SP_ASSETS_STRICT=1) turns those drops into exit code 1 for
// CI and packaging runs; by default the run only warns. No file is downloaded,
// dropped or kept differently either way, and a run whose files are all present
// produces the same manifest byte for byte.
//
// Beyond research 07's ids the plan covers what data/*.json adds: spawnable // Beyond research 07's ids the plan covers what data/*.json adds: spawnable
// enemies / tokens (data/enemies.json, data/tokens.json), the 自选 owned-6★ picks // enemies / tokens (data/enemies.json, data/tokens.json), the 自选 owned-6★ picks
// of data/backups.json `units` (art from 07's URL patterns) and their summons // of data/backups.json `units` (art from 07's URL patterns) and their summons
@@ -44,7 +54,8 @@
// //
// Usage: node tools/fetch-assets.mjs [--concurrency=16] [--force] [--offline] // Usage: node tools/fetch-assets.mjs [--concurrency=16] [--force] [--offline]
// [--dry-run] [--refresh-index] [--prune] // [--dry-run] [--refresh-index] [--prune]
// [--allow-shrink] [--add-only] [--local-spines] [--help] // [--allow-shrink] [--add-only] [--strict]
// [--local-spines] [--help]
import { readFile, writeFile, mkdir, rename, readdir, unlink } from 'node:fs/promises'; import { readFile, writeFile, mkdir, rename, readdir, unlink } from 'node:fs/promises';
import { existsSync, realpathSync } from 'node:fs'; import { existsSync, realpathSync } from 'node:fs';
@@ -103,6 +114,9 @@ const HELP = `Usage: node tools/fetch-assets.mjs [options]
(public/assets/local/** of tools/local-extract is never deleted); implies --allow-shrink (public/assets/local/** of tools/local-extract is never deleted); implies --allow-shrink
--allow-shrink write data/assets.json even when it loses entries the current one has --allow-shrink write data/assets.json even when it loses entries the current one has
(without it such a run keeps the current manifest, lists the entries and exits 1) (without it such a run keeps the current manifest, lists the entries and exits 1)
--strict exit 1 when a leaf was dropped for having no file on disk (the report's
droppedLeaves: an entry the plan resolved and the client would lose, e.g. a
sound that never plays); by default such a drop is only reported
--add-only download only files missing on disk; never re-download, rewrite or delete an existing --add-only download only files missing on disk; never re-download, rewrite or delete an existing
file, no font rebuild (a worktree sharing public/assets and public/fonts) file, no font rebuild (a worktree sharing public/assets and public/fonts)
--local-spines rewrite ${LOCAL_ENEMY_SPINES_FILE} and ${LOCAL_TOKEN_SPINES_FILE} from the --local-spines rewrite ${LOCAL_ENEMY_SPINES_FILE} and ${LOCAL_TOKEN_SPINES_FILE} from the
@@ -110,17 +124,28 @@ const HELP = `Usage: node tools/fetch-assets.mjs [options]
(public/assets/local/spine/enemy/, public/assets/local/spine/token/) (public/assets/local/spine/enemy/, public/assets/local/spine/token/)
--help this text --help this text
Environment: SP_ASSET_SOURCE sets the default source; SP_GITHUB_PROXY sets the Environment: SP_ASSET_SOURCE sets the default source; SP_GITHUB_PROXY sets the
HTTPS mirror prefix (default https://gh-proxy.com/; empty disables the proxy). HTTPS mirror prefix (default https://gh-proxy.com/; empty disables the proxy);
SP_ASSETS_STRICT=1 is the same as --strict (CI / packaging builds).
Mirror attempts have an 8 s response header timeout; response body has a separate idle timeout. Stops for this run after 3 consecutive Mirror attempts have an 8 s response header timeout; response body has a separate idle timeout. Stops for this run after 3 consecutive
failures. Only explicitly enabled GitHub downloads use the third-party proxy.`; failures. Only explicitly enabled GitHub downloads use the third-party proxy.`;
/**
* True for the usual truthy spellings of a boolean environment variable (`1`, `true`, `yes`, `on`); empty, `0`,
* `false`, `no` and `off` are false, and so is anything absent.
* @param {string|undefined} value
* @returns {boolean}
*/
function envFlag(value) {
return typeof value === 'string' && value !== '' && !/^(0|false|no|off)$/i.test(value);
}
/** /**
* Parse CLI flags. * Parse CLI flags.
* @param {string[]} argv * @param {string[]} argv
* @returns {{concurrency:number, force:boolean, offline:boolean, dryRun:boolean, refreshIndex:boolean, prune:boolean, allowShrink:boolean, addOnly:boolean, localSpines:boolean, voiceLang:string, voiceAll:boolean, help:boolean, source:string}} * @returns {{concurrency:number, force:boolean, offline:boolean, dryRun:boolean, refreshIndex:boolean, prune:boolean, allowShrink:boolean, strict:boolean, addOnly:boolean, localSpines:boolean, voiceLang:string, voiceAll:boolean, help:boolean, source:string}}
*/ */
export function parseArgs(argv) { export function parseArgs(argv) {
const o = { concurrency: 16, force: false, offline: false, dryRun: false, refreshIndex: false, prune: false, allowShrink: false, addOnly: false, localSpines: false, voiceLang: 'cn', voiceAll: false, help: false, source: process.env.SP_ASSET_SOURCE || 'direct' }; const o = { concurrency: 16, force: false, offline: false, dryRun: false, refreshIndex: false, prune: false, allowShrink: false, strict: envFlag(process.env.SP_ASSETS_STRICT), addOnly: false, localSpines: false, voiceLang: 'cn', voiceAll: false, help: false, source: process.env.SP_ASSET_SOURCE || 'direct' };
for (const a of argv) { for (const a of argv) {
const [k, v] = a.split('='); const [k, v] = a.split('=');
if (k === '--concurrency') o.concurrency = Math.max(1, Math.min(64, parseInt(v, 10) || 16)); if (k === '--concurrency') o.concurrency = Math.max(1, Math.min(64, parseInt(v, 10) || 16));
@@ -131,6 +156,7 @@ export function parseArgs(argv) {
else if (k === '--refresh-index') o.refreshIndex = true; else if (k === '--refresh-index') o.refreshIndex = true;
else if (k === '--prune') o.prune = true; else if (k === '--prune') o.prune = true;
else if (k === '--allow-shrink') o.allowShrink = true; else if (k === '--allow-shrink') o.allowShrink = true;
else if (k === '--strict') o.strict = true;
else if (k === '--add-only') o.addOnly = true; else if (k === '--add-only') o.addOnly = true;
else if (k === '--local-spines') o.localSpines = true; else if (k === '--local-spines') o.localSpines = true;
else if (k === '--voice-lang') { if (!VOICE_DIRS[v]) throw new Error(`unknown --voice-lang ${v} (cn | jp | en | kr)`); o.voiceLang = v; } else if (k === '--voice-lang') { if (!VOICE_DIRS[v]) throw new Error(`unknown --voice-lang ${v} (cn | jp | en | kr)`); o.voiceLang = v; }
@@ -184,6 +210,50 @@ export function shrinkGuard(prev, next, { allowShrink = false, prune = false } =
return { dropped, write: dropped.length === 0 || !!allowShrink || !!prune }; return { dropped, write: dropped.length === 0 || !!allowShrink || !!prune };
} }
/**
* The one summary line for the leaves the plan had resolved but whose every alternative file is missing on disk
* (resolveTemplate `droppedLeaves`). Each of them disappears from data/assets.json, so the client has no entry at all —
* for a sound, silence that no "the client uses fallbacks" wording explains. Printed on every run, empty or not: a
* green line is the check that nothing was dropped.
* @param {string[]} droppedLeaves dotted template paths
* @param {number} [max] how many paths the line names before "…"
* @returns {string}
*/
export function leafDropSummary(droppedLeaves, max = 8) {
const n = droppedLeaves.length;
if (!n) return '[assets] no leaf dropped: every planned leaf has a file on disk';
return `[assets] ${n} ${n === 1 ? 'leaf' : 'leaves'} dropped: no alternative on disk ` +
`(${droppedLeaves.slice(0, max).join(', ')}${n > max ? ', …' : ''})`;
}
/**
* The error line of a --strict / SP_ASSETS_STRICT run that dropped at least one leaf for having no alternative on
* disk. CI and packaging runs pass --strict so a manifest with such holes cannot ship (the entry is gone from the
* client's manifest; nothing retries it later, the next plain run only warns again).
* @param {string[]} droppedLeaves dotted template paths
* @param {number} [max] how many paths the line names before pointing at the report
* @returns {string}
*/
export function strictLeafDropError(droppedLeaves, max = 10) {
const n = droppedLeaves.length;
return `ERROR: ${n} ${n === 1 ? 'leaf' : 'leaves'} dropped for having no alternative on disk ` +
`(--strict / SP_ASSETS_STRICT=1): ${droppedLeaves.slice(0, max).join(', ')}` +
`${n > max ? `, … (all ${n} in ${relative(ROOT, REPORT)})` : ''}`;
}
/**
* Exit code of a finished run: 0 only when nothing failed — every required asset is present, no leaf was dropped
* while --strict / SP_ASSETS_STRICT is on, and the manifest was written (the shrink guard keeps the current one when
* the rebuild would lose entries).
* @param {{requiredMisses?:number, droppedLeaves?:number, strict?:boolean, manifestWritten?:boolean}} o
* @returns {number} 0 or 1
*/
export function runExitCode({ requiredMisses = 0, droppedLeaves = 0, strict = false, manifestWritten = true } = {}) {
if (requiredMisses) return 1;
if (strict && droppedLeaves) return 1;
return manifestWritten ? 0 : 1;
}
/** /**
* Files under public/assets the manifest does not reference (`--prune` deletes them). public/assets/local/** belongs to * Files under public/assets the manifest does not reference (`--prune` deletes them). public/assets/local/** belongs to
* tools/local-extract (data/local-assets.json) and is never an orphan: --prune used to delete all of it. Compared without * tools/local-extract (data/local-assets.json) and is never an orphan: --prune used to delete all of it. Compared without
@@ -411,11 +481,15 @@ async function main() {
const charIds = Object.keys(assets07.operators || {}); const charIds = Object.keys(assets07.operators || {});
const required = requiredMisses(manifest, charIds); const required = requiredMisses(manifest, charIds);
// Leaves the plan had resolved and the manifest loses here (no alternative of theirs on disk): reported on their own
// line below and kept in the report, whether or not they shrink the committed manifest.
const droppedLeaves = resolved.droppedLeaves;
const report = { const report = {
downloadedBytes: dl.totals.bytesDownloaded, downloadedBytes: dl.totals.bytesDownloaded,
totals: dl.totals, totals: dl.totals,
stats: manifest.stats, stats: manifest.stats,
requiredMisses: required, requiredMisses: required,
droppedLeaves,
misses: resolved.misses, misses: resolved.misses,
downloadErrors, // leaves whose primary failed transiently (fallbacks not tried; re-run to retry) downloadErrors, // leaves whose primary failed transiently (fallbacks not tried; re-run to retry)
fallbacks: resolved.fallbacks, fallbacks: resolved.fallbacks,
@@ -444,8 +518,9 @@ async function main() {
log(`fonts: ${Object.values(fonts.files).map((f) => f.woff2 || f.original).join(', ') || 'none'}`); log(`fonts: ${Object.values(fonts.files).map((f) => f.woff2 || f.original).join(', ') || 'none'}`);
if (resolved.fallbacks.length) { log(`fallbacks used (${resolved.fallbacks.length}):`); for (const f of resolved.fallbacks.slice(0, 20)) log(` ${f}`); } if (resolved.fallbacks.length) { log(`fallbacks used (${resolved.fallbacks.length}):`); for (const f of resolved.fallbacks.slice(0, 20)) log(` ${f}`); }
if (downloadErrors.length) log(`download errors (${downloadErrors.length}, re-run to retry): ${downloadErrors.slice(0, 10).join(', ')}`); if (downloadErrors.length) log(`download errors (${downloadErrors.length}, re-run to retry): ${downloadErrors.slice(0, 10).join(', ')}`);
log(leafDropSummary(droppedLeaves));
if (resolved.misses.length) { if (resolved.misses.length) {
log(`missing (${resolved.misses.length}, omitted from manifest; client uses fallbacks):`); log(`missing (${resolved.misses.length}, omitted from manifest; the client falls back only where the entry has another alternative):`);
for (const m of resolved.misses.slice(0, 40)) log(` ${m}`); for (const m of resolved.misses.slice(0, 40)) log(` ${m}`);
if (resolved.misses.length > 40) log(` … see ${REPORT}`); if (resolved.misses.length > 40) log(` … see ${REPORT}`);
} }
@@ -460,11 +535,12 @@ async function main() {
if (!guard.write) log(' re-run to retry the downloads (--refresh-index for the audio/model indexes), or pass --allow-shrink (or --prune) to write the smaller manifest'); if (!guard.write) log(' re-run to retry the downloads (--refresh-index for the audio/model indexes), or pass --allow-shrink (or --prune) to write the smaller manifest');
} }
log(`manifest: ${MANIFEST}${guard.write ? '' : ' (kept)'} · report: ${REPORT} · ${((Date.now() - t0) / 1000).toFixed(0)} s`); log(`manifest: ${MANIFEST}${guard.write ? '' : ' (kept)'} · report: ${REPORT} · ${((Date.now() - t0) / 1000).toFixed(0)} s`);
if (required.length) { if (required.length) log(`ERROR: ${required.length} required assets missing: ${required.slice(0, 10).join(', ')}`);
log(`ERROR: ${required.length} required assets missing: ${required.slice(0, 10).join(', ')}`); if (opts.strict && droppedLeaves.length) {
return 1; log(strictLeafDropError(droppedLeaves));
log(' a plain run downloads them again (--refresh-index if the upstream index lost them); without --strict / SP_ASSETS_STRICT this is only a warning');
} }
return guard.write ? 0 : 1; return runExitCode({ requiredMisses: required.length, droppedLeaves: droppedLeaves.length, strict: opts.strict, manifestWritten: guard.write });
} }
// run only as a script (tests import parseArgs / shrinkGuard) // run only as a script (tests import parseArgs / shrinkGuard)