Report missing asset leaves and add opt-in strict mode (#421)
This commit is contained in:
1 parent
174ccdf94d
commit
3e3919c917
6 files changed
+211
-14
No files matched your search
+23
-3
@@ -24,6 +24,7 @@ npm run assets # = node tools/vendor.mjs && node tools/fetch-assets.mjs
|
||||
| `--voice-all` | Plan every official voice slot, including the prep-only lines no battle plays (干员报到 / 编入队伍 / 任命队长 — 360 files, one per operator and slot). Off by default: nothing requests them, so planning them only makes every run download more. |
|
||||
| `--prune` | Delete files under `public/assets/` that the manifest no longer references, for example after a mapping change. Without this flag they are only listed in the report. `public/assets/local/` (written by `tools/local-extract`) is never pruned. Implies `--allow-shrink`. |
|
||||
| `--allow-shrink` | Write `data/assets.json` even when it loses entries the current one has (see "The manifest never shrinks by accident" below). |
|
||||
| `--strict` | Exit 1 when a leaf was dropped for having no file on disk, instead of only reporting it (see "A dropped leaf is reported" below). The same as the environment variable `SP_ASSETS_STRICT=1` — for CI and packaging builds. |
|
||||
| `--add-only` | For a checkout whose `public/assets/` and `public/fonts/` are shared with another one (a git worktree with symlinked asset folders): download only the files missing on disk and never re-download, rewrite or delete an existing file — atlases already on disk are left as they are, the fonts are not rebuilt (the manifest keeps its current `fonts`). Not with `--prune` / `--force`. |
|
||||
| `--local-spines` | Rewrite `tools/assets/local-enemy-spines.json` and `tools/assets/local-token-spines.json` (the metadata of the enemy and token models only the local client has, see "Enemy aliases" and "Token models from the local client") from the models `tools/local-extract/extract.py` extracted to `public/assets/local/spine/enemy/` and `public/assets/local/spine/token/`. Run it after a game update changed them; without it the committed files are used and a differing extraction only gets a warning. |
|
||||
|
||||
@@ -38,6 +39,23 @@ smaller manifest is intended, for example after a mapping change. Build fields (
|
||||
plan legitimately narrows — like the 干员战斗语音 default, which no longer plans the three prep-only slots (360 entries,
|
||||
DESIGN §21.30) — reports exactly those entries and needs `--allow-shrink` once; the list it prints is the check.
|
||||
|
||||
**A dropped leaf is reported, never silent.** `resolveTemplate` leaves out every entry none of whose alternative files
|
||||
is on disk — for a leaf that is an entry the client loses outright: a unit whose `attack` sound was never downloaded has
|
||||
no `audio.sfx.units[charId].attack` at all, so the sound is simply missing (nothing falls back; there is no URL to
|
||||
retry). That is a different thing from an entry whose *fallback* was used. The run therefore prints a summary naming at most eight such leaves
|
||||
and records the complete list as `droppedLeaves` (a subset of `misses`, which also lists the
|
||||
unresolved Spine models):
|
||||
|
||||
```
|
||||
[assets] 1 leaf dropped: no alternative on disk (audio.sfx.units.some_char.attack)
|
||||
```
|
||||
|
||||
A run with nothing dropped prints `[assets] no leaf dropped: every planned leaf has a file on disk`. This is only
|
||||
observability: which files are planned, downloaded and written does not change. `--strict` (or `SP_ASSETS_STRICT=1`)
|
||||
turns those drops into a failure (exit 1, the leaves named) so a CI or packaging run cannot ship a manifest with such
|
||||
holes — the committed manifest's shrink guard cannot see them once the entry is already gone. The manifest may
|
||||
already have been written before this check returns failure; strict mode does not roll it back.
|
||||
|
||||
The script is **idempotent**. A file on disk is kept, not re-downloaded, when any one of these holds:
|
||||
- its size matches the ledger entry from a previous download (`.cache/assets-ledger.json`);
|
||||
- its size matches the byte count recorded in research;
|
||||
@@ -59,10 +77,12 @@ Outputs:
|
||||
- `data/assets.json`: the manifest (committed).
|
||||
- `public/assets/**`: art and audio (git-ignored).
|
||||
- `public/fonts/*`: fonts and `fonts.css`.
|
||||
- `.cache/assets-report.json`: misses, fallbacks and notes from the last run.
|
||||
- `.cache/assets-report.json`: misses, fallbacks, `droppedLeaves` (the leaves left out of the manifest for having no file
|
||||
on disk) and notes from the last run.
|
||||
- `.cache/spine-info.json`: skeleton parse cache.
|
||||
|
||||
The run exits with code 1 if any pool operator is missing its avatar, its portrait or its Front Spine.
|
||||
The run exits with code 1 if any pool operator is missing its avatar, its portrait or its Front Spine, and — with
|
||||
`--strict` / `SP_ASSETS_STRICT=1` — if any leaf was dropped for having no file on disk.
|
||||
|
||||
Upstream indexes are cached under `.cache/`. They are downloaded when missing:
|
||||
- `.cache/gamedata/excel/audio_data.json`, from `Kengxxiao/ArknightsGameData` (zh_CN).
|
||||
@@ -100,7 +120,7 @@ The `stem` of a Spine model is the upstream file name. Two examples: `char_107_l
|
||||
|
||||
### Id scope
|
||||
|
||||
- **Operators:** all 138 pool charIds from `activity_table` (`charShopChessDatas[*].charId ∪ backupCharId`), including hidden chess and backup operators; plus every unit of `data/backups.json` research 07 does not list — the 71 自选 owned-6★ picks (`diy.ownedPool`, DATA.md §18; the collab operators are not in the data, and 焰狐龙梓兰's entries left the manifest when she left the pool in 0.2.0) — planned from 07's URL patterns (`tools/assets/plan.mjs patternOperator`, `tools/fetch-assets.mjs dataExtras`): avatar and portrait (E0–E1 and E2), the default-skin battle Spine Front / Back, the icon and skill sound of each skill, the sub-profession icon. 209 operators in all (206 with a Back model).
|
||||
- **Operators:** all 138 pool charIds from `activity_table` (`charShopChessDatas[*].charId ∪ backupCharId`), including hidden chess and backup operators; plus every unit of `data/backups.json` research 07 does not list — the 72 自选 owned-6★ picks (`diy.ownedPool`, DATA.md §18; the collab operators are not in the data, and 焰狐龙梓兰's entries left the manifest when she left the pool in 0.2.0) — planned from 07's URL patterns (`tools/assets/plan.mjs patternOperator`, `tools/fetch-assets.mjs dataExtras`): avatar and portrait (E0–E1 and E2), the default-skin battle Spine Front / Back, the icon and skill sound of each skill, the sub-profession icon. 209 operators in all (206 with a Back model).
|
||||
- **Tokens:** the 20 pool tokens, and the 38 summons of the 自选 picks (`data/backups.json tokens`) as extra tokens (below): their avatars; no dump carries the battle Spine of the 自选 summons (upstream has at most skin variants, which the default locations miss), so the web manifest has no model for them — 35 of them, and 4 pool summons, have the official model as an optional local-client overlay ("Token models from the local client"); 3 have no model in the game at all.
|
||||
- **Enemies:** 253 ids planned, 252 in the manifest (心烛 has no assets). The set is the union of:
|
||||
- the 07 enemy list;
|
||||
|
||||
@@ -87,3 +87,12 @@ Only that granted trait additionally listens to `dollSwap`, in both directions,
|
||||
Deployment and switches share the original cap, active-bond checks and grant deduplication. Other deployment
|
||||
traits keep their existing triggers. The 7 regression cases cover both real dollkeepers, natural and S2 returns,
|
||||
shared caps, duplicate grant holders, inactive bonds and unrelated timed windows.
|
||||
|
||||
### 28.6 Asset download visibility — PR #421
|
||||
|
||||
Ported from [PR #421](https://github.com/sganggs/Stronghold-Protocol/pull/421) by @LimitlessHPPK.
|
||||
The asset resolver records every planned leaf with no alternative on disk as `droppedLeaves` in its report;
|
||||
the terminal prints a bounded summary. `--strict` / `SP_ASSETS_STRICT=1` returns exit 1 for those omissions.
|
||||
The default remains a warning, and the existing manifest shrink guard remains in force. Strict mode changes
|
||||
the outcome code; it is not a transaction and does not roll back a manifest already written. Temporary test
|
||||
assets are cleaned up after each case. This is an opt-in tool check, not an added packaging requirement.
|
||||
@@ -443,10 +443,12 @@ describe('audio banks and plan id sets', () => {
|
||||
const r = resolveTemplate(tpl, { root: ROOT, spine: new Map([['k', { skel: '/assets/s.skel', atlas: '/assets/s.atlas', textures: [] }]]) });
|
||||
assert.deepEqual(Object.keys(r.value).sort(), ['keep', 'm']);
|
||||
assert.deepEqual(r.misses.sort(), ['a', 'b.c']);
|
||||
assert.deepEqual(r.droppedLeaves, ['a', 'b.c'], 'a leaf no alternative of which is on disk is a dropped leaf');
|
||||
assert.equal(collectLeaves(tpl).length, 2);
|
||||
const ok = resolveTemplate({ p: { alts: [{ rel: 'nope.png', urls: ['x'] }, { rel: 'package.json', urls: ['y'] }] } }, { root: ROOT, spine: new Map() });
|
||||
assert.equal(ok.value.p, '/assets/package.json');
|
||||
assert.equal(ok.fallbacks.length, 1);
|
||||
assert.deepEqual(ok.droppedLeaves, [], 'a fallback on disk is not a drop');
|
||||
});
|
||||
|
||||
test('an array whose lines are all missing is dropped, with the entry it empties (a voice slot never downloaded)', () => {
|
||||
@@ -462,6 +464,44 @@ describe('audio banks and plan id sets', () => {
|
||||
assert.deepEqual(r.value.voice, { b: { select: ['/assets/package.json'] } }, 'operator a has no line on disk: no entry at all');
|
||||
assert.deepEqual(r.value.empty, [], 'an array that was empty in the template stays');
|
||||
assert.deepEqual(r.misses.sort(), ['voice.a.place[0]', 'voice.a.select[0]', 'voice.a.select[1]', 'voice.b.select[0]']);
|
||||
assert.deepEqual(r.droppedLeaves.sort(), r.misses.slice().sort(), 'every dropped line is a dropped leaf too');
|
||||
});
|
||||
|
||||
// A unit's attack sound is a leaf like any other: when its file was never downloaded (or a checkout lost it) the whole
|
||||
// key vanishes from data/assets.json and the client plays nothing — there is no fallback and no URL to retry. The
|
||||
// resolution therefore reports it as a dropped leaf, which the run prints and --strict fails on
|
||||
// (tools/fetch-assets.mjs; docs/ASSETS.md "A dropped leaf is reported, never silent").
|
||||
test('move a resolved sound aside: the leaf leaves the manifest and is reported as dropped, never silently', async (t) => {
|
||||
const { mkdtemp, mkdir, writeFile, rm } = await import('node:fs/promises');
|
||||
const { tmpdir } = await import('node:os');
|
||||
const root = await mkdtemp(join(tmpdir(), 'sp-leaf-drop-'));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const rel = 'audio/sfx/player/p_atk/p_atk_somechar_n.mp3';
|
||||
const imp = 'audio/sfx/player/p_imp/p_imp_somechar_n.mp3';
|
||||
const ui = 'audio/sfx/general/g_ui/g_ui_btn_h.mp3';
|
||||
const leaf = (r) => ({ alts: [{ rel: r, urls: [`https://raw.githubusercontent.com/o/r/main/${r}`], kind: 'mp3' }] });
|
||||
const tpl = { audio: { sfx: { units: { char_x: { attack: leaf(rel), hit: leaf(imp) } }, ui: { click: leaf(ui) } } } };
|
||||
for (const r of [rel, imp, ui]) {
|
||||
await mkdir(join(root, dirname(r)), { recursive: true });
|
||||
await writeFile(join(root, r), 'ID3');
|
||||
}
|
||||
const before = resolveTemplate(tpl, { root, spine: new Map() });
|
||||
assert.equal(JSON.stringify(before.value), JSON.stringify({
|
||||
audio: {
|
||||
sfx: {
|
||||
units: { char_x: { attack: `/assets/${rel}`, hit: `/assets/${imp}` } },
|
||||
ui: { click: `/assets/${ui}` },
|
||||
},
|
||||
},
|
||||
}), 'every file on disk: the manifest is exactly the plan, byte for byte');
|
||||
assert.deepEqual([before.misses, before.droppedLeaves, before.fallbacks], [[], [], []]);
|
||||
|
||||
await rm(join(root, rel)); // the download never happened / the file went missing
|
||||
const after = resolveTemplate(tpl, { root, spine: new Map() });
|
||||
assert.equal(Object.hasOwn(after.value.audio.sfx.units.char_x, 'attack'), false, 'the key is gone from the manifest');
|
||||
assert.equal(after.value.audio.sfx.units.char_x.hit, `/assets/${imp}`, 'the roles whose file is on disk stay');
|
||||
assert.deepEqual(after.droppedLeaves, ['audio.sfx.units.char_x.attack'], 'and the drop is reported, not silent');
|
||||
assert.ok(after.misses.includes('audio.sfx.units.char_x.attack'), 'the wider miss list names it too');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -96,4 +96,42 @@ test('fetch-assets still runs as a script: --help lists --allow-shrink', () => {
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.match(r.stdout, /--allow-shrink/);
|
||||
assert.match(r.stdout, /--prune .*\n.*implies --allow-shrink/);
|
||||
assert.match(r.stdout, /--strict\s+exit 1 when a leaf was dropped/);
|
||||
assert.match(r.stdout, /SP_ASSETS_STRICT=1 is the same as --strict/);
|
||||
});
|
||||
|
||||
// A leaf whose every alternative file is missing on disk disappears from data/assets.json: the client has no entry at all
|
||||
// (a unit whose attack sound was never downloaded plays nothing), and the shrink guard cannot see it once it is gone from
|
||||
// the committed manifest. The run reports it as `droppedLeaves`; --strict / SP_ASSETS_STRICT=1 makes it a failure for CI
|
||||
// and packaging runs, which is the only part that changes a run's outcome (docs/ASSETS.md).
|
||||
test('--strict / SP_ASSETS_STRICT=1 fails a run that dropped a leaf for having no file on disk, instead of warning', async () => {
|
||||
const { parseArgs, runExitCode, leafDropSummary, strictLeafDropError } = await import('../tools/fetch-assets.mjs');
|
||||
const leaf = 'audio.sfx.units.char_x.attack';
|
||||
assert.equal(parseArgs([]).strict, false, 'a plain run only warns');
|
||||
assert.equal(parseArgs(['--strict']).strict, true);
|
||||
assert.throws(() => parseArgs(['--stricts']), /unknown option/);
|
||||
// the environment variable is the CI / packaging switch, like SP_ASSET_SOURCE
|
||||
const saved = process.env.SP_ASSETS_STRICT;
|
||||
try {
|
||||
for (const v of ['1', 'true', 'YES', 'on']) { process.env.SP_ASSETS_STRICT = v; assert.equal(parseArgs([]).strict, true, `SP_ASSETS_STRICT=${v}`); }
|
||||
for (const v of ['', '0', 'false', 'no', 'off']) { process.env.SP_ASSETS_STRICT = v; assert.equal(parseArgs([]).strict, false, `SP_ASSETS_STRICT=${v}`); }
|
||||
delete process.env.SP_ASSETS_STRICT;
|
||||
assert.equal(parseArgs([]).strict, false, 'unset');
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.SP_ASSETS_STRICT; else process.env.SP_ASSETS_STRICT = saved;
|
||||
}
|
||||
// the exit code: a dropped leaf fails a strict run and only warns otherwise
|
||||
assert.equal(runExitCode({ droppedLeaves: 1 }), 0, 'default: reported, the run still succeeds');
|
||||
assert.equal(runExitCode({ droppedLeaves: 48, strict: true }), 1, '--strict: the drop is a failure');
|
||||
assert.equal(runExitCode({ droppedLeaves: 0, strict: true }), 0, 'nothing dropped: a strict run succeeds');
|
||||
assert.equal(runExitCode({}), 0);
|
||||
assert.equal(runExitCode({ requiredMisses: 1 }), 1, 'unchanged: a missing avatar / portrait / Front Spine always fails');
|
||||
assert.equal(runExitCode({ manifestWritten: false }), 1, 'unchanged: the shrink guard keeps the run red');
|
||||
// and the messages name the leaf
|
||||
assert.match(strictLeafDropError([leaf]), /^ERROR: 1 leaf dropped for having no alternative on disk \(--strict \/ SP_ASSETS_STRICT=1\): audio\.sfx\.units\.char_x\.attack$/);
|
||||
assert.match(strictLeafDropError([leaf].concat(Array.from({ length: 12 }, (_, i) => `ui.g${i}`))), /all 13 in \.cache/);
|
||||
assert.equal(leafDropSummary([leaf]), '[assets] 1 leaf dropped: no alternative on disk (audio.sfx.units.char_x.attack)');
|
||||
assert.match(leafDropSummary(['a', 'b']), /^\[assets\] 2 leaves dropped: no alternative on disk \(a, b\)$/);
|
||||
assert.match(leafDropSummary(Array.from({ length: 9 }, (_, i) => `f${i}`)), /\(f0, f1, f2, f3, f4, f5, f6, f7, …\)$/, 'long lists are truncated on the line');
|
||||
assert.match(leafDropSummary([]), /^\[assets\] no leaf dropped: every planned leaf has a file on disk$/);
|
||||
});
|
||||
@@ -3,6 +3,13 @@
|
||||
// files that actually exist on disk into data/assets.json. Entries whose files
|
||||
// are missing are dropped (never emitted as broken URLs); a `literal(value)` node
|
||||
// is emitted as it is (no files behind it).
|
||||
//
|
||||
// A dropped entry is never silent: resolveTemplate reports every leaf none of whose
|
||||
// alternatives is on disk in `droppedLeaves` (a leaf whose file is missing is an
|
||||
// entry the client loses — a sound that never plays, not a broken URL it recovers
|
||||
// from), besides the wider `misses` list (leaves and unresolved Spine models). The
|
||||
// caller prints that list, keeps it in .cache/assets-report.json and can turn it
|
||||
// into a failure (fetch-assets.mjs --strict / SP_ASSETS_STRICT).
|
||||
|
||||
import { existsSync, statSync } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
@@ -80,10 +87,14 @@ export async function downloadLeaves(leaves, dl, root, label = 'files') {
|
||||
* @param {Map<string, any>} o.spine resolved Spine entries by model key
|
||||
* @param {(rel:string)=>string|undefined} [o.sourceOf] URL a file was downloaded from (ledger), to report
|
||||
* fallbacks that share the primary's path (e.g. an enemy icon taken from its base id)
|
||||
* @returns {{ value: any, misses: string[], fallbacks: string[], files: Set<string> }}
|
||||
* @returns {{ value: any, misses: string[], droppedLeaves: string[], fallbacks: string[], files: Set<string> }}
|
||||
* `droppedLeaves`: the dotted paths of the leaves no alternative of which is on disk, i.e. exactly the entries the
|
||||
* manifest loses here (a subset of `misses`, which also names the unresolved Spine models) — what a caller reports
|
||||
* as dropped, and what --strict fails on.
|
||||
*/
|
||||
export function resolveTemplate(template, { root, spine, sourceOf = () => undefined }) {
|
||||
const misses = [];
|
||||
const droppedLeaves = [];
|
||||
const fallbacks = [];
|
||||
const files = new Set();
|
||||
const walk = (node, path) => {
|
||||
@@ -103,6 +114,9 @@ export function resolveTemplate(template, { root, spine, sourceOf = () => undefi
|
||||
return assetUrl(a.rel);
|
||||
}
|
||||
}
|
||||
// No alternative on disk: the entry is left out of the manifest, which is a real loss (never a broken URL) — so
|
||||
// it is named as a dropped leaf, not only among the misses.
|
||||
droppedLeaves.push(path);
|
||||
misses.push(path);
|
||||
return undefined;
|
||||
}
|
||||
@@ -130,7 +144,7 @@ export function resolveTemplate(template, { root, spine, sourceOf = () => undefi
|
||||
};
|
||||
const isContainer = (v) => v && typeof v === 'object' && !isLeaf(v) && !isModelRef(v) && !isLiteral(v);
|
||||
const value = walk(template, '');
|
||||
return { value, misses, fallbacks, files };
|
||||
return { value, misses, droppedLeaves, fallbacks, files };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+85
-9
@@ -30,6 +30,16 @@
|
||||
// the entries it would drop and exits 1; --allow-shrink (or --prune) writes the
|
||||
// smaller manifest.
|
||||
//
|
||||
// Every leaf the plan had resolved but no alternative of which is on disk is also
|
||||
// reported on its own line — `[assets] N leaves dropped: no alternative on disk
|
||||
// (…, audio.sfx.units.some_char.attack, …)` — and kept as `droppedLeaves` in the
|
||||
// report, whether or not it shrinks the committed manifest: such a leaf is an
|
||||
// entry the client loses (a sound that never plays), not a URL it falls back
|
||||
// from. --strict (or SP_ASSETS_STRICT=1) turns those drops into exit code 1 for
|
||||
// CI and packaging runs; by default the run only warns. No file is downloaded,
|
||||
// dropped or kept differently either way, and a run whose files are all present
|
||||
// produces the same manifest byte for byte.
|
||||
//
|
||||
// Beyond research 07's ids the plan covers what data/*.json adds: spawnable
|
||||
// enemies / tokens (data/enemies.json, data/tokens.json), the 自选 owned-6★ picks
|
||||
// of data/backups.json `units` (art from 07's URL patterns) and their summons
|
||||
@@ -44,7 +54,8 @@
|
||||
//
|
||||
// Usage: node tools/fetch-assets.mjs [--concurrency=16] [--force] [--offline]
|
||||
// [--dry-run] [--refresh-index] [--prune]
|
||||
// [--allow-shrink] [--add-only] [--local-spines] [--help]
|
||||
// [--allow-shrink] [--add-only] [--strict]
|
||||
// [--local-spines] [--help]
|
||||
|
||||
import { readFile, writeFile, mkdir, rename, readdir, unlink } from 'node:fs/promises';
|
||||
import { existsSync, realpathSync } from 'node:fs';
|
||||
@@ -103,6 +114,9 @@ const HELP = `Usage: node tools/fetch-assets.mjs [options]
|
||||
(public/assets/local/** of tools/local-extract is never deleted); implies --allow-shrink
|
||||
--allow-shrink write data/assets.json even when it loses entries the current one has
|
||||
(without it such a run keeps the current manifest, lists the entries and exits 1)
|
||||
--strict exit 1 when a leaf was dropped for having no file on disk (the report's
|
||||
droppedLeaves: an entry the plan resolved and the client would lose, e.g. a
|
||||
sound that never plays); by default such a drop is only reported
|
||||
--add-only download only files missing on disk; never re-download, rewrite or delete an existing
|
||||
file, no font rebuild (a worktree sharing public/assets and public/fonts)
|
||||
--local-spines rewrite ${LOCAL_ENEMY_SPINES_FILE} and ${LOCAL_TOKEN_SPINES_FILE} from the
|
||||
@@ -110,17 +124,28 @@ const HELP = `Usage: node tools/fetch-assets.mjs [options]
|
||||
(public/assets/local/spine/enemy/, public/assets/local/spine/token/)
|
||||
--help this text
|
||||
Environment: SP_ASSET_SOURCE sets the default source; SP_GITHUB_PROXY sets the
|
||||
HTTPS mirror prefix (default https://gh-proxy.com/; empty disables the proxy).
|
||||
HTTPS mirror prefix (default https://gh-proxy.com/; empty disables the proxy);
|
||||
SP_ASSETS_STRICT=1 is the same as --strict (CI / packaging builds).
|
||||
Mirror attempts have an 8 s response header timeout; response body has a separate idle timeout. Stops for this run after 3 consecutive
|
||||
failures. Only explicitly enabled GitHub downloads use the third-party proxy.`;
|
||||
|
||||
/**
|
||||
* True for the usual truthy spellings of a boolean environment variable (`1`, `true`, `yes`, `on`); empty, `0`,
|
||||
* `false`, `no` and `off` are false, and so is anything absent.
|
||||
* @param {string|undefined} value
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function envFlag(value) {
|
||||
return typeof value === 'string' && value !== '' && !/^(0|false|no|off)$/i.test(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse CLI flags.
|
||||
* @param {string[]} argv
|
||||
* @returns {{concurrency:number, force:boolean, offline:boolean, dryRun:boolean, refreshIndex:boolean, prune:boolean, allowShrink:boolean, addOnly:boolean, localSpines:boolean, voiceLang:string, voiceAll:boolean, help:boolean, source:string}}
|
||||
* @returns {{concurrency:number, force:boolean, offline:boolean, dryRun:boolean, refreshIndex:boolean, prune:boolean, allowShrink:boolean, strict:boolean, addOnly:boolean, localSpines:boolean, voiceLang:string, voiceAll:boolean, help:boolean, source:string}}
|
||||
*/
|
||||
export function parseArgs(argv) {
|
||||
const o = { concurrency: 16, force: false, offline: false, dryRun: false, refreshIndex: false, prune: false, allowShrink: false, addOnly: false, localSpines: false, voiceLang: 'cn', voiceAll: false, help: false, source: process.env.SP_ASSET_SOURCE || 'direct' };
|
||||
const o = { concurrency: 16, force: false, offline: false, dryRun: false, refreshIndex: false, prune: false, allowShrink: false, strict: envFlag(process.env.SP_ASSETS_STRICT), addOnly: false, localSpines: false, voiceLang: 'cn', voiceAll: false, help: false, source: process.env.SP_ASSET_SOURCE || 'direct' };
|
||||
for (const a of argv) {
|
||||
const [k, v] = a.split('=');
|
||||
if (k === '--concurrency') o.concurrency = Math.max(1, Math.min(64, parseInt(v, 10) || 16));
|
||||
@@ -131,6 +156,7 @@ export function parseArgs(argv) {
|
||||
else if (k === '--refresh-index') o.refreshIndex = true;
|
||||
else if (k === '--prune') o.prune = true;
|
||||
else if (k === '--allow-shrink') o.allowShrink = true;
|
||||
else if (k === '--strict') o.strict = true;
|
||||
else if (k === '--add-only') o.addOnly = true;
|
||||
else if (k === '--local-spines') o.localSpines = true;
|
||||
else if (k === '--voice-lang') { if (!VOICE_DIRS[v]) throw new Error(`unknown --voice-lang ${v} (cn | jp | en | kr)`); o.voiceLang = v; }
|
||||
@@ -184,6 +210,50 @@ export function shrinkGuard(prev, next, { allowShrink = false, prune = false } =
|
||||
return { dropped, write: dropped.length === 0 || !!allowShrink || !!prune };
|
||||
}
|
||||
|
||||
/**
|
||||
* The one summary line for the leaves the plan had resolved but whose every alternative file is missing on disk
|
||||
* (resolveTemplate `droppedLeaves`). Each of them disappears from data/assets.json, so the client has no entry at all —
|
||||
* for a sound, silence that no "the client uses fallbacks" wording explains. Printed on every run, empty or not: a
|
||||
* green line is the check that nothing was dropped.
|
||||
* @param {string[]} droppedLeaves dotted template paths
|
||||
* @param {number} [max] how many paths the line names before "…"
|
||||
* @returns {string}
|
||||
*/
|
||||
export function leafDropSummary(droppedLeaves, max = 8) {
|
||||
const n = droppedLeaves.length;
|
||||
if (!n) return '[assets] no leaf dropped: every planned leaf has a file on disk';
|
||||
return `[assets] ${n} ${n === 1 ? 'leaf' : 'leaves'} dropped: no alternative on disk ` +
|
||||
`(${droppedLeaves.slice(0, max).join(', ')}${n > max ? ', …' : ''})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The error line of a --strict / SP_ASSETS_STRICT run that dropped at least one leaf for having no alternative on
|
||||
* disk. CI and packaging runs pass --strict so a manifest with such holes cannot ship (the entry is gone from the
|
||||
* client's manifest; nothing retries it later, the next plain run only warns again).
|
||||
* @param {string[]} droppedLeaves dotted template paths
|
||||
* @param {number} [max] how many paths the line names before pointing at the report
|
||||
* @returns {string}
|
||||
*/
|
||||
export function strictLeafDropError(droppedLeaves, max = 10) {
|
||||
const n = droppedLeaves.length;
|
||||
return `ERROR: ${n} ${n === 1 ? 'leaf' : 'leaves'} dropped for having no alternative on disk ` +
|
||||
`(--strict / SP_ASSETS_STRICT=1): ${droppedLeaves.slice(0, max).join(', ')}` +
|
||||
`${n > max ? `, … (all ${n} in ${relative(ROOT, REPORT)})` : ''}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Exit code of a finished run: 0 only when nothing failed — every required asset is present, no leaf was dropped
|
||||
* while --strict / SP_ASSETS_STRICT is on, and the manifest was written (the shrink guard keeps the current one when
|
||||
* the rebuild would lose entries).
|
||||
* @param {{requiredMisses?:number, droppedLeaves?:number, strict?:boolean, manifestWritten?:boolean}} o
|
||||
* @returns {number} 0 or 1
|
||||
*/
|
||||
export function runExitCode({ requiredMisses = 0, droppedLeaves = 0, strict = false, manifestWritten = true } = {}) {
|
||||
if (requiredMisses) return 1;
|
||||
if (strict && droppedLeaves) return 1;
|
||||
return manifestWritten ? 0 : 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Files under public/assets the manifest does not reference (`--prune` deletes them). public/assets/local/** belongs to
|
||||
* tools/local-extract (data/local-assets.json) and is never an orphan: --prune used to delete all of it. Compared without
|
||||
@@ -411,11 +481,15 @@ async function main() {
|
||||
|
||||
const charIds = Object.keys(assets07.operators || {});
|
||||
const required = requiredMisses(manifest, charIds);
|
||||
// Leaves the plan had resolved and the manifest loses here (no alternative of theirs on disk): reported on their own
|
||||
// line below and kept in the report, whether or not they shrink the committed manifest.
|
||||
const droppedLeaves = resolved.droppedLeaves;
|
||||
const report = {
|
||||
downloadedBytes: dl.totals.bytesDownloaded,
|
||||
totals: dl.totals,
|
||||
stats: manifest.stats,
|
||||
requiredMisses: required,
|
||||
droppedLeaves,
|
||||
misses: resolved.misses,
|
||||
downloadErrors, // leaves whose primary failed transiently (fallbacks not tried; re-run to retry)
|
||||
fallbacks: resolved.fallbacks,
|
||||
@@ -444,8 +518,9 @@ async function main() {
|
||||
log(`fonts: ${Object.values(fonts.files).map((f) => f.woff2 || f.original).join(', ') || 'none'}`);
|
||||
if (resolved.fallbacks.length) { log(`fallbacks used (${resolved.fallbacks.length}):`); for (const f of resolved.fallbacks.slice(0, 20)) log(` ${f}`); }
|
||||
if (downloadErrors.length) log(`download errors (${downloadErrors.length}, re-run to retry): ${downloadErrors.slice(0, 10).join(', ')}`);
|
||||
log(leafDropSummary(droppedLeaves));
|
||||
if (resolved.misses.length) {
|
||||
log(`missing (${resolved.misses.length}, omitted from manifest; client uses fallbacks):`);
|
||||
log(`missing (${resolved.misses.length}, omitted from manifest; the client falls back only where the entry has another alternative):`);
|
||||
for (const m of resolved.misses.slice(0, 40)) log(` ${m}`);
|
||||
if (resolved.misses.length > 40) log(` … see ${REPORT}`);
|
||||
}
|
||||
@@ -460,11 +535,12 @@ async function main() {
|
||||
if (!guard.write) log(' re-run to retry the downloads (--refresh-index for the audio/model indexes), or pass --allow-shrink (or --prune) to write the smaller manifest');
|
||||
}
|
||||
log(`manifest: ${MANIFEST}${guard.write ? '' : ' (kept)'} · report: ${REPORT} · ${((Date.now() - t0) / 1000).toFixed(0)} s`);
|
||||
if (required.length) {
|
||||
log(`ERROR: ${required.length} required assets missing: ${required.slice(0, 10).join(', ')}`);
|
||||
return 1;
|
||||
if (required.length) log(`ERROR: ${required.length} required assets missing: ${required.slice(0, 10).join(', ')}`);
|
||||
if (opts.strict && droppedLeaves.length) {
|
||||
log(strictLeafDropError(droppedLeaves));
|
||||
log(' a plain run downloads them again (--refresh-index if the upstream index lost them); without --strict / SP_ASSETS_STRICT this is only a warning');
|
||||
}
|
||||
return guard.write ? 0 : 1;
|
||||
return runExitCode({ requiredMisses: required.length, droppedLeaves: droppedLeaves.length, strict: opts.strict, manifestWritten: guard.write });
|
||||
}
|
||||
|
||||
// run only as a script (tests import parseArgs / shrinkGuard)
|
||||
|
||||
Reference in new issue
Block a user