Online ResourcesPrashant Chaudhary

Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance

Online Resource 2: Validity checks

Online Resource 2 to "Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance" (Prashant Chaudhary, October 2026). Section 4.6 of the paper lists eight checks: five that rest on deterministic evidence a reader can verify (Table 4) and three automated consistency audits run by separate automated reviewers (Table 4b). This document describes each check in the same order: the procedure, the inputs, the scope, the outcome as the paper gives it, and the paths of the evidence in the replication package. It also describes the mutation-fuzzing campaign (Section 4.4 of the paper) as a method, with its seed and its replay option. Paths are relative to the root of the replication repository.

1. Checks on deterministic evidence (Table 4)

1.1 Source tracing

1.2 Standalone reproduction

1.3 Driver audits

1.4 Determinism

1.5 Access check

2. Automated consistency audits (Table 4b)

Each audit was run by a separate automated reviewer, started from the environment that built and ran the harness. They are not human reviewers. Each reviewer was given only the inputs listed below and had no access to the study's reasoning. "Independent" therefore means that the reviewer did not see the author's reasoning (and, in audits 2.1 and 2.2, the results or findings); it does not mean a different tool or a different person. The audits are not evidence of inter-rater reliability. The verdicts rest on the evidence of Section 1.

2.1 Catalog review

2.2 Re-derivation, two clients

2.3 Re-derivation, three clients

3. Mutation fuzzing (Section 4.4 of the paper)

This section states the method as it stands. The results are in Section 5.3 and Appendix D of the paper.

3.1 Design

3.2 Seed check

The seed 20261017 regenerates all 1,442 stored cases in order: the corpus seed name and the mutation list of every case, every case id (f00001 to f01442), every round number, and the per-round case counts. The check runs only the generator; it starts no client and no server, and takes about 6 seconds. In the commands of this section, CHECKS stands for the folder of standalone checks in the archive and FUZZ for the folder of the stored fuzzing campaign:

cd <repository root>
nice -n 10 node "$CHECKS/fuzz-repros/fuzz-seed-check.mjs" "$FUZZ/cases.jsonl.gz" --seeds 20261017

Its output for the campaign is stored in fuzz-seed-check-r3.out in the archive's fuzzing analysis. The generator uses seed >>> 0, so seeds that differ by a multiple of 2^32 give the same stream. The default seed of a normal run (--seed 1) is a different stream and does not regenerate these cases.

3.3 Replay option

code/harness/fuzz/fuzz.mjs --replay reruns stored cases. cases.jsonl keeps only the corpus seed name and the mutation descriptions, and a description does not determine the mutated document ("delete <Tracking>" does not say which element). Replay therefore reruns the generator from the campaign seed (default 20261017 in replay mode), checks every rebuilt case against the stored descriptor, and stops with exit code 2 and a message if any case differs. Cases that are not selected are still drawn, so that the stream stays in step. Output in <out>/cases.jsonl has the same per-case record as a normal run (id and round are the stored ones), plus classes.json and examples/<id>/ for the first case of each class seen. The stop rule, --hours and --per-round do not apply to replay.

# all stored cases, in order (about 1 hour on 2 cores)
cd code/harness   # from the repository root
nice -n 10 node fuzz/fuzz.mjs /path/to/out --replay "$FUZZ/cases.jsonl.gz"

# chosen cases (comma list of stored ids), or the first N cases
nice -n 10 node fuzz/fuzz.mjs /path/to/out --replay "$FUZZ/cases.jsonl.gz" --only f00155,f00093
nice -n 10 node fuzz/fuzz.mjs /path/to/out --replay "$FUZZ/cases.jsonl.gz" --first 20

Options: --seed, --clients a,b, --only, --first.

Replay test: 59 cases (the first example case of each of the 9 root causes, taken from classes-verified.tsv in the archive's fuzzing analysis, and 50 other stored cases drawn by a seeded shuffle with seed 16). All 59 regenerated with the stored descriptor; all 59 have the same failure classes as the stored case (16 carry a class, 43 carry none, in both runs) and identical outcome fields for all four clients; all 9 root-cause examples reproduced their class, including the vast-client-js stalled-body hang (f00361). The test covers 59 of 1,442 cases. The hang and slow classes depend on the 30 second wall and on machine load, so a hang or slow case on a loaded machine can differ between runs.

Evidence: replay-mode-r3.jsonl in the archive's fuzzing analysis (replay output), replay-mode-r3-compare.json in the archive's fuzzing analysis (comparison); fuzz-repros/fuzz-replay-pick.mjs in the archive's standalone checks and fuzz-replay-compare.mjs; code/package/README.md (fuzzer subsection).

3.4 Saturation

Classes per round were 8, 2, 1, 1 and 0, and root causes per round 8, 1, 0, 0 and 0 (saturation-r3.tsv in the archive's fuzzing analysis). The empty final round is weak evidence that classes are exhausted: 3 classes occur in 1 case each, none in exactly 2, and the bias-corrected Chao1 estimate is 15.0 classes (95 percent interval 12.4 to 37.3) against 12 observed. Classes are message strings, so root causes are the unit reported. The last new root cause appeared at unique case 360, and rounds 3 to 5 (858 cases not seen before) added none. The bound holds only for this generator's seeds and operators, and it assumes independent cases. The class key depends on the detector's thresholds: the vast-client-js stalled-body class is a hang because its 54 second duration exceeds the 30 second wall. The detector sees only crashes, rejections, hangs, slow loads and internal errors. The "VAST version" mutation rewrites the first version="..." attribute, which is the XML declaration in the crafted vectors, so it changes the VAST version only in the 72 IAB samples.

The vast-client-js stalled-body class (C4) is a defect of the library and not an effect of load: its url_handler.js clears the timer when the headers arrive and reads the body with no timeout, so each of 9 Wrapper hops waits out the 6 second stall (302 at 54.19 seconds; with a 3 second stall, 27.10 seconds, which is 9 times the stall). Measurements: fuzz-repros/vc-hang/ in the archive's standalone checks.

3.5 Environment artifact

Class C9 is an unhandled rejection in video-ad-sdk's icon code that occurs only because Node.js 22's own CustomEvent reaches jsdom's event target. It is reproduced with the harness globals and absent with jsdom's own event classes. It is classified as an environment artifact and excluded from the library defects; the affected icon cells are browser-only, so no verdict changes. Evidence: fuzz-repros/vas-icon-event-artifact/ in the archive's standalone checks.