Performance

Valet streams with no file-size ceiling: volume, rules, compression and storage set speed; record limits, global-rule cardinality and the budget decide fit.

Measured 5 GB Parquet workload

• Timed end to end: startup, decoding, every check, output writes, manifest. Fixture generation excluded.
• Output: 5,400,820,356 bytes in 41 parts (before: ~5.15 GB in 156); faster Snappy trades size for CPU. Outputs were verified value-for-value against input.
• The sub-10-second target was not reached; past four workers gains are small (evaluation and output stay ordered single-owner stages) and memory rises.

Comparison with GX Core

Measured 2026-10-26, same workstation, with benchmarks/competitors:

• 10,000,000 rows: id int64 with nulls, amount decimal(18,2), quantity int64, status string.
• Checks: id not null, amount 0–10000, quantity 1–1000, status in a fixed set; ~1% of rows fail each.
• Median of three fresh processes, warm cache; both tools reported the expected failure counts.

Schema-only Parquet inspection

valet schema -p pipeline.yml -i input.parquet

Took 0.040 s on the same file, startup included. It reads only the footer: scope: schema, metadata_only: true, validation: not_evaluated, zero checked rows, unverified row count.

Parquet execution costs

• Full local validation keeps a baseline digest and a final replay digest; page verification is fused into row-group decoding, giving three source passes on the measured plan.
• S3 identity pinning adds no extra digest scan.
• MB/s depends on row width and compression; don’t extrapolate from small fixtures. No service-level guarantees.

What changes throughput

Capacity planning

Benchmark the exact pipeline in the target pod and storage with representative data. Keep memory_budget below the pod limit and GOMEMLIMIT with headroom (operations).

Opt-in scale suite:

go test -tags=scale ./benchmarks \
  -run '^TestScale100GiB$' \
  -timeout 0 \
  -v