Loading files into staging
Load only on exit 0. Rejects keep the original records.
What is Valet?
A validation gate for data files before they reach your warehouse.
Check CSV, Parquet, JSON or NDJSON files against your rules; the exit code decides whether ingestion proceeds.
One CLI binary
JSON + exit codes
Local + S3-compatible
$ valet validate \
-p examples/pipeline.yml \
-i examples/sample.csv
"validation": "fail"
"rule_counts": { "RECORD_ID_REQUIRED": 1 }
4rows checked
3rows passed
Exit 1failed checks
Supported
CSVJSONNDJSONParquetS3
Use cases
Where Valet fits.
Large uploads in a web service
Managed memory stays within a set budget, whatever the upload size.
Pipeline and CI gates
Airflow or CI blocks the next step on failed error-level checks.
Features
What you get.
More than format checks
Required values, ranges, keys, references, column relationships, dataset totals.
Rules as config
Versioned, reusable YAML or JSON; no per-input scripts.
Clear result
Aggregate JSON on stdout plus exit code; no files unless requested.
Row-level detail on request
Issues and Parquet/CSV outputs with provenance to source rows.
No extra service
Self-contained CLI; streaming shares one managed-memory budget.
Parquet schema only
Metadata-only command for names and types; values not checked.
Receive a filelocal data or an S3-compatible object
Run Valetcheck rules; return JSON and an exit code
Gate ingestionyour caller decides whether to upload
Your pipeline still owns ingestion.
Not an uploader, scheduler or storage platform; error-level rules block, warnings alone do not.