# valet documentation

> A validation gate for data files before they reach your warehouse.

Site: https://richardrh.github.io/valet/
Pipeline JSON Schema: https://richardrh.github.io/valet/schema/v1/pipeline.schema.json

Every documentation page follows in navigation order. Each starts with a
level-one heading and its source URL.

---

# Getting started

Source: https://richardrh.github.io/valet/getting-started/

## What is Valet?

A command-line validation gate for CSV, Parquet, NDJSON or JSON files (local or S3-compatible) before a lakehouse, warehouse or next job.
Checks live in YAML or JSON; by default you get one aggregate JSON report on stdout and an exit code, with no rewritten data or output files.

## Why Valet?

- **Readable ≠ correct:** checks schemas, missing values, ranges, duplicate keys, references and column relationships before problems propagate. Same versioned rules wherever the file enters.
- **Caller owns the next step:** continue on exit `0`, block on failed checks (`1`), handle execution failures separately (`2`). Warnings do not block by default; set mandatory rules to `level: error`.
- **Self-contained CLI, not a service:** no scheduling or lakehouse upload. Keep your uploader; validate the same immutable input before invoking it.

## Start here


## Choose the right command

| You want to… | Command | Result |
|---|---|---|
| Check Parquet columns and types without reading values | `valet schema -p pipeline.yml -i input.parquet` | Schema-only JSON report; no data outputs |
| Check values without publishing files | `valet validate -p pipeline.yml -i input.parquet` | One aggregate JSON summary on stdout; no manifest |
| Check row values, conversions, references and dataset rules | `valet validate -p pipeline.yml -i input.parquet -o results/` | Checked rows, issues and a terminal manifest |
| Preview configuration, schemas, passes and required state | `valet plan -p pipeline.yml -i input.parquet` | Execution-plan JSON; not a validation result |

Only `schema` is Parquet-only; full validation supports all four formats. `schema` and `plan` may succeed even when a row value would fail full validation — see [schema-only checks](../guides/schema/).

## Full validation lifecycle

1. Optional: `valet plan` to see the work and managed state.
2. `valet validate` reads, converts and checks the data.
3. Read results:
   - no output URI → aggregate stdout JSON; no files or manifest.
   - `-o`/`output.uri` → published `manifest.json` and its artifact references; partial artifacts without it are incomplete.

`valet schema` is separate: JSON report on stdout only; never publishes row buckets or a manifest.

## Exit codes

| Exit | `valet schema` | `valet validate` |
|---:|---|---|
| `0` | Schema compatible; row checks were not evaluated | Completed without error-level violations; warnings allowed |
| `1` | Error-level schema mismatch; inspect the JSON report | Completed with row, dataset or schema violations; inspect the summary |
| `2` | Configuration, format, metadata, storage or resource failure | Operational or presentation failure without a committed manifest |
| `3` | Not used: schema inspection commits no terminal manifest | Manifest committed, but stdout presentation failed |
| `128+N` | Interrupted by signal `N` | Interrupted by signal `N` before commit |


---

# Installation

Source: https://richardrh.github.io/valet/getting-started/installation/

Go 1.27.3+ is needed only for Go builds; no cgo or native libraries.

## Download a release

Get an archive and `checksums.txt` from [GitHub Releases](https://github.com/richardrh/valet/releases). `<tag>` is a full published tag, e.g. `v0.1.0-rc.2`; prereleases are not marked latest stable.

| Platform | Archive |
| --- | --- |
| Linux amd64 | `valet_<tag>_linux_amd64.tar.gz` |
| Linux arm64 | `valet_<tag>_linux_arm64.tar.gz` |
| Windows amd64 | `valet_<tag>_windows_amd64.zip` |
| macOS Intel | `valet_<tag>_darwin_amd64.tar.gz` |
| macOS Apple Silicon | `valet_<tag>_darwin_arm64.tar.gz` |

Archives contain `valet` (`valet.exe`), `README.md`, `CHANGELOG.md`, `LICENSE` (MIT; dependencies keep their licenses), `SECURITY.md`. Checksums catch corrupt downloads, not tampering; fetch both over HTTPS from this repo.

### Linux and macOS

In an empty directory (Linux amd64 shown):

```bash
tag=v0.1.0 # Replace with a published tag.
archive="valet_${tag}_linux_amd64.tar.gz" # Use linux_arm64 for ARM64.
release_url="https://github.com/richardrh/valet/releases/download/$tag"
curl --fail --location --remote-name "$release_url/$archive"
curl --fail --location --remote-name "$release_url/checksums.txt"
sha256sum --check --ignore-missing checksums.txt
tar -xzf "$archive"
mkdir -p "$HOME/.local/bin"
install -m 0755 valet "$HOME/.local/bin/valet"
"$HOME/.local/bin/valet" --version
```

Stop if the checksum fails; add `$HOME/.local/bin` to `PATH`. On macOS use the `darwin_amd64`/`darwin_arm64` archive and replace `sha256sum` with `shasum -a 256 "$archive"`, comparing against `checksums.txt`. Darwin builds are unsigned, not notarized, and not run on macOS in CI; macOS may ask for approval.

### Windows

PowerShell, in an empty directory:

```powershell
$tag = "v0.1.0" # Replace with a published tag.
$archive = "valet_${tag}_windows_amd64.zip"
$releaseUrl = "https://github.com/richardrh/valet/releases/download/$tag"
Invoke-WebRequest "$releaseUrl/$archive" -OutFile $archive
Invoke-WebRequest "$releaseUrl/checksums.txt" -OutFile checksums.txt
$entry = @(Get-Content checksums.txt | Where-Object {
    ($_ -split '\s+')[-1] -eq $archive
})
if ($entry.Count -ne 1) { throw "Missing or duplicate archive checksum" }
$expected = ($entry[0] -split '\s+')[0]
if ((Get-FileHash $archive -Algorithm SHA256).Hash -ne $expected) {
    throw "Archive checksum mismatch"
}
Expand-Archive $archive -DestinationPath valet-release
.\valet-release\valet.exe --version
```

Move `valet.exe` onto your `PATH`.

## Install with Go

```bash
go install github.com/richardrh/valet/cmd/valet@latest   # or @v0.1.0-rc.2 to pin
```

Installs to `GOBIN` (default `$(go env GOPATH)/bin`); add it to `PATH`. `--version` prints `dev`.

## Build from source

```bash
git clone https://github.com/richardrh/valet.git && cd valet
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=v0.1.0" -o dist/valet ./cmd/valet
```

Check out a tag first for a specific release; omit `-ldflags` for a development version. Cross-build with `GOOS`/`GOARCH` (e.g. `GOOS=windows GOARCH=amd64`, `-o dist/valet.exe`). Platform test coverage: [CI guide](../../guides/ci/).


---

# Quickstart

Source: https://richardrh.github.io/valet/getting-started/quickstart/

## Inspect the input

```bash
cat examples/sample.csv
```

```csv
record_id,location,amount
REC001,North,5000
REC002,South,500
,East,2000
REC004,West,1500000
```

Row 3 has no `record_id`.

## Plan the run

```bash
dist/valet plan \
  -p examples/pipeline.yml \
  -i examples/sample.csv
```

Expect: CSV source, `string`, `string`, `int64` columns, four rules, two [passes](../../concepts/execution/#passes). `pipeline.json` gives the same plan.

## Validate

```bash
dist/valet validate \
  -p examples/pipeline.yml \
  -i examples/sample.csv \
  > demo-summary.json

echo "exit=$?"
```

Expect `exit=1` (one error-level issue). Stdout is the JSON [report](../../concepts/results/#report); without `-o` no files are written.

```json
{
  "status": "completed",
  "validation": "fail",
  "total": 4,
  "pass": 3,
  "warn": 0,
  "error": 1,
  "rows_scanned": 8,
  "passes": 2
}
```

## Read the result

```bash
jq '{validation, total, pass, warn, error, rule_counts}' demo-summary.json
```

Gate on Valet's [exit code](../../concepts/results/#exit-codes), not `jq`'s; see [CI automation](../../guides/ci/).

## Optional: write Parquet and detailed issues

```bash
dist/valet validate \
  -p examples/pipeline.yml \
  -i examples/sample.csv \
  -o demo-output \
  > artifact-summary.json

echo "exit=$?"
```

Still `exit=1`; also writes Parquet pass/error buckets, issues and a [manifest](../../concepts/results/#manifest).

`--output-mode failures|report` trims row files; see [output modes](../../concepts/results/#output-modes).

```bash
MANIFEST="$(jq -r '.manifest_uri' artifact-summary.json)"
jq . "$MANIFEST"
```

The issues NDJSON (listed in `artifacts.ndjson`) contains:

```json
{
  "row": 3,
  "global_row": 3,
  "line": 4,
  "code": "RECORD_ID_REQUIRED",
  "severity": "error",
  "scope": "row",
  "column": "record_id"
}
```

## Try schema-only checks on Parquet

Check the **pass** Parquet part from the run above:

```bash
ARTIFACTS="$(jq -r '.artifacts.uri' "$MANIFEST")"
PARQUET="$(jq -r 'select(.kind == "pass" and .format == "parquet") | .uri' "$ARTIFACTS")"

dist/valet schema -p examples/parquet-schema.yml -i "$PARQUET"
```

Expect exit `0`: one file's schema matches (`record_id`, `location`, `amount`); the `AMOUNT_NONNEGATIVE` value rule is **not evaluated**.

```json
{
  "scope": "schema",
  "metadata_only": true,
  "validation": "not_evaluated",
  "compatible": true,
  "rows_checked": 0,
  "objects_checked": 1,
  "declared_rows": 3
}
```

No row values are checked and nothing is written. To check values too (exit `0`, three passing rows):

```bash
dist/valet validate -p examples/parquet-schema.yml -i "$PARQUET"
```

Next: [schema checks](../../guides/schema/), [rules cookbook](../../rules/), [runnable example](../../rules/#runnable-example).


---

# Concepts

Source: https://richardrh.github.io/valet/concepts/

How one `valet validate` run turns input files into a report and an exit code.

```mermaid
flowchart LR
  A[Read inputs<br/>local or S3] --> B[Convert to schema<br/>ingestion issues]
  B --> C[Run checks<br/>row, global, reference, schema]
  C --> D[Row buckets<br/>pass / warn / error]
  D --> E[Report on stdout<br/>outputs + manifest with -o]
  E --> F[Exit code<br/>0 / 1 / 2 / 3]
```

| Page | Covers |
|---|---|
| [Pipelines](pipelines/) | Pipeline file, inputs and formats, input modes, declared schemas, type names |
| [Rules and checks](rules-and-checks/) | Rules, levels, check scopes, conditions, row buckets, ingestion issues, dataset rules |
| [Results](results/) | Report, exit codes, output modes, issues and rejects, manifest |
| [Execution](execution/) | Memory budget, batches and workers, passes, malformed input |


---

# Pipelines

Source: https://richardrh.github.io/valet/concepts/pipelines/

A pipeline describes what to read, how to type it and which rules to run. Full reference: [pipeline configuration](../../guides/configuration/).

## Pipeline file

| Item | Detail |
|---|---|
| Formats | Local `.yaml`, `.yml` or `.json`; JSON is strict RFC syntax |
| Shared validation | Duplicate keys, shape, exact numbers, defaults, semantics |
| Top-level keys | `config_version`, `name`, `version`, `context`, `source`, `input`, `transforms`, `execution`, `output`, `rules`, `dataset_rules`, `stats` |
| `$schema` | Top-level key accepted and ignored in JSON and YAML; in YAML a `# yaml-language-server: $schema=…` modeline also works ([editor support](../../guides/configuration/#editor-support-and-ai-agents)) |
| CLI overrides | `-i`/`-o` replace input/output; explicit execution flags override the document |
| Relative paths | Sources, output, references, mappings and schema files resolve from the pipeline's directory |

## Inputs and formats

| Setting | Values |
|---|---|
| `source.uris` | Ordered list; local paths and `s3://` URIs can mix |
| `source.prefix` + `filter`, `recursive` | Deterministic, sorted discovery; exclusive with `uris` |
| `input.type` | `csv`, `json` (streamed top-level array of objects), `ndjson` (one object per nonblank line), `parquet` (flat schemas) |
| Type inference | From source suffix when omitted; trailing `.gz` ignored for text formats; explicit type wins |

All inputs form one logical dataset. Inputs must be replayable files or objects; stdin is not supported.

## Input modes

| `input.mode` | Behaviour |
|---|---|
| `infer_types` (default) | Infer types from up to `infer_schema_length` rows (default 1,000,000), then freeze; a later incompatible value is an input error |
| `raw_strings` | Keep non-null CSV values as UTF-8 strings; null markers (`null_values`, default `NULL`, `null`, `N/A`, `n/a`, `""`) stay distinct |

| Option | Values | Effect |
|---|---|---|
| `on_type_error` | `row_error` (default), `abort` | With a declared schema: conversion failure becomes an `INGEST_TYPE` row issue, or stops the run |
| `on_malformed` | `abort` (default), `reject` | See [malformed input](../execution/#malformed-input) |

## Declared schemas

`input.schema` sets exact target types instead of inference.

```yaml
input:
  schema:
    extra_fields: error   # or ignore
    columns:
      id: {type: string, required: true, nullable: false}
      amount: {type: decimal, precision: 18, scale: 2}
```

- Conversion failures become [ingestion issues](../rules-and-checks/#ingestion-issues) and rejects, never silent nulls.
- `extra_fields: error` rejects undeclared fields; `ignore` selects only declared columns.
- `valet plan --format pipeline` writes a schema block from a sample.

## Type names

Any letter case. C fixed-width names (without `_t`), Python and Arrow/Parquet names are accepted; reports and `valet plan` print the last column.

| Values | Write | Reported as |
|---|---|---|
| Signed integers | `int8`, `int16`, `int32`, `int64`; `int`, `integer` = `int64` | `int8` … `int64` |
| Unsigned integers | `uint8`, `uint16`, `uint32`, `uint64` | `uint8` … `uint64` |
| 32-bit floats | `float32` | `float32` |
| 64-bit floats | `float64`, `double`; `float` = `float64` | `float64` |
| Exact decimals | `decimal`, `decimal128` with `precision` (1–38) and `scale` | `decimal` |
| Booleans | `bool`, `boolean` | `bool` |
| Text | `str`, `string`, `utf8`, `large_string`, `large_utf8` | `string` |
| Bytes | `bytes`, `binary`, `large_binary` | `binary` |
| Dates | `date`, `date32`, `date64` | `date` |
| Timestamps | `datetime`, `timestamp` | `timestamp` |
| Nulls only | `null` | `null` |

- `int` and `float` follow Python (64-bit). In C, Arrow (pyarrow prints `float32` as `float`) and Parquet (`FLOAT`), `float` is 32-bit: write `float32` there.
- Names whose size or sign varies by platform (`long`, `char`, `short`, `unsigned`) are rejected with the spelling to use; `int32_t` → write `int32`:

```text
input.schema column "qty" type "long" is not a supported type: long is 32-bit on Windows and 64-bit on Linux and macOS; write int32 or int64
```


---

# Rules and checks

Source: https://richardrh.github.io/valet/concepts/rules-and-checks/

A rule groups one or more checks under a stable code. Examples: [rules cookbook](../../rules/).

## Rules

| Field | Meaning |
|---|---|
| `code` | Stable, unique across `rules` and `dataset_rules`; `INGEST_*` is reserved |
| `level` | `error` or `warn` |
| `column` | Default column; a check's `column`/`columns` overrides it |
| `checks` | Mapping of check name to arguments, e.g. `not_null: true`, `valid_values: [A, B]`; see [syntax](../../rules/) |
| `when` | Optional condition |
| `message`, `name` | Shown for failures; stored once in `rules.json` |

## Levels

| Level | Row bucket | Blocks (exit `1`) |
|---|---|---|
| `error` | `error` | Yes |
| `warn` | `warn` | No; exit `0` allows warnings |

## Check scopes

| Scope | Checks | Notes |
|---|---|---|
| row | `not_null`, `valid_range`, `matches_pattern`, `type_check`, … | Each row independently |
| global | `is_unique`, `is_unique_composite` | Prepass + replay; every duplicate, including the first, fails |
| reference | `referential_integrity`, `referential_integrity_composite` | Probe an index built from a reference file |
| schema | `required_columns`, `schema_matches`, `valid_schema_columns` | One dataset-wide outcome |
| dataset | `dataset_rules` | Aggregate metrics, see below |

## Conditions

`when` applies to row and reference checks.

```yaml
when:
  all:
    - {column: status, operator: eq, value: PAID}
    - {column: refunded, operator: eq, value: false}
```

- Leaf: `column`, `operator` (`eq`, `ne`, `lt`, `le`, `gt`, `ge`, `in`, `is_null`, `is_present`), `value`.
- Compose with `all`, `any`, `not`.
- False condition → rule not applicable; invalid operand → unevaluable.

## Row buckets

| Bucket | Row has |
|---|---|
| `pass` | No failures |
| `warn` | Only `warn`-level failures |
| `error` | Any `error`-level failure or ingestion issue |

`total = pass + warn + error`.

## Ingestion issues

Raised while reading and converting; always `error` severity.

| Code | Cause |
|---|---|
| `INGEST_TYPE` | Value cannot convert to the declared type |
| `INGEST_NULL` | Null in a `nullable: false` column |
| `INGEST_MISSING_FIELD` | `required: true` field missing |
| `INGEST_EXTRA_FIELD` | Undeclared JSON field with `extra_fields: error` |
| `INGEST_MALFORMED` | CSV row with wrong field count, `on_malformed: reject` |
| `INGEST_SCHEMA` | Schema scope: required column absent, type mismatch, undeclared or inconsistent column |

Checks on an invalid cell count as unevaluable rather than null.

## Dataset rules

```yaml
dataset_rules:
  - code: DATASET_NOT_EMPTY
    level: error
    metric: row_count
    min: 1
```

| Metric family | Examples |
|---|---|
| Row count | `row_count` |
| Missing/null/blank/conversion | counts and fractions, e.g. `null_fraction` |
| Rule failures | counts and fractions |
| Numeric | `sum`, `min`, `max`, `avg` |


---

# Results

Source: https://richardrh.github.io/valet/concepts/results/

Every run prints a report and returns an exit code; files appear only with an output URI. Details: [structured output](../../guides/output/).

## Report

One JSON summary on stdout (`--plain` for `key=value`; `--json-output PATH` adds a file).

| Key | Meaning |
|---|---|
| `report_version`, `run_id` | Report format (`2`) and this run's ID |
| `status`, `validation` | Terminal status; `pass` / `fail` (`not_evaluated` for schema-only). `completed` can pair with `fail` |
| `total`, `pass`, `warn`, `error` | Unique final-pass value-evaluated rows per bucket |
| `rows_scanned`, `passes` | Decoded row visits across all passes; passes run |
| `rule_counts`, `ingest_counts` | Failures per rule code and per `INGEST_*` code |
| `unevaluable`, `not_applicable` | Rows a rule could not evaluate, or skipped by `when` |
| `dataset_results`, `dataset_issues`, `schema_issues`, `stats` | Dataset rules, schema checks and configured aggregates |
| `verification`, `coverage`, `rule_coverage`, `trust_policy` | `values` or `metadata_assisted`; value vs footer-claimed rows |
| `scope` | `data` for `validate`, `plan` for `plan` |
| `output_mode`, `row_output` | Output mode and, per bucket, what was written or omitted |
| `duration_seconds`, `managed_peak_bytes` | Run time and peak managed memory |
| `manifest_uri` | Manifest URI, only when committed with `-o` (`manifest=` in `--plain`) |

## Exit codes

| Exit | Meaning |
|---:|---|
| `0` | Completed; no error-level failures (warnings allowed) |
| `1` | Completed with error-level row, dataset or schema failures |
| `2` | Operational or presentation failure; no committed manifest; see stderr |
| `3` | Manifest committed but stdout failed; consume the manifest |
| `130` / `143` | SIGINT / SIGTERM before commit (`128+N`); no terminal result |

## Output modes

| Mode | Default when | Writes |
|---|---|---|
| `report` | No output URI | Stdout only; with a URI also issues, indexes, summary, manifest |
| `all` | Output URI set | All row buckets + rejects + reports |
| `failures` | — | `warn`/`error` rows + rejects + reports |

Set with `output.mode` or `--output-mode`. `all`/`failures` without a URI is an error. See [validation modes](../../guides/validation-modes/).

## Issues and rejects

| Artifact | Contents |
|---|---|
| `issues/part-*.ndjson` | One line per issue: `code`, `severity`, `origin`, `scope`, `column`, `source` URI, 1-based `source_ordinal`/`row`/`global_row`/`line` |
| `rejects/part-*.ndjson` | Raw record, provenance, per-column `valid`/`missing`/`null`/`invalid`, ingestion diagnostics |

Fatal framing failures produce no row: exit `2`, diagnostics on stderr (`--log-format json`).

## Manifest

`manifest.json` is the commit point.

- Written last; references inputs, rules, artifacts and summary by URI, size, SHA-256 and record count.
- No manifest → ignore partial artifacts.
- Before commit, failures are operational (exit `2`); after, the result stands (exit `3` if stdout fails).


---

# Execution

Source: https://richardrh.github.io/valet/concepts/execution/

Valet streams input in bounded batches under a managed memory budget. Sizing: [operations](../../guides/operations/).

## Memory budget

| Setting | Default | Meaning |
|---|---|---|
| `memory_budget` | `512MiB` | Managed decoder, batch, rule-state, index and publication reservations; **not** a process RSS limit |
| `max_record_bytes` | `8MiB` | One CSV row or JSON record; Parquet pages are limited to the larger of this and `batch_target`; not the file size |

- Set `GOMEMLIMIT` (e.g. `768MiB` with a 512 MiB budget) for process headroom.
- Uniqueness and reference indexes grow with distinct keys; over budget they fail explicitly. No disk spill, no approximation.

## Batches and workers

| Setting | Default | Meaning |
|---|---|---|
| `batch_target` | `8MiB` | Target Arrow batch size, not a hard ceiling |
| `workers` | `1` (range 1–32) | Overlap stages and decode Parquet row groups in parallel |

- Rule evaluation and output stay single-owner and ordered.
- Workers share the budget; more workers do not raise it.

## Passes

| Pass | When |
|---|---|
| Inference | `infer_types` without a declared schema |
| Prepass | Global checks (`is_unique*`) collect exact keys; `schema_matches` castability gates are resolved |
| Reference load | Reference checks; each index built once, shared |
| Final pass | Evaluates rules; counts come from here only |

Inputs are re-read between passes; local files are digest-checked and S3 objects pinned by version/ETag.

## Malformed input

| Input | Default (`on_malformed: abort`) | `on_malformed: reject` |
|---|---|---|
| CSV row with wrong field count | Exit `2`, error names file and line | Row → `error` bucket, one `INGEST_MALFORMED`, raw fields in rejects |
| Unclosed/stray quotes, invalid UTF-8, blank lines, oversized records, malformed JSON/NDJSON | Exit `2` | Exit `2` |

Mapping files (`transforms`) always fail on malformed rows. Details: [malformed CSV rows](../../guides/configuration/#malformed-csv-rows).


---

# Guides

Source: https://richardrh.github.io/valet/guides/

Configure a pipeline, choose schema-only or full validation, then pick outputs and run it in production or CI.


---

# Pipeline configuration

Source: https://richardrh.github.io/valet/guides/configuration/

A pipeline is a local `.yaml`, `.yml` or `.json` file (strict RFC JSON). See [pipeline file](../../concepts/pipelines/#pipeline-file).

## Top-level structure

| Section | Key fields | Defaults |
|---|---|---|
| `source` | `uris` **or** `prefix` + `filter` + `recursive` | — |
| `input` | `type`, `schema`, `on_malformed`, `infer_schema_length`, `parquet.statistics` | `type` from suffix; `abort`; `1000000`; `off` |
| `execution` | `memory_budget`, `batch_target`, `max_record_bytes`, `workers` | `workers: 1` |
| `output` | `uri`, `mode`, `format`, `parquet.statistics` | `mode`: `report` without URI, else `all`; `none` |
| `rules`, `dataset_rules` | see [rules](../../rules/) | `[]` |

```yaml
name: Orders validation
version: "1.0"

source:
  uris: [orders.csv]

input:
  type: csv
  schema:
    extra_fields: error
    columns:
      order_id: {type: string}
      quantity: {type: int64}

execution:
  memory_budget: 512MiB
  batch_target: 8MiB
  max_record_bytes: 1MiB
  workers: 1

rules: []
dataset_rules: []
```

- `-i` / `-o` and execution flags override the document.
- Relative references, mappings and schema files resolve from the pipeline's directory.
- No output URI → stdout JSON report only. See [output modes](../../concepts/results/#output-modes).

## Sources

```yaml
source:
  uris:
    - data/part-001.csv
    - s3://bucket/data/part-002.csv
```

```yaml
source:
  prefix: s3://bucket/data/
  filter: csv
  recursive: true
```

`uris` and `prefix` are mutually exclusive; discovery is sorted deterministically.

## Input formats

`type`: `csv`, `json`, `ndjson` or `parquet`; inferred from the suffix if omitted. See [inputs and formats](../../concepts/pipelines/#inputs-and-formats).

### Malformed CSV rows

Default: a malformed record stops the run with exit `2`. To reject CSV rows with the wrong field count and continue:

```yaml
input:
  type: csv
  on_malformed: reject   # default: abort
```

| Rejected row | Result |
|---|---|
| Counts | In `total`, error bucket, every column invalid |
| Issue | One `INGEST_MALFORMED`, e.g. `found 3 fields; the header has 2` |
| Rejects output | Original fields as JSON array, e.g. `"raw": ["2","3","EXTRA"]` |
| Column rules | Not also failed |

Always fatal: bad quotes, invalid UTF-8, blank lines, oversized records, malformed JSON/NDJSON, malformed `transforms` mapping files. [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/spec.md)

## Explicit schemas

```yaml
input:
  schema:
    extra_fields: error
    columns:
      id: {type: string, required: true, nullable: false}
      amount: {type: decimal, precision: 18, scale: 2}
      effective_date: {type: date, formats: ["%F"]}
      observed_at:
        type: timestamp
        timezone: UTC
        formats: ["%Y-%m-%dT%H:%M:%S%:z"]
```

Failed conversions become ingestion issues and rejected records, not nulls.

### Generate a schema from a sample file

```bash
valet plan -p pipeline.yml -i sample.csv --format pipeline
```

```yaml
# Generated by valet plan. Review the types, then add rules.
input:
  type: csv
  schema:
    extra_fields: error
    columns:
      order id: {type: int64}
      amount: {type: float64}
      ok: {type: bool}
      note: {type: string}
```

| Format | Types |
|---|---|
| CSV, JSON | Inferred from `infer_schema_length` rows: `int64`, `float64`, `bool`, `string` only |
| Parquet | File's own types, incl. decimal precision/scale and time zones |

Review before use: change money to `{type: decimal, precision: …, scale: …}` and dates to `date`/`timestamp` with `formats`.

### Choose schema inspection or value conversion

`valet validate` converts values to `input.schema`; `valet schema` only compares it with Parquet footer types. See [schema-only Parquet checks](../schema/).

```bash
valet schema -p pipeline.yml -i input.parquet
```

### Output policy and optional Parquet statistics

```yaml
input:
  parquet:
    statistics: off       # off (default) or explicitly trusted
output:
  uri: ./results          # explicitly request artifact publication
  mode: all               # all, failures, report
  format: parquet
  parquet:
    statistics: none      # none (default), bounded
```

`all`/`failures` need a destination. Override with `--output-mode` and `--parquet-stats`; see [validation modes](../validation-modes/).

## Execution limits

| Field | Limits |
|---|---|
| `memory_budget` | Valet-managed memory; not process RSS |
| `batch_target` | Target batch size |
| `max_record_bytes` | One record or Parquet page, **not** the file |
| `workers` | 1–32, default 1; does not raise the budget |

Starting point for a 1 GiB pod, with `GOMEMLIMIT=768MiB`:

```yaml
execution:
  memory_budget: 512MiB
  batch_target: 8MiB
  max_record_bytes: 1MiB # maximum one CSV row, not the CSV file
  workers: 1
```

Uniqueness and reference indexes fail at the budget. See [memory budget](../../concepts/execution/#memory-budget).

## Editor support and AI agents

The [pipeline JSON Schema](https://richardrh.github.io/valet/schema/v1/pipeline.schema.json) gives YAML/JSON editors completion and typo checks.

```yaml
# yaml-language-server: $schema=https://richardrh.github.io/valet/schema/v1/pipeline.schema.json
```

```json
{
  "$schema": "https://richardrh.github.io/valet/schema/v1/pipeline.schema.json",
  "rules": []
}
```

Valet ignores `$schema`; `valet plan` still runs cross-field and data checks.

| For agents | Content |
|---|---|
| [`llms.txt`](https://richardrh.github.io/valet/llms.txt) | Page index |
| [`llms-full.txt`](https://richardrh.github.io/valet/llms-full.txt) | All docs, one file |
| `…/guides/configuration.md` | Any page as Markdown |

```sh
uvx --from mcpdoc mcpdoc --urls "Valet:https://richardrh.github.io/valet/llms.txt"
```

Next: [rules cookbook](../../rules/), [operations](../operations/), [performance](../performance/).


---

# Schema-only Parquet checks

Source: https://richardrh.github.io/valet/guides/schema/

`valet schema` checks **Parquet column names and types without reading rows**. CSV/JSON, and any value checks, need `valet validate`.

## Run a schema check

```bash
valet schema -p examples/parquet-schema.yml -i input.parquet
valet schema -p examples/parquet-schema.yml -i s3://my-bucket/data/input.parquet
valet schema -p examples/parquet-schema.yml -i input.parquet > schema-report.json
valet schema -p examples/parquet-schema.yml -i input.parquet --format text
```

- Uses [`examples/parquet-schema.yml`](https://github.com/richardrh/valet/blob/main/examples/parquet-schema.yml); S3 uses the AWS SDK chain and `source.s3`.
- No `-o`, `summary.json` or manifest. `--format text` groups differences by source.

## Declare the expected columns and types

```yaml
input:
  type: parquet

rules:
  - code: EXPECTED_SCHEMA
    level: error
    checks:
      schema_matches:
        mode: strict
        columns:
          record_id: string
          location: string
          amount: int64
        require_columns: true
        allow_extra_columns: true
```

| Setting | Effect |
|---|---|
| `mode: strict` | Compares types from metadata; default `castable` needs row data |
| `require_columns: true` | Missing listed column fails |
| `allow_extra_columns: false` | Listed columns must be the full set |
| `level: error` | Mismatch → exit `1`; `warn` only reports |

Names only:

```yaml
rules:
  - code: COLUMNS_PRESENT
    level: error
    checks:
      required_columns: [record_id, location, amount]
```

`input.schema.columns` also works: types, decimal/timestamp parameters, `required: true` and `extra_fields: error` are compared after renames. The bundled pipeline uses `extra_fields: ignore` so `__valet_*` provenance columns pass.

With no declared schema, Valet only checks that metadata is readable and, for several files, consistent with the first.

## Which checks run?

| Configuration or check | What `valet schema` establishes |
|---|---|
| `input.schema.columns` | Declared types match actual file types; required columns are present |
| `input.schema.extra_fields: error` | No undeclared input columns are present |
| `required_columns` | The named columns are present |
| `schema_matches`, `mode: strict` | The configured name/type schema and required/extra-column policies match |
| `schema_matches`, `mode: castable` | Presence/extra-column policies only; value castability remains not evaluated |
| `type_check`, `not_null`, `not_blank`, ranges, patterns and dates | Not evaluated: these are value checks |
| Uniqueness, references, `valid_schema_columns`, statistics and dataset assertions | Not evaluated: these need row or reference data |

> [!IMPORTANT]
> A **required column** must exist; its values may still be null. Pages are not decoded, so a passing file can still fail full validation.

## Can Parquet statistics check values without a scan?

Not with `valet schema`: value checks are listed as not evaluated. `valet validate --parquet-stats trusted` can use footer statistics in report or failures-only modes; see [output modes and metadata trust](../validation-modes/). Missing statistics mean unknown, not pass.

### Trust and tampering

> [!WARNING]
> Footer statistics are producer claims, not proof. Stale or tampered statistics look valid.

ETags, digests, CRCs and signatures do not prove statistics are correct. Keep the default `off` unless you trust the producer. [Validation modes spec](https://github.com/richardrh/valet/blob/main/docs/engineering/validation-modes-spec.md)

## Read the JSON report

```json
{
  "report_version": 2,
  "scope": "schema",
  "metadata_only": true,
  "validation": "not_evaluated",
  "compatible": true,
  "rows_checked": 0,
  "objects_checked": 1,
  "declared_rows": 3
}
```

| Field | Meaning |
|---|---|
| `compatible` | No error-level schema mismatch; not row-rule results |
| `checks` | Metadata-check outcomes |
| `schema_issues` | Mismatches: source, column, severity, reason |
| `differences` | Expected/actual type attributes |
| `not_evaluated` | Checks needing data, incl. row rules |
| `declared_rows` | Footer row count; `rows_checked` stays `0` |

| Exit | Meaning |
|---:|---|
| `0` | Schema is compatible; inspect the report for warnings and skipped checks |
| `1` | An error-level schema mismatch was found; inspect `schema_issues` |
| `2` | Configuration, unsupported format, metadata, storage or resource failure; inspect stderr |

No validation manifest is written. Logs go to stderr.

## Check every file, not just a sample

```yaml
source:
  prefix: s3://my-bucket/data/
  filter: parquet
  recursive: true

input:
  type: parquet
```

Run `valet schema -p pipeline.yml` **without** `-i`; `-i` checks only that one object.

## Run the value checks too

```bash
valet validate -p examples/parquet-schema.yml -i input.parquet -o results/
```

This runs row rules such as `AMOUNT_NONNEGATIVE` and publishes outputs and a manifest. See [structured output](../output/) or the [quickstart](../../getting-started/quickstart/#try-schema-only-checks-on-parquet).


---

# Output modes and metadata trust

Source: https://richardrh.github.io/valet/guides/validation-modes/

Choose **which rows to write** separately from **what to validate**. Terms: [output modes](../../concepts/results/#output-modes).

## Choose which rows to write

| Mode | Row files | Also written |
|---|---|---|
| `all` (default with URI) | pass, warn, error | issues, raw rejects, indexes, summary, manifest |
| `failures` | warn, error | issues, raw rejects, indexes, summary, manifest |
| `report` (default without URI) | none, no raw rejects | issues, indexes, summary, evidence, manifest |

```bash
valet validate -p pipeline.yml -i input.parquet -o results/ --output-mode failures
valet validate -p pipeline.yml -i input.parquet -o reports/ --output-mode report
```

- Without a URI, only the stdout summary is produced; `all`/`failures` are rejected.
- Every mode runs full validation. `row_output` reports omitted buckets and row counts.
- `report` is not anonymization: names, schemas and messages may be sensitive.

```yaml
output:
  uri: results/
  format: parquet
  mode: failures
```

## Use Parquet statistics only when you trust the producer

> [!WARNING]
> Statistics can be wrong or tampered with. Keep the default `off` for untrusted data.

`--parquet-stats trusted` lets a row group pass from footer null counts and bounds without decoding.

```bash
valet validate -p numeric-checks.yml -i input.parquet -o reports/ \
  --output-mode report --parquet-stats trusted --log-format json
```

```yaml
input:
  type: parquet
  parquet:
    statistics: off
  schema:
    columns:
      amount: {type: int64, required: true, nullable: false}

output:
  mode: report

rules:
  - code: AMOUNT_POSITIVE
    column: amount
    level: error
    checks:
      valid_range: {min: 0, exclusive_min: true}
```

| Situation | Result |
|---|---|
| Null count 0, minimum > 0 | Group passes from statistics |
| Missing stats, nulls, unprovable bound, warn-level violation | Group is scanned |
| `--output-mode all` | Always scans |
| Conversions, conditions, references, uniqueness, global checks | Always scans |
| `valet schema --parquet-stats` | Rejected |

Valet-written files are never trusted automatically. See [trust and tampering](../schema/#trust-and-tampering); proof rules: [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/validation-modes-spec.md).

## Read coverage, not just the outcome

| Field | Meaning |
|---|---|
| `total`, `pass`, `warn`, `error`, `coverage.value_rows` | Rows decoded and checked |
| `coverage.metadata_claimed_rows` | Rows passed from statistics |
| `coverage.accounted_rows` | Sum of both |
| `verification` | `values`, `metadata_assisted`, `not_evaluated` |
| `trust_policy`, `ingestion_verification` | Accepted assumptions; ingestion coverage |
| `rule_coverage` | Both counts per rule |

A `metadata_assisted` pass depends on trusting the producer, and `total` can be 0. Trusted runs also write an evidence ledger ([Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/validation-modes-spec.md)).

## Write bounded statistics for downstream consumers

Off by default. Writes integer, decimal and boolean statistics only:

```yaml
output:
  format: parquet
  parquet:
    statistics: bounded
```

## Inspect performance without mixing it into result JSON

```bash
valet validate -p pipeline.yml -i input.parquet -o results/ \
  --json-output - --log-format json --log-level debug
```

| Stream/event | Content |
|---|---|
| stdout | Result JSON |
| stderr | NDJSON logs |
| `run.phase` | Per-phase work (phases overlap; don't sum) |
| `run.finished` | Outcome, time, memory, bytes, coverage |
| `parquet.stats_decision` (debug) | Skip/scan reason |

Exit `3`: use the committed manifest; don't rerun.


---

# Structured output

Source: https://richardrh.github.io/valet/guides/output/

What `valet validate` writes. Terms: [report](../../concepts/results/#report), [manifest](../../concepts/results/#manifest); modes: [output modes](../validation-modes/).

| Stdout flag (`validate` only; see [CLI reference](../../reference/cli/)) | Effect |
|---|---|
| (default) | JSON summary |
| `--plain` | `key=value` |
| `--stats` | Plain statistics unless JSON requested |
| `--json-output -` | JSON; conflicts with `--plain` |
| `--json-output PATH` | Also write summary to a file |

## Schema-only output is different

`valet schema` writes JSON to stdout only; no artifacts or manifest.

```bash
valet schema -p pipeline.yml -i input.parquet > schema-report.json
```

Use `compatible` for the outcome; see [schema report](../schema/#read-the-json-report).

## Publication tree

Written only with `-o`/`output.uri`. Read `manifest.json` first; ignore runs without one.

```text
<output>/<run-id>/
├── manifest.json
├── summary.json
├── inputs.ndjson
├── rules.json
├── artifacts.ndjson
├── pass/part-*.csv|parquet
├── warn/part-*.csv|parquet
├── error/part-*.csv|parquet
├── issues/part-*.ndjson
├── rejects/part-*.ndjson
└── evidence/part-*.ndjson
```

| Path | Holds |
|---|---|
| `manifest.json` | URI, size, SHA-256 of every artifact; `row_output` |
| `rules.json` | Rule messages and definitions |
| `pass/` `warn/` `error/` `rejects/` | Only if the mode writes them and they're nonempty |
| `evidence/` | Trusted-statistics runs only |

CSV/Parquet buckets follow `output.format`; everything else is JSON/NDJSON. Integrity details: [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/validation-modes-spec.md).

## Summary

Status, outcome, row and per-rule counts, coverage, memory peak and the manifest URI.

- `status: completed` + `validation: fail` means error-level issues, not a crash.
- A metadata-assisted pass can have `total: 0`; read `verification` and `coverage`.

## Issues

```json
{
  "report_version": 2,
  "run_id": "...",
  "source": "orders.csv",
  "source_ordinal": 1,
  "row": 2,
  "global_row": 2,
  "line": 3,
  "code": "QUANTITY_POSITIVE",
  "severity": "error",
  "origin": "rule",
  "scope": "row",
  "column": "quantity"
}
```

Rows and lines are 1-based. Parquet adds `row_group` (0-based) and `row_in_group` (1-based).

## Rejects

| Failure | Result |
|---|---|
| Recoverable | Raw record, provenance, per-column `valid`/`missing`/`null`/`invalid` |
| Fatal framing | Exit `2`, no manifest; diagnostics on stderr |

## Full-validation exit codes

| Exit | Do |
|---:|---|
| `0` | Pass; warnings allowed |
| `1` | Error-level row, dataset or schema failures |
| `2` | No manifest; read stderr |
| `3` | Manifest committed, stdout failed; use it |
| `128+N` | Interrupted; no result |


---

# Operations

Source: https://richardrh.github.io/valet/guides/operations/

Running Valet in pods and batch jobs: memory, storage, signals, logs.

## Kubernetes memory

`execution.memory_budget` is a managed limit, not an RSS cap; leave headroom for the Go runtime, SDK and buffers ([memory budget](../../concepts/execution/#memory-budget)).

> [!IMPORTANT]
> Limits are per **decompressed row or record**, not per file. A 5 GB CSV with ordinary rows fits the 1 GiB profile (slower, not 5 GB of memory).

| Pod limit | `GOMEMLIMIT` | Managed budget | Batch target | Max one CSV row | Max one JSON record | Max one Parquet row/page |
|---:|---:|---:|---:|---:|---:|---:|
| 512 MiB | 384 MiB | 256 MiB | 4 MiB | 512 KiB | 2 MiB | 4 MiB |
| 1 GiB | 768 MiB | 512 MiB | 8 MiB | 1 MiB | 4 MiB | 8 MiB |
| 2 GiB | 1.5 GiB | 1 GiB | 8–16 MiB | 2 MiB | 8 MiB | 8–16 MiB |

```yaml
resources:
  requests:
    memory: 768Mi
  limits:
    memory: 1Gi
env:
  - name: GOMEMLIMIT
    value: 768MiB
```

| What | Memory behaviour |
|---|---|
| `max_record_bytes` | A large ceiling does not by itself mean a large allocation; CSV reservations grow with actual record bytes. |
| Workers | Share the same managed budget. |
| Row-local checks | Stay bounded as file size grows. |
| Exact uniqueness, reference indexes | Grow with distinct keys; can exhaust the budget and fail explicitly (no spilling or approximation). |
| Parquet row groups | Keep uncompressed ≤ 32 MiB for the 1 GiB profile; large groups can exhaust the guard. |

- Profiles are starting points: measure RSS with your real compression, schemas, output, cardinality and storage before setting production limits.
- Small-fixture MB/s does not predict runtime; see [Performance](../performance/).

Internals (decoder buffers, queues, reorder state): [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/spec.md).

## S3

Uses the standard AWS SDK credential chain; never put static credentials in pipeline documents.

| Topic | Behaviour |
|---|---|
| Settings | Endpoint, region, path-style addressing and profile, set separately for source and output. |
| Publication | Unique run prefixes; writes never overwrite. |
| Multipart | Ambiguous uploads are aborted; constrained budgets upload serially. |
| Reads | Adjacent range reads are coalesced; schema-only inspection reads just the footer. |
| Digests | Publication and artifact references share one completed digest. |

Configure an abort-incomplete-multipart lifecycle rule as a backstop. Write reconciliation and part sizing: [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/spec.md).

## Local publication

Files are published without replacement and fsynced; `manifest.json` is written last and marks the commit ([Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/spec.md)).

## Metadata trust is not data integrity

- Schema-only inspection checks metadata, not payload; Parquet statistics can be false in a valid footer.
- Version pinning and ETags do not prove the statistics match the values.
- Valet reads values by default. `--parquet-stats trusted` accepts eligible producer claims with reduced coverage, disclosed in the result and evidence ledger. Read [trust and tampering](../schema/#trust-and-tampering) and [output modes](../validation-modes/) first.
- Keep trusted digests, signatures and producer permissions outside the data's write boundary; an attacker who can replace both file and checksum can fake the evidence.

## Signals and cancellation

| Signal | Effect | Exit code |
|---|---|---|
| SIGINT | Cancel, close readers/writers, abort known multipart uploads, no terminal manifest | `130` |
| SIGTERM | Same | `143` |

## Logging

Logs go to stderr; stdout carries only requested output.

```bash
valet validate ... --log-format json --log-level info
valet validate ... --log-level debug
```

| Item | Detail |
|---|---|
| All run records | `run_id`; URIs redacted; row values and credentials never logged. |
| JSON records | `log_version`, `event`, run-local `seq`, command identity. |
| `run.phase` | Work/queue-wait timings; phases overlap, so not additive CPU. |
| `decode` phase | CSV/JSON/NDJSON parsing and batch conversion, once per batch. |
| `run.finished` | Exit/commit outcome, byte counters, managed peak, coverage. |
| `parquet.stats_decision` | Per row group, debug level; raw cells and min/max never logged. |
| Logging failure | Does not abort validation; exit `3` still reports the committed manifest. |

Event schema: [validation modes spec](https://github.com/richardrh/valet/blob/main/docs/engineering/validation-modes-spec.md).

### Why a run failed

With text logs, a failed run (exit `2`) ends with an error and, where known, a hint:

```text
valet: error: malformed CSV in data/orders.csv at line 3: found 4 fields; the header has 3
valet: hint: quote any field that contains ",", and check this line for a missing or extra ","; set input.on_malformed: reject to reject such rows and check the rest of the file
```

With `--log-format json`, the same information is in the `command failed` record:

- `error_class`: `input_format`, `resource`, `not_found`, `access_denied`, `canceled`, `deadline` or `operation`.
- Pipeline load/validation errors show the full message (e.g. `unknown field config.rules[0].chekcs (line 5)`); other errors show class and typed fields only, to avoid leaking row values.

Malformed input (`error_class: input_format`) is reported by position and reason, never record contents:

| Field | Meaning |
|---|---|
| `input_format` | `csv`, `json` or `ndjson` |
| `input_source` | The file or object, without credentials or query string |
| `input_line`, `input_column` | Where the record starts, when known |
| `input_offset` | Byte offset, when known |
| `framing_reason` | `field_count`, `quote`, `bare_quote`, `csv_syntax`, `blank_line`, `missing_header`, `invalid_utf8`, `record_too_large`, `json_syntax`, `truncated`, `not_array`, `not_object`, `trailing_data`, `multiple_records`, `multiline_record`, `duplicate_key` or `too_deep` |
| `expected_fields`, `found_fields` | Header and record field counts, for `field_count` |
| `hint` | How to fix the input |

Malformed records stop the run; `input.on_malformed: reject` rejects only wrong-field-count CSV rows ([malformed CSV rows](../configuration/#malformed-csv-rows)).

## Health checks for batch jobs

See [exit codes](../../concepts/results/#exit-codes) for definitions.

| Exit | Treat as | Read |
|---|---|---|
| `0` | Completed, warnings allowed | stdout JSON or manifest |
| `1` | Completed, checks failed | stdout JSON or manifest |
| `2` | Failed, no committed manifest | stderr |
| `3` | Manifest committed, stdout failed | manifest |
| `130`, `143` | Interrupted by signal | — |


---

# Performance

Source: https://richardrh.github.io/valet/guides/performance/

Valet streams with no file-size ceiling: volume, rules, compression and storage set speed; record limits, global-rule cardinality and the budget decide fit.

> [!NOTE]
> `max_record_bytes: 1MiB` allows a multi-gigabyte CSV if each row is at most 1 MiB.

## Measured 5 GB Parquet workload

| Setup | Value |
|---|---|
| Date, machine | 2026-10-19; Intel Core Ultra X7 358H, 16 logical CPUs, Linux amd64, Go 1.27.2 |
| Storage | Disk-backed local (not RAM-backed `/tmp`) |
| Runs | Single timed runs, not medians or cold-cache |
| Input | **5,003,192,038 bytes, 10,215,424 rows**; 512-byte string ID, `int64` amount, string status; Snappy |
| Pipeline | Explicit schema; `not_blank`, positive range, allowed status; 512 MiB budget, 8 MiB batches, 1 MiB record limit; all rows pass |

| Build | Configured workers | Full CLI elapsed | Peak process RSS |
|---|---:|---:|---:|
| Before optimizations | 1 | 65.91 s | 58.7 MiB |
| Optimized | 1 | 24.61 s | 89.4 MiB |
| Optimized | 4 | 17.57 s | 147.5 MiB |
| Optimized | 8 | 17.19 s | 224.9 MiB |

- Timed end to end: startup, decoding, every check, output writes, manifest. Fixture generation excluded.
- Output: 5,400,820,356 bytes in 41 parts (before: ~5.15 GB in 156); faster Snappy trades size for CPU. Outputs were verified value-for-value against input.
- The sub-10-second target was **not reached**; past four workers gains are small (evaluation and output stay ordered single-owner stages) and memory rises.

## Comparison with GX Core

Measured 2026-10-26, same workstation, with [`benchmarks/competitors`](https://github.com/richardrh/valet/tree/main/benchmarks/competitors):

- 10,000,000 rows: `id` int64 with nulls, `amount` decimal(18,2), `quantity` int64, `status` string.
- Checks: `id` not null, `amount` 0–10000, `quantity` 1–1000, `status` in a fixed set; ~1% of rows fail each.
- Median of three fresh processes, warm cache; both tools reported the expected failure counts.

| Input | Tool | Wall time | Peak RSS |
|---|---|---:|---:|
| CSV, 251 MiB | Valet, 1 worker | 5.37 s | 53 MiB |
| CSV | Valet, 4 workers | 3.60 s | 74 MiB |
| CSV | GX Core 1.24.1 + pandas 3.1.0 | 2.71 s | 1,467 MiB |
| Parquet, 93 MiB | Valet, 1 worker | 3.04 s | 112 MiB |
| Parquet | Valet, 4 workers | 2.53 s | 280 MiB |
| Parquet | GX Core + pandas | 17.22 s | 2,654 MiB |

| | GX Core | Valet |
|---|---|---|
| Memory | Whole file in a pandas DataFrame | Streams within the managed budget |
| Notes | Parquet decimals become Python `Decimal` (slow); CSV `amount` compared as float64 | No rules, 1 worker: 3.8 s CSV, 1.4 s Parquet (CSV reading ≈ 70%) |
| `abc` in decimal column | Range expectation errors instead of counting | Row fails, cell counted as `INGEST_TYPE`, run continues |
| Wrong CSV field count | `pandas.read_csv` raises | Exits `2` |

## Schema-only Parquet inspection

```bash
valet schema -p pipeline.yml -i input.parquet
```

Took **0.040 s** on the same file, startup included. It reads only the footer: `scope: schema`, `metadata_only: true`, `validation: not_evaluated`, zero checked rows, unverified row count.

## Parquet execution costs

- Full local validation keeps a baseline digest and a final replay digest; page verification is fused into row-group decoding, giving three source passes on the measured plan.
- S3 identity pinning adds no extra digest scan.
- MB/s depends on row width and compression; don't extrapolate from small fixtures. No service-level guarantees.

## What changes throughput

| Factor | Effect |
|---|---|
| Schema inference | Extra pass; explicit CSV schema avoids it |
| Uniqueness, global checks | Replay plus cardinality-dependent memory |
| Wide strings, decimals, timestamps, regexes, conversion failures | Higher per-row cost |
| Output format, compression, part size | Sink cost |
| Cold storage, S3 latency, TLS, range requests | Can dominate compute |
| Gzip | Lower input MB/s |
| Small files | Dominated by startup and publication |

## Capacity planning

Benchmark the exact pipeline in the target pod and storage with representative data. Keep `memory_budget` below the pod limit and `GOMEMLIMIT` with headroom ([operations](../operations/)).

Opt-in scale suite:

```bash
go test -tags=scale ./benchmarks \
  -run '^TestScale100GiB$' \
  -timeout 0 \
  -v
```


---

# CI automation

Source: https://richardrh.github.io/valet/guides/ci/

Gate on the exit code and stdout JSON ([exit codes](../../concepts/results/#exit-codes)).

| Exit | `valet validate` | `valet schema` ([schema-only checks](../schema/)) |
|---|---|---|
| `0` | No error-level failures (warnings allowed) | Compatible |
| `1` | Failed checks | Error-level schema mismatch |
| `2` | Operational/presentation failure; no manifest | Operational/configuration failure |
| `3` | Artifacts committed; stdout failed | — |

Without `-o`/`output.uri`, `validate` writes only the stdout summary. `schema` never publishes a validation manifest.

## GitHub Actions example

The bundled fixture exits `1` (missing record ID). Omit `output.uri` for this aggregate-only path.

```yaml
name: Validate data

on:
  pull_request:
  workflow_dispatch:

jobs:
  validate:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache-dependency-path: go.sum

      - name: Build Valet
        run: go build -trimpath -o dist/valet ./cmd/valet

      - name: Validate
        id: validate
        shell: bash
        run: |
          set +e
          dist/valet validate \
            -p examples/pipeline.yml \
            -i examples/sample.csv \
            --log-format json \
            > validation-summary.json \
            2> validation.log.ndjson
          code=$?
          set -e

          echo "exit_code=$code" >> "$GITHUB_OUTPUT"
          if test "$code" -gt 1; then
            exit "$code"
          fi

      - name: Upload validation report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: validation-result
          path: |
            validation-summary.json
            validation.log.ndjson
          if-no-files-found: warn

      - name: Enforce data quality
        if: steps.validate.outputs.exit_code == '1'
        run: exit 1
```

## Consume the summary

```bash
jq '{validation, error, rule_counts}' validation-summary.json
```

- Parse only after exit `0`/`1`; otherwise read stderr.
- Run lakehouse ingestion after **Enforce data quality**, on the same immutable input.
- Mandatory checks need `level: error`; warnings still exit `0`.
- Artifacts: add `-o validation-output`, upload `validation-output/`; the summary gains `manifest_uri`. Read `kind: issues`/`kind: rejects` entries from `manifest.json`, never by listing files ([results](../../concepts/results/#manifest)).
- `--output-mode report` publishes issues and reports without rewriting row data.

## JSON logs

```bash
valet validate ... --log-format json 2>valet.log.ndjson
```

Framing, configuration, I/O and resource failures exit `2` with no manifest.

## S3 jobs

AWS SDK credential chain (workload identity, OIDC, secret provider); no static credentials in the pipeline. Source and output may differ in endpoint, region, profile.

## Run checks locally

```bash
go test ./...
go test -tags=integration ./integration -timeout 20m
```

Object-store cases skip without endpoint and credentials. Real S3-compatible acceptance:

```bash
bash .github/scripts/test-s3-compatible.sh minio
bash .github/scripts/test-s3-compatible.sh rustfs
```

- Needs Go, a C compiler, Docker, curl, jq. Rootless Podman: prefix `CONTAINER_RUNTIME=podman`.
- Loopback bindings, local-only credentials; cleans up containers and data; any selected skip fails. `VALET_S3_LOG_DIR` keeps test events and server logs.

## Repository CI and release publishing

Workflows in `.github/workflows/` use Go from `go.mod` (1.27.3); don't lower it for a runner. Test-gate rationale: [Engineering spec](https://github.com/richardrh/valet/blob/main/docs/engineering/spec.md).

### CI contract

`go.yml` runs on PRs, pushes to `main`, manual dispatch and release calls (not other branch or tag pushes). Docs-only PRs (`docs/content`, layouts, Hugo config, engineering docs, `*.md`, `docs.yml`) skip every Go job.

| Lane | When | Jobs |
|---|---|---|
| PR | Every code PR | `lint`: modules, gofmt, vet, staticcheck, `govulncheck` v1.9.0, `actionlint` v1.7.12 · `test`: `go test -race ./...` · `cli`: release executable + whole integration suite, S3 included, against MinIO `07c3a429bfed` |
| Full | `main`, releases, manual runs, PRs labelled `full-ci` | adds `native` (Linux arm64, Windows amd64), `darwin` (amd64/arm64 cross-built and tests compiled; not run, signed or notarized), `s3` (MinIO + RustFS 1.0.3, release and race executables; logs `s3-compatibility-*`), `go-install` (isolated `GOBIN`; tags also `go install …@<tag>`) |
| Manual | Input `run_scale` | 10 GiB and 100 GiB Arrow streams |

- **`CI result`** is the single required check: it passes when every job that had to run passed.
- **Caching:** each job restores its own Go build, test and staticcheck caches, saved on every `main` commit; the MinIO build is cached by its pinned commit and Go version. Integration runs against a built executable always use `-count=1`.
- **AWS:** optional opt-in; not a release gate. **`docs.yml`:** Hugo Extended 0.156.0 (checksum-verified) for docs PRs, `main`, manual.

Release publication runs the full lane first. `docs.yml` PR builds have `contents: read` only, use the repository's default Pages URL as base URL and never touch Pages (they work before Pages exists; separate concurrency groups can't cancel deploys). Publishing runs only on `main` push/manual and use the Pages base URL (custom domain included).

### One-time GitHub setup

Repository settings; the workflows don't change them for you.

| # | Where | Do |
|---|---|---|
| 1 | Settings → Actions → General | Enable Actions; read-only default token; no PAT/secret needed |
| 1a | Org policy | Allow `contents: write`, `pages: write`, `id-token: write`; fork PRs may need approval |
| 2 | Ruleset on `main` | Require review and the **CI result** check |
| 2a | — | Don't require individual jobs, scale or the path-filtered docs check (skipped jobs would leave PRs pending) |
| 3 | Settings → Pages | Source **GitHub Actions**; don't commit `docs/public/`. Not enabled automatically: without it the first publishing run fails (PR builds don't) |
| 4 | Environments → **github-pages** | **Selected branches and tags**: `main` only; optional reviewers |
| 5 | Actions → **Documentation** | Run on `main`, check the URL; other branches only build |
| 6 | Tag ruleset | Only trusted maintainers create release tags; review workflow changes. The release job's `GITHUB_TOKEN` gets `contents: write`; Pages gets `pages: write` + `id-token: write`; PR builds get neither |

### Publish a version

Update `CHANGELOG.md`, review `main`, then:

```bash
git switch main
git pull --ff-only
git tag -a v0.1.0 -m "Release v0.1.0"
git push origin v0.1.0
```

Tags: `vMAJOR.MINOR.PATCH` (`v0.1.0`), optional hyphenated prerelease (`v0.1.0-rc.1`); no `+build.1` metadata.

`release.yml` validates the tag, runs full Go CI, creates a draft, uploads assets, then publishes via `GITHUB_TOKEN` (no PAT):

- `valet_<tag>_linux_amd64.tar.gz`, `valet_<tag>_linux_arm64.tar.gz`
- `valet_<tag>_darwin_amd64.tar.gz`, `valet_<tag>_darwin_arm64.tar.gz`
- `valet_<tag>_windows_amd64.zip`
- `pipeline.schema.json`: the pipeline JSON Schema for that version
- `checksums.txt`: SHA-256 per asset (not a signature or attestation)

Archives hold the executable, `README.md`, `CHANGELOG.md`, `SECURITY.md`, MIT `LICENSE`; dependencies keep their own licenses. Verify downloads: [installation guide](../../getting-started/installation/).

- Prerelease tags never become latest; stable ones do.
- Existing release or draft: refused. Failed draft: inspect, delete if safe, rerun the same tag.
- Never move or reuse a published tag; release a new version.


---

# Rules cookbook

Source: https://richardrh.github.io/valet/rules/

Check schemas, individual rows and whole datasets. A row violates a rule when any of its checks fails; at most one issue is emitted per rule and row.

```yaml
rules:
  - code: STABLE_RULE_CODE      # stable, unique across rules and dataset_rules
    name: Human-readable name
    level: error                # error: fails validation; warn: row stays a warning
    column: default_column
    message: Explanation shown for failures
    checks:
      not_null: true                       # no arguments: true
      valid_values: [OPEN, PAID]           # one main argument: short form
      valid_range: {min: 0, max: 10000}    # several arguments: a mapping
```

- Each key under `checks` is a check name; checks run in document order.
- The value is one of:
  - `true` (or empty, or `{}`) for a check without arguments;
  - the check's main argument, for checks with a short form (see *Short form* below), e.g. `matches_pattern: '^[A-Z]+$'`;
  - a mapping of arguments, e.g. `valid_range: {min: 0, exclusive_min: true}`.
- `false` is rejected: remove the check instead.
- Check-level `column`/`columns` overrides the rule's `column`; the two are mutually exclusive. Unknown checks and arguments are rejected.
- A list is still accepted, and is needed only to use the same check twice in one rule:

  ```yaml
  checks:
    - {type: matches_pattern, pattern: '^[A-Z]'}
    - {type: matches_pattern, pattern: '[0-9]$'}
  ```

## Check reference

Scopes: see [Check scopes](../concepts/rules-and-checks/#check-scopes). `column*` = `column` or `columns` (or the rule's `column`). *Short form* is what a non-mapping value sets.

| Check | Scope | Parameters | Short form | Null | Notes |
|---|---|---|---|---|---|
| `not_null` | row | `column*` | `true` | fails | null or missing fails |
| `not_blank` | row | `column*` | `true` | fails | also empty / Unicode-whitespace-only strings |
| `is_present` | row | `column*` | `true` | explicit null passes | only whether the JSON field/path exists |
| `valid_length` | row | `column*`, `length` | `length` | fails | Unicode code points |
| `min_length` | row | `column*`, `min_length` | `min_length` | fails | code points |
| `max_length` | row | `column*`, `max_length` | `max_length` | fails | code points |
| `valid_range` | row | `column*`, `min`, `max`, `exclusive_min`, `exclusive_max` | — | passes | inclusive by default; `exclusive_min` needs `min`, `exclusive_max` needs `max` |
| `equals` | row | `column*`, `value` | `value` | fails | fails on mismatch |
| `not_equals` | row | `column*`, `value` | `value` | passes | fails only on equality |
| `all_equal` | row | `columns` (required), `value` | — | fails | every selected column equals `value` |
| `has_null_keys` | row | `columns` (required) | `columns` | fails | |
| `valid_values` | row | `column*`, `values` | `values` | passes | allowed set |
| `matches_pattern` | row | `column*`, `pattern` | `pattern` | fails | Go regexp; searches unless anchored |
| `is_unique` | global | `column*` | `true` | fails | every duplicate occurrence fails, incl. the first |
| `is_unique_composite` | global | `columns` (required) | `columns` | fails | as above |
| `compare_columns` | row | `column`, `other_column`, `operator` | — | fails | `==` `!=` `<` `<=` `>` `>=` |
| `referential_integrity` | reference | `column`, `reference_file`, `reference_column` | — | distinct from empty | |
| `referential_integrity_composite` | reference | `columns` (required), `reference_file`, `reference_columns` | — | distinct from empty | ordered column lists |
| `required_columns` | schema | `columns` (required) | `columns` | — | schema gate at `error`; extra columns allowed |
| `schema_matches` | schema | `columns` (name → type map), `mode`, `require_columns`, `allow_extra_columns` | — | — | schema gate at `error` |
| `valid_schema_columns` | schema | `reference_file`, `reference_column` | — | — | |
| `type_check` | row | `column*`, `expected_type` | `expected_type` | target-dependent | not run by `valet schema` |
| `date_format` | row | `column*`, `format` or `formats` | `format` | passes | |
| `timestamp_window` | row | `column*`, `min`, `max`, `timezone` | — | passes | inclusive bounds; offset-less wall times need IANA `timezone` |

- Invalid conversion makes a check unevaluable, not null. Uniqueness state counts against the [memory budget](../concepts/execution/#memory-budget).
- Relative `reference_file` paths resolve from the pipeline file.
- `compare_columns`: declare an input schema so numbers, decimals and dates compare by type, not as strings.

## Conditions

`when` limits a rule to matching rows. Allowed only on `row` and `reference` scope checks (conditional global/schema checks are rejected).

- Operators: `eq`, `ne`, `lt`, `le`, `gt`, `ge`, `in`, `is_null`, `is_present`; compose with `all: [...]`.
- False condition: rule not applicable. Invalid operand: counted as unevaluable.

```yaml
- code: CLOSED_AT_REQUIRED
  level: error
  column: closed_at
  when: {column: status, operator: eq, value: CLOSED}
  checks: {not_null: true}

# composed
when:
  all:
    - {column: status, operator: eq, value: PAID}
    - {column: refunded, operator: eq, value: false}
```

## Dataset metrics

`dataset_rules` are separate from row rules: `code`, `level`, `metric`, then `min` / `max` / `equals` (`equals` excludes `min`/`max`; `min` ≤ `max`).

| Metric | Needs | Bounds |
|---|---|---|
| `row_count` | no `column`, no `rule_code` | nonnegative integer |
| `missing_count`, `null_count`, `blank_count`, `conversion_error_count` | `column` | nonnegative integer |
| `missing_fraction`, `null_fraction`, `blank_fraction`, `conversion_error_fraction` | `column` | 0–1 |
| `rule_failure_count` / `rule_failure_fraction` | `rule_code` of a known row rule; no `column` | integer / 0–1 |
| `sum`, `min`, `max`, `avg` | `column` declared numeric in an explicit input schema | any number; `tolerance` (finite, ≥ 0) only with `equals` on float columns |

```yaml
dataset_rules:
  - {code: DATASET_NOT_EMPTY, level: error, metric: row_count, min: 1}
  - {code: CUSTOMER_NULL_RATE, level: error, metric: null_fraction, column: customer_id, max: 0.001}
  - {code: ORDER_TOTAL, level: error, metric: sum, column: subtotal, min: 0}
```

## Recipes

One example per pattern; other checks follow the same shape.

```yaml
rules:
  - code: CUSTOMER_ID_REQUIRED
    level: error
    column: customer_id
    checks: {not_null: true}
  - code: QUANTITY_POSITIVE          # > 0; null passes, add not_null to forbid it
    level: error
    column: quantity
    checks:
      valid_range: {min: 0, exclusive_min: true}
  - code: SUBTOTAL_EXCEEDS_DISCOUNT  # rule-level column not needed
    level: error
    checks:
      compare_columns: {column: subtotal, other_column: discount, operator: ">"}
  - code: ORDER_LINE_UNIQUE
    level: error
    checks:
      is_unique_composite: [order_id, line_number]
  - code: CUSTOMER_EXISTS
    level: error
    column: customer_id
    checks:
      referential_integrity: {reference_file: references/customers.csv, reference_column: customer_id}
  - code: QUANTITY_IS_INTEGER        # row-level type check, e.g. with mode: raw_strings
    level: error
    column: quantity
    checks:
      type_check: int64
```

### Columns and schema

```yaml
  - code: EXACT_SCHEMA               # exact names and types, none missing or extra
    level: error
    checks:
      schema_matches:
        mode: strict
        require_columns: true
        allow_extra_columns: false
        columns: {order_id: string, customer_id: string, quantity: int64, unit_price: decimal, created_at: timestamp}
```

With Parquet input, schema checks run without row reads: `valet schema -p pipeline.yml -i input.parquet`.

| `schema_matches` mode | Behaviour |
|---|---|
| `strict` | Compares logical types; does not prove non-null, valid-format or in-range values |
| `castable` (default) | Needs full validation; `valet schema` reports it not evaluated (presence/extra-column policies still checked) |

- `valet schema` compares actual file types without conversion; `valet validate` may apply `input.schema` first.
- `input.schema` `extra_fields: error` / `required: true` check columns, not values; they do not replace `not_null`.
- Type control: `valet schema` (Parquet types, no row scan) · `input.schema` (conversion; failures are ingestion rejects) · `type_check` (rule issue; not run by `valet schema`). Type names and schemas: [Pipelines](../concepts/pipelines/#type-names). Report fields and exit codes: [schema-only guide](../guides/schema/).

## Runnable example

The complete orders cookbook is in `examples/cookbook/`:

```bash
go build -o dist/valet ./cmd/valet

dist/valet plan \
  -p examples/cookbook/orders.yml \
  -i examples/cookbook/orders.csv

dist/valet validate \
  -p examples/cookbook/orders.yml \
  -i examples/cookbook/orders.csv \
  -o cookbook-output \
  --json-output cookbook-summary.json
```

Exit `1` is expected: the example contains invalid rows. Follow `manifest_uri` in `cookbook-summary.json`, then the manifest's artifact index, to the structured issues.


---

# Reference

Source: https://richardrh.github.io/valet/reference/

Lookup pages for authoring and operating pipelines.


---

# CLI reference

Source: https://richardrh.github.io/valet/reference/cli/

```text
valet validate -p PIPELINE [-i URI] [-o URI] [options]
valet plan     -p PIPELINE [-i URI] [options]
valet schema   -p PIPELINE [-i URI] [options]
valet --help | --version
```

| Command | Does |
|---|---|
| `validate` | Validates the data. Prints a JSON summary to stdout; with `-o`, also publishes artifacts and a manifest |
| `plan` | Describes schemas, passes and state without validating. `--format pipeline` prints an `input.schema` block inferred from the data |
| `schema` | Checks Parquet footer and schema compatibility without reading rows. See [schema-only checks](../../guides/schema/) |

CLI values override the pipeline file, which overrides defaults. Logs and diagnostics go to stderr.

## Flags

Each command accepts only its own flags.

| Flag | Value | `validate` | `plan` | `schema` |
|---|---|:-:|:-:|:-:|
| `-p`, `--pipeline` | Local `.yaml`, `.yml` or `.json` file (required) | ✓ | ✓ | ✓ |
| `-i`, `--input` | File or `s3://` object; replaces the configured source. Not stdin | ✓ | ✓ | ✓ |
| `-o`, `--output` | Directory or `s3://` prefix for artifacts | ✓ | ✓ | |
| `--memory-budget` | Size, e.g. `512MiB` (default `512MiB`) | ✓ | ✓ | ✓ |
| `--batch-target` | Size, e.g. `8MiB` | ✓ | ✓ | |
| `--workers` | Number of validation workers (default `1`) | ✓ | ✓ | |
| `--output-mode` | `all`, `failures` or `report`. Default `report` without `-o`, `all` with it | ✓ | ✓ | |
| `--parquet-stats` | `off` (default) or `trusted` | ✓ | ✓ | |
| `--format` | `plan`: `json` (default) or `pipeline`; `schema`: `json` (default) or `text` | | ✓ | ✓ |
| `--plain` | Print the summary as `key=value` lines | ✓ | | |
| `--stats` | Include per-rule failure counts and configured statistics | ✓ | | |
| `--json-output` | Also write the summary JSON to a path; `-` selects JSON on stdout | ✓ | | |
| `-v`, `--verbose` | Debug logging, same as `--log-level debug` | ✓ | ✓ | ✓ |
| `--log-level` | `debug`, `info` (default), `warn` or `error` | ✓ | ✓ | ✓ |
| `--log-format` | `text` (default) or `json` | ✓ | ✓ | ✓ |

`-h`/`--help` and `--version` work without reading any storage.

## Flag errors

Flag errors exit `2` before any input is read.

| Input | Message |
|---|---|
| `schema -p p.yml -o out` | `-o is not a schema option; schema accepts --pipeline, --input, --format, …` |
| `validate --bogus` | `unknown flag --bogus; see valet --help` |
| `--output-mode some` | `--output-mode must be all, failures or report, not "some"` |
| `--workers many` | `--workers must be a number, not "many"` |
| `-i -` | `--input - (stdin) is unsupported: inputs must be replayable files or s3:// objects` |
| `-v --log-level info` | `--verbose selects debug logging and conflicts with --log-level info` |
| `--plain --json-output -` | `--plain conflicts with --json-output -` |

## Exit codes

| Code | Meaning |
|---|---|
| `0` | Passed; warnings allowed |
| `1` | Validation or schema failure |
| `2` | Configuration, input, I/O or resource failure before anything was committed |
| `3` | The result was committed, but printing it failed |
| `130` | Interrupted with Ctrl-C (SIGINT); no result committed |
| `143` | Terminated (SIGTERM), e.g. by a CI timeout or `kill`; no result committed |

`130` and `143` follow the shell convention of 128 + signal number. See [signals and cancellation](../../guides/operations/#signals-and-cancellation) for what is cleaned up.