Pipeline configuration

A pipeline is a local .yaml, .yml or .json file (strict RFC JSON). See pipeline file.

Top-level structure

Section	Key fields	Defaults
`source`	`uris` or `prefix` + `filter` + `recursive`	—
`input`	`type`, `schema`, `on_malformed`, `infer_schema_length`, `parquet.statistics`	`type` from suffix; `abort`; `1000000`; `off`
`execution`	`memory_budget`, `batch_target`, `max_record_bytes`, `workers`	`workers: 1`
`output`	`uri`, `mode`, `format`, `parquet.statistics`	`mode`: `report` without URI, else `all`; `none`
`rules`, `dataset_rules`	see rules	`[]`

name: Orders validation
version: "1.0"

source:
  uris: [orders.csv]

input:
  type: csv
  schema:
    extra_fields: error
    columns:
      order_id: {type: string}
      quantity: {type: int64}

execution:
  memory_budget: 512MiB
  batch_target: 8MiB
  max_record_bytes: 1MiB
  workers: 1

rules: []
dataset_rules: []

-i / -o and execution flags override the document.
Relative references, mappings and schema files resolve from the pipeline’s directory.
No output URI → stdout JSON report only. See output modes.

Sources

source:
  uris:
    - data/part-001.csv
    - s3://bucket/data/part-002.csv

source:
  prefix: s3://bucket/data/
  filter: csv
  recursive: true

uris and prefix are mutually exclusive; discovery is sorted deterministically.

Input formats

type: csv, json, ndjson or parquet; inferred from the suffix if omitted. See inputs and formats.

Malformed CSV rows

Default: a malformed record stops the run with exit 2. To reject CSV rows with the wrong field count and continue:

input:
  type: csv
  on_malformed: reject   # default: abort

Rejected row	Result
Counts	In `total`, error bucket, every column invalid
Issue	One `INGEST_MALFORMED`, e.g. `found 3 fields; the header has 2`
Rejects output	Original fields as JSON array, e.g. `"raw": ["2","3","EXTRA"]`
Column rules	Not also failed

Always fatal: bad quotes, invalid UTF-8, blank lines, oversized records, malformed JSON/NDJSON, malformed transforms mapping files. Engineering spec

Explicit schemas

input:
  schema:
    extra_fields: error
    columns:
      id: {type: string, required: true, nullable: false}
      amount: {type: decimal, precision: 18, scale: 2}
      effective_date: {type: date, formats: ["%F"]}
      observed_at:
        type: timestamp
        timezone: UTC
        formats: ["%Y-%m-%dT%H:%M:%S%:z"]

Failed conversions become ingestion issues and rejected records, not nulls.

Generate a schema from a sample file

valet plan -p pipeline.yml -i sample.csv --format pipeline

# Generated by valet plan. Review the types, then add rules.
input:
  type: csv
  schema:
    extra_fields: error
    columns:
      order id: {type: int64}
      amount: {type: float64}
      ok: {type: bool}
      note: {type: string}

Format	Types
CSV, JSON	Inferred from `infer_schema_length` rows: `int64`, `float64`, `bool`, `string` only
Parquet	File’s own types, incl. decimal precision/scale and time zones

Review before use: change money to {type: decimal, precision: …, scale: …} and dates to date/timestamp with formats.

Choose schema inspection or value conversion

valet validate converts values to input.schema; valet schema only compares it with Parquet footer types. See schema-only Parquet checks.

valet schema -p pipeline.yml -i input.parquet

Output policy and optional Parquet statistics

input:
  parquet:
    statistics: off       # off (default) or explicitly trusted
output:
  uri: ./results          # explicitly request artifact publication
  mode: all               # all, failures, report
  format: parquet
  parquet:
    statistics: none      # none (default), bounded

all/failures need a destination. Override with --output-mode and --parquet-stats; see validation modes.

Execution limits

Field	Limits
`memory_budget`	Valet-managed memory; not process RSS
`batch_target`	Target batch size
`max_record_bytes`	One record or Parquet page, not the file
`workers`	1–32, default 1; does not raise the budget

Starting point for a 1 GiB pod, with GOMEMLIMIT=768MiB:

execution:
  memory_budget: 512MiB
  batch_target: 8MiB
  max_record_bytes: 1MiB # maximum one CSV row, not the CSV file
  workers: 1

Uniqueness and reference indexes fail at the budget. See memory budget.

Editor support and AI agents

The pipeline JSON Schema gives YAML/JSON editors completion and typo checks.

# yaml-language-server: $schema=https://richardrh.github.io/valet/schema/v1/pipeline.schema.json

{
  "$schema": "https://richardrh.github.io/valet/schema/v1/pipeline.schema.json",
  "rules": []
}

Valet ignores $schema; valet plan still runs cross-field and data checks.

For agents	Content
`llms.txt`	Page index
`llms-full.txt`	All docs, one file
`…/guides/configuration.md`	Any page as Markdown

uvx --from mcpdoc mcpdoc --urls "Valet:https://richardrh.github.io/valet/llms.txt"

Next: rules cookbook, operations, performance.

Quickstart Parquet schema checks