DartAframes

Native Polars and Apache Arrow data for Dart.

dartaframes_polars runs lazy Polars queries through a Rust native library and brings the results back through a Dart-native Arrow data model. It includes:

• lazy CSV and Parquet scans with projection and predicate pushdown;
• eager DataFrame and Series operations;
• owned Arrow arrays, schemas, values, and record batches implemented in Dart;
• zero-copy-compatible Arrow C Data and C Stream interchange; and
• bounded record-batch streaming.

This is the first stable release with partial Polars 0.55.2 coverage. The Arrow layer is also intentionally scoped: it is a Dart port of the Arrow columnar data model and interchange interfaces used by this package, not a port of the complete Apache Arrow compute ecosystem. See API coverage for the current surface.

Quick start

The package and native binaries are published as dartaframes_polars 0.1.0. Install it from pub.dev, or build the native library with mise for local development.

import 'package:dartaframes_polars/polars.dart';

void main() {
  final polars = Polars.open('/path/to/libdartaframes_polars_ffi.dylib');

  final people = polars.scanCsv('people.csv').head(3).collectSync();
  print(const OwnedBatchJsonCodec().encode(people.exportSync()));
}

The same lazy API works with Parquet:

final sales = polars
    .scanParquet('sales.parquet')
    .select([polars.col('region'), polars.col('revenue')])
    .collectSync();

After the first release, verified prebuilt binaries will make the usual setup:

final polars = Polars.native();

See Getting started for query examples and Build from source for the current setup.

Arrow for Dart

The repository contains an owned, strongly typed Arrow representation written in Dart. Polars frames and series can be copied into RecordBatch and ArrowArray values, or exchanged with compatible native libraries through the standard Arrow C Data and C Stream interfaces.

import 'package:dartaframes_polars/arrow.dart';

Use this focused entrypoint when only the Dart Arrow value model is needed. package:dartaframes_polars/polars.dart re-exports it for Polars applications. Read Arrow interoperability for scope, ownership, and examples.

import 'package:dartaframes_polars/arrow.dart'; // Pure Dart, no native library.

Use Polars independently for native file scans and computation:

import 'package:dartaframes_polars/polars.dart';

final polars = Polars.open('/path/to/libdartaframes_polars_ffi.dylib');
final frame = polars.scanParquet('sales.parquet').collectSync();

Bridge the two APIs with frame.exportSync() and polars.fromRecordBatchSync(batch).

Write files with Polars

CSV and Parquet writers accept typed options on eager frames and lazy sinks. Parsing, computation, and serialization are delegated to upstream Polars.

frame.writeCsvSync(
  'output.csv',
  options: const CsvWriteOptions(separator: ';'),
);
frame.writeParquetSync(
  'output.parquet',
  options: const ParquetWriteOptions(compression: ParquetCompression.zstd),
);

Writers use the destination directly, without a binding-owned temporary-file replacement layer. Failed writes can leave partial output.

This package exposes Polars APIs and the Arrow interchange needed to move values across FFI. It does not implement database connectors, workbook conversion, or a separate query engine. SqlContext remains available for upstream Polars SQL over registered DataFrames and LazyFrames.

Resource lifetime

Native-backed objects have finalizers, so normal application code can stay idiomatic and does not need nested try/finally blocks. close() remains available and idempotent when deterministic release matters, especially in servers, loops, benchmarks, and batch-stream consumers.

final query = polars.scanCsv('events.csv');
final frame = query.collectSync();

// Use frame...

frame.close();
query.close();

Derived native values own independent handles. Values from different Polars runtimes cannot be mixed. Pure Dart Arrow values such as RecordBatch do not need to be closed.

Public entrypoints

import 'package:dartaframes_polars/polars.dart'; // Polars plus the Arrow value API.
import 'package:dartaframes_polars/arrow.dart';  // Arrow values without Polars APIs.

Most applications need only polars.dart.

Supported native targets

Android, iOS, other native targets, and the web are not supported. See Native binaries for distribution and verification details.

Development

Requirements: Dart 3.13 or later and the pinned Rust toolchain.

dart pub get --enforce-lockfile
dart format --output=none --set-exit-if-changed .
dart analyze --fatal-infos
dart test --coverage=coverage
dart test tool/crap_gate_test.dart
python3 -m unittest discover -s tool -p 'test_native_*.py'
cargo test --locked --workspace

See Contributing, the documentation index, and the release process.

Security and license

Report vulnerabilities privately through GitHub Security Advisories, not a public issue. See SECURITY.md.

Licensed under the Apache License 2.0.