Data & Dataframes

Arrow and Parquet are Apache specifications rather than Rust projects — the format publishes a feature matrix across its official implementations, and Rust is one column of itSource ¹: arrow.apache.org. Around that column sit the query engines, columnar formats and streaming systems that are Rust top to bottom.

Several of these are Rust implementations whose documented API is the Python one, and several others are servers you deploy rather than crates you depend on. Entries are tagged with which.

arrow.apache.org — the format's own implementation matrix Back to reference 1 in the text

Dataframes & query engines comprehensive

Dataframes and the query engines under them. Production-grade and Rust-native, though several are Rust cores with Python-first APIs — check that the Rust API is first-class before you build on it.

Featured: polars [crate · repo · docs]

The dataframe engine, with a funded company behind it and a lazy query optimizer that is genuinely good. It is filed here as a Rust core with a Python API, and that is not a slight — it is what the project's own artifacts say. Python Polars is 1.44 with a written versioning policy and a 2.0 upgrade guide; the Rust crate is 0.55, has no stability guarantee documented anywhere, and the 65 KB upgrade guide contains the words "rust", "crate" and "cargo" exactly zero times. You can absolutely build on it from Rust — plenty do — but expect every minor bump to break you and expect the docs to be written for someone else. If you want a Rust-first engine with a policy attached, that is DataFusion.

DataFrame library based on Apache Arrow

docsPolars' versioning policy — which describes the Python package only postpolars #25959: a maintainer proposing to rename the fundamental Rust constructors

version: 0.55.2
downloads: 2.6M / 90d
stars: 39.7k
license: MIT
last commit: 2026-09-14
published: 2026-08-06
dependents: 43 owners · 242 repos

Featured: datafusion [crate · repo · docs]

An extensible SQL and DataFrame query engine, an Apache top-level project, and the clearest counter-example to the pattern on the rest of this hub: it treats Rust callers as the customer and writes down what it promises them in a published API-health policy, with per-version upgrade guides. That is why it is embedded in InfluxDB 3, Cloudflare's R2 query engine, dbt's Fusion engine and Rerun rather than only used as a server. If you need to run SQL over Arrow data inside your own Rust process, this is it.

DataFusion is an in-memory query engine that uses Apache Arrow as the memory model

docsDataFusion's API health policy — what it promises Rust callers productionDataFusion's list of named production users

version: 55.1.0
downloads: 5M / 90d
stars: 9.3k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-09-11
dependents: 31 owners · 194 repos

Eventual-Inc/Daft [repo]

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

stars: 5.8k
license: Apache-2.0
last commit: 2026-09-11

duckdb [crate · repo · docs]

Ergonomic wrapper for DuckDB

version: 1.10505.0
downloads: 1.5M / 90d
stars: 961
license: MIT
last commit: 2026-09-14
published: 2026-07-22
dependents: 33 owners · 11 repos

lakehq/sail [repo]

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

stars: 3.4k
license: Apache-2.0
last commit: 2026-09-12

connectorx [crate · repo · docs]

Load data from databases to dataframes, the fastest way.

version: 0.4.5
downloads: 13k / 90d
stars: 2.6k
license: MIT
last commit: 2026-09-14
published: 2026-01-18
dependents: 5 owners · 4 repos

New & experimental (1)

Collected, but not clearing the published listing bar — too new, too quiet, or with no use by anyone but its author that we can see. Plenty of good software is here, some of it years old: this says what we can measure, not what it is worth.

ndatafusion [crate · repo · docs]

Extensions and support for linear algebra in DataFusion

version: 0.1.1
downloads: 46 / 90d
stars: 19
license: Apache-2.0
last commit: 2026-07-28
published: 2026-04-15

Archive (1)

Kept so you stop looking for them. Archives are computed on every build and reverse themselves if the reason clears. Where a human has named the reason or a replacement, it cites the public issue behind that.

utah [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

dataframe structure and operations

version: 0.1.2
downloads: 17 / 90d
stars: 145
license: MIT
last commit: 2018-07-24
published: 2016-12-28
dependents: 1 repo

Lakehouse & columnar formats comprehensive

Columnar formats and the table layers above them. Arrow is the interchange format that lets everything else on this page talk to everything else, and its Rust implementation is one the Apache project maintains itself.

Featured: arrow [crate · repo · docs]

Apache Arrow in Rust, on a stable semver train in the fifties — and the interchange format that lets everything else on this hub talk to everything else. Polars, DataFusion, Lance, Vortex, delta-rs, iceberg-rust and the Python bindings all speak it, which means a column you produce in one can be read by another without a copy. parquet from the same repository is the on-disk half. If you are moving tabular data across a boundary in Rust, the answer is almost always Arrow.

Rust implementation of Apache Arrow

docsarrow-rs API documentation productionInfluxDB completes its move from Go to Rust, onto Arrow and DataFusion (505 points, r/rust, 2023-09-29) postDaft on abandoning the unmaintained arrow2 for arrow-rs (2025-12-03)

version: 59.3.0
downloads: 18M / 90d
stars: 3.6k
license: Apache-2.0
last commit: 2026-09-11
published: 2026-09-01
dependents: 17 owners · 757 repos

Featured: vortex [crate · repo · docs]

A columnar format built for the shapes Parquet handles badly — random access, wide multimodal rows, cascading compression chosen per chunk — and, unusually for this hub, one that treats Rust as the customer: the README leads with cargo add vortex, the file format has been stable since 0.36, and it publishes a formal MSRV policy. It was donated to LF AI & Data, so it is not one vendor's format. Moved from spiraldb/vortex.

Vortex file format with all builtin codecs and a sampling compressor.

docsVortex documentation and format specification productionSpice AI, whose Cayenne engine ships Vortex in production

version: 0.86.1
downloads: 145k / 90d
stars: 3.2k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-09-11
dependents: 12 owners

parquet [crate · repo · docs]

Apache Parquet implementation in Rust

version: 59.3.0
downloads: 16.1M / 90d
stars: 3.6k
license: Apache-2.0
last commit: 2026-09-11
published: 2026-09-01
dependents: 47 owners · 599 repos

iceberg [crate · repo · docs]

Apache Iceberg Rust implementation

version: 0.10.1
downloads: 1.1M / 90d
stars: 1.4k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-08-01
dependents: 24 owners

deltalake [crate · repo · docs]

Native Delta Lake implementation in Rust

version: 0.32.4
downloads: 452k / 90d
stars: 3.3k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-06-07
dependents: 23 owners · 41 repos

lance [crate · repo · docs]

A columnar data format that is 100x faster than Parquet for random access.

version: 11.0.0
downloads: 1.3M / 90d
stars: 7.1k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-08-30
dependents: 26 owners · 1 repo

hudi [crate · repo · docs]

The native Rust implementation for Apache Hudi

version: 0.5.0
downloads: 877 / 90d
stars: 279
license: Apache-2.0
last commit: 2026-09-10
published: 2026-09-10

New & experimental (1)

minarrow [crate · repo · docs]

Apache Arrow-compatible, Rust-first columnar data library for high-performance computing, native streaming, and embedded workloads. Minimal dependencies, ultra-low-latency access, automatic 64-byte SIMD alignment, and fast compile times. Great for real-time analytics, HPC pipelines, and systems integration.

version: 0.18.1
downloads: 1k / 90d
stars: 100
license: Apache-2.0
last commit: 2026-09-10
published: 2026-09-10
dependents: 2 owners

Archive (1)

arrow2 [crate · repo · docs]

Archived: repo archived. Detected automatically, and it un-archives itself if that changes.

Unofficial implementation of Apache Arrow spec in safe Rust

version: 0.18.0
downloads: 566k / 90d
stars: 1.1k
license: Apache-2.0
last commit: 2024-01-29
published: 2023-09-17
dependents: 28 owners · 495 repos

Datasets, formats & loading partial

Getting data into a model: file formats, array stores on object storage, and the Hugging Face Hub. The ordinary formats are covered; dataloaders and data versioning are much thinner.

Featured: hf-hub [crate · repo · docs]

How Rust reaches the Hugging Face Hub: resolve a repo and revision, download weights and tokenizers with caching, and get Xet-accelerated transfer for free because hf-xet is an unconditional dependency. It reached 1.0 in July 2026, and tokenizers, candle, mistral.rs, fastembed and lancedb all depend on it. If your program loads a model by name rather than by path, this is the crate doing it.

Rust client for the Hugging Face Hub API

productionLanceDB depends on hf-hub to resolve models by name productionmistral.rs depends on hf-hub across its workspace

version: 1.0.0
downloads: 7.7M / 90d
stars: 330
license: Apache-2.0
last commit: 2026-08-17
published: 2026-07-10
dependents: 36 owners · 1 repo

csv [crate · repo · docs]

Fast CSV parsing with support for serde.

version: 1.4.0
downloads: 47.2M / 90d
stars: 2k
license: Unlicense/MIT
last commit: 2026-08-04
published: 2025-10-17
dependents: 53 owners · 10109 repos

image [crate · repo · docs]

Imaging library. Provides basic image processing and encoders/decoders for common image formats.

version: 0.25.10
downloads: 48.9M / 90d
stars: 5.9k
license: MIT OR Apache-2.0
last commit: 2026-08-28
published: 2026-03-10
dependents: 43 owners · 15485 repos

npyz [crate · repo · docs]

NumPy file format (de-)serialization. Fork of outdated npy-rs.

version: 0.9.1
downloads: 1.7M / 90d
stars: 33
license: MIT
last commit: 2026-05-03
published: 2026-05-02
dependents: 27 owners · 172 repos

netcdf [crate · repo · docs]

High-level NetCDF bindings for Rust

version: 0.12.1
downloads: 42k / 90d
stars: 105
license: MIT OR Apache-2.0
last commit: 2026-08-12
published: 2026-07-26
dependents: 26 owners · 16 repos

zarrs [crate · repo · docs]

A library for the Zarr storage format for multidimensional arrays and metadata

version: 0.23.14
downloads: 72k / 90d
stars: 271
license: MIT OR Apache-2.0
last commit: 2026-09-13
published: 2026-08-14
dependents: 29 owners

hdf5-metno [crate · repo · docs]

The live HDF5 bindings: a fork of aldanor/hdf5-rust renamed so it can publish. Use this rather than the hdf5 crate.

version: 0.14.1
downloads: 847k / 90d
stars: 83
license: MIT OR Apache-2.0
last commit: 2026-09-07
published: 2026-08-12
dependents: 28 owners

ndarray-npy [crate · repo · docs]

.npy and .npz file format support for ndarray

version: 0.10.0
downloads: 1.9M / 90d
stars: 71
license: MIT OR Apache-2.0
last commit: 2025-12-15
published: 2025-12-15
dependents: 30 owners · 64 repos

burn-dataset [crate · repo · docs]

Library with simple dataset APIs for creating ML data pipelines

version: 0.22.0-pre.3
downloads: 214k / 90d
stars: 15.9k
license: MIT OR Apache-2.0
last commit: 2026-09-11
published: 2026-08-25
dependents: 3 owners · 19 repos

icechunk [crate · repo · docs]

Transactional storage engine for Zarr designed for use on cloud object storage

version: 2.2.0
downloads: 7k / 90d
stars: 675
license: Apache-2.0
last commit: 2026-09-10
published: 2026-09-02
dependents: 3 owners

probminhash [crate · repo · docs]

Minhash algorithms for weighted Jaccard index

version: 0.1.12
downloads: 389k / 90d
stars: 31
license: MIT OR Apache-2.0
last commit: 2026-01-22
published: 2025-06-11
dependents: 2 owners · 1 repo

xvc [crate · repo · docs]

An MLOps tool to manage data files and pipelines on top of Git

version: 0.7.1-alpha.3
downloads: 176 / 90d
stars: 77
license: GPL-3.0
last commit: 2026-08-14
published: 2026-01-24
dependents: 1 owner · 1 repo

liboxen [crate · repo · docs]

Oxen is a fast data version control system, built with machine learning training data in mind. Designed to handle terabytes of data with ease, using a workflow similar to git. Version both structured and unstructured data of any modality: text, images, video, audio, CSV, Parquet, JSONL, model checkpoints, and more. liboxen is the embeddable core library behind the oxen CLI and server, which power fine tuning and inference pipelines for multimodal LLMs, image models, and video models on Oxen.ai.

version: 0.56.1
downloads: 1k / 90d
stars: 1.2k
license: non-standard
last commit: 2026-09-11
published: 2026-09-02
dependents: 1 owner · 1 repo

New & experimental (2)

beowolx/rensa [repo]

High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datasets

stars: 364
license: MIT
last commit: 2026-09-06

ai-dataloader [crate · repo · docs]

Rust implementation to the PyTorch DataLoader

version: 0.6.2
downloads: 225 / 90d
stars: 31
license: MIT OR Apache-2.0
last commit: 2024-10-27
published: 2024-09-14

Archive (6)

hdf5 [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

The original HDF5 bindings — hdf5-metno is the fork that publishes current versions.

version: 0.8.1
downloads: 158k / 90d
stars: 347
license: MIT OR Apache-2.0
last commit: 2024-05-30
published: 2021-11-21
dependents: 24 owners · 42 repos

tfrecord [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

TFRecord de/serialize for TensorBoard

version: 0.15.0
downloads: 10k / 90d
stars: 25
license: non-standard
last commit: 2023-09-09
published: 2023-09-09
dependents: 2 owners · 2 repos

arff [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

ARFF file format serializer and deserializer

version: 0.3.0
downloads: 105 / 90d
stars: 3
license: MIT/Apache-2.0
last commit: 2019-07-26
published: 2018-06-20
dependents: 1 owner · 1 repo

cuticula [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

Data Preprocessing library for Machine Learning

version: 0.2.0
downloads: 38 / 90d
stars: 89
license: MIT OR Apache-2.0
last commit: 2016-04-17
published: 2016-04-18
dependents: 1 repo

rusty-data [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

A data handling library (designed for machine learning).

version: 0.0.3
downloads: 51 / 90d
stars: 4
license: MIT
last commit: 2016-01-27
published: 2016-01-27

openml [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

A rust interface to [OpenML](http://openml.org/).

version: 0.1.2
downloads: 12 / 90d
stars: 12
license: MIT/Apache-2.0
last commit: 2018-06-29
published: 2018-06-27

Pipelines & streaming partial

Stream processing engines and data pipelines. Several of the best-known streaming systems are written in Rust, but they are servers you deploy rather than crates you build with.

vectordotdev/vector [repo]

A high-performance observability data pipeline.

stars: 22.6k
license: MPL-2.0
last commit: 2026-09-11

MaterializeInc/materialize [repo]

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

stars: 6.4k
last commit: 2026-09-13

risingwavelabs/risingwave [repo]

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

stars: 9.3k
license: Apache-2.0
last commit: 2026-09-14

Eventual-Inc/Daft [repo]

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

stars: 5.8k
license: Apache-2.0
last commit: 2026-09-11

rdkafka [crate · repo · docs]

Rust wrapper for librdkafka

version: 0.39.0
downloads: 6.4M / 90d
stars: 2k
license: MIT
last commit: 2026-06-14
published: 2026-01-25
dependents: 41 owners · 457 repos

fluvio-community/fluvio [repo]

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

stars: 5.3k
license: Apache-2.0
last commit: 2026-04-28

cocoindex-io/cocoindex [repo]

Incremental engine for long horizon agents 🌟 Star if you like it!

stars: 11.5k
license: Apache-2.0
last commit: 2026-09-13

feldera/feldera [repo]

The Feldera Incremental Computation Engine

stars: 2.1k
license: MIT
last commit: 2026-09-14

ArroyoSystems/arroyo [repo]

Distributed stream processing engine in Rust

stars: 5k
license: Apache-2.0
last commit: 2026-09-09

lakehq/sail [repo]

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

stars: 3.4k
license: Apache-2.0
last commit: 2026-09-12

timely [crate · repo · docs]

A low-latency data-parallel dataflow system in Rust

version: 0.31.0
downloads: 78k / 90d
stars: 3.6k
license: MIT
last commit: 2026-09-12
published: 2026-07-14
dependents: 15 owners · 33 repos

differential-dataflow [crate · repo · docs]

An incremental data-parallel dataflow platform

version: 0.25.1
downloads: 68k / 90d
stars: 3k
license: MIT
last commit: 2026-07-29
published: 2026-07-15
dependents: 11 owners · 23 repos

iggy [crate · repo · docs]

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.

version: 0.11.0-edge.8
downloads: 79k / 90d
stars: 4.9k
license: Apache-2.0
last commit: 2026-09-14
published: 2026-09-11
dependents: 10 owners

arkflow-rs/arkflow [repo]

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.

stars: 1.3k
license: Apache-2.0
last commit: 2026-08-04

Archive (1)

rettle [crate · repo · docs]

Archived: dormant. Detected automatically, and it un-archives itself if that changes.

A multithreaded ETL with inspiration drawn from Keras.

version: 2.0.0
downloads: 279 / 90d
stars: 25
license: MIT OR Apache-2.0
last commit: 2020-01-08
published: 2020-01-08
dependents: 1 owner