Rust is a reasonable place to run an open-weights model, and increasingly the language the thing running it is written in: NVIDIA's Dynamo and Cloudflare's Infire are Rust-cored, and the constrained-decoding engine several Python and C++ servers embed is a Rust crate. That is not a recommendation to do your AI work in Rust — the highest-scoring reply in the standing thread on why Rust is uncommon in AI says the opposite1.

Calling a model you did not host works for every major provider; most of those crates are community work rather than published by the vendor whose API they wrap.

To build an application on top of a model, see Agents, Tools & Context.

  1. users.rust-lang.org, 2025-07-24 — the thread's highest-scoring reply