ONNX Runtime & Edge AI

Cloud inference is convenient, but it introduces latency, network dependency, recurring costs, and privacy concerns. ONNX Runtime flips that equation: it takes models trained in any major framework and runs them directly on the device where the data lives — a laptop, a phone, a browser, an IoT sensor, or a GPU workstation. The Open Neural Network Exchange (ONNX) format is the universal adapter that makes this possible. This guide covers the full pipeline from model conversion through optimization to cross-platform edge deployment.

We will work through ONNX model export from PyTorch, TensorFlow, and scikit-learn; run inference in Python, C#, and JavaScript; apply quantization and graph optimization to shrink models by 4x; deploy to Windows, Android, iOS, and the browser; and configure hardware-specific execution providers for CUDA, TensorRT, DirectML, CoreML, and more.

The ONNX ecosystem at a glance

ONNX Runtime sits at the center of a pipeline that decouples where you train from where you run. Models flow from training frameworks through the ONNX format into a single high-performance runtime that targets every major hardware backend. The following diagram shows how these layers connect:

The key insight is separation of concerns: data scientists train in whatever framework they prefer, export once to ONNX, and the deployment team picks the right execution provider for each target without rewriting any inference code.

What is ONNX and why it matters

ONNX (Open Neural Network Exchange) is an open-source format for representing machine learning models. Originally co-developed by Microsoft and Meta, it defines a common set of operators and a standard file format that any framework can export to and any runtime can consume. Think of it as the PDF of machine learning: write once in any tool, read everywhere.

An ONNX file (.onnx) is a serialized protobuf that contains three things:

• Graph definition — a directed acyclic graph of computational nodes (Conv, MatMul, Relu, Softmax, etc.)
• Operator set version — the specific opset (currently v21) that determines the semantics of each operator
• Weights and metadata — trained parameters, tensor shapes, data types, and optional model metadata

Converting models to ONNX

Every major ML framework provides a path to ONNX. The conversion step captures the computational graph and weights into a framework-independent representation. Below are production-ready examples for the three most common sources.

PyTorch to ONNX

PyTorch’s torch.onnx.export traces the model with sample input and serializes the resulting graph. The dynamo_export API (PyTorch 2.1+) uses TorchDynamo for more reliable capture of dynamic control flow.

import torch
import torch.onnx
from torchvision import models

# Load a pretrained ResNet-50
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.eval()

# Create dummy input matching expected shape
dummy_input = torch.randn(1, 3, 224, 224)

# Export to ONNX with dynamic batch size
torch.onnx.export(
    model,
    dummy_input,
    "resnet50.onnx",
    opset_version=17,
    input_names=["image"],
    output_names=["logits"],
    dynamic_axes={
        "image":  {0: "batch_size"},
        "logits": {0: "batch_size"}
    }
)

# Verify the exported model
import onnx
onnx_model = onnx.load("resnet50.onnx")
onnx.checker.check_model(onnx_model)
print("ONNX model validated successfully")

TensorFlow / Keras to ONNX

The tf2onnx converter handles TensorFlow SavedModel and Keras .h5 formats. It maps TF ops to ONNX ops and folds constants to produce a cleaner graph.

# Install the converter
pip install tf2onnx

# Convert a SavedModel directory
python -m tf2onnx.convert \
  --saved-model ./saved_model_dir \
  --output model.onnx \
  --opset 17

# Convert a Keras .h5 file
python -m tf2onnx.convert \
  --keras ./model.h5 \
  --output model.onnx \
  --opset 17

scikit-learn to ONNX

The skl2onnx library converts traditional ML models (classifiers, regressors, pipelines) into ONNX. This is valuable for deploying sklearn models in environments without Python.

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

# Train a simple model
X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=100)
clf.fit(X, y)

# Define input schema: 4 float features
initial_type = [("features", FloatTensorType([None, 4]))]

# Convert to ONNX
onnx_model = convert_sklearn(
    clf,
    initial_types=initial_type,
    target_opset=17
)

# Save the model
with open("rf_iris.onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())

ONNX Runtime inference

Once you have an .onnx file, ONNX Runtime (ORT) runs it. The same model file works across Python, C#, C++, Java, JavaScript, Objective-C, and Swift. ORT automatically applies graph optimizations at session creation and routes operators to the best available execution provider.

Python inference

import onnxruntime as ort
import numpy as np
from PIL import Image
from torchvision import transforms

# Create an inference session
session = ort.InferenceSession(
    "resnet50.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)

# Preprocess an image
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])
img = Image.open("photo.jpg")
input_tensor = transform(img).unsqueeze(0).numpy()

# Run inference
input_name = session.get_inputs()[0].name
results = session.run(None, {input_name: input_tensor})

# Get top-5 predictions
logits = results[0][0]
top5 = np.argsort(logits)[-5:][::-1]
print("Top-5 class indices:", top5)

C# inference (.NET)

The Microsoft.ML.OnnxRuntime NuGet package provides the same capabilities for .NET applications. This is especially useful for integrating ML into existing enterprise C# services.

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

// Create session with GPU acceleration
var sessionOptions = new SessionOptions();
sessionOptions.AppendExecutionProvider_DML(); // DirectML for Windows GPU

using var session = new InferenceSession("resnet50.onnx", sessionOptions);

// Prepare input tensor (1 x 3 x 224 x 224)
var inputTensor = new DenseTensor<float>(
    new[] { 1, 3, 224, 224 }
);
// ... populate tensor with preprocessed image data ...

// Run inference
var inputs = new List<NamedOnnxValue>
{
    NamedOnnxValue.CreateFromTensor("image", inputTensor)
};

using var results = session.Run(inputs);
var output = results.First().AsTensor<float>();

// Get predicted class
int predictedClass = output
    .ToArray()
    .Select((val, idx) => (val, idx))
    .OrderByDescending(x => x.val)
    .First().idx;

Console.WriteLine($"Predicted class: {predictedClass}");

JavaScript inference (ONNX Runtime Web)

ONNX Runtime Web runs models directly in the browser using WebAssembly or WebGPU as the backend. No server round-trip required.

import * as ort from 'onnxruntime-web';

// Configure WebGPU backend (falls back to WASM)
ort.env.wasm.numThreads = 4;

async function runInference() {
  // Load the ONNX model
  const session = await ort.InferenceSession.create(
    './model.onnx',
    { executionProviders: ['webgpu', 'wasm'] }
  );

  // Create input tensor from Float32Array
  const inputData = new Float32Array(1 * 3 * 224 * 224);
  // ... fill with preprocessed pixel data ...

  const feeds = {
    image: new ort.Tensor('float32', inputData, [1, 3, 224, 224])
  };

  // Run inference
  const results = await session.run(feeds);
  const output = results.logits.data;

  console.log('Prediction complete', output);
}

runInference();

Model optimization

Raw ONNX models exported from training frameworks carry overhead: redundant operations, full 32-bit precision, and unoptimized graph structure. ONNX Runtime provides tools to strip that overhead without sacrificing meaningful accuracy.

Graph optimization

ORT automatically applies graph-level optimizations when creating a session. You can control the optimization level and save the optimized model for reuse:

import onnxruntime as ort

# Configure session with maximum graph optimization
options = ort.SessionOptions()
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
options.optimized_model_filepath = "resnet50_optimized.onnx"

# Creating the session triggers optimization and saves result
session = ort.InferenceSession(
    "resnet50.onnx",
    sess_options=options,
    providers=["CPUExecutionProvider"]
)

# Optimizations applied:
# - Constant folding (precompute static subgraphs)
# - Operator fusion (Conv+BN+Relu -> single fused op)
# - Redundant node elimination
# - Layout transformation (NCHW -> NHWC for CPU)

Quantization: INT8 and INT4

Quantization reduces model size and accelerates inference by converting weights and activations from 32-bit floating point to lower-precision integers. The onnxruntime.quantization module supports static (calibrated), dynamic, and GPTQ/AWQ quantization.

from onnxruntime.quantization import (
    quantize_dynamic,
    quantize_static,
    QuantType,
    CalibrationDataReader
)

# --- Dynamic quantization (no calibration data needed) ---
quantize_dynamic(
    model_input="resnet50.onnx",
    model_output="resnet50_int8_dynamic.onnx",
    weight_type=QuantType.QInt8
)

# --- Static quantization (requires representative dataset) ---
class ImageCalibrationReader(CalibrationDataReader):
    def __init__(self, calibration_images):
        self.data = iter(calibration_images)

    def get_next(self):
        try:
            return {"image": next(self.data)}
        except StopIteration:
            return None

calibration_reader = ImageCalibrationReader(calibration_set)

quantize_static(
    model_input="resnet50.onnx",
    model_output="resnet50_int8_static.onnx",
    calibration_data_reader=calibration_reader,
    quant_format=QuantFormat.QDQ,  # Quantize-Dequantize nodes
    weight_type=QuantType.QInt8,
    activation_type=QuantType.QInt8
)

INT4 quantization for LLMs

Large language models benefit enormously from 4-bit quantization. The ONNX Runtime GenAI toolchain integrates GPTQ and AWQ methods to compress multi-billion-parameter models to a fraction of their original size.

from onnxruntime.quantization import matmul_4bits_quantizer

# Quantize a large model to INT4 (block-wise)
quantizer = matmul_4bits_quantizer.MatMul4BitsQuantizer(
    model=onnx_model,
    block_size=32,            # 32-element blocks
    is_symmetric=True,
    accuracy_level=4          # Highest accuracy mode
)
quantizer.process()
quantizer.model.save_model_to_file("phi3_int4.onnx")

ONNX Runtime for LLMs (GenAI)

The ONNX Runtime GenAI library adds autoregressive text generation on top of standard ORT inference. It handles the KV-cache, sampling strategies (greedy, beam search, top-k, top-p), and tokenization so you can run models like Phi-3, Llama 3, and Mistral entirely on-device.

import onnxruntime_genai as og

# Load a quantized Phi-3-mini model
model = og.Model("models/phi-3-mini-int4-onnx")
tokenizer = og.Tokenizer(model)

# Configure generation parameters
params = og.GeneratorParams(model)
params.set_search_options(
    max_length=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

# Tokenize the prompt
prompt = "Explain edge AI in three sentences."
input_tokens = tokenizer.encode(prompt)
params.input_ids = input_tokens

# Generate token by token (streaming)
generator = og.Generator(model, params)
output_tokens = []

while not generator.is_done():
    generator.compute_logits()
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    output_tokens.append(new_token)
    print(tokenizer.decode(new_token), end="", flush=True)

print()  # newline after streaming output

Running Phi-3 with C# GenAI

using Microsoft.ML.OnnxRuntimeGenAI;

// Load the ONNX model and tokenizer
using var model = new Model("models/phi-3-mini-int4-onnx");
using var tokenizer = new Tokenizer(model);

var prompt = "What are the benefits of on-device inference?";
var sequences = tokenizer.Encode(prompt);

using var genParams = new GeneratorParams(model);
genParams.SetSearchOption("max_length", 256);
genParams.SetSearchOption("temperature", 0.7);
genParams.SetInputSequences(sequences);

// Stream generated tokens
using var tokenizerStream = tokenizer.CreateStream();
using var generator = new Generator(model, genParams);

while (!generator.IsDone())
{
    generator.ComputeLogits();
    generator.GenerateNextToken();
    Console.Write(tokenizerStream.Decode(
        generator.GetSequence(0)[^1]
    ));
}

Platform-specific deployment

ONNX Runtime ships pre-built packages for every major platform. Each deployment target has its own SDK, but they all consume the same .onnx file. The differences lie in which execution providers are available and how you package the model with your app.

Windows: DirectML & Windows ML

On Windows, DirectML is the recommended GPU execution provider. It runs on any DirectX 12-compatible GPU (NVIDIA, AMD, Intel, Qualcomm) without vendor-specific drivers. Windows ML provides a higher-level WinRT API that integrates directly with UWP and WinUI apps.

// NuGet: Microsoft.ML.OnnxRuntime.DirectML
var options = new SessionOptions();
options.AppendExecutionProvider_DML(deviceId: 0);
options.EnableMemoryPattern = true;
options.EnableCpuMemArena = true;

using var session = new InferenceSession("model.onnx", options);

// For NPU acceleration on Snapdragon X / Copilot+ PCs:
// options.AppendExecutionProvider("QNN");

Mobile: Android and iOS

The ONNX Runtime Mobile package strips unused operators to reduce binary size. On Android it uses NNAPI for hardware acceleration; on iOS it delegates to CoreML.

// Android (Kotlin) — add onnxruntime-android to build.gradle
val env = OrtEnvironment.getEnvironment()
val options = OrtSession.SessionOptions()
options.addNnapi()  // Enable Android NNAPI acceleration

val session = env.createSession(
    modelBytes,  // loaded from assets
    options
)

// Prepare input
val inputTensor = OnnxTensor.createTensor(
    env,
    FloatBuffer.wrap(inputData),
    longArrayOf(1, 3, 224, 224)
)

// Run
val results = session.run(
    mapOf("image" to inputTensor)
)
val output = (results[0].value as Array<FloatArray>)[0]

Web: WebAssembly & WebGPU

ONNX Runtime Web compiles the runtime to WebAssembly with optional WebGPU acceleration. Models run entirely in the browser with zero server dependencies.

# Install the npm package
npm install onnxruntime-web

# For WebGPU support (experimental)
npm install onnxruntime-web@latest

Hardware acceleration: execution providers

Execution providers (EPs) are ONNX Runtime’s abstraction for hardware backends. You specify a priority list; ORT routes each operator to the highest-priority EP that supports it and falls back down the list for anything unsupported.

import onnxruntime as ort

# List available providers on this machine
print("Available:", ort.get_available_providers())

# Configure a priority chain: TensorRT > CUDA > CPU
session = ort.InferenceSession(
    "model.onnx",
    providers=[
        ("TensorrtExecutionProvider", {
            "trt_max_workspace_size": 2147483648,  # 2 GB
            "trt_fp16_enable": True,
            "trt_engine_cache_enable": True,
            "trt_engine_cache_path": "./trt_cache"
        }),
        ("CUDAExecutionProvider", {
            "device_id": 0,
            "arena_extend_strategy": "kSameAsRequested",
            "cudnn_conv_algo_search": "EXHAUSTIVE"
        }),
        "CPUExecutionProvider"
    ]
)

# Verify which provider is handling each node
for node in session.get_providers():
    print("Active provider:", node)

ONNX Runtime capabilities

Runtime comparison

ONNX Runtime is not the only inference engine. Here is how it compares against alternatives on key dimensions:

Performance benchmarks

The following benchmarks illustrate the impact of ONNX Runtime optimizations on common model architectures. All measurements use batch size 1 on representative hardware.

End-to-end optimization pipeline

The following script demonstrates the full workflow: load a model, apply graph optimization, quantize to INT8, and benchmark the result. This is the pattern you would use in a CI/CD pipeline to produce deployment-ready models.

import onnxruntime as ort
import numpy as np
import time
from onnxruntime.quantization import quantize_dynamic, QuantType

# Step 1: Graph optimization
opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
opts.optimized_model_filepath = "model_optimized.onnx"
_ = ort.InferenceSession("model.onnx", sess_options=opts)

# Step 2: Dynamic INT8 quantization
quantize_dynamic(
    model_input="model_optimized.onnx",
    model_output="model_int8.onnx",
    weight_type=QuantType.QInt8
)

# Step 3: Benchmark
def benchmark(model_path, runs=100):
    session = ort.InferenceSession(model_path)
    input_name = session.get_inputs()[0].name
    input_shape = session.get_inputs()[0].shape
    # Replace dynamic dims with 1
    shape = [s if isinstance(s, int) else 1 for s in input_shape]
    dummy = np.random.randn(*shape).astype(np.float32)

    # Warmup
    for _ in range(10):
        session.run(None, {input_name: dummy})

    # Timed runs
    start = time.perf_counter()
    for _ in range(runs):
        session.run(None, {input_name: dummy})
    elapsed = (time.perf_counter() - start) / runs
    return elapsed * 1000  # ms

original_ms = benchmark("model.onnx")
optimized_ms = benchmark("model_int8.onnx")

print(f"Original:  {original_ms:.1f} ms")
print(f"Optimized: {optimized_ms:.1f} ms")
print(f"Speedup:   {original_ms / optimized_ms:.1f}x")

Next steps

1. Export one of your existing models to ONNX and verify it with onnx.checker. Start with the framework you already use.
2. Run it through ONNX Runtime with graph optimization enabled. Measure the baseline latency improvement over native framework inference.
3. Apply dynamic INT8 quantization and compare accuracy on your validation set. If the drop is unacceptable, switch to static quantization with a calibration dataset.
4. Choose your target platform and install the matching ORT package (DirectML for Windows GPU, NNAPI for Android, CoreML for iOS, WASM for browser).
5. Try ONNX Runtime GenAI with a quantized Phi-3 or Llama model to add local LLM capabilities without cloud dependencies.
6. Integrate into your CI/CD pipeline by automating the optimization and benchmarking steps. Use the benchmarking script above as a starting point for regression testing.