GPU Inference Example - Onnx Typing CST Test
Loading…
GPU Inference Example — Onnx Code
Running ONNX model inference on GPU using ONNX Runtime with CUDA provider.
import onnxruntime as ort
import numpy as np
# Load model on GPU
session = ort.InferenceSession('model.onnx', providers=['CUDAExecutionProvider'])
input_name = session.get_inputs()[0].name
input_data = np.random.rand(1,4).astype(np.float32)
# Run inference
outputs = session.run(None, {input_name: input_data})
print('GPU inference output:', outputs)Onnx Language Guide
ONNX (Open Neural Network Exchange) is an open-source format and ecosystem for representing machine learning models, enabling interoperability between frameworks like PyTorch, TensorFlow, and scikit-learn, and allowing deployment across diverse platforms.
Primary Use Cases
- ▸Exporting models from PyTorch, TensorFlow, or other frameworks
- ▸Cross-framework deployment without retraining
- ▸Hardware-accelerated inference on CPUs, GPUs, and specialized accelerators
- ▸Optimizing models with ONNX Runtime for production
- ▸Edge AI and mobile deployment of ML models
Notable Features
- ▸Framework-agnostic model format
- ▸Supports both deep learning and classical ML operators
- ▸ONNX Runtime for high-performance inference
- ▸Quantization and optimization tools for deployment
- ▸Extensible operator set for custom layers
Origin & Creator
ONNX was co-developed by Microsoft and Facebook in 2017 to unify model representation and interoperability between deep learning frameworks.
Industrial Note
ONNX is widely used in production pipelines where models need to be transferred between frameworks, optimized for inference, or deployed on resource-constrained devices like mobile phones or edge servers.