A Gentle Introduction to Agile Robots and Idealworks Edge AI Stack for maximum compatibility.
/ 7 min read1. Hardware
In 2018, Idealworks chose the NVIDIA Jetson AGX Xavier as the main compute platform for its robots. At the time, this was a forward-looking decision; it later scaled to production deployments across multiple sites.
In 2026, the question is whether older hardware can still contribute to modern AI workloads. Our answer is yes: by supercharging existing devices with software, we reduce hardware churn while extending the useful life of deployed platforms.
While extending the life of existing Xavier deployments remains a key focus, we are also preparing for next-generation platforms such as NVIDIA Jetson AGX Thor. Maintaining a common software foundation across hardware generations allows us to reuse deployment pipelines while enabling increasingly capable edge AI workloads.
2. BSP and Custom OS
Yocto-based custom OS using the community-maintained meta-tegra BSP repository reduces overhead, improves memory usage, and strengthens security. As of July 2026, NVIDIA’s official support for meta-tegra in JetPack 7.2 accelerated integration.
3. Inferencing Pipeline
The deployment stack is optimized for performance and resource efficiency. The deployment stack uses modern build and packaging practices designed for reproducibility and efficient edge deployment.
The group also contributes open-source Bazel modules through https://github.com/idealworks/bazel-public-registry.
The inference workflow has two phases:
- Prototyping: Developers iterate quickly in Python, using TensorRT for CUDA-level optimizations. On Xavier, TensorRT 8.5.2 had to run alongside ROS2 Jazzy and Python 3.12, so the team built a custom TensorRT version from open source for tighter integration: https://github.com/nvidia/tensorrt.
- Productionization: Final inference products are implemented in C++ for performance, memory control, and native CUDA kernel support. Runtime libraries are fetched, linked, packaged, and deployed through Bazel.
4. Inferencing Framework
TensorRT, together with Isaac ROS NITROS, is the preferred inference framework for high-performance robotics workloads. The key challenge is supporting modern architectures, such as Vision Transformers, on older Xavier hardware. These same optimization principles remain relevant on newer platforms. While devices such as Jetson AGX Thor provide significantly more compute, efficient inference through TensorRT, quantization, graph optimization, and model-aware acceleration continue to be essential for real-time robotics applications.
Rather than abandoning TensorRT performance or rewriting everything as custom operations, the team combined TensorRT acceleration with targeted plugins for unsupported graph operations.
1. The Problem
VLMs on Jetson AGX Xavier can suffer from high latency and low throughput, often below 10 tokens per second. To make older compute useful, the team explored splitting model layers: keep what Xavier can handle locally and offload bottlenecks, starting with vision encoders, to newer inference platforms.
During the PyTorch → ONNX → TensorRT conversion path, the pipeline failed because TensorRT 8.5.2, the maximum supported version in our Xavier setup, did not recognize the ONNX LayerNormalization operation.
2. The Solution
The solution was to keep TensorRT’s optimized operators and add a custom nvinfer plugin for LayerNormalization. For simplicity, the example builds the plugin with CMake and CUDA while preserving TensorRT’s tactic selection for supported operations.
1. Choosing the Plugin interface version
For TensorRT 8.5.2 on Xavier, the plugin extends IPluginV2 through IPluginV2DynamicExt. Newer TensorRT releases move toward IPluginV3, but Xavier compatibility requires the older interface. Reference: https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/plugins-api-migration.html.
2. Identifying the problematic operation in the ONNX Graph
Netron was used to inspect the ONNX graph, locate unsupported operations, and capture the required inputs, outputs, attributes, shapes, and data types: https://github.com/lutzroeder/netron.
3. Implementing our custom plugin class
The plugin implementation extends two main classes:
- IPluginV2DynamicExt: Defines the plugin behavior, including tensor shapes, supported formats, serialization, and the enqueue() method that launches the CUDA kernel. It supports dynamic shapes and broadcast behavior, which are useful for vision workloads.
- IPluginCreator: Registers the plugin with TensorRT and recreates it when loading a serialized engine. It exposes metadata such as name and version, implements createPlugin(), and connects the creator to TensorRT’s plugin registry.
The C++ wrapper connects TensorRT to the CUDA implementation. The enqueue() method launches the LayerNormalization kernel, which performs the mathematical operation directly on the GPU.
The CUDA kernel implements LayerNormalization, and the plugin library is compiled with nvcc through CMake.
Once compiled, the plugin is passed to TensorRT during conversion, enabling normal tactic selection for supported layers and fallback to the custom plugin for LayerNormalization.
4. TensorRT conversion and runtime
The engine exports and runs successfully by passing the compiled plugin library, for example: --plugin=libsiglip2_layernorm_plugin.so.
At runtime, the plugin creator must be explicitly loaded. In Python, the .so file can be loaded with ctypes; no pybind wrapper is required.
5. Deployment Optimization
Hardware upgrades alone are rarely enough to achieve production-grade performance. Modern generative AI workloads can often be accelerated through inference-aware optimizations that reduce redundant computation. For example, cache-assisted approaches such as MagCache can significantly reduce denoising iterations in diffusion models while maintaining output quality. Combined with TensorRT and modern edge hardware, these techniques help bring advanced generative AI closer to real-time robotic applications.
With the transition to the Jetson AGX Thor (128 GB LPDDR5X, up to 2,070 FP4 TFLOPS), significantly greater compute headroom opens up for resource-intensive Physical AI models: optimizations such as MagCache reduce denoising steps by 50% with no observable degradation in quality, enabling tasks to be reliably completed in under three minutes (2:37 min) with an average full-chunk denoising time of 1.5 seconds.
6. Operators Benchmarking
Operator performance was profiled with NVIDIA Nsight Systems on standalone kernels: https://docs.nvidia.com/nsight-systems/UserGuide/index.html. CUDA kernel guidance is available at https://docs.nvidia.com/cuda/cuda-programming-guide/index.html.
The evolution of edge AI is not only about deploying faster hardware. Equally important is continuously improving the software stack to seamlessly integrate state-of-the-art third-party models. At Agile Robots and Idealworks, our approach centers on this adoption: extending the capabilities of existing platforms to accommodate these external models while preparing for next-generation systems through efficient deployment pipelines, optimized inference frameworks, and modern AI acceleration techniques tailored for third-party AI integration.
About the Authors
Jad Yammine – Senior AI Engineer, Idealworks
Laurenz von Czettritz – Team Lead Product Management AI, Agile Robots
Anthony Rizk – Co-founder & Research and Engineering Manager, AI, Robotics & Simulation, Idealworks