feature request: edge device CUDA backend

[mindbeast]

Does it make sense for executorch to have a mobile cuda backend? There are many edge devices in the Jetson lineup from nvidia that have a cuda gpu, but can benefit from not wanting to link an enormous libtorch dependence.

[bionictoucan]

+1 to this, would be nice to get comparable performance to TensorRT without having to export models to ONNX etc. first!

[hietalajulius]

+1

[mergennachin]

@mindbeast @bionictoucan @hietalajulius

Hi, thanks for the comment.

Yes, that makes sense in general.

Right now, for ExecuTorch, we are integrating Vulkan into ExecuTorch. The reason is that it is a suitable solution for mobile GPUs. Enabling mobile use-cases is our primary goal at the moment.

We will revisit Cuda, but perhaps, in the second half in the year. Curious, what are your current product needs?

[DzAvril]

Apologies for opening a similar feature request in #5263.

Curious, what are your current product needs?

@mergennachin We want to deploy LLMs in cars, but Python-based inference frameworks like vLLM and SGLang are not suitable for edge devices.

We will revisit Cuda, but perhaps, in the second half in the year.

Nearly five months have passed, is there any progress on this?

[digantdesai]

Thank you for following up @DzAvril.

We want to deploy LLMs in cars, but Python-based inference frameworks like vLLM and SGLang are not suitable for edge devices.

I guess this is using a platform similar to Jetson?

Nearly five months have passed, is there any progress on this?

No update yet on CUDA backend for ET at the moment. We can get back to you here once we plan something.

[DzAvril]

I guess this is using a platform similar to Jetson?

@digantdesai Yes, Jetson Orin for now, and possibly Thor in the future.

Looking forward to your update.

[DuinoDu]

For mobile cuda backend, does torch_tensorrt satisfy the requirement?

[mindbeast]

@DuinoDu My expectation is that compatibility is poor with torch_tensorrt. I expect a more compliant backend in executorch would help a lot of developers.

[yrik]

Any update on this? What is the best alternative so far to run CUDA on jetson? Directly Torch? Onnx?

[shoumikhin]

@Gasoonjia may like to update this discussion?

[larryliu0820]

We have a WIP cuda backend, backend by AOTInductor: https://docs.pytorch.org/docs/stable/torch.compiler_aot_inductor.html. We have enabled some popular models (whisper, voxtral, gemma3 etc), please checkout this README.md to give whisper a try! You can also find voxtral instructions here: https://github.com/pytorch/executorch/tree/main/examples/models/voxtral#readme, gemma3 instructions here: https://github.com/pytorch/executorch/blob/main/examples/models/gemma3/README.md

[mindbeast]

You should actually link to the backend:

executorch/backends/cuda/cuda_backend.py

Line 31 in 6014129

cuda_decomposition_table = {

Why is executorch involved at all if you are just using the AOTInductor backend from torch? Any overview of design here?

I don't know what whisper models are and don't see how that is related to the CUDA backend.