Integrate TrixiCUDA.jl with Enzyme.jl for Differentiable Programming

[junyixu]

I propose integrating the GPU version of Trixi.jl with Enzyme.jl for differentiable programming.

Benefits:

• Differentiable Programming: Allows for gradient-based optimizations in CFD simulations.
• Unified Workflow: Provides a seamless experience for differentiable programming.

Note: Jacobian matrices computed on CPU and GPU may differ due to:

• Precision Differences: GPUs often use lower precision (e.g., FP32) compared to CPUs (e.g., FP64), affecting numerical accuracy.
• Parallelism Effects: GPUs handle parallel computations differently, potentially introducing slight numerical differences.

[huiyuxie]

Thanks for your advice @junyixu! You mentioned about Jacobian matrices - are they the only parts that could benefit from auto differentiation on GPU?

[junyixu]

Some potential benefits: When computing large-scale Jacobians, we avoid data transfer overhead between CPU and GPU. In DG methods, with 100k elements, that matrix gets too big to store on CPU. If we compute derivatives directly on GPU, we can use matrix-free methods and compute them on the fly when needed.

This is especially useful for optimization problems - when studying how inlet conditions affect shock positions, or doing mesh adaptation where gradients are needed frequently. Computing directly on GPU is much faster than shuffling data back and forth.

Though it is trickier to implement than on CPU, and requires more careful memory management. Some existing CPU code might need rewriting.

[huiyuxie]

How does this issue relate to your work here https://github.com/junyixu/TrixiEnzyme.jl?

[junyixu]

Enzyme.jl allows direct AD on GPU, avoiding CPU-GPU data transfer.