Rebuilding FUN3D with AOCC + AOCL to unlock GCP H4D

FUN3D is NASA Langley's unstructured-grid CFD suite — a node-centered, finite-volume compressible/incompressible RANS solver used across aerospace for everything from launch-vehicle ascent to rotorcraft. It's mostly modern Fortran with a C layer, parallelized with MPI and partitioned with ParMETIS. It's also export-controlled (ITAR): you need authorization to obtain and run it, and nothing in this post reproduces any of its source. This is purely a build-and-tuning methodology.

A customer had migrated their FUN3D campaigns to GCP H4D — the HPC-optimized VM family built on 5th-gen AMD EPYC (“Turin”, Zen 5) with 12-channel DDR5 and Cloud RDMA over the Titanium offload. They were running the distro-default build: gfortran/gcc at -O2, reference BLAS/LAPACK, stock libm. It worked. It was also leaving 30–40% on the table. Here's what we did about it.

Why the stock build underperforms on EPYC

FUN3D, like most unstructured CFD, is dominated by two things: sparse linear-algebra kernels (a point-implicit / GMRES solve per timestep) and irregular gather/scatter over the mesh. That makes it heavily memory-bandwidth bound with poor cache locality. Three defaults hurt on Turin:

• Generic codegen. A -O2, -mtune=generic build doesn't schedule for Zen 5, doesn't reliably emit FMAs, and under-uses the 256-bit datapaths that matter for the flux and gradient loops.
• Reference math. Netlib BLAS/LAPACK and stock libm are correctness references, not performance libraries. The transcendental-heavy thermodynamic and turbulence routines pay for that on every cell.
• No memory/NUMA discipline. A bandwidth-bound code with ranks floating across NUMA domains and 4 KiB pages spends its life waiting on remote memory and TLB misses.

The toolchain swap: AOCC + AOCL

The core move is to rebuild with AMD's tuned toolchain:

• AOCC — AMD's LLVM-based compiler (Clang for C/C++, Flang for Fortran), which knows how to schedule and vectorize for znver5.
• AOCL — AMD Optimizing CPU Libraries: BLIS (BLAS), libFLAME (LAPACK), AOCL-LibM (vector math), and AOCL-FFTW. These replace the reference kernels the solver leans on.

A word of realism on Fortran: FUN3D exercises a lot of modern-Fortran surface area, and the Flang frontend has historically been the fragile part of an AMD build. Budget time for it. In practice we built the C/C++ components with AOCC Clang, compiled the Fortran with Flang where it was clean, and kept a gfortran fallback for the handful of modules that tripped the frontend — all linked against AOCL either way. The math-library swap delivers most of the win regardless of which Fortran frontend compiles a given file.

Rebuild the binaries from source

The whole rebuild is driven from the environment: make the MPI compiler wrappers call AOCC underneath, hand FUN3D's configure the AOCL BLAS/LAPACK, and the resulting nodet_mpi comes out linked against the AMD math stack.

# 0. Obtain FUN3D (ITAR: requires an approved NASA software usage agreement)
tar xzf fun3d-14.x.tar.gz && cd fun3d-14.x

# 1. Load AMD compiler + math library, and make mpicc/mpif90 use AOCC
module load aocc/5.0.0 aocl/5.0.0 openmpi/5.0-aocc
export AOCL_ROOT=$AOCL_HOME
export OMPI_CC=clang OMPI_CXX=clang++ OMPI_FC=flang   # wrappers now call AOCC
export CC=mpicc CXX=mpicxx FC=mpif90

# 2. Zen 5 (Turin) codegen + safe FMA contraction. znver4 on Genoa-class nodes.
#    NOTE: no -Ofast / -ffast-math on a CFD solver (see the warning below).
ARCH="-O3 -march=znver5 -mtune=znver5 -funroll-loops -ffp-contract=fast"
export CFLAGS="$ARCH" CXXFLAGS="$ARCH" FCFLAGS="$ARCH"

# 3. Configure FUN3D against the AMD math library (BLIS + libFLAME) instead
#    of reference BLAS/LAPACK. This is the "matlib" swap that does the heavy lifting.
./configure \
  --prefix=$PWD/install \
  --with-blas="-L$AOCL_ROOT/lib -lblis" \
  --with-lapack="-L$AOCL_ROOT/lib -lflame" \
  --enable-optimized

# 4. Build and install the solver binaries
make -j"$(nproc)" 2>&1 | tee build.log
make install

Do not reach for -Ofast or -ffast-math on a CFD solver. They imply reciprocal math, reassociation, and finite-math assumptions that change residual convergence, break conservation, and can silently move your answer. We use -ffp-contract=fast for FMAs (well-behaved) and stop there — a faster path to the same validated solution, not a different one.

Confirm the binary actually picked up the AMD math library before you trust any timing — ldd is the one-second sanity check:

$ ldd install/bin/nodet_mpi | grep -Ei 'blis|flame|alm|amdlibm'
        libblis.so.4    => /opt/aocl/5.0.0/lib/libblis.so.4
        libflame.so.5   => /opt/aocl/5.0.0/lib/libflame.so.5
        libalm.so       => /opt/aocl/5.0.0/lib/libalm.so      # AOCL-LibM

Keep the stock build around too — copy it aside as nodet_mpi.gcc and the tuned one as nodet_mpi.aocc — so the A/B tests below run the exact same case through both. For pure-MPI runs link single-threaded -lblis and let MPI own the cores; only use -lblis-mt for hybrid MPI+OpenMP, or you'll oversubscribe.

Quick tests that prove it

Before spending node-hours on a full case, three cheap CLI checks confirm each layer of the win: the BLAS kernels, the vector math, and the actual solve.

1. DGEMM: reference BLAS vs AOCL BLIS

Same benchmark binary, swapped underneath with LD_PRELOAD — isolates the math-library effect with zero recompilation:

# single-threaded, 4096^3 double-precision GEMM
$ OMP_NUM_THREADS=1 ./dgemm_bench 4096
  reference BLAS : 4096  ->   38.7 GFLOP/s

$ OMP_NUM_THREADS=1 LD_PRELOAD=$AOCL_ROOT/lib/libblis.so ./dgemm_bench 4096
  AOCL BLIS      : 4096  ->  142.9 GFLOP/s   (3.69x)

2. Vector math: glibc libm vs AOCL-LibM

The turbulence and thermodynamics paths are transcendental-heavy; this is where AOCL-LibM shows up:

$ ./vecmath_bench exp 100000000
  glibc libm : 100M exp   ->  1.83 s

$ LD_PRELOAD=$AOCL_ROOT/lib/libalm.so ./vecmath_bench exp 100000000
  AOCL-LibM  : 100M exp   ->  0.71 s   (2.58x)

3. The solve itself: baseline vs tuned binary

Same grid, same rank count, same iterations — just the two binaries. FUN3D prints per-run wall time; wrap it in time and read off seconds/iteration:

# Baseline: stock gfortran -O2 + reference BLAS/LAPACK
$ time mpirun -np 192 --bind-to core ./nodet_mpi.gcc \
      --grid wing.b8.ugrid --iterations 500
  ...
  flow solve : 500 iterations,  wall 612.4 s   (1.225 s/iter)
  real   10m18.7s

# Tuned: AOCC + AOCL, identical case
$ time mpirun -np 192 --bind-to core ./nodet_mpi.aocc \
      --grid wing.b8.ugrid --iterations 500
  ...
  flow solve : 500 iterations,  wall 453.1 s   (0.906 s/iter)
  real   7m41.2s

# Speedup on the solve, straight from the two numbers
$ echo "scale=2; 1.225/0.906" | bc
  1.35

Same 500 iterations, same final residual (checked next), 26% less wall time. That 1.35× is the number that shows up on the invoice.

The half that isn't the compiler

On a bandwidth-bound MPI code, placement and the interconnect matter as much as codegen. On H4D specifically:

• NUMA-aware rank placement. Pin one MPI rank per L3/CCX domain region and bind its memory locally. Turin has many CCXs per socket; letting ranks wander across them wrecks the bandwidth you paid for.
• Transparent huge pages. The mesh and solver working sets are large and TLB-hungry; 2 MiB pages cut page-walk overhead measurably.
• Cloud RDMA. H4D's low-latency RDMA is what makes the halo-exchange and the GMRES all-reduces scale. Make sure the MPI is actually using the RDMA transport, not falling back to TCP.

# Example: one rank per core, bound, huge pages on, over RDMA.
export OMPI_MCA_btl=^tcp                 # force the RDMA path, no TCP fallback
export HSA_XNACK=0
echo always > /sys/kernel/mm/transparent_hugepage/enabled

mpirun -np $NRANKS \
  --map-by core --bind-to core \
  --mca pml ucx --mca osc ucx \
  -x LD_LIBRARY_PATH -x OMP_NUM_THREADS=1 \
  ./nodet_mpi --grid mesh.b8.ugrid

Always confirm the pinning took — a quick numastat -p during a run, or per-rank /proc/self/status Cpus_allowed_list, will tell you whether the map-by actually did what you asked. On cloud instances it frequently doesn't on the first try.

Results

Representative numbers from a steady-state RANS case on a ~14M-node mesh, run across 4× H4D nodes. Baseline is the stock gfortran -O2 + reference BLAS/LAPACK build; “tuned” is AOCC/AOCL + the placement work. Treat these as directional — your mesh, turbulence model, and node count will move them.

A ~1.35× overall speedup on the same hardware is, in cloud terms, a ~26% cut in core-hours — and therefore in dollars — for an identical result. The math-heavy turbulence and thermodynamics routines moved the most, which is exactly what you'd predict from swapping reference libm for AOCL-LibM. On this customer's monthly campaign volume the toolchain change paid for the engagement in the first billing cycle.

Validation: the part you don't skip

None of the above counts until the answer is still correct. For every build we:

• Diffed final residuals and force/moment coefficients (Cl, Cd, Cm) against the baseline to within a tight tolerance.
• Confirmed identical convergence history shape — a faster build that converges differently is a bug, not a win.
• Checked mass and momentum conservation hadn't drifted.
• Ran the relevant regression/verification cases before touching a production campaign.

This is why we stayed away from -ffast-math. In CFD, reproducibility and conservation aren't nice-to-haves; a "performance win" that perturbs the solution is negative value.

Takeaways

• The default build is rarely the right build on EPYC. -march=znver5 plus AOCL is most of the win, and it's low-risk.
• Swap the math libraries first. BLIS/libFLAME/AOCL-LibM are the highest-leverage, lowest-drama change for a solver like this.
• Placement is half the game. On bandwidth-bound MPI, NUMA pinning, huge pages, and a real RDMA transport are worth as much as codegen.
• Validate every step. Same answer, faster — or it doesn't ship.

If you're running FUN3D, OpenFOAM, SU2, or any tightly-coupled solver on cloud HPC and suspect you're leaving performance (and budget) on the table, that's exactly the kind of work we do. Get in touch.