OpenFOAM

OpenFOAM (Open-source Field Operation And Manipulation) is well-known software in computational fluid mechanics (CFD). It contains several standard solvers for simulating a broad range of continuum flows. Using the OpenFOAM toolbox, users can build their own solver to solve specific problems. OpenFOAM is open-source software under the GNU General Public License Version 3.

Official website: https://www.openfoam.com/ and https://openfoam.org/

Updated: Sep 2024

1 1. Case directory
2 2. Job submission script
- 2.1 I. RunFunction commands
- 2.2 II. Explicit commands
3 3. Job submission
4 4. Additional note
5 5. Post-processing

Modules

Module name	Description	Note

Module name	Description	Note
OpenFOAM/v2212-cpeCray-23.03	From https://www.openfoam.com/	MPI only, more tools
OpenFOAM/10-cpeCray-23.03	From https://openfoam.org/	MPI only, slightly faster

In addition to OpenFOAM,

Gmsh (Gmsh/4.12.2-cpeCray-23.03-NoGUI) --- a finite-element mesh generator
ParaView (ParaView/5.12.1-cpeCray-23.03-NoGUI) --- a scientific visualization tool

are also available on LANTA.

1. Case directory

A case directory is needed to run OpenFOAM. For new users, visit https://www.openfoam.com/documentation/tutorial-guide or https://doc.cfd.direct/openfoam/user-guide-v10/index.

After loading an OpenFOAM module, several examples can be found at $FOAM_TUTORIALS. Before running, copy them to your local path cp -r $FOAM_TUTORIALS . .

2. Job submission script

Two approaches for submitting OpenFOAM jobs are presented below. These approaches can be adapted to any OpenFOAM variant; however, some commands may have slightly different names.

1 I. RunFunction commands
2 II. Explicit commands

I. RunFunction commands

Below is an example of OpenFOAM submission script (submitFoam.sh) using $WM_PROJECT_DIR/bin/tools/RunFunctions. It can be created using vi submitFoam.sh.

The total number of tasks (nodes x ntasks-per-node) must be equal or larger than to the number of subdomains specified in system/decomposeParDict (or from getNumberOfProcessors command).
LANTA compute partition has 128 CPU cores per node. Therefore, ntasks-per-node <= 128.

#!/bin/bash
#SBATCH -p compute             # Partition
#SBATCH --nodes=1              # Number of nodes
#SBATCH --ntasks-per-node=32   # Number of MPI processes per node 
#SBATCH -t 5-00:00:00          # Job runtime limit
#SBATCH -J OpenFOAM            # Job name
#SBATCH -A ltxxxxxx            # Account *** {USER EDIT} *** 

module purge
module load OpenFOAM/v2212-cpeCray-23.03

ulimit -s unlimited

# *** OpenFOAM steps *** {USER EDIT} ***
source $WM_PROJECT_DIR/bin/tools/RunFunctions   # Import RunFunctions

runApplication surfaceFeatureExtract
runApplication blockMesh
runApplication snappyHexMesh -overwrite
runApplication decomposePar
runParallel renumberMesh -overwrite
runParallel checkMesh
restore0Dir -processor
runParallel $(getApplication)
runApplication reconstructPar

II. Explicit commands

Another example of OpenFOAM submission script (submitFoam.sh) is shown below.

#!/bin/bash
#SBATCH -p compute             # Partition
#SBATCH --nodes=1              # Number of nodes
#SBATCH --ntasks-per-node=32   # Number of MPI processes per node 
#SBATCH -t 5-00:00:00          # Job runtime limit
#SBATCH -J OpenFOAM            # Job name
#SBATCH -A ltxxxxxx            # Account *** {USER EDIT} *** 

module purge
module load OpenFOAM/v2212-cpeCray-23.03

ulimit -s unlimited

# *** OpenFOAM steps *** {USER EDIT} ***
cp -r ./0.orig ./0
srun -N1 -n1 surfaceFeatureExtract 2>&1
srun -N1 -n1 blockMesh 2>&1
srun -N1 -n1 snappyHexMesh -overwrite 2>&1
srun -N1 -n1 decomposePar -copyZero 2>&1
srun renumberMesh -parallel -overwrite 2>&1
srun checkMesh -parallel 2>&1
srun xxxFoam -parallel 2>&1
srun -N1 -n1 reconstructPar 2>&1

3. Job submission

To submit jobs to the SLURM queuing system, execute

sbatch submitFoam.sh

4. Additional note

Out of memory
If users encounter
slurmstepd: error: Detected ... oom-kill event(s) ... killed by the cgroup out-of-memory handler
in any log file, try increasing RAM per CPU core by adding
- #SBATCH --mem-per-cpu=3800M – while having ntasks-per-node <= 64 or
- #SBATCH --mem-per-cpu=7600M – while having ntasks-per-node <= 32
to your submitFoam.sh.
Floating-point exception trapping
By default, the floating-point exception trapping is disabled. To enable, add export FOAM_SIGFPE=true to your script.
Building custom executable
Please use
- FOAM_USER_APPBIN in place of FOAM_APPBIN and FOAM_SITE_APPBIN
- FOAM_USER_LIBBIN in place of FOAM_LIBBIN and FOAM_SITE_LIBBIN
to avoid permission denied.
[v2212] To manually specify those paths, see $WM_PROJECT_DIR/bin/tools/change-userdir.sh

5. Post-processing

PareView is recommended for visualizing software outputs directly on LANTA (see ParaView-on-LANTA). The following are our notes when opening an OpenFOAM case.

For ParaView to read an OpenFOAM case directory, an empty file (.foam) representing the directory is required to be presented inside.
Please pay attention when selecting Labal Size and Scalar Size. If your case was run on LANTA, then the Label Size is 32 bits while the Scalar Size is 64 bits. If inconsistent numbers were chosen, you could get a black layout screen.
When opening a large case, especially a deconstructed case, it is common that a “Not responding“ message window would appear. Before clicking any button, please monitor the progress bar at the bottom for a while, if it increases little by little, then let the warnings be.

Contact Us
ThaiSC support service : thaisc-support@nstda.or.th