Install your own WRFDA

Updated: 04 January 2024

1. Load required modules and set environment variables

Paste the whole recipe into your terminal.

module purge
module load cpeIntel/23.09
module load cray-hdf5-parallel/1.12.2.7
module load cray-netcdf-hdf5parallel/4.9.0.7
module load cray-parallel-netcdf/1.12.3.7
module load libpng/1.6.39-cpeIntel-23.09
module load JasPer/1.900.1-cpeIntel-23.09
module load ADIOS2/2.9.1-cpeIntel-23.09

export WRF_EM_CORE=1    # Explicitly select ARW core developed by NCAR
export WRF_NMM_CORE=0   # Not NMM core
export WRF_CHEM=0       # Select WRF-Chem

export NETCDF=${CRAY_NETCDF_HDF5PARALLEL_PREFIX}
export NETCDFPAR=${CRAY_NETCDF_HDF5PARALLEL_PREFIX}    
export PNETCDF=${CRAY_PARALLEL_NETCDF_PREFIX}           
export HDF5=${CRAY_HDF5_PARALLEL_PREFIX}
export ADIOS2=${EBROOTADIOS2}

export JASPERINC=${EBROOTJASPER}/include              # (Optional)
export JASPERLIB=${EBROOTJASPER}/lib                  # (Optional)
export WRFIO_NCD_LARGE_FILE_SUPPORT=1                 # (Optional)
export PNETCDF_QUILT=1                                # (Optional)

### Common mistake :: insert spaces before or after =

2. Set install location directory (DIR)

Let set an install path, DIR.

# While being in your desired install location directory, execute
export DIR=$(pwd)

3. Download WRF source code

An example for WRF 4.5.1 is as shown below.

cd $DIR
wget https://github.com/wrf-model/WRF/releases/download/v4.5.1/v4.5.1.tar.gz
tar xzf v4.5.1.tar.gz
mv WRFV4.5.1 WRF

For other version, check the WRF stable branches on the GitHub repository and its official website. For WRF 3.X, users have to add ./WRFV3/chem folder with these.

4. Enable GRIB2 IO (Optional)

Must set JASPERINC and JASPERLIB in Step 1

Open ${DIR}/WRF/arch/Config.pl (or Config_new.pl) using a text editor such as vi
- Change the parameter $I_really_want_to_output_grib2_from_WRF from "FALSE" to "TRUE"

vi ${DIR}/WRF/arch/Config.pl                  # or Config_new.pl
# Change $I_really_want_to_output_grib2_from_WRF = "FALSE" ; 
#   to   $I_really_want_to_output_grib2_from_WRF = "TRUE" ;

5. Install RTTOV (Optional)

WRFDA 4d-var version 4.5.1 supports up to RTTOV 12.1

Download the source code from https://nwp-saf.eumetsat.int/site/software/rttov/ (registration required)
Make a directory RTTOV, extract the source code and go inside its build directory.

mkdir ${DIR}/RTTOV
cd ${DIR}/RTTOV
tar xf XXXXX            # where XXXXX is the path to your downloaded source code
cd build

Edit the below lines in the Makefile.local.

HDF5_PREFIX  = ${CRAY_HDF5_PARALLEL_PREFIX}
FFLAGS_HDF5  = -D_RTTOV_HDF
LDFLAGS_HDF5 = 

NETCDF_PREFIX  = ${CRAY_NETCDF_HDF5PARALLEL_PREFIX}
FFLAGS_NETCDF  = -D_RTTOV_NETCDF
LDFLAGS_NETCDF =

The flags for HDF5, NetCDF and LAPACK library will added by the Cray compiler wrapper (cc, ftn).

Create ./arch/cray-ifort-lanta file that contains the following recipe
(vi ${DIR}/RTTOV/build/arch/cray-ifort-lanta)

FC=ftn
FC77=ftn
CC=cc
LDFLAGS_ARCH= -static-intel
CFLAGS_ARCH=
FFLAGS_ARCH=-fPIC -O3 -fp-model source -static-intel
AR=ar r

# Loop unrolling causes ifort v13 and later to take a long time to compile these subroutines
FFLAGS_ARCH_rttov_opdep_9_ad=-fPIC -O3 -unroll0 -fp-model source
FFLAGS_ARCH_rttov_opdep_9_k=-fPIC -O3 -unroll0 -fp-model source
FFLAGS_ARCH_rttov_opdep_13_ad=-fPIC -O3 -unroll0 -fp-model source
FFLAGS_ARCH_rttov_opdep_13_k=-fPIC -O3 -unroll0 -fp-model source

# -fp-model source ensures more consistent floating point results

Run the ./rttov_compile.sh script
1. Type ‘cray-ifort-lanta' → Press ‘Enter'
2. (Leave blank) → Press ‘Enter'
3. Type ‘y’ → Press ‘Enter’
4. Type '-j 2' → Press 'Enter’
5. Type 'y' → Press 'Enter’
  (This step will take around 10-15 minutes.)
Set the RTTOV environment variable

export RTTOV=${DIR}/RTTOV

6. Compile WRFPLUS (Optional, only for 4dvar)

Copy the source code and go inside the directory

cp -r WRF WRFPLUS
cd WRFPLUS

Run ./confiugre wrfplus
→ Type ‘26' + Press 'Enter’
Open ./configure.wrf
- Change icc to cc -Wno-implicit-function-declaration -Wno-implicit-int
- Append -Wno-implicit-function-declaration -Wno-implicit-int to existing cc
- Prepend -fp-model precise -fp-speculation=safe to FCBASEOPTS_NO_G.
Compile the program

./compile wrfplus 2>&1 | tee compile.wrfplus.log        # Take around 1 hour on LANTA

Run ls ./main/*.exe to check if wrfplus.exe exists.
Set the WRFPLUS_DIR environment variable

export WRFPLUS_DIR=$(pwd)

7. Compile WRFDA 4dvar

Copy the source code and go inside the directory

cp -r WRF WRFDA
cd WRFDA

Run ./confiugre 4dvar
→ Type ‘26' + Press ‘Enter’
(Note: for 3dvar, run ./configure wrfda -> Type '50' + Press 'Enter')
Open ./configure.wrf
- Change icc to cc -Wno-implicit-function-declaration -Wno-implicit-int
- Append -Wno-implicit-function-declaration -Wno-implicit-int to existing cc
- Prepend -fp-model precise -fp-speculation=safe to FCBASEOPTS_NO_G.
Run

 ./compile all_wrfvar 2>&1 | tee compile.wrfda.log         # Take around 10 minutes on LANTA

Run ls ./var/build/*.exe | wc -l to check if 43 executables were successfully compiled.
Run ls ./var/obsproc/src/*.exe to check if obsproc.exe exists.

8. Example: Job submission script

#!/bin/bash
#SBATCH -p compute              # Partition
#SBATCH -N 1                    # Number of nodes
#SBATCH --ntasks-per-node=128   # Number of MPI processes per node
#SBATCH -t 5-00:00:00           # Job runtime limit
#SBATCH -J WRFDA                # Job name
#SBATCH -A ltxxxxxx             # Account *** {USER EDIT} ***

module purge
module load cpeIntel/23.09
module load cray-hdf5-parallel/1.12.2.7
module load cray-netcdf-hdf5parallel/4.9.0.7
module load cray-parallel-netcdf/1.12.3.7
module load libpng/1.6.39-cpeIntel-23.09
module load JasPer/1.900.1-cpeIntel-23.09
module load ADIOS2/2.9.1-cpeIntel-23.09

export OMP_NUM_THREADS=1

ulimit -s unlimited

...

Contact Us
ThaiSC support service : thaisc-support@nstda.or.th