Versions Compared

Version Old Version 1 New Version 2
Changes made by

Somrath Kanoksirirath

Somrath Kanoksirirath

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Updated: 15 December September 2024

Table of Contents
minLevel1
maxLevel4
stylecircle

...

Info

This guide is largely derived from https://opentopas.readthedocs.io/en/latest/getting-started/Debian.html#. It is recommended to visit the page for more detailed explanation of each [Step X] .

1. Load required modules and set environment variables [Step 1-4]

Paste the below lines into your terminal. (Putting Do NOT put these directly in ~/.bashrc is not recommended.)

Code Block
languagebash
module purge
module load expat/2.5.0
module load module purgeX11/20231019-cpeCray-23.03
module load libGLU/9.0.3-cpeCray-23.03
module load ParallelIOHarfBuzz/8.2.6.2-cpeIntelcpeCray-23.0903
module load METIS/5.1.0-cpeIntel-23.09

export PIO=${EBROOTPARALLELIO}
export NETCDF=${CRAY_NETCDF_HDF5PARALLEL_PREFIX}
export PNETCDF=${CRAY_PARALLEL_NETCDF_PREFIX}
export OPENMP=true

### Common mistake :: insert spaces before or after =

2. Download MPAS source code

An example for MPAS 8.0.1 is as shown below.

Code Block
languagebash
wgetCMake/3.26.4

export CC=cc CXX=CC FC=ftn F77=ftn F90=ftn

2. Download Geant4 [Step 5]

Code Block
languagebash
mkdir $HOME/Applications/GEANT4
cd $HOME/Applications/GEANT4
wget https://gitlab.cern.ch/geant4/geant4/-/archive/v11.1.3/geant4-v11.1.3.tar.gz
tar -zxf geant4-v11.1.3.tar.gz

3. Install Geant4 [Step 7]

Code Block
languagebash
cd $HOME/Applications/GEANT4/
rm -rf geant4-install geant4-build
mkdir geant4-{build,install}
cd geant4-build
cmake ../geant4-v11.1.3 -DGEANT4_INSTALL_DATA=OFF -DGEANT4_BUILD_MULTITHREADED=ON -DCMAKE_INSTALL_PREFIX=../geant4-install -DGEANT4_USE_QT=OFF -DGEANT4_USE_OPENGL_X11=ON -DGEANT4_USE_RAYTRACER_X11=ON
make -j8 install

cd $HOME/Applications/GEANT4/geant4-install
ln -s lib64 lib
Note

  • -DCMAKE_PREFIX_PATH=/usr/lib/qt5 -DGEANT4_USE_QT=ON is changed to -DGEANT4_USE_QT=OFF.

  • /geant4-install/lib is symbolically linked to /geant4-install/lib64.

4. Download G4DATA [Step 6]

Note

As of 10 September 2024, there is a connection issue between transfer.lanta.nstda.or.th and https://cern.ch/.Therefore, G4DATA was previously downloaded by ThaiSC admin and temporarily made available at /project/common/G4DATA.

Code Block
languagebash
cd $HOME/Applications/GEANT4/
cp -r /project/common/G4DATA .

5. Download OpenTOPAS [Step 8.1]

Code Block
languagebash
mkdir $HOME/Applications/TOPAS
cd $HOME/Applications/TOPAS
git clone https://github.com/MPAS-Dev/MPAS-Model/archive/refs/tags/v8.0.1.tar.gz
tar xzf v8.0.1.tar.gz
mv MPAS-Model-8.0.1 MPAS
cd MPAS
Info

For other version, check https://github.com/MPAS-Dev/MPAS-Model/releases.

3. Compile MPAS

Compile each MPAS core with your desired floating-point precision; for example,

Code Block
languagebash
make -j4 intel-nersc CORE=init_atmosphere PRECISION=single AUTOCLEAN=true
make -j4 intel-nersc CORE=atmosphere PRECISION=single AUTOCLEAN=true
...
Info

User still has to use make clean CORE=XXX before recompiling each core.

Additional notes:

  1. CORE=ocean only supports double precision.

  2. CORE=seaice only supports double precision and only serial build (-j1) works.

  3. Users may have to add parentheses for the print function, i.e., print xxx → print(xxx), in

    • ./src/core_ocean/BGC/makedep.py

    • ./src/core_ocean/cvmix/makedep.py

    to make these files compatible with Python3.

...

OpenTOPAS/OpenTOPAS.git

6. Install GDCM [Step 8.2-8.3]

If $HOME/Applications/GDCM already exists, rename it to another name such as mv $HOME/Applications/GDCM $HOME/Applications/GDCM-OLD before proceed.

Code Block
languagebash
mkdir $HOME/Applications/GDCM
cd $HOME/Applications/TOPAS/OpenTOPAS
mv gdcm-2.6.8.tar.gz ../../GDCM
cd ../../GDCM
tar -zxf gdcm-2.6.8.tar.gz

rm -rf gdcm-install gdcm-build
mkdir gdcm-{build,install}
cd gdcm-build
cmake ../gdcm-2.6.8 -DGDCM_BUILD_SHARED_LIBS=ON -DGDCM_BUILD_DOCBOOK_MANPAGES:BOOL=OFF  -DCMAKE_INSTALL_PREFIX=../gdcm-install 
make -j8 install

7. Install OpenTOPAS [Step 8.4]

Code Block
languagebash
cd $HOME/Applications/TOPAS
rm -rf OpenTOPAS-install OpenTOPAS-build
mkdir OpenTOPAS-{build,install}
cd OpenTOPAS-build

export Geant4_DIR=$HOME/Applications/GEANT4/geant4-install
export GDCM_DIR=$HOME/Applications/GDCM/gdcm-install
cmake ../OpenTOPAS -DCMAKE_INSTALL_PREFIX=../OpenTOPAS-install
make -j8 install

8. Create a wrapper script of topas [Step 9.1.b]

Code Block
languagebash
mkdir $HOME/Applications/shellScripts
cd $HOME/Applications/shellScripts

cat << Eof > topas
#!/bin/bash

module purge
module load expat/2.5.0
module load X11/20231019-cpeCray-23.03
module load libGLU/9.0.3-cpeCray-23.03
module load HarfBuzz/8.2.2-cpeCray-23.03

export TOPAS_G4_DATA_DIR=$HOME/Applications/GEANT4/G4DATA
export LD_LIBRARY_PATH=$HOME/Applications/TOPAS/OpenTOPAS-install/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/Applications/GEANT4/geant4-install/lib:$LD_LIBRARY_PATH

$HOME/Applications/TOPAS/OpenTOPAS-install/bin/topas $1
Eof

chomd +x topas
echo 'export PATH=~/Applications/shellScripts:$PATH' >> $HOME/.bashrc

9. Run OpenTOPAS on LANTA

Note

Since OpenTOPAS does NOT installed with Qt, b:Ts/UseQt = "False" must always be used.

Two approaches for running OpenTOPAS on LANTA are presented below.

  • The first approach using sbatch command is generally recommended. However, users MUST ensure that b:Ts/PauseBeforeQuit is set to "False", otherwise your job will run on the system until time out, wasting your SHr!

  • The second approach using sinteract command can be useful. However, this method is not recommended if your internet connection is poor or you have to run topas for a long time.

A. Job submission script

Below is a template of OpenTOPAS job script. It can be created using vi submitTOPAS.sh. Edit it to meet your demands.

Code Block
languagebash
#!/bin/bash
#SBATCH -p compute             # Partition
#SBATCH -N 1                   # Number of nodesnode
#SBATCH --ntasks-per-node=641    # Number of MPI processes per node
#SBATCH --cpus-per-task=216      # Number of OpenMP threads per MPI process
#SBATCH -t 5-00120:00:00           # Job runtime limit
#SBATCH -J MPAS TOPAS               # Job name
#SBATCH -A ltxxxxxx            # Account *** {USER EDIT} ***


module purge
module load ParallelIO/2.6.2-cpeIntel-23.09
module load METIS/5.1.0-cpeIntel-23.09

export OMP_STACKSIZE="32M"
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

ulimit -s unlimited

srun -n${SLURM_NTASKS} -c${SLURM_CPUS_PER_TASK} ./init_atmosphere_model
srun -n${SLURM_NTASKS} -c${SLURM_CPUS_PER_TASK} ./atmosphere_model
Info

MPAS static geographical datasets are available at /project/common/MPAS_Static/ ,
while WPS static datasets are available at /project/common/WPS_Static/geog .

Users can choose one of them as config_geog_data_path in namelist.init_atmosphere.
source $HOME/.bashrc
echo "Use topas = $(which topas)"

topas xxx.txt                  # Your case *** {USER EDIT} ***
Note

In your case file, b:Ts/PauseBeforeQuit MUST set to "False", otherwise your job will run indefinitely until time out.

To submit this job script to the SLURM queuing system, execute

Code Block
languagebash
sbatch submitTOPAS.sh

(Also see https://thaisc.atlassian.net/wiki/spaces/LANTA/pages/447021064/Quick+Start+Guide#6.-Running-your-job)

B. Interactive session

You can directly access LANTA computing nodes by requesting an interactive session. For example, execute

Code Block
languagebash
sinteract -p compute -c 16 -t 06:00:00

(You could add -X option to enable X11 forwarding for OpenGL-related features such as CopyOpenGLToSVG. Please visit Interactive session before using this capability)

After the requested resources are allocated, you can use this session to run topas and interactively execute Geant4 commands afterwards. For example,

Code Block
languagebash
topas $HOME/Applications/TOPAS/OpenTOPAS-install/examples/Basic/TwoBeams.txt

...

Contact Us
ThaiSC support service : thaisc-support@nstda.or.th

...