Versions Compared

Version Old Version 1 New Version Current
Changes made by

Somrath Kanoksirirath

Somrath Kanoksirirath

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Updated: 15 December 20 September 2024

Table of Contents
minLevel1
maxLevel4
stylecircle

...

  • This interactive approach can be useful. However, this method is not recommended if your internet connection is poor or you have to run topas for a long time.

  • On a HPC, running OpenTOPAS in batch mode is preferred. Please visit OpenTOPAS (Batch, Intel) on LANTA to install OpenTOPAS that suitable for batch mode instead.

Info

This guide is largely derived from https://opentopas.readthedocs.io/en/latest/getting-started/Debian.html#. It is recommended to visit the page for more detailed explanation of each [Step X] .

1. Load required modules and set environment variables [Step 1-4]

Paste the below lines into your terminal. (Putting Do NOT put these directly in ~/.bashrc is not recommended.)

Code Block
languagebash
module purge
module purgeload expat/2.5.0
module load X11/20231019-cpeCray-23.03
module load ParallelIOlibGLU/29.60.23-cpeIntelcpeCray-23.0903
module load METISHarfBuzz/58.12.02-cpeIntelcpeCray-23.0903
module 
export PIO=${EBROOTPARALLELIO}load CMake/3.26.4

export NETCDF=${CRAY_NETCDF_HDF5PARALLEL_PREFIX}
export PNETCDF=${CRAY_PARALLEL_NETCDF_PREFIX}
export OPENMP=true

### Common mistake :: insert spaces before or after =

2. Download MPAS source code

...

CC=cc CXX=CC FC=ftn F77=ftn F90=ftn

2. Download Geant4 [Step 5]

Code Block
languagebash
mkdir $HOME/Applications
mkdir $HOME/Applications/GEANT4
cd $HOME/Applications/GEANT4
wget https://github.com/MPAS-Dev/MPAS-Modelgitlab.cern.ch/geant4/geant4/-/archive/refs/tags/v8.0.1v11.1.3/geant4-v11.1.3.tar.gz
tar xzf v8.0.1-zxf geant4-v11.1.3.tar.gz
mv MPAS-Model-8.0.1 MPAS
cd MPAS
Info

For other version, check https://github.com/MPAS-Dev/MPAS-Model/releases.

3. Compile MPAS

Compile each MPAS core with your desired floating-point precision; for example,

Code Block
languagebash
make -j4 intel-nersc CORE=init_atmosphere PRECISION=single AUTOCLEAN=true
make -j4 intel-nersc CORE=atmosphere PRECISION=single AUTOCLEAN=true
...
Info

User still has to use make clean CORE=XXX before recompiling each core.

Additional notes:

  1. CORE=ocean only supports double precision.

  2. CORE=seaice only supports double precision and only serial build (-j1) works.

  3. Users may have to add parentheses for the print function, i.e., print xxx → print(xxx), in

    • ./src/core_ocean/BGC/makedep.py

    • ./src/core_ocean/cvmix/makedep.py

    to make these files compatible with Python3.

4. Example: Job submission script

Code Block
languagebash
#!/bin/bash
#SBATCH -p compute             # Partition
#SBATCH -N 1                   # Number of nodes
#SBATCH --ntasks-per-node=64   # Number of MPI processes per node
#SBATCH --cpus-per-task=2      # Number of OpenMP threads per MPI process
#SBATCH -t 5-00:00:00          # Job runtime limit
#SBATCH -J MPAS                # Job name
#SBATCH -A ltxxxxxx            # Account *** {USER EDIT} ***

module purge
module load ParallelIO/2.6.2-cpeIntel-23.09
module load METIS/5.1.0-cpeIntel-23.09

export OMP_STACKSIZE="32M"
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

ulimit -s unlimited

srun -n${SLURM_NTASKS} -c${SLURM_CPUS_PER_TASK} ./init_atmosphere_model
srun -n${SLURM_NTASKS} -c${SLURM_CPUS_PER_TASK} ./atmosphere_model
Info

MPAS static geographical datasets are available at /project/common/MPAS_Static/ ,
while WPS static datasets are available at /project/common/WPS_Static/geog .

Users can choose one of them as config_geog_data_path in namelist.init_atmosphere.
3. Install Geant4 [Step 7]
 Code Block
languagebash
cd $HOME/Applications/GEANT4/
rm -rf geant4-install geant4-build
mkdir geant4-{build,install}
cd geant4-build
cmake ../geant4-v11.1.3 -DGEANT4_INSTALL_DATA=OFF -DGEANT4_BUILD_MULTITHREADED=ON -DCMAKE_INSTALL_PREFIX=../geant4-install -DGEANT4_USE_QT=OFF -DGEANT4_USE_OPENGL_X11=ON -DGEANT4_USE_RAYTRACER_X11=ON
make -j8 install

cd $HOME/Applications/GEANT4/geant4-install
ln -s lib64 lib
 Note
  • -DCMAKE_PREFIX_PATH=/usr/lib/qt5 -DGEANT4_USE_QT=ON is changed to -DGEANT4_USE_QT=OFF.
  • /geant4-install/lib is symbolically linked to /geant4-install/lib64.
4. Download G4DATA [Step 6]
 Note
As of 10 September 2024, there is a connection issue between transfer.lanta.nstda.or.th and https://cern.ch/.Therefore, G4DATA was previously downloaded by ThaiSC admin and temporarily made available at /project/common/G4DATA. 
 Code Block
languagebash
cd $HOME/Applications/GEANT4/
cp -r /project/common/G4DATA . 
5. Download OpenTOPAS [Step 8.1]
 Code Block
languagebash
mkdir $HOME/Applications/TOPAS
cd $HOME/Applications/TOPAS
git clone https://github.com/OpenTOPAS/OpenTOPAS.git
6. Install GDCM [Step 8.2-8.3]
If $HOME/Applications/GDCM already exists, rename it to another name such as mv $HOME/Applications/GDCM $HOME/Applications/GDCM-OLD before proceed.
 Code Block
languagebash
mkdir $HOME/Applications/GDCM
cd $HOME/Applications/TOPAS/OpenTOPAS
mv gdcm-2.6.8.tar.gz ../../GDCM
cd ../../GDCM
tar -zxf gdcm-2.6.8.tar.gz

rm -rf gdcm-install gdcm-build
mkdir gdcm-{build,install}
cd gdcm-build
cmake ../gdcm-2.6.8 -DGDCM_BUILD_SHARED_LIBS=ON -DGDCM_BUILD_DOCBOOK_MANPAGES:BOOL=OFF  -DCMAKE_INSTALL_PREFIX=../gdcm-install 
make -j8 install
7. Install OpenTOPAS [Step 8.4]
 Code Block
languagebash
cd $HOME/Applications/TOPAS
rm -rf OpenTOPAS-install OpenTOPAS-build
mkdir OpenTOPAS-{build,install}
cd OpenTOPAS-build

export Geant4_DIR=$HOME/Applications/GEANT4/geant4-install
export GDCM_DIR=$HOME/Applications/GDCM/gdcm-install
cmake ../OpenTOPAS -DCMAKE_INSTALL_PREFIX=../OpenTOPAS-install
make -j8 install
 Info
Alternatively, you could append -DTOPAS_WITH_PTL_GEANT4:BOOL=ON to the cmake step to compile without graphic support. 
8. Create a wrapper script of topas [Step 9.1.b]
 Code Block
languagebash
mkdir $HOME/Applications/shellScripts
cd $HOME/Applications/shellScripts

cat << Eof > topas
#!/bin/bash

module purge
module load expat/2.5.0
module load X11/20231019-cpeCray-23.03
module load libGLU/9.0.3-cpeCray-23.03
module load HarfBuzz/8.2.2-cpeCray-23.03

export TOPAS_G4_DATA_DIR=$HOME/Applications/GEANT4/G4DATA
export LD_LIBRARY_PATH=$HOME/Applications/TOPAS/OpenTOPAS-install/lib:\$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/Applications/GEANT4/geant4-install/lib:\$LD_LIBRARY_PATH

$HOME/Applications/TOPAS/OpenTOPAS-install/bin/topas \$1
Eof

chmod +x topas
echo 'export PATH=~/Applications/shellScripts:$PATH' >> $HOME/.bashrc
source $HOME/.bashrc
9. Run OpenTOPAS on LANTA
 Note
Since OpenTOPAS does NOT installed with Qt, the option b:Ts/UseQt won’t take any effects and should be set to "False".
You can request an interactive session by executing, for example, 
 Code Block
languagebash
sinteract --X11 -p compute -c 16 -t 06:00:00
The X11 forwarding feature may require further setup. 
  • If you use MobaXterm, the X11 forwarding should be supported by default. 
  • If you login from a Linux terminal, set ForwardX11Trusted in /etc/ssh/ssh_config to yes and then login using ssh -Y username@lanta.nstda.or.th.
  • If you login from a Windows/Mac terminal, you need to install and run a X11 server, such as Xming and XQuartz, while using. Then login using ssh -Y username@lanta.nstda.or.th.
Please visit Interactive session for more details.
After the requested resources are allocated, you can use this session to run topas and interactively execute commands afterwards. A graphic window should appears when objects are rendered. For example, run
 Code Block
languagebash
topas $HOME/Applications/TOPAS/OpenTOPAS-install/examples/Basic/OneBox.txt
...
Contact Us
ThaiSC support service : thaisc-support@nstda.or.th
...