Summary for experienced HPC user

To access TARA, ssh to [username]@tara.nstda.or.th, where [username] is your registered TARA username.
Each user is allocated with 50 GB home directory ($HOME).
The project directory(s) (/tarafs/data/project/[projxxxx]-[short-name]/) has 200 GB of space to be shared among all users in the project; where [projxxxx] is project id and [short-name] is its short name.
There is a scratch directory (/tarafs/scratch/[projxxxx]-[short-name]/) which is a fast read & write via read-intensive SSD. For temporary file only, files will be removed after 30 days of last access.
Software are managed as modules using Environment module system (Lmod).
TARA usage is subjected to fairshare policy and managed by Slurm job scheduler. Examples of Slurm batch script for specific applications can be found at การติดตั้งและการงานใช้ Application.

Cheat Sheet

View file

name	20200102-tara-cheat-sheet-v3.pdf

Download this document as a PDF

View file

name	01-getting_started_windows-vj-ed3.pdf

TARA System Overview

An up-to-date overview of TARA system can be found at Resources – ThaiSC.

Workflow

This is a suggested workflow for users when using TARA

Table of Contents

minLevel	2
maxLevel	7	style	none

1. Logging in to TARA

Info
The connection through VPN is required to access TARA. The instruction how to setup and connect to the VPN can be found in your email.

Accessing TARA via ssh protocol, user can use SSH client such as PuTTy, MobaXterm, etc.. In this document, we demonstrate how to log in to TARA using MobaXterm.

More information about MobaXterm is available at https://mobaxterm.mobatek.net/.

For macOS or Linux, you can use ssh -Y [username]@tara.nstda.or.th command in the Terminal (YouTube : https://youtu.be/kr0WR46UGuc).

To access TARA, you can follow the steps below (YouTube : https://youtu.be/T6earis8Hqo).

1. Open MobaXterm, a SSH client program

2. Click on Session button to initiate the session page

3. Click on the SSH button to use SSH protocol and set the configuration as the following details.

Remote host = tara.nstda.or.th

Specify username = [username]

Port = 22

Where [username] is your TARA username.

4. Click OK button to accept the settings. The SSH terminal session will start and prompt for a password. Enter your TARA password.

If this is your first time logging in using MobaXterm, MobaXterm will ask whether you want to save your password.

Info

For a first time login to TARA, you will be asked to set your new password.

enter your Current password
enter your New password
re-enter your New password

5. The connection between SSH session and TARA is now established. You will see the welcome message. All commands executed in this session are run on TARA.

2. Transferring file to TARA

Each user is allocated with 50 GB home directory ($HOME) and project directory(s) (/tarafs/data/project/proj[xxxx]-[short-name]/) of 200 GB to be shared among all users in the project.

Note: [projxxxx] is project id and [short-name] is your project short name.

To transfer files to TARA with Graphic User Interface (GUI), use can use FTP client such as FileZilla, WinSCP, MobaXterm, etc..

For macOS or Linux, you can use scp command in the Terminal (YouTube : https://youtu.be/ovQj9dzwE-w).

In this document, we demonstrate this process using MobaXterm (YouTube : https://youtu.be/aSwjYtY1KrA).

1. Open MobaXterm and click at Session button to open Session page.

2. Click at SFTP button to use SFTP protocol and set the configuration as shown.

Remote host = tara.nstda.or.th

Username = [username]

Port = 22

3. Click OK, then SFTP session will start. Now, you can drag and drop files to transfer files between TARA and your local machine.

3. Checking Software availability

TARA has reasonable number of software installed. These software are managed by module framework using Environment module system (Lmod). (YouTube : https://youtu.be/NAkJv2uVQZA)

The commands in this part are executed in SSH session. If you transferring file using SFTP session, switch to the SSH session by click at SSH tap with the key icon.

To list all available software

Command : module avail

Example :

Code Block

[hpcuser@tara-frontend-1 ~]$ module avail
----------------------------------------------------------------------------------------------------- /tarafs/utils/modules/modules/all -----------------------------------------------------------------------------------------------------
   ANSYS/19.3                                 EasyBuild/4.1.0                            Mako/1.1.0-GCCcore-8.3.0                                 ScaLAPACK/2.0.2-gompic-2019b    (D)    fosscuda/2019b
   Advisor/2019_update5                       FFTW/3.3.8-gompi-2019b                     Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4                  Singularity/3.3.0                      gcccuda/2019b
   Autoconf/2.69-GCCcore-8.3.0                FFTW/3.3.8-gompic-2019b            (D)     NASM/2.14.02-GCCcore-8.3.0                               Singularity/3.4.2               (D)    gnuplot/5.2.8-GCCcore-8.3.0
   Autoconf/2.69                       (D)    FriBidi/1.0.5-GCCcore-8.3.0                NLopt/2.6.1-GCCcore-8.3.0                                Szip/2.1.1-GCCcore-8.3.0               gompi/2019b
   Automake/1.15.1-GCCcore-8.3.0              GATK/4.1.4.1-GCCcore-8.3.0-Java-11         NSPR/4.21-GCCcore-8.3.0                                  Tcl/8.6.9-GCCcore-8.3.0                gompic/2019b
...

To load software

Command : module load [software]

Example :

Code Block
[hpcuser@tara-frontend-1 ~]$ module load Python/3.7.4-GCCcore-8.3.0 [hpcuser@tara-frontend-1 ~]$ python --version Python 3.7.4

To list all loaded software

Command : module list

Example :

Code Block

[hpcuser@tara-frontend-1 ~]$ module list

Currently Loaded Modules:
  1) GCCcore/8.3.0                       7) Tcl/8.6.9-GCCcore-8.3.0
  2) zlib/1.2.11-GCCcore-8.3.0     (H)   8) SQLite/3.29.0-GCCcore-8.3.0
  3) binutils/2.32-GCCcore-8.3.0         9) XZ/5.2.4-GCCcore-8.3.0
  4) bzip2/1.0.8-GCCcore-8.3.0          10) GMP/6.1.2-GCCcore-8.3.0
  5) ncurses/6.1-GCCcore-8.3.0     (H)  11) libffi/3.2.1-GCCcore-8.3.0
  (H)
  6) libreadline/8.0-GCCcore-8.3.0 (H)  12) Python/3.7.4-GCCcore-8.3.0

  Where:
   H:  Hidden Module

To unload all loaded software

Command : module purge

Example :

Code Block
[hpcuser@tara-frontend-1 ~]$ module purge [hpcuser@tara-frontend-1 ~]$ module list No modules loaded

Info
For more information about LMOD system please see การใช้งาน module ใน TARA Cluster

4. Software/Application installation (if applicable)

If TARA does not have the software(s) you need installed, you can install them by yourself in either your home directory or your project directory. Please make sure that you have the software license and the license term allows to run the software on TARA.

We support EasyBuild (https://easybuild.io/), see การใช้งาน local module ใน TARA Cluster for the step by step instruction to install software as a local module. You can also check การติดตั้งและการงานใช้ Application for some specific software installation/setup instruction.

5. Running Jobs

Check TARA HPC status

Before submitting your job to run on TARA, you can check available computational resources or status of partitions by using sinfo command.

Example :

Code Block

[hpcuser@tara-frontend-1 ~]$ sinfo
PARTITION      AVAIL  TIMELIMIT  NODES  STATE NODELIST
devel             up    2:00:00      2   idle tara-c-[001-002]
compute*          up 5-00:00:00      1    mix tara-c-034
compute*          up 5-00:00:00     57  alloc tara-c-[003-033,035-060]
memory            up 5-00:00:00      5    mix tara-m-[001-003,005-006]
memory            up 5-00:00:00      2  alloc tara-m-[004,007]
memory            up 5-00:00:00      1   idle tara-m-008
memory-preempt    up 5-00:00:00      5    mix tara-m-[001-003,005-006]
memory-preempt    up 5-00:00:00      2  alloc tara-m-[004,007]
memory-preempt    up 5-00:00:00      1   idle tara-m-008
gpu               up 5-00:00:00      2   idle tara-g-[001-002]
dgx               up 5-00:00:00      1   idle tara-dgx1-003
dgx-preempt       up 5-00:00:00      2   idle tara-dgx1-[001-002]

where

Item	Description
PARTITION	partition name
AVAIL	partition status (up : working, down : not working)
TIMELIMIT	maximum time that a job can run on this partition
NODES	number of nodes in the partition
STATE	partition availability (idle : free, alloc : busy, mix : some core(s) in node is free)
NODELIST	list of nodes in the partition

Job submission

TARA usage is subjected to Fair-share policy and managed by Slurm job scheduler. To submit a job follows the following steps.

1. Create bash script using vi sbatch_script.sh (or gedit sbatch_script.sh , nano sbatch_script.sh)

2. Specify the details in the bash script according to the template provided below.

Code Block

#!/bin/bash

#SBATCH -p compute                  # Specify the partition or machine type used [compute/memory/gpu]
#SBATCH -N 1 --ntasks-per-node=40   # Specify the number of nodes and the number of core per node
#SBATCH --mem=16GB                  # Specify memory to use in the node or specify
#SBATCH -t 00:10:00                 # Specifies the maximum time limit (hour: minute: second)
#SBATCH -J my_job                   # Specify the name of the Job
#SBATCH -A projxxxx                 # Specify Project account which will be received after Register

## purge and load modules
module purge                        # unload all modules as they may have previously been loaded.
module load intel                   # Load the module that you want to use. This example is intel

## your script
srun  my_hpc_script                 # Run your program or executable code

3. Use sbatch command followed by your bash script name to submit your job to SLURM

Code Block
sbatch sbatch_script.sh

Note
`sbatch` ,`sh`, `bash` are not the same command. To submit the job, it is an obligation to use `sbatch` .

Info

If you want to try to submit a sample job we recommend to check out our https://youtu.be/8k-NMCIAlZw or https://youtu.be/eZEmCK5MwlI. For DGX partition, you can try https://youtu.be/JwdK7lifI_w.

The example of Slurm batch script for specific applications can be found in the การติดตั้งและการงานใช้ Application.

6. Checking job status

Check job status

To check your job status, using myqueue command.

Example :

Code Block

[hpcuser@tara-frontend-1 ~]$ myqueue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            151602   compute    test3  hpcuser PD       0:00      1 (Priority)
            140215   compute    test2  hpcuser  R   03:25:08      1 tara-c-055
            149496   compute    test1  hpcuser  R 2-02:11:27      1 tara-c-042

Item	Description
JOBID	ID of the specific job
PARTITION	partition which job is running/queuing on
NAME	name of the specific job
USER	user who owns/run the job
ST	Job status (PD : pending, R : running)
TIME	elapsed time of the running job
NODE	the number of node used in the run
NODELIST(REASON)	list of node used to run (reason why the job is pending) see สถานะ Pending (PD) for more information

Cancel job

To cancel the job in the queue, using scancel [JOBID] command

Example : scancel 140215

7. Transferring files to local machine

To transfer files from TARA to your local machine, you can use FTP client such as FileZilla, WinSCP, MobaXterm, etc.. In this document, we demonstrate this process using MobaXterm.

1. Open SFTP session in the MobaXterm (see session 2 Transferring files to TARA for instruction)

2. Drag file/directory from TARA and drop at your local machine

8. Checking resource usage and remaining quota

Service Unit

You can see the usage quantity and the remaining amount of the service unit (SU) by using sbalance. You can see SU usage by users by using -d flag (i.e. sbalance -d)

Code Block

[hpcuser@tara-frontend-1 ~]$ sbalance
   Account  Description  Allocation(SU)  Remaining(SU)  Remaining(%)     Used(SU)
-----------------------------------------------------------------------------------
    proj0001        latte        30000000    23140501.00         77.14     23140501
    proj0002        cappu        70000000     1040270.00          1.49     68959730
    proj0003        espre         1000000      990400.00         99.04         9600

Disk quota

You can see the amount of space used and remaining using myquota

Example :

Code Block

[hpcuser@tara-frontend-1 ~]$ myquota
                           Disk Space Limits         File Count Limits
                          usage        quota        usage        quota
                     -----------------------  ------------------------
Home                     25.34G          50G        17841       300000
proj0001-latte           111.3G         200G       943239      2000000
proj0002-cappu           151.3G         200G      1213381      2000000
proj0003-espre            71.3G         200G       647175      2000000

Note
Caution : If your disk usage reach the quota limits either disk space limits or file count limits, you are NOT allowed to create a new file or copy/move a file to your home and project directory.

9. Removing unnecessary files

You can free up your disk space by removing unnecessary files. You can remove file by using rm [file] command and remove directory by using rm -r [directory] command.

Example :

Code Block

[hpcuser@tara-frontend-1 tmp]$ ls -l
total 1
drwxr-xr-x 2 tarauser tarausers 4096 Mar 12 14:15 myfolder
-rw-r--r-- 1 tarauser tarausers   15 Mar 12 14:15 test_file.dat
[hpcuser@tara-frontend-1 tmp]$ rm test_file.dat
[hpcuser@tara-frontend-1 tmp]$ ls -l
total 1
drwxr-xr-x 2 tarauser tarausers 4096 Mar 12 14:15 myfolder
[hpcuser@tara-frontend-1 tmp]$ rm -r myfolder/
[hpcuser@tara-frontend-1 tmp]$ ls -l
total 0

ls -l is the command to list files or directories in the current location

Version	Old Version 5	New Version Current
Changes made by	Viwan Jarerattanachat	Viwan Jarerattanachat
Saved on	Jun 13, 2022	Jul 19, 2022

Versions Compared

Key

Summary for experienced HPC user

TARA System Overview

Workflow

1. Logging in to TARA

2. Transferring file to TARA

3. Checking Software availability

To list all available software

To load software

To list all loaded software

To unload all loaded software

4. Software/Application installation (if applicable)

5. Running Jobs

Check TARA HPC status

Job submission

6. Checking job status

Check job status

Cancel job

7. Transferring files to local machine

8. Checking resource usage and remaining quota

Service Unit

Disk quota

9. Removing unnecessary files

Related articles

Page Comparison

Versions Compared

Key

Summary for experienced HPC user

TARA System Overview

Workflow

1. Logging in to TARA

2. Transferring file to TARA

3. Checking Software availability

To list all available software

To load software

To list all loaded software

To unload all loaded software

4. Software/Application installation (if applicable)

5. Running Jobs

Check TARA HPC status

Job submission

6. Checking job status

Check job status

Cancel job

7. Transferring files to local machine

8. Checking resource usage and remaining quota

Service Unit

Disk quota

9. Removing unnecessary files

Related articles