/
TARA Getting started [EN]

TARA Getting started [EN]

Summary for experienced HPC user

  1. To access TARA, ssh to [username]@tara.nstda.or.th, where [username] is your registered TARA username.

  2. Each user is allocated with 50 GB home directory ($HOME).

  3. The project directory(s) (/tarafs/data/project/[projxxxx]-[short-name]/) has 200 GB of space to be shared among all users in the project; where [projxxxx] is project id and [short-name] is its short name.

  4. There is a scratch directory (/tarafs/scratch/[projxxxx]-[short-name]/) which is a fast read & write via read-intensive SSD. For temporary file only, files will  be removed after 30 days of last access.

  5. Software are managed as modules using Environment module system (Lmod).

  6. TARA usage is subjected to fairshare policy and managed by Slurm job scheduler. Examples of Slurm batch script for specific applications can be found at การติดตั้งและการงานใช้ Application.

Cheat Sheet

Download this document as a PDF

TARA System Overview

An up-to-date overview of TARA system can be found at Resources – ThaiSC.

Workflow

This is a suggested workflow for users when using TARA

1. Logging in to TARA

The connection through VPN is required to access TARA. The instruction how to setup and connect to the VPN can be found in your email.

Accessing TARA via ssh protocol, user can use SSH client such as PuTTy, MobaXterm, etc.. In this document, we demonstrate how to log in to TARA using MobaXterm. 

More information about MobaXterm is available at https://mobaxterm.mobatek.net/.

For macOS or Linux, you can use ssh -Y [username]@tara.nstda.or.th command in the Terminal (YouTube : https://youtu.be/kr0WR46UGuc).

To access TARA, you can follow the steps below (YouTube : https://youtu.be/T6earis8Hqo).

1. Open MobaXterm, a SSH client program

2. Click on Session button to initiate the session page

MobaXterm user interface

3. Click on the SSH button to use SSH protocol and set the configuration as the following details.

Remote host = tara.nstda.or.th  

Specify username = [username]  

Port = 22

Where [username] is your TARA username.

The SSH setting

4. Click OK button to accept the settings. The SSH terminal session will start and prompt for a password. Enter your TARA password.

If this is your first time logging in using MobaXterm, MobaXterm will ask whether you want to save your password.

For a first time login to TARA, you will be asked to set your new password.

  1. enter your Current password

  2. enter your New password

  3. re-enter your New password

5. The connection between SSH session and TARA is now established. You will see the welcome message. All commands executed in this session are run on TARA.

TARA welcome message

2. Transferring file to TARA

Each user is allocated with 50 GB home directory ($HOME) and project directory(s) (/tarafs/data/project/proj[xxxx]-[short-name]/) of 200 GB to be shared among all users in the project.

Note: [projxxxx] is project id and [short-name] is your project short name.

To transfer files to TARA with Graphic User Interface (GUI), use can use FTP client such as FileZilla, WinSCP, MobaXterm, etc..

For macOS or Linux, you can use scp command in the Terminal (YouTube : https://youtu.be/ovQj9dzwE-w).

In this document, we demonstrate this process using MobaXterm (YouTube : https://youtu.be/aSwjYtY1KrA).

1. Open MobaXterm and click at Session button to open Session page.

2. Click at SFTP button to use SFTP protocol and set the configuration as shown.

Remote host = tara.nstda.or.th

Username = [username]

Port = 22

3. Click OK, then SFTP session will start. Now, you can drag and drop files to transfer files between TARA and your local machine.

The SFTP session. The arrow indicates drag and drop file transfer.

3. Checking Software availability

TARA has reasonable number of software installed. These software are managed by module framework using Environment module system (Lmod). (YouTube : https://youtu.be/NAkJv2uVQZA)

The commands in this part are executed in SSH session. If you transferring file using SFTP session, switch to the SSH session by click at SSH tap with the key icon.

To list all available software

Command : module avail

Example :

[hpcuser@tara-frontend-1 ~]$ module avail ----------------------------------------------------------------------------------------------------- /tarafs/utils/modules/modules/all ----------------------------------------------------------------------------------------------------- ANSYS/19.3 EasyBuild/4.1.0 Mako/1.1.0-GCCcore-8.3.0 ScaLAPACK/2.0.2-gompic-2019b (D) fosscuda/2019b Advisor/2019_update5 FFTW/3.3.8-gompi-2019b Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4 Singularity/3.3.0 gcccuda/2019b Autoconf/2.69-GCCcore-8.3.0 FFTW/3.3.8-gompic-2019b (D) NASM/2.14.02-GCCcore-8.3.0 Singularity/3.4.2 (D) gnuplot/5.2.8-GCCcore-8.3.0 Autoconf/2.69 (D) FriBidi/1.0.5-GCCcore-8.3.0 NLopt/2.6.1-GCCcore-8.3.0 Szip/2.1.1-GCCcore-8.3.0 gompi/2019b Automake/1.15.1-GCCcore-8.3.0 GATK/4.1.4.1-GCCcore-8.3.0-Java-11 NSPR/4.21-GCCcore-8.3.0 Tcl/8.6.9-GCCcore-8.3.0 gompic/2019b ...

To load software

Command : module load [software]

Example :

[hpcuser@tara-frontend-1 ~]$ module load Python/3.7.4-GCCcore-8.3.0 [hpcuser@tara-frontend-1 ~]$ python --version Python 3.7.4

To list all loaded software

Command : module list

Example :

[hpcuser@tara-frontend-1 ~]$ module list Currently Loaded Modules: 1) GCCcore/8.3.0 7) Tcl/8.6.9-GCCcore-8.3.0 2) zlib/1.2.11-GCCcore-8.3.0 (H) 8) SQLite/3.29.0-GCCcore-8.3.0 3) binutils/2.32-GCCcore-8.3.0 9) XZ/5.2.4-GCCcore-8.3.0 4) bzip2/1.0.8-GCCcore-8.3.0 10) GMP/6.1.2-GCCcore-8.3.0 5) ncurses/6.1-GCCcore-8.3.0 (H) 11) libffi/3.2.1-GCCcore-8.3.0 (H) 6) libreadline/8.0-GCCcore-8.3.0 (H) 12) Python/3.7.4-GCCcore-8.3.0 Where: H: Hidden Module

To unload all loaded software

Command : module purge

Example :

[hpcuser@tara-frontend-1 ~]$ module purge [hpcuser@tara-frontend-1 ~]$ module list No modules loaded

For more information about LMOD system please see การใช้งาน module ใน TARA Cluster

4. Software/Application installation (if applicable)

If TARA does not have the software(s) you need installed, you can install them by yourself in either your home directory or your project directory. Please make sure that you have the software license and the license term allows to run the software on TARA.

We support EasyBuild (https://easybuild.io/), see การใช้งาน local module ใน TARA Cluster for the step by step instruction to install software as a local module. You can also check การติดตั้งและการงานใช้ Application for some specific software installation/setup instruction.

5. Running Jobs

Check TARA HPC status

Before submitting your job to run on TARA, you can check available computational resources or status of partitions by using sinfo command.

Example :

[hpcuser@tara-frontend-1 ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST devel up 2:00:00 2 idle tara-c-[001-002] compute* up 5-00:00:00 1 mix tara-c-034 compute* up 5-00:00:00 57 alloc tara-c-[003-033,035-060] memory up 5-00:00:00 5 mix tara-m-[001-003,005-006] memory up 5-00:00:00 2 alloc tara-m-[004,007] memory up 5-00:00:00 1 idle tara-m-008 memory-preempt up 5-00:00:00 5 mix tara-m-[001-003,005-006] memory-preempt up 5-00:00:00 2 alloc tara-m-[004,007] memory-preempt up 5-00:00:00 1 idle tara-m-008 gpu up 5-00:00:00 2 idle tara-g-[001-002] dgx up 5-00:00:00 1 idle tara-dgx1-003 dgx-preempt up 5-00:00:00 2 idle tara-dgx1-[001-002]

where

Item

Description

Item

Description

PARTITION

partition name

AVAIL

partition status (up : working, down : not working)

TIMELIMIT

maximum time that a job can run on this partition

NODES

number of nodes in the partition

STATE

partition availability (idle : free, alloc : busy, mix : some core(s) in node is free)

NODELIST

list of nodes in the partition

Job submission

TARA usage is subjected to Fair-share policy and managed by Slurm job scheduler. To submit a job follows the following steps.

1. Create bash script using vi sbatch_script.sh (or gedit sbatch_script.sh , nano sbatch_script.sh)

2. Specify the details in the bash script according to the template provided below.

#!/bin/bash #SBATCH -p compute # Specify the partition or machine type used [compute/memory/gpu] #SBATCH -N 1 --ntasks-per-node=40 # Specify the number of nodes and the number of core per node #SBATCH --mem=16GB # Specify memory to use in the node or specify #SBATCH -t 00:10:00 # Specifies the maximum time limit (hour: minute: second) #SBATCH -J my_job # Specify the name of the Job #SBATCH -A projxxxx # Specify Project account which will be received after Register ## purge and load modules module purge # unload all modules as they may have previously been loaded. module load intel # Load the module that you want to use. This example is intel ## your script srun my_hpc_script # Run your program or executable code

3. Use sbatch command followed by your bash script name to submit your job to SLURM

sbatch sbatch_script.sh

sbatch ,sh, bash are not the same command. To submit the job, it is an obligation to use sbatch .

If you want to try to submit a sample job we recommend to check out our https://youtu.be/8k-NMCIAlZw or https://youtu.be/eZEmCK5MwlI. For DGX partition, you can try https://youtu.be/JwdK7lifI_w.

The example of Slurm batch script for specific applications can be found in the การติดตั้งและการงานใช้ Application.

6. Checking job status

Check job status

To check your job status, using myqueue command.

Example :

[hpcuser@tara-frontend-1 ~]$ myqueue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 151602 compute test3 hpcuser PD 0:00 1 (Priority) 140215 compute test2 hpcuser R 03:25:08 1 tara-c-055 149496 compute test1 hpcuser R 2-02:11:27 1 tara-c-042

Item

Description

Item

Description

JOBID

ID of the specific job

PARTITION

partition which job is running/queuing on

NAME

name of the specific job

USER

user who owns/run the job

ST

Job status (PD : pending, R : running)

TIME

elapsed time of the running job

NODE

the number of node used in the run

NODELIST(REASON)

list of node used to run (reason why the job is pending) see สถานะ Pending (PD) for more information

Cancel job

To cancel the job in the queue, using scancel [JOBID] command

Example : scancel 140215

7. Transferring files to local machine

To transfer files from TARA to your local machine, you can use FTP client such as FileZilla, WinSCP, MobaXterm, etc.. In this document, we demonstrate this process using MobaXterm.

1. Open SFTP session in the MobaXterm (see session 2 Transferring files to TARA for instruction)

2. Drag file/directory from TARA and drop at your local machine

8. Checking resource usage and remaining quota

Service Unit

You can see the usage quantity and the remaining amount of the service unit (SU) by using sbalance. You can see SU usage by users by using -d flag (i.e. sbalance -d)

[hpcuser@tara-frontend-1 ~]$ sbalance Account Description Allocation(SU) Remaining(SU) Remaining(%) Used(SU) ----------------------------------------------------------------------------------- proj0001 latte 30000000 23140501.00 77.14 23140501 proj0002 cappu 70000000 1040270.00 1.49 68959730 proj0003 espre 1000000 990400.00 99.04 9600

Disk quota

You can see the amount of space used and remaining using myquota

Example :

[hpcuser@tara-frontend-1 ~]$ myquota Disk Space Limits File Count Limits usage quota usage quota ----------------------- ------------------------ Home 25.34G 50G 17841 300000 proj0001-latte 111.3G 200G 943239 2000000 proj0002-cappu 151.3G 200G 1213381 2000000 proj0003-espre 71.3G 200G 647175 2000000

Caution : If your disk usage reach the quota limits either disk space limits or file count limits, you are NOT allowed to create a new file or copy/move a file to your home and project directory.

9. Removing unnecessary files

You can free up your disk space by removing unnecessary files. You can remove file by using rm [file] command and remove directory by using rm -r [directory] command.

Example :

[hpcuser@tara-frontend-1 tmp]$ ls -l total 1 drwxr-xr-x 2 tarauser tarausers 4096 Mar 12 14:15 myfolder -rw-r--r-- 1 tarauser tarausers 15 Mar 12 14:15 test_file.dat [hpcuser@tara-frontend-1 tmp]$ rm test_file.dat [hpcuser@tara-frontend-1 tmp]$ ls -l total 1 drwxr-xr-x 2 tarauser tarausers 4096 Mar 12 14:15 myfolder [hpcuser@tara-frontend-1 tmp]$ rm -r myfolder/ [hpcuser@tara-frontend-1 tmp]$ ls -l total 0

ls -l is the command to list files or directories in the current location

Related articles