Running ANSYS on TARA -- though RSM
- Somrath Kanoksirirath
I. Setup ANSYS RSM
The following steps are to establish connections between a local machine (Windows OS) and the TARA HPC for submitting ANSYS jobs through ANSYS Remote Solve Manager.
Official guide: ANSYS Remote Solve Manager User's Guide
Last update: Nov 2023
1. Checklist
1.1 ANSYS Remote Solve Manager (RSM)
Please confirm that ANSYS RSM has been installed on your local machine by typing 'RSM' in the search field on your taskbar. It should be the same version as ANSYS Fluent/Mechanical.
1.2 PuTTY (with plink.exe, pscp.exe and puttygen.exe)
Please confirm that the software have been installed on your local machine by typing 'plink' and 'putty key' in the search field on your taskbar.
1.3 An access to TARA terminal
We will assume that you can open a TARA terminal. If not, please follow TARA Getting started [EN] | 1. Logging in to TARA.
2. Passwordless connection for RSM
This step is to generate a SSH key pair, which will be used by RSM in accessing to TARA without requesting password from users.
2.1 Generate a SSH key pair
Open PuTTY Key Generator.
Change key type to ‘ECDSA' and change curve to ‘nistp521'.
Click 'Generate' then moving your mouse randomly.
Click 'Save private key' and 'OK' to save your private key (.ppk) to a directory without specifying any passphrase.
For security reason, we recommend creating 'C:\Users\XXX\.ssh\' folder and putting it inside, where xxx is your username on your local machine.
As a backup, you could save the corresponding public key (.pub) to the same location.
Copy the whole text generated inside the 'Key' box under 'Public key for pasting into OpenSSH authorized_keys file', then follow the next step.
Path to the private key (.ppk) and its name MUST NOT contain spaces.
2.2 Put your public key to TARA
Open a TARA terminal.
(If not already exists) Create $HOME/.ssh directory by typing
mkdir $HOME/.ssh, then press ENTER.Edit $HOME/.ssh/authorized_keys by, for example, typing
nano $HOME/.ssh/authorized_keysMake a new line and paste the copied public key from the last step of 2.1.
'Ctrl + O' to save it, then 'Ctrl+X’ to exit.
2.3 Define KEYPATH environment variable to your private key for RSM
Typing 'env' in the search field on your taskbar then select 'Edit environment variables for your account'.
Click 'New…' inside User variables box (preferred for security reasons).
Define a new variable name 'KEYPATH'
Click 'Browse File…' and go to your private key (.ppk) generated in Step 2.1.
Click 'Open' then 'OK' to save the setting.
2.4 Check that the KEYPATH is working
To check that the above steps were done correctly, open a cmd (Windows Command Prompt) and type
plink.exe -i %KEYPATH% your-tara-username@tara.nstda.or.thwhere ‘your-tara-username' is the one used to access TARA in Step 2.2, then Enter (repeatedly if needed).
If you can access to the TARA HPC without using any password, then enter 'exit' to log out and proceed to the next step.
If 'Host does not exist' is displayed, please ensure that the local machine is connected to NSTDA VPN.
If any password is requested by cmd, press 'Ctrl+C' to stop, then enter command
echo %KEYPATH%→ If exactly '%KEYPATH%' is displayed, then close the cmd and redo Step 2.3 again.
→ If a path is shown but is not the path to your private key (.ppk) then close the cmd and correct it in Step 2.3.
→ If the path to your private key is shown but contain spaces, then move and/or rename your private key such that the resulting path does not contain spaces. Also, close the cmd and update KEYPATH variable in Step 2.3.
→ If the path to your private key is shown and it does not contain any spaces, then you may have define a passphrase for your SSH key pair in Step 2.1, please redo Step 2.1-2.3 again or Contact Us if the problem persists.
3. Loading ANSYS by default
The below steps are required for RSM to see ANSYS program when accessing to TARA HPC.
Open a TARA terminal.
Edit $HOME/.bashrc.
nano $HOME/.bashrc'Alt + /' to go to the last line.
Make a new line, then copy and paste the below text.
function loadANSYS(){
module purge
module load motif/2.3.8-GCCcore-10.3.0 # Required by Mechanical
module load libGLU/9.0.1-GCCcore-10.3.0 # Required by Mechanical
module load ANSYS/2022.2
export KMP_AFFINITY=disabled # Required by ansys222 in DM, Hybrid mode
export AWP_ROOT231=${AWP_ROOT222} # Required by Ansys.Solvers.Explicit.Launcher.exe
#export ANS_SEE_RUN_COMMAND=1
#export ANS_MPI_VERBOSE=1
}
loadANSYS
'Ctrl + O' to save it, then 'Ctrl+X’ to exit.
Adding # to comment a line. User is advised to add # in front of 'loadANSYS' when not using RSM.
If you would like to use other ANSYS version on TARA, i.e., rather than 2022 R2, please change line 5 and remove line 8 accordingly.
4. RSM configuration
Open 'RSM configuration’ of the ANSYS version you specified in Step 3.
4.1 Add ‘TARA-Devel’
Click 'Add HPC Resource…' icon as pointed in the figure above.
4.1.1 HPC Resource
Enter the information as shown below.
Substitute in ‘your-tara-account’ (HPC project) for SU to be charged from.
It should either look like ‘preXXXX' or ‘projXXXX’.
After substituting it in, there should be a space both before '-t’ and after '-A’.Substitute in 'your-tara-username' that used and verified in Step 2.2, 2.4 and 3.0.
Click ‘Apply' at the bottom.
4.1.2 File Management
Click 'File Management' at the top.
Enter the information as shown below and substitute in ‘your-project-home-directory', which should look like '/tarafs/data/project/XXXXX’, or other directory/disk space for ANSYS to run.
Checking the 'Keep job files in staging directory when job is complete’ box is optional.
Click ‘Apply' at the bottom.
4.1.3 Queues
Click 'Queues' at the top.
Click 'Import/Refresh HPC Queues' (If an error occurs, please revisit Step 2.4 and Step 3).
Check ‘Devel', then click 'Apply' at the bottom.
4.1.4 Submitting a test job
You should be able to click the ‘Submit' button of the column ‘Test’ and row 'Devel’.
Click 'Submit' to send a test job by RSM to TARA.
If the job is completed successfully, the blue question mark will be replaced by a green check mark.
If the job ends unsuccessfully, please click the adjacent report cell to view the log then correct the previous steps accordingly, or Contact Us if the problem persists.
4.2 Add 'TARA-Compute'
Repeat Step 4.1 again, but type/check
‘TARA-Compute' instead of 'TARA-Devel’ on HPC Resource page
'-t 120:00:00' instead of '-t 02:00:00’ on HPC Resource page
'Compute' instead of 'Devel’ on Queues page
‘TARA-Devel' is recommended for testing, before submitting actual jobs to ‘TARA-Compute’.
4.3 Add ‘TARA-Memory’ (Optional)
Repeat Step 4.2 again, but type/check
‘TARA-Memory' instead of 'TARA-Compute’ on HPC Resource page
‘Memory' and 'Memory-preempt' instead of 'Compute’ on Queues page
'TARA-Memory' is NOT recommended as SU will be charged 1.25 times more.
5. SSH communication among TARA nodes
This step is crucial for running ANSYS across TARA nodes. Otherwise, your job will stuck at an initial stage.
The below steps are required for running ANSYS across TARA nodes, e.g., requesting >40 compute cores.
Open a TARA terminal (the same username as in Step 3)
Generate a key pair on TARA
cd $HOME
ssh-keygen -t rsa -b 4096 -C "@tara.nstda.or.th"Press Enter repeatedly.
(That is, do not set any passphrase, except type 'y' if it asks we want to replace the existing key or not)
Use ls -l $HOME/.ssh/ to check that the key pair was successfully generated.
Execute the command below, after replacing '[username]'
ssh-copy-id -i .ssh/id_rsa.pub [username]@tara.nstda.or.thEnter your password, used when opening the TARA terminal, of that username.
II. Running ANSYS Fluent
Example 1: A setup for 40 CPU cores on TARA-Compute
Example 2: A setup for doing parametric analysis on TARA-Compute: 2 jobs per run, 3 tasks per job and 4 CPU cores per task, which will use 3*4+1 = 13 CPU cores per job.
III. Running ANSYS Mechanical
Example 1: A setup for 40 CPU cores, 8 processes with 5 threads per process, on TARA-Compute.
For user supports, contact your ANSYS provider such as
CAD-IT Consultants Pte Ltd
202 Le Concord Tower, Unit 1602, 16th Floor, Ratchadapisek Rd, HuayKwang, Bangkok 10310, Thailand.
Tel. 02-645-3127 9 Email: thailand@caditglobal.com
For technical supports regarding HPC job submission,
Contact Us
ThaiSC support service : thaisc-support@nstda.or.th