ANSYS Mechanical Installation FAQs

This Guide will server as a general overview on how to configure and use ANSYS Mechanical version 12 running on Microsoft Windows HPC 2008. This document will assume that you have already installed and configured Microsoft Windows HPC 2008.

TABLE OF CONTENTS

Post Installation Steps (After ANSYS is installed)

Overview of Windows HPC Server 2008 and ANSYS

Submission Methods

Remote Solve Manager

 Post Installation Steps

After the ANSYS software is installed on the head node, perform the following post installation steps.

WARNING Windows 2008 Server HPC 2008 uses the Windows Vista system of permissions which means to run any sort of Clusrun command you will need to run the program in administrator mode (this includes the command prompt). This is even if you as a user are an Administrator. You do this by right clicking on the program you want to use and selecting run as administrator.

  1. Turn of USER ACCOUNT CONTROL (UAC) (This batch file will not run if you have UAC enabled)
    1. Control Panel
    2. User Accounts
    3. User Turn User Account Control On or Off
  2. Run the C:\Program Files\Ansys Inc\v120\ansys\winHPC\Config_ANSYSMech.bat script from the head node to configure ANSYS Mechanical on all compute nodes.

NOTE: The value of the ANSYS120_PRODUCT environment variable may require modification depending on what products are licensed to you. It is currently set to ANE3FL in the Config_ANSYSMech.bat .

NOTE: It is recommended that you perform a reboot of the entire cluster after the above step so that the ANSYS environment variables are recognized by all ANSYS applications on all compute nodes.

 Testing the Configuration

For general usage instructions please see the section entitled Windows HPC Server 2008 GUI with XML Templates.

Each ANSYS Mechanical user should run the following commands to set up their working directory on all nodes.

  • clusrun mkdir C:\Temp\%USERNAME%\Work
  • Cache your password by running the setmpipassword.bat script.
  • Copy the C:\Program Files\Ansys Inc\v120\ansys\winHPC\ RUNANSYS.xml to a directory of your choice and copy the runansys.bat, spar.inp and pcg.inp files to C:\Temp\%USERNAME% on the head node.
  • Run the HPC Cluster Manager and click on Job Management or run the HPC Job Manager and click on "Create New Job from Description File". Open the RUNANSYS.xml file and click on "Submit".

NOTE: C:\Temp\%USERNAME%\Work will contain the ANSYS scratch files. Any files that you want to save should be copied back to the head node, e.g. \\%CCP_SCHEDULER%\Temp\%USERNAME%.

NOTE : The procedure described here is also used to run LS-DYNA MPP.  In this case, replace RUNANSYS.xml and runansys.bat in the above procedure with RUNLSDYNA.xml and  runlsdyna.bat.

 Overview of Windows HPC Server 2008 and ANSYS

 Windows HPC Server 2008 Workflow

Windows HPC Server 2008 has the following workflow for the submission of jobs to a cluster:

In this guide the Job Template box highlighted in Red is the functionality we are showing you how to use. If you assistance implementing the functionality contained in the Submission Filters or Activation Filters then you will need to contact your Microsoft Support Representative.

 How Job Submission works

Windows HPC Server 2008 splits the job submission process into 2 parts:

  • Jobs
  • Tasks

The Job is an overall unit of submission. It will determine the overall amount of resources that are available to the tasks below it. You can have multiple tasks per job.

A Task is an individual process with a job. In ANSYS Mechanical this would be a single analysis submission.

  • You can have multiple tasks per job.
  • Tasks use the basic unit of submission that you assigned to them with so if your job uses nodes you will assign individual tasks in the unit of nodes.
  • Tasks can be independent of each other and so can run side by side within the resources defined by the job as long as those resources are available.
  • You can create dependent tasks. Dependent tasks work in groups so group ones task must be completed before group twos tasks can start.

 When to Use Nodes, Sockets or Cores

In Windows HPC Server 2008 you can choose one of three basic units of allocation (CCS 2003 only had Core allocation). Microsoft refers to this as “resource type” or “granularity”.

  • Core: Refers to a single processing unit capable of performing computations.  A core is the smallest unit of allocation available in HPC Server 2008.
  • Socket: Refers to collection of cores with a direct pipe to memory.  Each socket contains one or more cores.  Note that this does not necessarily refer to a physical socket, but rather to the memory architecture of the machine, which will depend on your chip vendor.
  • Node: Refers to an entire compute node.  Each node contains one or more sockets

In general, the rule as specified by Microsoft is:

  • Use core allocation if the application is CPU intensive; the more processors you can throw at it the better!
  • Use socket allocation if memory access is what bottlenecks your application's performance.  Since how much data can come in from memory is what limits the speed of the job, running more tasks on the same memory bus won't result in speed-up since all of those tasks are fighting over the path to memory.
  • Use node allocation if some node-wide resource is what bottlenecks your application.  This is the case with applications that are relying heavily on access to disk or to networks resources.  Running multiple tasks per node won't result in a speed-up since all of those tasks are waiting for access to the same disk or network pipe. Setting node allocation makes the task running on that node exclusive so no other task can use it at the same time.

However, Microsoft also added many commands to MPIEXEC (This is the executable file which is the mechanism to start a parallel MPI application) which can be used to gain finer control over the resource allocation.

Some examples of these flags are listed below:

Flag\Switch Explanation
-n Specifies a number of processors per unit picked. So if you pick cores and picked 4 cores and assigned –n to 8 then it would put 2 processes on each core. If you picked 4 sockets and set -n to 8 it would put 2 processes per socket. If you picked 4 nodes and set -n to 16 it would assign the cores in a round robin fashion around the nodes.
-cores Specifies a “high water” of the number of processes that can be used on a single node. If you specify it by itself it will run that many cores per node.
-affinity Each rank will get affinity to one of the cores on its assigned node, so that the two ranks sharing a node cannot step on each other's toes.  If the nodes in question have a NUMA architecture, the ranks on each node will automatically be placed on separate NUMA nodes.

 Submission Methods

 Submitting a Mechanical Job at the Command Line (per Microsoft)

Job Submit /numnodes:4 mpiexec [mpiexec arguments] [ANSYS path]ANSYS120.exe [ANSYS arguments]

However we do not give our customers direct access to MPIEXEC. So you cannot call it directly as per the above example! However, with version 12.0 we have implemented two flags and an environment variable which should help you to be able to use MPIEXEC.

Flag\Environment Variable Explanation
-np Same as Microsoft –n implementation (see above) when used with Windows HPC Server 2008.
-cores Same as Microsoft –cores or –c implementation (as above). Only applicable on Windows HPC Server 2008. This can only be used when Nodes has been picked as your unit of allocation.
(Environment Variable) MPIOPTS

This is the preferred method: Allows the user to set an environment variable called MPIOPTS which will pass a string directly to MPIEXEC (you need no flag on the command line as it will just pick it up)

We recommend NOT using –np or the –cores flags if you are using this environment variable

 Submitting a Mechanical Job at the Command Line (per ANSYS)

Job Submit /numnodes:4 [ANSYS path]ANSYS120.exe [ANSYS Arguments]

Question: Under what circumstances would you want to more finely control how the resources are being used?

Answer: This comes down to two major issues:

  1. Memory bandwidth on the processors
  2. Amount of memory available on the nodes

If you have the current generation of processors, Nehalem and Shanghai, ignore this section. If you are running on the older XEON processors, for example, Woodcrest, Harpertown, etc., these processors do not have enough memory bandwidth to deal with all of the cores on a quad core processor being used all at once. Hence, using all 4 cores produces, in the best case, a reduced speedup for the job than is expected and in worst case actually slows performance dramatically. To get around this problem it is better to NOT use all the cores on a dual or quad core system but instead to use half the cores on the processor. For a more in-depth discussion on this subject please see the Performance Guide located in ANSYS Help > Mechanical ADPL (Formerly ANSYS) > Performance Guide

In other circumstances, it may be that there is not enough memory per node to run your problems. In cases like this it may be better to more finely control your resource allocation so you use less cores per node so that you have enough memory per node to run your larger jobs.

 Submission through ANSYS Workbench Simulation

  1. Open Workbench and load the relevant project and enter simulation.
  2. Click on the arrow next to the lightening icon in the Menu bar and pick your remote server name (in the below example “Cray Cluster” is our HPC2008 cluster name that we defined when adding remote in the above instructions). This will send the job to that cluster.
  3. You can then check in the RSM Manager how the job is progressing.
  4. When the job has finished it should look like this in the RSM Manager
  5. In your simulation environment you should now get some green down arrows indicating that the results are ready to be fetched. Right click on solution and pick “Get Results” to retrieve them.
  6. In the RSM Manager it should now show you that it has retrieved the files.
  7. You can now post process in Simulation.

 ANSYS Mechanical Job Guidelines

  • Core allocation should only be used with the new generation of processors such as Nehalem or Shanghai and where you are sure you have enough memory per core to run your job.   We do not recommend at this time stacking more than 1 job on a single node and therefore recommend you make any jobs exclusive when using cores allocation.
  • Socket allocation should be used to easily space large jobs on a limited amount of cores per node. You can customize this allocation using –np and –affinity.
  • Node allocation should be used in most circumstances as it gives the greatest amount of Flexibility using –cores, -np and –affinity.

 Examples of using Cores, Sockets or Nodes

NOTE: If you are only specifying a base “unit of allocation” then you do not need to pass –np and/or –cores to the command line or set anything in the MPIOPTS Environment Variable. For example, if you want to use 2 processors but want the unit of allocation to be sockets, simply set sockets in the GUI to 2 and it will spawn ANSYS in Parallel with 2 processes running. This works the same for core allocation and node allocation.

Example 1:

You have the older generation processors that do not have enough memory bandwidth to use all 4 processors in a quad core effectively. You have 4 nodes with 32 processors total but because of this limitation you want to use 4 per node and make sure they are spaced 2 per quad core.

To do this you would do the following through the GUI:

  1. Set the Resource Allocation to Nodes and specify 4 nodes.
  2. Add a task and specify all 4 nodes
  3. Setup the working directories and output files.

In the task command line add:

  1. Type at the command line:
    • Set MPIOPTS=-affinity
    • [ANSYS path]ANSYS120.exe –cores 4 [additional ANSYS Arguments]

Example 2:

You have older generation processors that do not have enough memory bandwidth to use all 4 processors in a quad core effectively. You want to assign the cores in a round robin fashion (so 1 to 1 node, then one to another and then loop around again) to a maximum of 16 processors over the 4 nodes.

To do this you would do the following through the GUI:

  1. Set the Resource Allocation to Nodes and specify 4 nodes.
  2. Add a task and specify all 4 nodes
  3. Setup the working directories and output files.
  4. Type at the command line:
    • Set MPIOPTS=-affinity
    • [ANSYS path]ANSYS120.exe –np 16 [additional ANSYS Arguments]

Example 3:

You have older generation processors that do not have enough memory bandwidth to use all 4 processors in a quad core effectively. You have 4 nodes with 32 processors total but because of this limitation you want to use 4 cores per node and make sure they are spaced 2 per quad core. You have the additional limitation of only having 12 Mechanical HPC Licenses available for use (total 14 processors). So you need to specify 14 processors total but a maximum of 4 per node.

To do this you would do the following through the GUI:

  1. Set the Resource Allocation to Nodes and specify 4 nodes.
  2. Add a task and specify all 4 nodes
  3. Setup the working directories and output files.
  4. Type at the command line:
    • Set MPIOPTS=-cores 4 –n 14 -affinity
    • [ANSYS path]ANSYS120.exe [additional ANSYS Arguments]

Example 4:

You have the new generation of processors that do have enough memory bandwidth but each node only has 8 GB RAM and the amount of RAM that each core needs for the run is ~2 GB (which is greater than 1 GB available). This would mean it would not be able to run fully in-core. You want to use 16 cores and each blade has 8 cores.

To do this you would do the following through the GUI:

  1. Set the Resource Allocation to Nodes and specify 4 nodes.
  2. Add a task and specify all 4 nodes
  3. Setup the working directories and output files.
  4. Type at the command line:
    • Set MPIOPTS=-cores 4
    • [ANSYS path]ANSYS120.exe [additional ANSYS Arguments]

 Submission methods with ANSYS and Windows HPC Server 2008

We support the following options to be used with Windows HPC Server 2008:

  • Windows HPC Server 2008 GUI with XML Templates
  • Command Line with XML Templates
  • RSM with Simulation in Workbench

We do not support custom scripting with the Windows HPC Server 2008 API. You will need to contact Microsoft for information on how to do this and we will not be able to assist with any problems associated with it.  

 Submission Basics

The recommended way of submitting jobs to the cluster is to utilize XML Templates and use a file called RUNANSYS.XML . Note: This file resides in the ANSYS installation directory, ANSYS Inc\v120\ansys\WinHPC .

We recommend using XML Templates because it is very easy to make a typing mistake when submitting jobs which can lead to frustration in trying debug. After importing an XML template you can always change any values manually and create another template from the new values.

 Windows HPC Server 2008 GUI with XML Templates

To submit a Mechanical job using the Windows HPC Server 2008 GUI follow the following steps:

  1. Open the HPC Job Manager
  2. Under the Actions Menu, Job Submission, select “Create New Job from Description File” and select the XML file, RUNANSYS.xml .
  3. Verify Job Details and Task List Settings and then click “Submit”

NOTE: After the job is complete, you may want to clean up your working directory on all nodes BY running clus_scrub_workdir.bat file. This file resides in the ANSYS installation directory, ANSYS Inc\v120\ansys\WinHPC.

NOTE: C:\Temp\%USERNAME%\Work will contain the ANSYS scratch files. Any files that you want to save should be copied back to the head node, e.g. \\%CCP_SCHEDULER%\Temp\%USERNAME%.

NOTE : The procedure described here is also used to run LS-DYNA MPP.  In this case, replace RUNANSYS.xml and runansys.bat in the above procedure with RUNLSDYNA.xml and  runlsdyna.bat.

 Command Line with XML Templates

To submit jobs using the Command Line use following steps:

  1. Open the command prompt
  2. Type: job submit –jobfile [insert xml file here]

For a full list of commands that can be used with Job submit type “job submit -?”

You can also use the HPC PowerShell with similar commands to do this as well but that will not be discussed in this document. Please refer to the Microsoft documentation for instructions.

 Configuring the Remote Solver Manager and ANSYS Workbench for Windows HPC Server 2008

 RSM Setup

For instructions on how to setup the RSM Manager please refer to the documentation

Choose the Start menu -> All Programs -> ANSYS 12.0 -> Help -> ANSYS Help and then go to the section in the contents entitled “The Remote Solve Manager (RSM)” and follow the setup procedure from there on the head node of your cluster.

For specific discussion on HPC2008 see the subsection “Integration with Microsoft Compute Cluster”.

 Configuring Multiple Network Interface Cards (NIC)

If after step 4 below you still cannot see the server (it will have a cross through the server name in the RSM client menu) and it can see it over the network this is probably due to you having Multiple Network Interface Cards. This will probably be the norm for most of the Windows HPC 2008 clusters as you will have at least an enterprise and private network (if not an enterprise, private and application if you are running infiniband). Please see ANSYS Help > Remote Solve Manager (RSM) > Administration > Configuing Multiple Network Interface Cards (NIC)

 Connecting a Client Machine RSM to the Cluster Head nodes RSM

  1. Open the RSM on the client Machine by going to the Start menu -> All Programs -> ANSYS v12.0 -> RSM -> RSM Manager
  2. Click on the Tools menu choose “options”.
  3. Under the “Solver Managers” type in the name of the head node of the cluster and click the add button (make sure when it is created it is ticked otherwise it won't show in the RSM menu).
  4. Click OK to return to the RSM window. You should now see the same layout as you do on the server.
  5. Go into Workbench Simulation and click on the Tools menu and choose “Solve Process Settings”.
  6. Click Add Remote and give the remote server a name.
  7. In the box labeled “Computer Settings” add the name of the name of the head node.
  8. You should then click on the box labeled “Queue” and pick the queue you assigned on the RSM Server and pick the relevant license that you the user to run with.
  9. In the Advanced menu choose the box labeled “Distributed ANSYS Solution (if possible)”.
  10. If you want to change the number of processors used (this is for specific jobs when the user wants more or less processors as well in the future) you can change this in the above menu as well.

 Password Reminder

RSM requires that your Windows password be cached with MSCC. If not, MSCC will prompt at job submission. There is currently no way to enter a password in RSM to respond to the prompt. The job will eventually fail after a time-out period.

 Caching the Password

  1. Log into the head node computer and running the following command:
[rsm_install_dir]\bin\mccsubmit.exe localhost

This command submits a test job to the cluster. You are prompted for a password and then asked if the password should be cached. You must repeat this process whenever your Windows password is changed .

Home

Updated - June 08, 2009