Learning Forum

[helen.durand]

We are having an issue with the computation time of our ANSYS simulation on a high-performance computing cluster. Here is some data:

Using 1 node with a different number of cores, the result below indicates that 32 cores would be the fastest.

---------------------

64 cores: 5m30s

60 cores: 6m28s

48 cores: 4m54s

32 cores: 3m57s

24 cores: 4m33s

16 cores: 5m11s

12 cores: 5m12s

8 cores: 6m33s

4 cores: 8m41s

---------------------

If we double the number of nodes and cores, the result below indicates that this would make the simulation slower.

---------------------------------------

1 node 24 cores 1m45s

2 nodes 48 cores 5m59s

---------------------------------------

Is it expected that ANSYS performs much worse like this as more cores are used? Are there things we can do to take advantage of the number of

[Rob]

In general more cores means faster simulations up to a point. That point being where the network traffic increase outweighs the extra computational power. For the above did you use the same case for both tests? If so, why does the 24 core test show so much variation? Is anyone else doing anything on the system, eg virus scans, backup, running another job?

[helen.durand]

Thank you for the suggestions, Rob.
With another run of a simulation, our cluster engineer stated that on one of the nodes:
36 cores ran in 2m1.921s
24 cores ran in 1m36.592s
16 cores ran in 1m57.335s
This is another newer Intel node that has a peak around 24 or so cores...this really looks to me like you have a limit to how much this model (the one weÔÇÖve been benchmarking) can be parallelized. IÔÇÖve never seen these modern nodes have problems with inter-processor communication on linearly scaling work."
Do you know if there are things that could lead to an ANSYS simulation not parallelizing well in the setup we have done? For example, would certain geometries be split up in bad ways that do not parallelize effectively, or is there anything else we might have done wrong in the setup that is preventing our model from being able to be run effectively on many cores?
Thank you Research Group Discovery

[Rob]

Partitioning can have an effect but Metis is usually pretty good. Multiphase may not parallelise quite as well depending on what's going on (ie where the free surface is, where the particles are etc).
The most common issue is a message/memory bottle neck on the hardware. They're not designed to carry the data traffic so as you pass a certain point the chips appear to slow down. We used to see this as we passed 8 or 12 cores on a chip: they're now putting 24-48 cores on a chip but still haven't improved the message passing.