April 11, 2022 at 9:06 amhelen.durandSubscriber
We are having an issue with the computation time of our ANSYS simulation on a high-performance computing cluster. Here is some data:
Using 1 node with a different number of cores, the result below indicates that 32 cores would be the fastest.
64 cores: 5m30s
60 cores: 6m28s
48 cores: 4m54s
32 cores: 3m57s
24 cores: 4m33s
16 cores: 5m11s
12 cores: 5m12s
8 cores: 6m33s
4 cores: 8m41s
If we double the number of nodes and cores, the result below indicates that this would make the simulation slower.
1 node 24 cores 1m45s
2 nodes 48 cores 5m59s
Is it expected that ANSYS performs much worse like this as more cores are used? Are there things we can do to take advantage of the number ofApril 11, 2022 at 10:26 amRobForum ModeratorIn general more cores means faster simulations up to a point. That point being where the network traffic increase outweighs the extra computational power. For the above did you use the same case for both tests? If so, why does the 24 core test show so much variation? Is anyone else doing anything on the system, eg virus scans, backup, running another job?
April 22, 2022 at 3:14 amhelen.durandSubscriberThank you for the suggestions, Rob.
With another run of a simulation, our cluster engineer stated that on one of the nodes:
36 cores ran in 2m1.921s
24 cores ran in 1m36.592s
16 cores ran in 1m57.335s
This is another newer Intel node that has a peak around 24 or so cores...this really looks to me like you have a limit to how much this model (the one weÔÇÖve been benchmarking) can be parallelized. IÔÇÖve never seen these modern nodes have problems with inter-processor communication on linearly scaling work."
Do you know if there are things that could lead to an ANSYS simulation not parallelizing well in the setup we have done? For example, would certain geometries be split up in bad ways that do not parallelize effectively, or is there anything else we might have done wrong in the setup that is preventing our model from being able to be run effectively on many cores?
Thank you Research Group Discovery
April 25, 2022 at 1:53 pmRobForum ModeratorPartitioning can have an effect but Metis is usually pretty good. Multiphase may not parallelise quite as well depending on what's going on (ie where the free surface is, where the particles are etc).
The most common issue is a message/memory bottle neck on the hardware. They're not designed to carry the data traffic so as you pass a certain point the chips appear to slow down. We used to see this as we passed 8 or 12 cores on a chip: they're now putting 24-48 cores on a chip but still haven't improved the message passing.
Viewing 3 reply threads
Ansys Innovation Space
- The topic ‘Question on Fluent Scaling on High-Performance Computing Cluster’ is closed to new replies.
Boost Ansys Fluent Simulations with AWS
Computational Fluid Dynamics (CFD) helps engineers design products in which the flow of fluid components is a significant challenge. These different use cases often require large complex models to solve on a traditional workstation. Click here to join this event to learn how to leverage Ansys Fluids on the cloud, thanks to Ansys Gateway powered by AWS.
Earth Rescue – An Ansys Online Series
The climate crisis is here. But so is the human ingenuity to fight it. Earth Rescue reveals what visionary companies are doing today to engineer radical new ideas in the fight against climate change. Click here to watch the first episode.
Subscribe to the Ansys Blog to get great new content about the power of simulation delivered right to your email on a weekly basis. With content from Ansys experts, partners and customers you will learn about product development advances, thought leadership and trends and tips to better use Ansys tools. Sign up here.Trending discussions
- Licensing – Request name does not exist in the licensing pool
- 2022 R1 Ansys Mechanical APDL: QXcbConnection: Failed to initialize XRandr
- error “ansyslmd: Cannot connect to license server system. (-15,10032)
- The analysis systems in Workbench are missing
- Student license renewal
- Mechanical APDL Launcher Options
- how to open files created from newer version by using the older version of ansys
- Problems using 2022/R2
- Ansys Helic Tools Installation Issue: Missing Helic_PDK directory
- Results Probe in ANSYS Workbench
Top Rated Tags
© 2023 Copyright ANSYS, Inc. All rights reserved.Ansys does not support the usage of unauthorized Ansys software. Please visit www.ansys.com to obtain an official distribution.