May 24, 2023 at 7:11 amPernilla Ekborg-TannerSubscriber
I'm running Lumerical on a HPC cluster and get terrible scalability when running multiple concurrent simulations. If I run something like this on a compute node:
mpirun -n 3 ~/opt/lumerical/v231/bin/fdtd-engine-ompi-lcl -t 1 test1.fsp &
mpirun -n 3 ~/opt/lumerical/v231/bin/fdtd-engine-ompi-lcl -t 1 test2.fsp &
I can see that 6 processes are started that each run on about 50% CPU efficiency. If I where to run 4 simulations at once, the CPU would drop down to 25%. As a result, the wall time per simulation is greaty increased. I run on a single node, and this happens no matter of how many cores I have available. I do not have the same issue on my local workstation where I use the built-in MPICH2 and the MPICH2-nem fdtd engine. Are there any known issues using OpenMPI? Do you have any idea of what might be the issue? Me and one of the cluster technicians have been at this problem for weeks.
The cluster runs RHEL8 and OpenMPI 4.1.4 and I run Lumerical 23.1.
Thanks in advance and best regards,
May 26, 2023 at 10:26 pmLito YapAnsys Employee
Have you tried running with the bundled MPICH2 nemesis on the cluster?
Or run with the FDTD executable to run all your simulations at once: (i.e. run 4 simulations concurrently on 1 node with 16 cores)
/[installpath]/lumerical/v231/bin/fdtd-engine-mpich2nem -t 4 sweepfile1.fsp sweepfile2.fsp sweepfile3.fsp sweepfile4.fsp
Or run with Intel MPI: > Running simulations using terminal in Linux (Using Intel MPI) – Ansys Optics
Otherwise, the best way to run concurrent FDTD simulations is to run 1 simulation on 1 machine/node.
- You must be logged in to reply to this topic.
Boost Ansys Fluent Simulations with AWS
Computational Fluid Dynamics (CFD) helps engineers design products in which the flow of fluid components is a significant challenge. These different use cases often require large complex models to solve on a traditional workstation. Click here to join this event to learn how to leverage Ansys Fluids on the cloud, thanks to Ansys Gateway powered by AWS.
Earth Rescue – An Ansys Online Series
The climate crisis is here. But so is the human ingenuity to fight it. Earth Rescue reveals what visionary companies are doing today to engineer radical new ideas in the fight against climate change. Click here to watch the first episode.
Subscribe to the Ansys Blog to get great new content about the power of simulation delivered right to your email on a weekly basis. With content from Ansys experts, partners and customers you will learn about product development advances, thought leadership and trends and tips to better use Ansys tools. Sign up here.
- Why am I getting “process exited without calling finalize”, and how do I fix it?
- “Import optical generation” or “delta generation rate”?
- Error: addfdtd is not a valid function or a variable name
- Error on Lumerical device
- Using a license file on a new license server
- Ansys Insight: Transmission results greater than one
- Ansys Insight: Diverging Simulations
- Finding your Ansys (or Lumerical) account number
- Is there a Lumerical script command to output the Simulation and Memory requirements?
- Ansys Insight: Why my simulation result is different from published paper or experiment?
© 2023 Copyright ANSYS, Inc. All rights reserved.