February 21, 2023 at 8:57 pmBoqin ZhaoSubscriberHi! We have recently encountered a weird issue when running Lumerical FDTD simulations on our server. When we run multiple simulations simultaneously (either by the same user or multiple different users), the simulation time is significantly increased. For example, running 2 simulations at the same time, each simulation will take more than 10 times longer to finish compared to running one simulation alone. In theory, the simulation time should only double since each simulation occupies 50% CPU, and was the case previously.
February 22, 2023 at 12:14 amGuilin SunAnsys Employee
May I know how many available licenses? It seems the simulation files share the same one license. Another possibility is, the memory might be at its maximum, when two or more simulations are run simutameously.
February 22, 2023 at 12:36 amLito YapAnsys Employee
Which version is having this issue when running concurrent FDTD simulation as compared to which older version as per your post;
In theory, the simulation time should only double since each simulation occupies 50% CPU, and was the case previously.
FDTD performance and utilization is not linear. FDTD Is very CPU and memory intensive. It will try to take up all of the available memeory/system bandwidth on the machine. It may be only running with 50% of the CPU cores but it will be running with all avialable memory bandwidth on the machine.
Sometimes, depending on the simulation/sweeps, it would be faster to run them in sequence than running multiple simulations concurrently. We have this KB guide below that might come handy.
February 22, 2023 at 12:59 amBoqin ZhaoSubscriber
Thanks for your replys.
@Guilin Sun and @Lito Yap,
We have two licenses now. The problem occured when two users run their respective simulations at the same time, or one user run two simulations at the same time.
We had only one license a year ago. Back then, Running 2 simulations at the same time simply means that the simulation time would double. It was still working like that after we upgraded to 2 licenses, until maybe 1-2 months ago. We have been and is still using FDTD version 2022 R1.
No matter how many simulations are being run, CPU is being used ~100%. And I can confirm that the memory is not overwhelmed.
Thanks for your attention.
February 22, 2023 at 1:03 amBoqin ZhaoSubscriber
However, you reminded me that perhaps for some reason not all the cores are being used when running multiple simulations, despite the 100% CPU usage. I am not very familiar with these technical terms, but will follow the instruction and check that.
February 22, 2023 at 2:34 amLito YapAnsys Employee
FDTD will try to use all the available memory bandwidth even if it will not use all of the memory or not use all CPU cores. When all of the system's bandwidth is used, it will slow down other running processes on the machine.
A single user can run 2 or more parameter sweeps (concurrent parametric computing) on a single machine - for simple/small sweeps than runs on a few cores and taking only a few MB of memory.
It is not recommended to have more than 1 user run FDTD simulations on the same machine. If 2 users run FDTD, both simulations will slow down significantly depending on the simulation's components/design/requirements.
February 22, 2023 at 6:22 amBoqin ZhaoSubscriber
Okay, thanks. I am a bit disappointed because having two users run simulations at the same time is one important reason why we bought 2 licenses.
However, I am still curious why the simulations will slow down significantly. Could you please elaborate a little bit more? Maybe I can try to avoid certain setup/type of simulations when I want to run multiple at the same time in the future. Thanks!
February 22, 2023 at 5:45 pmLito YapAnsys Employee
Yes you can certainly have 2 user run 2 simulations at the same time but just not recommended to run at the same machine. Each user should be running their simulation on their own local desktop. The license can be shared across other users/computers that is connected to the same local network as the license server running the Ansys License Manager:
>>>How to obtain a floating license – Ansys Optics
This is just how FDTD behaves. It takes up alot of your CPU and bandwidth. And increasing the number of MPI processes/cores to run the simulation is not linear. This will depend on the simulation itself and the machine as shown in our KB:
>>>FDTD Performance Benchmarks – Ansys Optics
You may be able to run small/less complex concurrent simulation/parameter sweeps but larger and more complex simulation jobs that requires more memory and resources should run by itself on one machine.
February 23, 2023 at 10:08 pmBoqin ZhaoSubscriber
I have done a few testings following your suggestions and seem to understand what was going on.
I am not a computer hardware expert. Please forgive me if I say anything wrong.
Our CPU on the server is Intel Xeon E5-2695 V2. It has 48 processors in total. We have been running simulations with the number of processes set to 48, which means that it will use all processes. If another simulation with 48 processes is being run at the same time, each processor will be responsible for both simulations. However, one processor can only handle one task at a time so it has to switch between two simulations. The switching may be slow which causes both simulations to also slow down significantly.
I tried running 2 simulations with each one limited to 24 processes. It turned out that the speed for each process (in Mnode/s) is not significantly slower than one sim with 48 processes.
I think our best strategy moving forward is to limit 24 processes per simulation.
Thanks for your help again.
- You must be logged in to reply to this topic.
Boost Ansys Fluent Simulations with AWS
Computational Fluid Dynamics (CFD) helps engineers design products in which the flow of fluid components is a significant challenge. These different use cases often require large complex models to solve on a traditional workstation. Click here to join this event to learn how to leverage Ansys Fluids on the cloud, thanks to Ansys Gateway powered by AWS.
Earth Rescue – An Ansys Online Series
The climate crisis is here. But so is the human ingenuity to fight it. Earth Rescue reveals what visionary companies are doing today to engineer radical new ideas in the fight against climate change. Click here to watch the first episode.
Subscribe to the Ansys Blog to get great new content about the power of simulation delivered right to your email on a weekly basis. With content from Ansys experts, partners and customers you will learn about product development advances, thought leadership and trends and tips to better use Ansys tools. Sign up here.
- “Import optical generation” or “delta generation rate”?
- Why am I getting “process exited without calling finalize”, and how do I fix it?
- Using a license file on a new license server
- Ansys Insight: Diverging Simulations
- Error: addfdtd is not a valid function or a variable name
- Questions about the calculation of the cross-polarization conversion efficiency of metasurface
- Finding your Ansys (or Lumerical) account number
- Error on Lumerical device
- Ansys Insight: About override mesh in FDTD: its use and settings
- Lumerical – error message when trying to open from Linux terminal
© 2023 Copyright ANSYS, Inc. All rights reserved.