Learning Forum

[Boqin Zhao]

Hi! We have recently encountered a weird issue when running Lumerical FDTD simulations on our server. When we run multiple simulations simultaneously (either by the same user or multiple different users), the simulation time is significantly increased. For example, running 2 simulations at the same time, each simulation will take more than 10 times longer to finish compared to running one simulation alone. In theory, the simulation time should only double since each simulation occupies 50% CPU, and was the case previously.

[Guilin Sun]

May I know how many available licenses? It seems the simulation files share the same one license. Another possibility is, the memory might be at its maximum, when  two or more simulations are run simutameously.

[Lito Yap]

 

@Boqin Zhao,

Which version is having this issue when running concurrent FDTD simulation as compared to which older version as per your post;

In theory, the simulation time should only double since each simulation occupies 50% CPU, and was the case previously.

FDTD performance and utilization is not linear. FDTD Is very CPU and memory intensive. It will try to take up all of the available memeory/system bandwidth on the machine. It may be only running with 50% of the CPU cores but it will be running with all avialable memory bandwidth on the machine. 

>>> Information on Hardware Specifications – Ansys Optics

Sometimes, depending on the simulation/sweeps, it would be faster to run them in sequence than running multiple simulations concurrently. We have this KB guide below that might come handy. 

>>>Determining the Optimum Resource Configuration to Run FDTD Simulations – Ansys Optics

 

[Boqin Zhao]

Thanks for your replys.

@Guilin Sun and @Lito Yap,

We have two licenses now. The problem occured when two users run their respective simulations at the same time, or one user run two simulations at the same time.

We had only one license a year ago. Back then, Running 2 simulations at the same time simply means that the simulation time would double. It was still working like that after we upgraded to 2 licenses, until maybe 1-2 months ago. We have been and is still using FDTD version 2022 R1.

No matter how many simulations are being run, CPU is being used ~100%. And I can confirm that the memory is not overwhelmed.

Thanks for your attention.

[Boqin Zhao]

However, you reminded me that perhaps for some reason not all the cores are being used when running multiple simulations, despite the 100% CPU usage. I am not very familiar with these technical terms, but will follow the instruction and check that.

[Lito Yap]

@Boqin Zhao,

FDTD will try to use all the available memory bandwidth even if it will not use all of the memory or not use all CPU cores. When all of the system's bandwidth is used, it will slow down other running processes on the machine. 

A single user can run 2 or more parameter sweeps (concurrent parametric computing) on a single machine - for simple/small sweeps than runs on a few cores and taking only a few MB of memory.

It is not recommended to have more than 1 user run FDTD simulations on the same machine. If 2 users run FDTD, both simulations will slow down significantly depending on the simulation's components/design/requirements. 

[Boqin Zhao]

Okay, thanks.  I am a bit disappointed because having two users run simulations at the same time is one important reason why we bought 2 licenses.

However, I am still curious why the simulations will slow down significantly. Could you please elaborate a little bit more? Maybe I can try to avoid certain setup/type of simulations when I want to run multiple at the same time in the future. Thanks!

[Lito Yap]

Yes you can certainly have 2 user run 2 simulations at the same time but just not recommended to run at the same machine. Each user should be running their simulation on their own local desktop. The license can be shared across other users/computers that is connected to the same local network as the license server running the Ansys License Manager:
>>>How to obtain a floating license – Ansys Optics

This is just how FDTD behaves. It takes up alot of your CPU and bandwidth. And increasing the number of MPI processes/cores to run the simulation is not linear. This will depend on the simulation itself and the machine as shown in our KB: 
>>>FDTD Performance Benchmarks – Ansys Optics 

You may be able to run small/less complex concurrent simulation/parameter sweeps but larger and more complex simulation jobs that requires more memory and resources should run by itself on one machine.   

[Boqin Zhao]

I have done a few testings following your suggestions and seem to understand what was going on.

I am not a computer hardware expert. Please forgive me if I say anything wrong.

Our CPU on the server is Intel Xeon E5-2695 V2. It has 48 processors in total. We have been running simulations with the number of processes set to 48, which means that it will use all processes. If another simulation with 48 processes is being run at the same time, each processor will be responsible for both simulations. However, one processor can only handle one task at a time so it has to switch between two simulations. The switching may be slow which causes both simulations to also slow down significantly.

I tried running 2 simulations with each one limited to 24 processes. It turned out that the speed for each process (in Mnode/s) is not significantly slower than one sim with 48 processes.

I think our best strategy moving forward is to limit 24 processes per simulation.

Thanks for your help again.