Learning Forum

[arcadianleericky]

Hi,

I've been using lumapi with python to optimize a structure in FDTD, and I'm running my task on our HPC enabled with distributed computing (4 computing nodes). I'm submitting the task with Slurm coupled with MPICH2. The command used to run the code:

python myCode.py 

where distributed computing across 4 nodes is enabled by:

/opt/lumerical/v221/mpich2/nemesis/bin/mpiexec -hosts node01:4,node02:8,node03:16,node04:8 /opt/lumerical/v221/bin/fdtd-engine-mpich2nem -logall -t 1 $HOME/working/simulation.fsp

The above last line of command is written inside the Resources tab in FDTD. simulation.fsp is loaded by myCode.py.

In myCode.py, "runsetup" and "runanalysis" are used to first initialize the photonic structure and then run the analysis (Q analysis and modal volume specifically), respectively.

My issue is, runsetup is using all 4 nodes with no problem, however, runanalysis somehow is only using 1 of the 4 nodes.

Does runanalysis not use distributed computing by default? If not, what could be the fix for this issue here?

Thanks! @greg_baethge @Lito Tagging the experts for a faster response, hope yall wouldn't mind :)

[Greg Baethge]

Thanks for your post. Unfortunately, if the FDTD engine is fully parallelized and supports both multithreading and multiprocessing (via MPI), allowing for distributed computing on multiple nodes, the GUI (where the scripts are interpreted) only supports multithreading, so it can only run on a single node. Moreover, not all script commands are multithreaded and the parallelization efficiency greatly depends on the command used.

[arcadianleericky]

Hi Greg Thanks for your prompt answer.
Based on your reply, I deduce that "runanalysis" simply doesn't support multi-node computing as a result of the GUI only supporting multithreading and not multiprocessing. Is there a fix or workaround for this issue? If not, which script command, other than runanalysis, can be used to run the analysis that also supports multi-process computing on multiple nodes?
some of the other experts are tagged, hope you guys can help out here :)

[Greg Baethge]

Apologies for the delayed response, it's been a crazy week! Unfortunately, this is a limitation of our scripting environment. I can't really see any easy way to work around this. If the analysis could be segmented in individual tasks, you could possibly run them on different machines but that would require multiple licenses.
The alternative would be to run the analysis in a different language, that supports multi-node tasks.