Using Inverse Design and Lumopt with cloud computing

amnonjwamnonjw Member Posts: 7

Hi,

I need to optimize a 3D design of grating coupler, and I'm considering the inverse design features.

The model I use requires great computation resources, so in normal simulations I use Lumerical on a cloud service with parallel computing: a single job runs in the cloud 6x faster than on my local computer, and I can run up to 15 jobs in parallel.

Let's say I prepare a python script that given an fsp file it:

  1. uploads the file to the cloud via ssh
  2. runs the simulation on the cloud
  3. once the simulation is finished - downloads the file

With the adjoint method - if in each iteration we call Optimization.run() twice - is there a way to prepare two fsp files, and overload the Optimization.run() command such that we send both of them to the cloud service to run in parallel (for each iteration) using my python script?

Answers

  • kjohnsonkjohnson Posts: 230Ansys Employee

    Hello @amnonjw ,

    I don't think trying to rewrite the lumopt methods would be a good approach. It would be much easier to run the inverse design scripts directly on the cloud. Do you have access to the GUI on the cloud?

    When lumopt runs the inverse design optimization it will already prepare two fsp files, one for the forward and one for the adjoint simulation, and submit both files as separate jobs to the solver. You can run these two simulations in parallel if you set the "Capacity" of your Resource Configuration to 2 on your cloud instance:

    Both jobs will now run in parallel:

    So running the forward and adjoint simulations in parallel is essentially the same as running multiple jobs during a parameter sweep or PSO optimization. Do you think this approach will work for you?

  • amnonjwamnonjw Posts: 15Member
    edited October 10

    Hi,

    If we use the desktop app directly on the cloud's server - it closes after a short period of time by admins (< 1hour). We instead need to submit jobs to queue, which is usually isn't full (in general my algorithm should handle the possibility jobs are waiting to be executed).

    The jobs use the mpi version of lumerical exe to perform the heavy simulations

    /usr/local/lumerical-2021R1/mpich2/nemesis/bin/mpiexec

    /usr/local/lumerical-2021R1/bin/fdtd-engine-mpich2nem

    It allow us to utilize number of cores from the cloud service for shorter simulation time.

  • kjohnsonkjohnson Posts: 230Ansys Employee

    Hello @amnonjw ,

    Which job scheduler are you using on your cluster? It might be possible to configure your resources to automatically submit your jobs to the queue from your local FDTD, as described on this page: https://support.lumerical.com/hc/en-us/articles/360034620113

  • amnonjwamnonjw Posts: 15Member

    Hi,

    Our queue system is PBS. I think it's an extension of Torque so perhaps your preset and scripts can help, like the Torque.py script I found in my "Lumerical\v212\scripts\job_schedulers" folder.

    From first glance - it looks to me very complicated to setup inverse design + remote job scheduler. Do I have to modify the py scripts or only the resource manager in the made IDE? - because it looks like the inverse design py files don't open the main IDE when they run the simulations.

    Perhaps you can expand one of your inverse design examples - show how to modify the py scripts and/or the IDE settings to use a remote host with job scheduler like Torque.

    I attach an example of the sh script (upload as txt file) that I use:

    I was told to submit it with command like "qsub script.sh"


    Thanks in advance.

  • amnonjwamnonjw Posts: 15Member

    UPDATE: I managed to configure a remote job scheduler from a local IDE to our cluster server.

    Lumerical has a list of 4 predefined clusters: SGE, Slurm, Torque, and LSF. Our cluster is PBS which is a variation of Torque (I think). I tried adding a PBS.py script to the "scripts\job_schedulers" folder, but the IDE didn't recognize it as a 5th cluster option. I decided to overwrite the Torque.py instead, and I had to tweak it a little to match actual status reports from the cluster manager.

    What will happen if I update the IDE on my computer - will it revert the Torque.py file?

  • kjohnsonkjohnson Posts: 230Ansys Employee

    Hi @amnonjw ,

    If you update the Lumerical software it will overwrite the Torque.py script, since it is in the install folder. So you will have to save the script somewhere else, update the software, then overwrite the Torque.py file with your version again.

Sign In or Register to comment.