Have you tried running with the bundled MPICH2 nemesis on the cluster?
Or run with the FDTD executable to run all your simulations at once: (i.e. run 4 simulations concurrently on 1 node with 16 cores)
/[installpath]/lumerical/v231/bin/fdtd-engine-mpich2nem -t 4 sweepfile1.fsp sweepfile2.fsp sweepfile3.fsp sweepfile4.fsp
Or run with Intel MPI: > Running simulations using terminal in Linux (Using Intel MPI) – Ansys Optics
Otherwise, the best way to run concurrent FDTD simulations is to run 1 simulation on 1 machine/node.