Error: "unable to locate checkpoint file" when running simple FDTD training simmulation

lelma87lelma87 Member Posts: 1

I use recent version of Lumerical Suite on Windows. When running training simulation set up accordingly to FDTD100 tutorial video, simulation crashes at approx. 15%, with an error: "Process number 0 unable to locate checkpoint file." Can anyone point me to the solution of the problem? Log file attached.


Answers

  • LitoLito Posts: 215Ansys Employee

    @lelma87, Is the simulation file saved in a location that you have write access? Try to save the simulation example into your user account/Downloads or Desktop folder.

    Copy and paste the log results, we are unable to download files from ALF.

  • lelma87lelma87 Posts: 3Member

    Dear Lito, thank you for taking interest in my question. I tried various directories, I have admin privileges. Interrupted simmulation generates movie file and log file in simulation file directory. Results are always the same, as described. Log results below and prtscr of job details:

    Ansys Lumerical 2021 R1.2 FDTD Solver Version 8.25.2621 (Windows 64bit)

    expires (month/day/year): Permanent

    using config file: Ansys Licensing set with ANSYSLMD_LICENSE_FILE

    license host: 1055@10.0.3.13

    current time: Fri Mar 26 09:14:20 2021

    Running on host: 49244_CEZ

    host available memory (GB): 60.266922

    All processes are communicating and ready to proceed to simulation...

    Running fsp file: C:\Program Files\Lumerical\v211\bin\fdtd-engine-msmpi.exe -t 1 -remote -resume C:/Users/m.lelit/Documents/Lumerical training/training simulation 1.fsp

    This simulation has already been run, will run again.

    The program terminated due to an error:Unable to locate any file matching 'C:\Users\m.lelit\Documents\Lumerical training\training simulation 1_*_p0.fsp.ckpt'

    Error: Process number 0 unable to locate checkpoint file.


  • LitoLito Posts: 215Ansys Employee
    edited May 26

    @lelma87, try to disable checkpoint on the simulation.

    Can you try and run the mwp.fsp file from this example? Does the issue happen to this example, mwp.fsp?

  • lelma87lelma87 Posts: 3Member

    @Lito, thank ou for this suggestion. This worked but for some reason the other way around: at first, chceckpoint option was already disabled and error occured. However, when I checked both "checkpoint during simulation" and "checkpoint at schutoff", checkpoint files were generated and simmulation completed cuccessfuly.

    mwp.fsp works fine regardless of checkpoint options - simmulation completed cuccessfuly with and without checkpoints.

  • LitoLito Posts: 215Ansys Employee

    @lelma87, Glad to hear that this has been resolved.

  • RyanChungRyanChung Posts: 18Member

    Hi, I recently got the similar issue on the linux cluster with limited walltime of the queue. Is there any way to enable the checkpoint feature over terminal via fdtd-engine-impi-lcl command?

  • LitoLito Posts: 215Ansys Employee
    edited May 21

    @RyanChung

    Starting a simulation job with checkpoint:

    • Enable checkpoint in FDTD.

    • Run the simulation as you would without the -resume option.
    • This will create the checkpoint files based on the number of processes and checkpoint period.
    • Ensure that the checkpoint files are created and not deleted when the simulation terminates.

    To resume the simulation based on the last checkpoint:

    • Do not change or edit the simulation.fsp file.
    • Ensure that the checkpoint files are found together with your simulation.fsp file.
    • Run the simulation with the -resume option. (do not use the "-resume" option if this is the first time you are running the simulation and you are not "resuming" a simulation job)
    ./fdtd-engine-<executable>  -resume  -fullinfo  simulationfile.fsp
    

    Best,

    Lito

  • RyanChungRyanChung Posts: 18Member

    Thanks. I’ll enable it via CAD on the local PC before uploading it onto the CAD-free cluster.

  • LitoLito Posts: 215Ansys Employee

    @RyanChung

    You are welcome.

    Best,

    Lito

Sign In or Register to comment.