Learning Forum

[pawar002]

Hello,

I am simulating fluent job on Linux based HPC. I am using 4 nodes with 128 processors. But after certain timestep, my simulation slows down to overcome this I have to submit the job again with last saved data file and again it runs faster and after some timesteps, it slows down again could you please help me out with this issue. below I have attached some screenshots of residuals to give more details about the problem.

Initially when submitting the job initial speed

after some timesteps

 

Please help me out with an issue how can I eliminate t this fluctuation in my simulation.

 

 

Thank you.

[Rob]

Can you check system side too in case there's a memory leak. What are you solving, it's possible you need to rebalance the load as the solution changes. 

[DrAmine]

Hi to add to the comments: that time plotted is very coarse estimate how long it might take to finish the time step.I do not recommend to rely on that if you want doing benchmarking.

[pawar002]

Hi rwoolhou,

 

I am simulating evacuated tube solar collector. geometry is quite big having 1.8 m length and outer dia of 0.058 m. approx number of nodes is 700,000. could you please guide me on how to rebalance the load. moreover, I am simulating it on Linux based cluster so, I don't know how to figure out memory leak prob.

 

Hi abenhadj,

I am comparing my result with the experimental result. total flow time is 86400 second and simulating with time-step of 0.1-sec. with 128 processors its taking 14-15 days to finish the job.

[DrAmine]

Do not rely on that time/iter in Fluent console to know how long it takes to run X number of time steps. That information is wrong.

[pawar002]

Hi Abenhadj,

I am not relying on that time. But when I am running any case with a single node my job runs smoothly without any issue. But as soon as I increase the number of nodes, after a few hours of simulation (or CPU time) it takes a longer time to finish the one time step. Then I have to stop my job and I have to again submit the job on cluster and simulation runs smoothly. I believe that it is a problem in either cluster or my PBS script file if anyone has an idea then please let me know.

 

Thank you.

[DrAmine]

Benchmark a case by running 10 to 100 time steps or certain number of iterations without any I/O (report files, etc...) check the parallel performance of the cluster and make a comparison. Ensure that the affinity is set correctly so that your run is leveraging the whole resources and not concurring with other open Fluent sessions. GEt in touch with your ASC and IT to find some ANSYS cases for benchmarking.

https://www.ansys.com/en-gb/solutions/solutions-by-role/it-professionals/platform-support/benchmarks-overview

[pawar002]

Thank you so much abenhadj for sharing the link. I will run the Benchmark case and will update you.

Thank you. 

[pawar002]

Hi rwoolhou, abenhadj

 

The reason behind my job is getting slower is

"Memory occupation by larger remaining Linux I/O buffer caches can (and most likely will) lead to an unbalanced distribution of ANSYS Fluent's memory allocations with respect to the launched ANSYS Fluent tasks on the dual socket compute node, i.e. a number of ANSYS Fluent tasks need to permanently access their corresponding data not from the pürocessors own local memory but from the memory of the other neighbouring processor on the dual-socket compute node, thereby encountering less efficient memory access. This potentially can lead to ANSYS Fluent performance degradation in the order of up to 10-12% based on the measurable increase in simulation run time. For a 48 hours simulation run 10-12% increase in simulation time are about 5 hours plus."

 

does anyone knows how to purged Linux I/O buffer caches while running the simulation or is it possible to use  "(flush-cache)" after every certain timestep. 

[Rob]

You can run commands using the Execute Commands in Fluent. I'd be careful using flush type system commands incase it flushes something important: but I think it'll run commands after autosave so worst case is you need to restart from the last saved data.