Fluids

Fluids

Running out of memory when running a very coarse mesh but with several mass and momentum sources

    • peter_beshay
      Subscriber

      Hi,

      I am trying to run a case on HPC, the cell count is very low (300k), and the case runs fine if i turn off the source terms (mass and momentum in a fluid zone). However, when i turn on the source terms (which are added using "named expressions"), the case runs for sometime then terminates with an error "exceeded allocated memory, even for 160GB memory allocated).

      Could someone help me please in this issue?

      Is there like a PBS command to clear virtual memory from time to time?

      The problem only appears with activating source terms

    • Karthik R
      Administrator
      Hello This is weird behavior. How many source terms are we talking about here? It seems to be happening when Fluent is updating the animation. What happens when you turn off animation? Do you still get the same error?
      Karthik
    • peter_beshay
      Subscriber

      Thank you so much for your reply.
      The case consists of 4 fluid domains, 3 zones are hedges, for each hedge, 1 mass source, 3 momentum sources (x, y, z), 1 turbulent K.E. and one dissipation (so a total of 18 source terms defines).
      I've turned off animation, still runs out of memory.
      I really appreciate your help, this is a very urgent project and i'm stuck with this error. Thank you
      I've tried "execute command" --> (cflush), to clear virtual memory every few time steps (maybe 100 or so), it worked few times as shown below:
      but then, it terminates, showing the following error:
      When, i turned off the execute command (cflush) and the animation, the case runs fine for some times steps, then slows down extremely and eventually terminates sowing the below error:
      or this error:






    • Rob
      Ansys Employee
      The source terms shouldn't cause a memory leak like that. What calculations are done to find the source terms? Can you monitor RAM usage from the IT side? Also what's using the RAM as we've seen rogue (or deliberate!) IT tasks kill jobs before, typically the virus scanner tries to do something clever.
    • peter_beshay
      Subscriber
      Hi Mr Rob Below is an example of a source term (the case has 18 source terms like this):
      How do i monitor the RAM from the IT side? is there a PBS command ?
      I do not understand this: "Also what's using the RAM as we've seen rogue (or deliberate!) IT tasks kill jobs before, typically the virus scanner tries to do something clever."
      My IT advised me to look for a fluent internal command to clear/flush cache memory every specified number of time steps, but i dont know how.
      As you can see below, i've requested 160GB of ram, and it still terminated after slightly over 5000 time steps:


    • Rob
      Ansys Employee
      It's almost certainly the expression step that's leaking RAM as it's evaluated for every cell in every iteration. I know we've been doing more work on expressions since 2021R1 so I'll see if it's a known issue, and if so whether it's fixed in 2022R1 (due "soon").
      The IT comment is based on experience. A few users at various companies lost jobs when running over some nights but not others. We eventually traced this to a deep virus scan on the clusters that was trying to check an open Fluent log file: as the file kept changing the scanner kept restarting the scan whilst not closing the previous scan. By the time the user returned in the morning the cluster had killed the task with a RAM error. The virus scanner then completed the tasks as it restarted automatically and didn't report an error......
    • peter_beshay
      Subscriber
      Dear Mr Rob Yes please check for me if there is any solution for this RAM issue with expressions.
      I tried "(cflush)" as exectue command every 100 time steps, it worked few times, then the solution crashed too, as you can see in the above messages.
      Could this solve the problem? but why it generated "cortex error" ? and sometimes it doesnt even work
      Thank you so much
    • Karthik R
      Administrator
      Hello Some more thoughts:
      You can go to the parallel menu in Fluent, choose, CPU Info -> Memory Usage to understand the overall usage (assuming you are running using the Fluent launcher). You should be able to find the corresponding TUI for this.
      You probably may need to increase your system limits. Please check with your IT on how to do this. You may also ask them to increase the limits in your PBS script used to launch Fluent. There are many resources online. Google the following keywords: open pbs stack size unlimited
      Thanks.
      Karthik
    • peter_beshay
      Subscriber

      I am running Fluent on HPC, so i cannot check the overall memory usage .
      I tried contacting the IT to increase memory usage, but the maximum is 160GB for 24cores, and this is huge memory in fact, the case is only 300k cells, the problem with the many source terms which consume memory, so i need to clear the stored unnecessary memory every few hundred time steps. and i dont know how.
      i will google the keywords you suggested, hopefully i can solve this issue.
      Please if you have any more ideas do let me know
    • Rob
      Ansys Employee
      Just a thought for a short term bodge. If the solver works for (say) 250 time steps can you set a batch script that'll launch Fluent, read case & data run, write case & data then close; open new session, read in the recently saved files and continue? I've not needed to try this in years but did in my early career as I needed to run 10 simulations overnight on one core.
    • peter_beshay
      Subscriber
      Hi Mr Rob That seems way too advanced for me, i do it manually thou, i resume running the case on HPC using the last saved data file, its kinda working but its a hassle to run all cases with that procedure.
      It will be a lot easier if there is a script or command to clear the RAM every now and then.
      Thank you for that idea

    • Rob
      Ansys Employee
      It's a memory leak - and may be fixed in 2021R2 and 2022R1: this assumes you've found the same issue as the earlier one.
    • peter_beshay
      Subscriber
      Mr Rob, the Ansys version we have on our HPC is 2021 R1, this cannot be solved in this version? isn't there anything i can do using this version to solve the memory leak issue?
      if i download the student version 2021R2 or 2022R1, it solved in this version? do i have to make any further steps? or it is automatically solved?
    • Rob
      Ansys Employee
      As far as I can see it's version specific, so updating should fix it. Note, Student won't play well with any installs of the normally licenced software so be wary of putting it on a Uni machine. 2022R1 is due soon so it may be a good time to talk to the IT department about an update. You can run multiple versions of the Ansys software on one machine (I've usually got 5-6 different release of Fluent installed) so it shouldn't impact any other users.
    • peter_beshay
      Subscriber
      Thank you so much Mr Rob I will talk to the IT for an update.

Viewing 14 reply threads
  • You must be logged in to reply to this topic.