Learning Forum

[soloviev]

Hello,

 

We have run two of the same models with the only difference being the amount of processors the model was run on.

Specifically, we are using the VOF to DPM model and modeling particles. We also use mesh adaption. 

When we run the model on 264 processors we see a large generation of particles (~60,000) while when run on 528 processors we see around 5000 particles. The settings are exactly the same.

Why is this?

 

Thanks,
Alex

[DrAmine]

Can you add a detailed description of the case?

Can you confirm a depth convergence in both runs?

[soloviev]

The case is a square domain initialized with air and water. The water is initialized with waves. We set mesh adaption based on the water volume fraction and curvature. We used the same values for coarsening and refining. We then set the vof to dpm transitions and injections, etc. We also set periodic boundaries and a wind stress from the top boundary. 

 

What do you mean by a depth convergence?

 

Thanks,

Alex

[DrAmine]

If the case is deeply converged: small imbalances residuals monotonic monitors,..

Please check the partition boundaries if they are lying on the free surface: that can be an issue.

Moreover check transcript file and scrutinize the outcome of particles If the get lost somewhere.

It's definitely something which we cannot support per remote without looking into the case.

[soloviev]

Hello,

Looking at the model, we have the time step set to variable, so that it can self adjust. Usually these time steps self adjust to a very small number (i.e. 2e^-. Looking at the history of residuals, for continuity for example we have it set to convergence criteria of 0.001, but the residuals only show it going to 0.1 and then moving on to the next time step. How does this occur if we have the convergence criteria set to a smaller number?

 

Thanks,
Alex

[DrAmine]

Why should that not occur from the numerical point of view?

I guess either the number of iterations per cycle is not enough but at that tiny temporal increment I would rather say the whole case is stiff and probably you need to explain more and add details about the run.

[soloviev]

I was under the impression that that means that the model is not converging if it is not reaching the convergence criteria, but I am also relatively new when it comes to Fluent. 

 

We leave the max iterations set to 20, but yes, it does not go more than a few iterations usually before moving to the next time step.

What more information would be useful?

 

Thanks,
Alex

[soloviev]

Hello,

Would it possibly be helpful or is there a way to send the case so that the settings can be easily seen by someone at ANSYS? Maybe this would allow for more insight into the issues with the model. 

 

Thanks,

Alex

[DrAmine]

You wrote

it does not go more than a few iterations usually before moving to the next time step

Does it mean it goes to the next time step before the whole 20 is reached?

What you can try at first before thinking about looking into the case is that you disable convergence by residuals under Residual and run the case once again for a certain number of time steps. Run it at first on say N than N/2 nodes. Please switch off load balancing andv reorder domain Fluent is doing when adapting mesh. Do that as calculation activity every x steps.

[soloviev]

Yes, the model moves to the next time step before the whole 20 iterations is reached.

I am going to test using your recommendations. When you say switch off load balancing, what do you mean? Currently in my partitioning and load balancing dialog box under dynamic load balancing only mesh adaption is checked and the threshold is at 5% (This is my first time using this dialog box, I usually set partitioning using the console). What do you mean by reorder domain? It is set to method: metis with the number of partitions set to N nodes. 

 

Thanks,
Alex

[DrAmine]

It is most efficient to switch off auto load balancing as it is too frequent for VOF-to-DPM (Uncheck box for MEsh Adaption). Instead of that you can execute the partitioning via calculation activities. You execute below commands every 50 time steps for example. Pleae check in Fluent and do not copy paste might be I am making an error here.

parallel/partition/method metis,

parallel/partition/reorder-partitions-to-architecture

parallel/partition/use-stored-partition

[soloviev]

Thank you for the suggestion. I just implemented this into my model. I am going to run it first on N nodes then N/2 with the changes and see if this makes a difference. 

 

Thanks,
Alex

[DrAmine]

Welcome!

[soloviev]

When running with the settings suggested the model runs into the following error:

*** Error in `/cm/shared/apps/ansys_inc/v192/fluent/fluent19.2.0/lnamd64/3ddp_node/fluent_mpi.19.2.0': free(): corrupted unsorted chunks: 0x00000000053e1190 ***"

BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 431458 RUNNING AT node011
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

 

Thanks, Alex

[DrAmine]

Dear Alex please use a new release whenever you are working with vof to DPM especially adaption has been enhanced.

Is the error reproduceable ?

[soloviev]

Each time I run the model it produces the same error. Other models do not though. I will update to the newest version of fluent though, as we are currently running 19.2.

 

Thanks,

Alex

[DrAmine]

Please use the latest possible version (2019R2). Please let us know if you still face the issue in the latest release.

[Raef.Kobeissi]

Very interesting topic! Thanks abenhadj!

[soloviev]

Hello,

 

I have updated to the newest release. When adapting my mesh I see the new option under advanced controls for 'additional refinement layers' what does this mean? Is this the same feature as 'max levels of refinement' from 19.2?

 

Thanks,

Alex

[soloviev]

Also, when I unchecked mesh adaption under load balancing my mesh stopped adapting. The mesh adaption is crucial to my model.

 

Thanks, Alex

[DrAmine]

I recommend having a look into the new VOF to DPM tutorial in the learning hub. You need to define the three commands mentioned earlier to be executed every 50 steps.
The additional layers get refined for cells which are not marked as to be defined. I would not use it now.

[soloviev]

Yes, I set up the commands mentioned earlier, but I'm not sure how to determine them to be executed at a certain time step. I unfortunately do not have access to the learning hub. 

 

Thank you,

Alex

[soloviev]

I am now receiving the following error when running in 2019R2:

Warning: ST_Realloc: out of memory.realloc_storage: unable to realloc 

 

I received this for various things including face area, velocity, volume, etc. 

 

Thanks,
Alex

[soloviev]

Hello,

 

Is there any update to this issue?

 

Thanks,
Alex

[DrAmine]

Warning indicates that your case is too big for the machine memory, because of which it runs out of memory, hence the 'out of memory' message comes. 

[DrAmine]

Check your refinement criteria and limit the minimum cell volume /use lower number of refinement levels

[soloviev]

Thank you. I am running on an HPC with 192 GB of ram per compute node, and typically run on either 6 or 12 nodes. Would you recommend increasing this RAM? Or should this be sufficient with a reduction in refinement to generate accurate results in Fluent?

 

Thanks,
Alex

[Rob]

You should have enough RAM, can you monitor incase it's one node that's trying to do all of the adaption: this could cause an overload depending on how the cluster is set up. Reducing the initial mesh resolution & refinement will help, but you'll need to check it's not effecting accuracy. 

[DrAmine]

Are you still using the automatic load balancing or relying on the three commands earlier in different post? You might need to do that more frequent perhaps. Moreover bear in mind that 1 million cells would require at least for double precision 2 GBs.

[soloviev]

I have switched over to the command based load balancing as you suggested. When adapting it says it is dividing domain into 264 partitions, so does this indicate that it is adapting across all the nodes?

 

Thanks,
Alex