Learning Forum

[naibaf]

Dear community,

I have a problem in parallel simulation. I want to simulate fluid flow in a porous media representing a generic underground mine. Therefore, is set up the geometry and assign porous media. Other specifications of my model are:

·        VOF model used.

·        porous media with spatially varying porosity and permeability in the domain

·        two-phase (air, water) flow with compressible liquid for both phase

·        laminar flow conditions

·        UDF with mass source term for air-phase

·        14 Solver processes with activated hyperthreading, partition method X-Cartesian

·        Ryzen 7 3700X processor

The code works in serial however I must investigate physics in more detail to check plausibility of the results. Serial simulation is very slow, so I wanted to start a parallel calculation. I parallelize my UDF’s, and the mesh is partitioned. But at some point, the calculation gives incorrect results. At a partition interface the volume fraction of water vanishes ahead of the fully water-filled area. Before there was a significant VOF of water. Additionally, a high negative static pressure evolves at the partition interface (see pictures). In serial mode there is no problem at this point.

Does anybody faced such a problem in the past and may explain the reason for these 'artefact' and how I remove it? So far, I think there is a problem in communication between the node processes or with the water surface at the partition interface. The problem occurs even if I deactivate all my UDF's. When changing the partiton method the problem occurs at a different location due to changed location of the partition interface.

[DrAmine]

So running with single node or serial you do not see that issue?

[naibaf]

Dear DrAmine thank you for your reply. The problem does not occur when simulating with one node (select one solver process in Fluent launcher)

Regards

[Rob]

What does the mass source do?

[naibaf]

Dear Rob Thank you for your reply. The mass source ÔÇ£absorbsÔÇØ the surplus air mass in the cells due to water rising/flow (a bid similar to the ÔÇ£degassing boundary udfÔÇØ in the manual). I cannot use an outlet-boundary condition in the model since the water should stay inside the domain and in theory has the possibility to flow into nearly every part of the model. At first, I use DEFINE_ADJUST to calculate mass and momentum sources for each cell in the model. Values are saved in UDM. The DEFINE_SOURCE access UDM and takes the value for the source in each cell. I also tried to use PRF_GSYNC(). However, the issue in parallel calculation occurs with or without my UDFs.
The UDFs Looks like:
DEFINE_ADJUST(calcSource,d)
{
#if!RP_HOST
Thread*cthread,*ts;
cell_tc;
thread_loop_c(cthread,d)
{
begin_c_loop_int(c,cthread)
{
ts=THREAD_SUB_THREAD(cthread,1);
C_UDMI(c,cthread,0)=-(C_R(c,ts)-1.225)*C_VOF(c,ts)/CURRENT_TIMESTEP*C_POR(c,cthread);
C_UDMI(c,cthread,1)=-(C_R(c,ts)-1.225)*C_VOF(c,ts)/CURRENT_TIMESTEP*C_U(c,cthread)*C_POR(c,cthread);
C_UDMI(c,cthread,2)=-(C_R(c,ts)-1.225)*C_VOF(c,ts)/CURRENT_TIMESTEP*C_V(c,cthread)*C_POR(c,cthread);
C_UDMI(c,cthread,3)=-(C_R(c,ts)-1.225)*C_VOF(c,ts)/CURRENT_TIMESTEP*C_W(c,cthread)*C_POR(c,cthread);
}
end_c_loop_int(c,cthread)
}
#endif
}

DEFINE_SOURCE(DensitySource,cell,thread,dS,eqn)
{
#if!RP_HOST
dS[eqn]=0.0;
returnC_UDMI(cell,thread,0);
#endif
}

DEFINE_SOURCE(x_recoil,cell,t,dS,eqn)
{
#if!RP_HOST
dS[eqn]=0.0;
returnC_UDMI(cell,t,1);
#endif
}

DEFINE_SOURCE(y_recoil,cell,t,dS,eqn)
{
#if!RP_HOST
dS[eqn]=0.0;
returnC_UDMI(cell,t,2);
#endif
}

DEFINE_SOURCE(z_recoil,cell,t,dS,eqn)
{
#if!RP_HOST
dS[eqn]=0.0;
returnC_UDMI(cell,t,3);
#endif
}

Thank you in advance!

[DrAmine]

It is not clear for me where in your VOF picture the porous domain is starting. Can you highlight that? Have you tried without any porous domain?

[naibaf]

Dear DrAmine the whole domain is defined as porous media, since rock and backfill material is porous. So therefore I did not try without porous media since the porosity of rock has to be considered.
In tests on a much smaller column model with constant porosity and permeability the issue doesn't occur within parallel calculation.
Regards

[DrAmine]

We are trying to help you debugging the case: Will you see the issue if you disable the porous domain? Are you using physical or superficial velocity?

[naibaf]

Thank you very much for your help.
I am using superficial velocity. And I try to perform the simulation without porous media model. However, convergence issues occur when disabling pororus media. I think this is due to absent permeability of the medium. Therefore my calculatiuon settings etc. are not sufficient anymore.
Regards

[naibaf]

For a first look, the calculation is very unstable without porous media. But it seems that the problem remains.
Fig 1 Vanishing volume fraction of water in the area of partition interface, ahead of the fully water-filled part.
Fig 2 Residuals showing poor convergence of the simulation without porous media model.

[Rob]

You're trying to turn the flow in a very small (coarse mesh) space. What happens if you increase the resolution?

[naibaf]

Dear Rob increasing resolution might be a good point for further investigation, thank you. But (only for better understanding of the topic) what is the difference in calculating on a single node or calculating on several nodes, related to mesh resolution? I do not understand why the single node simulation doesn't show the issue.
Regards

[Rob]

It shouldn't, but you also have a parallelised UDF. What do the UDM values show when compared to the single core model?

[naibaf]

At the beginning of the calculation the UDM values for the source term are nearly identical in both cases. Slightliy differences at the partition boundary are visible. With increasing simulation duration the error between the UDM values at the partiton interface increases (see Figures below). The calculated minimum mass source (outflow) is the same in both cases. But there is a significant maximum mass source (inflow) in the parallel case.
But I am wondering since the issue remains even if I unhook and unload all the UDF's . As expected the UDMs without UDFs are zero everywhere. So no source should be applied, but the issue with the vanishing volume fraction of water remains.
Fig 1 UDM value for mass source in parallel case.
Fig 2 UDM value for mass source in single node case.

Thank you in advance!

[DrAmine]

Please summarize now (still bit surprising for me how a non-porous case using where the porous one is using superficial velocity is having convergence issue).
Can yo confirm these statements

1/The issue occurs without UDF#s? If Yes go to 2/ If No: We cannot help
2/The issue does not occur on Single CPU If Yes go to 3/
3/Post for the same time step size the VOF fraction field on a plane without any node values and use the same range

Depending on the answers above we will try to respond.

[naibaf]

The convergence issue may arise from my modelling approach. I initialize an "excess pressure" in a specified area in the mine. The excess pressure evolves due to a mechanical deformation, which I calculate within another software on the same grid. ANSYS Fluent should simulate the relaxation of the pressurized fluid to a nearly hydrostatic equilibrium. The initial pressure is therefore maybe too high for simulation without a viscous resistance. So, I think setting-changes are needed for such a case (time-step size etc.).
I can confirm (to the best of my knowledge and conscience) that:
1/ yes, it occurs even without UDFs
2/ yes, the issue does not occur with one solver process
3/I hope these are the figures you expected.
The Figures show the VOF for water in the underground rooms and drifts as plane view. In Fig 1 for the parallel case the vof drops, right at the partition interface. If the simulation runs further the spatial distribution of the decreasein the vof becomes larger. In Fig 2 for single node the VOF do not drop. If the calculation on the single node is continued, the chambers are getting filled with water more and more. For the parallel case, the water does not flow further. Both figures are saved after fluid-time = 221146.63 s after 40 time steps are calculated. I use adaptive time advancement to gradually increase the time step. The time advancement settings are the same in both cases.

Fig 1 Volume fraction of water for the parallel case at t = 221146.63 s after 40 time steps.

Fig 2 Volume fraction of water for the single node case at t = 221146.63 s after 40 time steps.
Fig. 3 Distribution of partitions.

Thank you very much for your help and time!

[Rob]

Repartition onto a different number of nodes or with y-direction so the partition is in a different place. What happens? What is the flow field like?

[naibaf]

Dear Rob for my first test with the default METIS-partitioning-method. The issue was occuring on a different location but also at the partition. So I changed the partitioning method to X-Cartesian and the issue occured at a different time at the new partition locations.

[naibaf]

So from my point of view, the fluid flow is ok until it reaches the partition interfaces.

[Rob]

OK, so we have a gradient issue with a very coarse mesh at the parallel partition. Please refine the mesh and see what happens.

[naibaf]

Dear Rob ok thank you for the help. I will try a mesh refinement to check results in such a case. I will post my findings here asap. Maybe it helps other users.

[Mohammad Haikal Md Fouzi]

Hi Rob, I've facing similar issue as naibaf where i'm trying to access c_udmi value in the exterior cell of a partition. My intention is to calculate the average value of C_UDMI for a cell_t c at thread t based on the neighboring cell (if the given cell_t c has 6 faces, that means it has 6 neighboring cells). The averaging function works if the cell zone was in the same partition, but will give the wrong value if the cell zone have more than one partition.

Below images is the correct contour of user defined memory as the cell zone is not separated by any partition.

 

Below is the wrong contour (wrong averaging calculation) as the cell zone was splitted into two different partition.

Is there a way to access user defined memory on the exterior cell of a partition mesh so that i could get the averaging calculation right.