Parallelizing “DEFINE ON_DEMAND” & “DEFINE_SOURCE” UDF to apply x-momentum source from a text file
Hello,
This problem is related to fluent UDF.
I have a text file storing x-momentum source at every cell center of the domain. I store the source term in order of global cell index in the text file. i.e. value in first line correspond to source of cell with cell index 0, value of second line correspond to source of cell with cell index 1 and so on. I want to read this text file and add source term to x-momentum equation. My UDF for serial run is following (which is working fine)
#include "udf.h"
float xSource[194875]; //global array variable
DEFINE_ON_DEMAND(read_source)
{
int i;
FILE *xSourceFile; // declare a FILE pointer
xSourceFile = fopen("Fx.txt", "r"); // open file for reading
Message("xSourceFile file opened\n");
/* loop to read file */
i=0; // initialize before to loop
for (i=0;i<194875;i++)
{
fscanf (xSourceFile, "%f", xSource+i); // one value per line
}
fclose(xSourceFile);
Message("xSourceFile file closed\n");
}
DEFINE_SOURCE(xmom_source,c,t,dS,eqn)
{
dS[eqn]=0.0;
return xSource[c]; //xSource array has values in global index order
}
The above code works fine in serial. To run the same case in parallel I have made following udf:
#include "udf.h"
float xSource[194875]; //global array variable
DEFINE_ON_DEMAND(read_source)
{
#if !RP_NODE
int i;
FILE *xSourceFile; // declare a FILE pointer
xSourceFile = fopen("Fx.txt", "r"); // open file for reading
Message("xSourceFile file opened\n");
/* loop to read file */
i=0; // initialize before to loop
for (i=0;i<194875;i++)
{
fscanf (xSourceFile, "%f", xSource+i); // one value per line
}
fclose(xSourceFile);
Message("xSourceFile file closed\n");
#endif
host_to_node_float(xSource,194875);
}
DEFINE_SOURCE(xmom_source,c,t,dS,eqn)
{
#if !RP_HOST
dS[eqn]=0.0;
return xSource[c]; //xSource array has values in global index order
#endif
}
My observations are
- First udf works fine when I run case in serial and gives perfect results
- Second udf works fine when I run case in serial and gives perfect results
- Second udf does not work properly when I run case in parallel. It gives warning as "warning C4716: xmom_source': must return a value" . And, results are also not good (not similar to first two cases )
Can anyone please identify what is the problem with second udf when I run it in Parallel ?.
It would be really helpful to me.
Thanks in advance
Comments
The host and nodes must read the define source .. ds(..) must be seen by both. Remove the if statement
Define source does not require parallisation in general.
Hello @YasserSelima & @DrAmine ,
Thanks a ton for your response !!
Is this what you are suggesting ? Because I already tried this and its not giving proper results.
DEFINE_ON_DEMAND(read_source)
{
#if !RP_NODE
int i;
FILE *xSourceFile; // declare a FILE pointer
xSourceFile = fopen("Fx.txt", "r"); // open file for reading
Message("xSourceFile file opened\n");
/* loop to read file */
i=0; // initialize before to loop
for (i=0;i<194875;i++)
{
fscanf (xSourceFile, "%f", xSource+i); // one value per line
}
fclose(xSourceFile);
Message("xSourceFile file closed\n");
#endif
host_to_node_float(xSource,194875);
}
DEFINE_SOURCE(xmom_source,c,t,dS,eqn)
{
dS[eqn]=0.0;
return xSource[c]; //xSource array has values in global index order
}
I already tried this . When I compile , warning "warning C4716: xmom_source': must return a value" does not appear . But it does not give results similar to the one I get in serial run using the first UDF.
If you have any other suggestion please comment.
A quick doubt- Is the global cell index different in serial and parallel run ? or it will remain same even when the mesh is partitioned to computing nodes
Thanks again
That might cause an issue as it starts from zero for each partition
try to print a message with c value. Probably the problem is there
Hello @DrAmine & @YasserSelima ,
As per your suggestion I wrote following UDF to check which cells are being passed to DEFINE_SOURCE and at what index. I also write them in a text file after one iteration using DEFINE_ON_DEMAND udf.
//There are total 194875 cells in the domain
#include "udf.h"
int indexC[194875]; //To store the index which will be passed in Define_Source
float Xc[194875]; //x-coordinate of cell being passed to Define_Source
float Yc[194875]; //y-coordinate of cell being passed to Define_Source
float Zc[194875]; //z--coordinate of cell being passed to Define_Source
int j=0;
//This udf does nothing except storing the index and coordinates of cells being passed to define source by solver
DEFINE_SOURCE(xmom_source,c,t,dS,eqn)
{
real x[ND_ND];
C_CENTROID(x, c, t);
indexC[j]=c; //Storing cell index in global array
Xc[j]=x[0]; //Storing cell x-coordinate in global array
Yc[j]=x[1]; //Storing cell y-coordinate in global array
Zc[j]=x[2]; //Storing cell z-coordinate in global array
j++; //increasing global cell counter
dS[eqn]=0.0;
return 0;
}
DEFINE_ON_DEMAND(write_data)
{
int i;
FILE *CellFile;
CellFile = fopen("cF.txt","w");
for (i=0;i<j;i++)
{
fprintf(CellFile,"%d ",indexC[i]);
fprintf(CellFile,"%f ",Xc[i]);
fprintf(CellFile,"%f ",Yc[i]);
fprintf(CellFile,"%f ",Zc[i]);
fprintf(CellFile,"\n");
}
fclose(CellFile);
}
So when I run this UDF in serial
When I run this UDF in Parallel
And coordinates corresponding to index 0 in serial and parallel run are different and similarly for other indexes. @DrAmine This proves your point that cells start from 0 in each partition. But
I remember I have seen in the manual a macro that gets the index. Not sure of its name ... if you find it, this would solve your problem
Hello @YasserSelima ,
The macro you are suggesting is C_ID which output the global cell indexes. But I am not sure if that will give cell indexes in parallel since all the computing nodes are having cell index ranging from 0,1,2......
The problem is that I will have to read source from a text file which has got data is specific order i.e storing data in order of cell index and it worked for serial run but not in parallel. (I can of course change the order according to my need). But the problem is in what order I should put the data in text file and read in into an array. I hope I am clear.
Again thanks you so much for giving your time on this thread !!
Your other problem is that cell ID in Fluent can alter every time you do something such as partitioning with a different number of cores. So you may need to switch to using a location, and looking data up from there. As you're not bothered if a cell is set twice the parallelisation of the function can be neglected.
Using locations is a nice idea. Or you can attach the values to a UDM and now eveynode would read same values for each call.
Hello @Rob & @YasserSelima ,
Based on your comments I am trying to modify my codes. I will post back If I am not able to solve it. Or if I solve it then I will post the solution step by step.
Thanks for giving your valuable time !!
Hello @DrAmine @YasserSelima & @Rob ,
I have solved this problem by doing a simple exercise on how cells are being passed to the define_source by the solver. The step are: (Explanation might seem obvious and lengthy but I am posting it so whoever faces same issue can get help).
To understand how cell are being passed by the solver to DEFINE_SOURCE udf. I make a cube domain and mesh it with 2*2*2 cell. (2 cells in each direction -> Total 8 cells). And I use following DEFINE_SOURCE udf which only print a message as:
#include "udf.h"
DEFINE_SOURCE(xmom_source,c,t,dS,eqn)
{
real x[ND_ND];
C_CENTROID(x, c, t);
Message("%lf %lf %lf %d %d %d \n",x[0],x[1],x[2],myid,c,C_ID(c, t));
dS[eqn]=0.0;
return 0;
}
Here x[0], x[1], x[2] are coordinate of centroid, myid is the id of the computing node, “c” is the local index of cell being passed by solver to DEFINE_SOURCE and C_ID(c, t) is the global index of the cell being passed to define_source udf by the solver.
1) I run this case in serial, compile the udf, apply the source term to x-momentum equation and run the solution for 1 iteration. The console output is:
That means, In serial the cell index "c" is the only index and there is nothing like global cell index. Computing node is same for all the cells and global cell index defined by macro C_ID(c,t) is also zero for all the cells. (Note cell index start from zero)
2) I run this case in parallel with two computing nodes, compile udf, apply source to equation and run solution for 1 iteration. The console output is: (Note in parallel 8 cell has become 16 since all the cells are shared by each node or 2 nodes)
That mean in parallel, the index “c” of cell being passed to the udf is different than global cell index. And solver first call cells of 0th node then of 1st node and so on. And global cell index does not change during partition. (Global cell index start from 1 while local cell index start from 0)
This exercise gives the idea that If I can get the Computing node id, Global Cell Id and Corresponding cell coordinates in a text file. Then I can interpolate the source term data onto these coordinate and apply source term by comparing the computing node id and global cell id. Thus, instead of using single dimension array, I used 2d array (where first index was computing node id and second index was global cell id and applied the source term. Or one can even solve this problem by using single dimension array using global cell index "C_ID(c,t)" which does not change during partition.
I hope its clear and right
Thanks everyone for giving your suggestions !!
Thanks for summary. The undocumented C_ID (use it on your own risk) is the way to go + some other was to search for the cell with certain coordinates.
If you have already your source terms defined via X,Y,Z why not directly using Cell Profiles. With 21R1 you can access the profiles in Expression and you can then use the expressions for source terms (might work) :)
@kavatar Thank you for letting us know the solution.
@DrAmine C_ID is mentioned in the documentation 20R1 and R2 ... not sure about 21