Parallelizing “DEFINE ON_DEMAND” & “DEFINE_SOURCE” UDF to apply x-momentum source from a text file

Hello,

This problem is related to fluent UDF.

I have a text file storing x-momentum source at every cell center of the domain. I store the source term in order of global cell index in the text file. i.e. value in first line correspond to source of cell with cell index 0, value of second line correspond to source of cell with cell index 1 and so on. I want to read this text file and add source term to x-momentum equation. My UDF for serial run is following (which is working fine)


#include "udf.h"

float xSource[194875]; //global array variable


DEFINE_ON_DEMAND(read_source)

{

int i;

FILE *xSourceFile; // declare a FILE pointer

xSourceFile = fopen("Fx.txt", "r"); // open file for reading

Message("xSourceFile file opened\n");


/* loop to read file */

i=0; // initialize before to loop

for (i=0;i<194875;i++)

{

fscanf (xSourceFile, "%f", xSource+i); // one value per line

}

fclose(xSourceFile);

Message("xSourceFile file closed\n");

}

DEFINE_SOURCE(xmom_source,c,t,dS,eqn)

{

dS[eqn]=0.0;

return xSource[c]; //xSource array has values in global index order

}


The above code works fine in serial. To run the same case in parallel I have made following udf:


#include "udf.h"

float xSource[194875]; //global array variable


DEFINE_ON_DEMAND(read_source)

{

#if !RP_NODE

int i;

FILE *xSourceFile; // declare a FILE pointer

xSourceFile = fopen("Fx.txt", "r"); // open file for reading

Message("xSourceFile file opened\n");


/* loop to read file */

i=0; // initialize before to loop

for (i=0;i<194875;i++)

{

fscanf (xSourceFile, "%f", xSource+i); // one value per line

}

fclose(xSourceFile);

Message("xSourceFile file closed\n");

#endif

host_to_node_float(xSource,194875);

}

DEFINE_SOURCE(xmom_source,c,t,dS,eqn)

{

#if !RP_HOST

dS[eqn]=0.0;

return xSource[c]; //xSource array has values in global index order

#endif

}


My observations are

  • First udf works fine when I run case in serial and gives perfect results
  • Second udf works fine when I run case in serial and gives perfect results
  • Second udf does not work properly when I run case in parallel. It gives warning as "warning C4716: xmom_source': must return a value" . And, results are also not good (not similar to first two cases )


Can anyone please identify what is the problem with second udf when I run it in Parallel ?.


It would be really helpful to me.


Thanks in advance

Comments

  • The host and nodes must read the define source .. ds(..) must be seen by both. Remove the if statement

  • DrAmineDrAmine GermanyForum Coordinator

    Define source does not require parallisation in general.

  • Hello @YasserSelima & @DrAmine ,


    Thanks a ton for your response !!


    Is this what you are suggesting ? Because I already tried this and its not giving proper results.


    DEFINE_ON_DEMAND(read_source)

    {

    #if !RP_NODE

    int i;

    FILE *xSourceFile; // declare a FILE pointer

    xSourceFile = fopen("Fx.txt", "r"); // open file for reading

    Message("xSourceFile file opened\n");


    /* loop to read file */

    i=0; // initialize before to loop

    for (i=0;i<194875;i++)

    {

    fscanf (xSourceFile, "%f", xSource+i); // one value per line

    }

    fclose(xSourceFile);

    Message("xSourceFile file closed\n");

    #endif

    host_to_node_float(xSource,194875);

    }

    DEFINE_SOURCE(xmom_source,c,t,dS,eqn)

    {

    dS[eqn]=0.0;

    return xSource[c]; //xSource array has values in global index order

    }


    I already tried this . When I compile , warning "warning C4716: xmom_source': must return a value"  does not appear . But it does not give results similar to the one I get in serial run using the first UDF.

    If you have any other suggestion please comment.


    A quick doubt- Is the global cell index different in serial and parallel run ? or it will remain same even when the mesh is partitioned to computing nodes


    Thanks again

  • DrAmineDrAmine GermanyForum Coordinator

    That might cause an issue as it starts from zero for each partition

  • YasserSelimaYasserSelima Member
    edited January 13

    try to print a message with c value. Probably the problem is there

  • Hello @DrAmine & @YasserSelima ,


    As per your suggestion I wrote following UDF to check which cells are being passed to DEFINE_SOURCE and at what index. I also write them in a text file after one iteration using DEFINE_ON_DEMAND udf.


    //There are total 194875 cells in the domain

    #include "udf.h"


    int indexC[194875]; //To store the index which will be passed in Define_Source

    float Xc[194875]; //x-coordinate of cell being passed to Define_Source

    float Yc[194875]; //y-coordinate of cell being passed to Define_Source

    float Zc[194875]; //z--coordinate of cell being passed to Define_Source


    int j=0;


    //This udf does nothing except storing the index and coordinates of cells being passed to define source by solver

    DEFINE_SOURCE(xmom_source,c,t,dS,eqn)

    {

    real x[ND_ND];

    C_CENTROID(x, c, t);


    indexC[j]=c; //Storing cell index in global array

    Xc[j]=x[0]; //Storing cell x-coordinate in global array

    Yc[j]=x[1]; //Storing cell y-coordinate in global array

    Zc[j]=x[2]; //Storing cell z-coordinate in global array

    j++; //increasing global cell counter

    dS[eqn]=0.0;

    return 0;

    }


    DEFINE_ON_DEMAND(write_data)

    {

    int i;

    FILE *CellFile;

    CellFile = fopen("cF.txt","w");

    for (i=0;i<j;i++)

    {

    fprintf(CellFile,"%d ",indexC[i]);

    fprintf(CellFile,"%f ",Xc[i]);

    fprintf(CellFile,"%f ",Yc[i]);

    fprintf(CellFile,"%f ",Zc[i]);

    fprintf(CellFile,"\n");

    }

    fclose(CellFile);

    }


    So when I run this UDF in serial

    • I get all 194875 cells with index and coordinate in cF.txt file (index goes from 0 to 194875)

    When I run this UDF in Parallel

    • I get only 51870 cells with index and coordinates in cF.txt file (index goes from 0 to 51870)


    And coordinates corresponding to index 0 in serial and parallel run are different and similarly for other indexes. @DrAmine This proves your point that cells start from 0 in each partition. But

    1. Why I am only getting 51870 cells when I run in parallel ? The file cF.txt should give all 194875 cells data. Though the index should be something like 0,1,2---51870, 0,1 2,-----51870, 0 1, 2------ and so on till 194875th column
    2. Can you suggest any other way to sail through this problem . So lets say If I have a text file containing source terms for all the cells. Each line of this text file has has four entries (X, Y, Z, Fx). Here X,Y,Z are cell centre of the mesh and Fx is x-source term ?
  • I remember I have seen in the manual a macro that gets the index. Not sure of its name ... if you find it, this would solve your problem

  • Hello @YasserSelima ,


    The macro you are suggesting is C_ID which output the global cell indexes. But I am not sure if that will give cell indexes in parallel since all the computing nodes are having cell index ranging from 0,1,2......


    The problem is that I will have to read source from a text file which has got data is specific order i.e storing data in order of cell index and it worked for serial run but not in parallel. (I can of course change the order according to my need). But the problem is in what order I should put the data in text file and read in into an array. I hope I am clear.


    Again thanks you so much for giving your time on this thread !!

  • RobRob UKForum Coordinator

    Your other problem is that cell ID in Fluent can alter every time you do something such as partitioning with a different number of cores. So you may need to switch to using a location, and looking data up from there. As you're not bothered if a cell is set twice the parallelisation of the function can be neglected.

  • Using locations is a nice idea. Or you can attach the values to a UDM and now eveynode would read same values for each call.

  • kavatarkavatar Member
    edited January 14

    Hello @Rob & @YasserSelima ,


    Based on your comments I am trying to modify my codes. I will post back If I am not able to solve it. Or if I solve it then I will post the solution step by step.

    Thanks for giving your valuable time !!

  • kavatarkavatar Member
    edited January 15

    Hello @DrAmine @YasserSelima & @Rob ,

    I have solved this problem by doing a simple exercise on how cells are being passed to the define_source by the solver. The step are: (Explanation might seem obvious and lengthy but I am posting it so whoever faces same issue can get help).

    To understand how cell are being passed by the solver to DEFINE_SOURCE udf. I make a cube domain and mesh it with 2*2*2 cell. (2 cells in each direction -> Total 8 cells). And I use following DEFINE_SOURCE udf which only print a message as:

    #include "udf.h"

    DEFINE_SOURCE(xmom_source,c,t,dS,eqn)

    {

                   real x[ND_ND];

                   C_CENTROID(x, c, t);

                   Message("%lf %lf %lf %d %d %d \n",x[0],x[1],x[2],myid,c,C_ID(c, t));

     

                   dS[eqn]=0.0;

                   return 0;

    }

    Here x[0], x[1], x[2] are coordinate of centroid, myid is the id of the computing node, “c” is the local index of cell being passed by solver to DEFINE_SOURCE and  C_ID(c, t) is the global index of the cell being passed to define_source udf by the solver. 

    1)     I run this case in serial, compile the udf, apply the source term to x-momentum equation and run the solution for 1 iteration. The console output is:

    That means, In serial the cell index "c" is the only index and there is nothing like global cell index. Computing node is same for all the cells and global cell index defined by macro C_ID(c,t) is also zero for all the cells. (Note cell index start from zero)

    2)      I run this case in parallel with two computing nodes, compile udf, apply source to equation and run solution for 1 iteration. The console output is: (Note in parallel 8 cell has become 16 since all the cells are shared by each node or 2 nodes)

    That mean in parallel, the index “c” of cell being passed to the udf is different than global cell index. And solver first call cells of 0th node then of 1st node and so on. And global cell index does not change during partition. (Global cell index start from 1 while local cell index start from 0)

    This exercise gives the idea that If I can get the Computing node id, Global Cell Id and Corresponding cell coordinates in a text file. Then I can interpolate the source term data onto these coordinate and apply source term by comparing the computing node id and global cell id. Thus, instead of using single dimension array, I used 2d array (where first index was computing node id and second index was global cell id and applied the source term. Or one can even solve this problem by using single dimension array using global cell index "C_ID(c,t)" which does not change during partition.

    I hope its clear and right

    Thanks everyone for giving your suggestions !!

  • DrAmineDrAmine GermanyForum Coordinator

    Thanks for summary. The undocumented C_ID (use it on your own risk) is the way to go + some other was to search for the cell with certain coordinates.

    If you have already your source terms defined via X,Y,Z why not directly using Cell Profiles. With 21R1 you can access the profiles in Expression and you can then use the expressions for source terms (might work) :)

  • @kavatar Thank you for letting us know the solution.

    @DrAmine C_ID is mentioned in the documentation 20R1 and R2 ... not sure about 21

Sign In or Register to comment.