Discrepancy between high core count and low core count simulation results.
Hello ANSYS Community, I have been having an issue where I get different results based on the number of cores I use to solve a problem in ANSYS CFX. I am trying to solve a thermo-electric problem, and the main variables that change with different core counts are the values for heat in, heat out, and heat generated. The error of these values varies from 0.2% to 3%. I have tested core counts from 1 core to 56 cores, and I have found that the best result occurs when using about 6 cores. The study I am doing is a grid convergence study, and this issue occurs regardless of the mesh size that I used. The fine mesh that I am using has 43.2 million elements, the medium is at 5.4 million, and the coarse is at 675 thousand. Also my residual target is 1e-12.
I have tried running the case on multiple Windows machines, as well as a supercomputer, and the same problem occurs. I have enabled the large problem setting and the double precision settings everywhere where possible, and this did not improve the results. Additionally, I have tried different partition settings, mainly Metis RCB, and DRCB. The high core count tests that I have been running are with 24 Cores and the low core count is with 6 cores. For DRCB I have been playing around with the refinement levels and directions, and found that different refinement levels still experience the same discrepancy between the high core count and low core count. However, the discrepancy between results was better with DRCB compared to Metis. What exactly do the refinement level and direction settings do? Whenever I used 3 as a value for directions, the results were off by like 6%. Metis RCB didn’t even run on 24 cores, it experiences an ENFORCE_BOUNDS error, so I couldn’t compare that to the 6 core result that did run. I also tested if it was something to do with the uncertainty of the issue. I made 3 separate Def files and ran the cases, but they all got the same results. I tried Intel vs. IBM MPI settings, and I even tried making the Def file on different machines. These tests did not improve the error at all, and they experienced the same discrepancy between different core count results.
I have data for the things that I have tried and I can provide that if you would like. Anyway, what settings would I need to enable to get consistent results between core counts? Also, why would CFX give different results based on the number of cores used to solve the problem? Thank you for your time.