Learning Forum

[Florian Coors]

Hello everybody,

 

I am simulating a 3d, dp, eulerian phase + particle, transient case using fluent 21.2.

The simulation is working and I am running several timesteps, writing a .dat file of the results of the last timestep, trying to read the same file and fluent gives me the following error:

 

 

I am thankful for any suggenstions and help

BR Florian

[Rob]

Did you write case and data at the end of the run? Are any of the intermediate saves OK? 

[Florian Coors]

Overall, the case is supposed to run as a job array and I had cases were I couldn't open the Autosave files as well. 

[Florian Coors]

I am Sorry, I am only writing data at the end of my simulation. I saved the case before start

[Rob]

Has the mesh or any boundary type changed? The data read suggests the particles are being passed to cells, which I wouldn't expect so I wonder if something's got scrambled between case & data. It's usually good practice to save case & data for results in case something gets changed. 

[Florian Coors]

No nothing has changed. In fact, I wrote the data file and immediately tried to read it again without closing the case or changing anything and fluent was not able to read it.

And I also tried to save the data first and writing the case file afterwards. 

I tried to run this in batch mode (thats where the problem originally ocurred) and this gave me some error log file. Maybe there is some information that could help you and I don't understand.

Thanks for your help!

 

myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14c660247c20]
 /lib64/libc.so.6(+0x39ad7) [0x14c652682ad7]
 /lib64/libc.so.6(on_exit+0) [0x14c652682c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14c660247c20]
 /lib64/libc.so.6(__sched_yield+0xb) [0x14c65272c6bb]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x14c650810923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14c650976df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14c6507cfb84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14c6507d4ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14c662eefaf7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14c6631a359f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14c663158aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14c66314e8ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14c65266c493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x147deb87cc20]
 /lib64/libc.so.6(+0x39ad7) [0x147dddcb7ad7]
 /lib64/libc.so.6(on_exit+0) [0x147dddcb7c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x147deb87cc20]
 /lib64/libc.so.6(__sched_yield+0xb) [0x147dddd616bb]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x147ddbe45923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x147ddbfabdf0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x147ddbe04b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x147ddbe09ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x147dee524af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x147dee7d859f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x147dee78daa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x147dee7838ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x147dddca1493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14a2a596bc20]
 /lib64/libc.so.6(+0x39ad7) [0x14a297da6ad7]
 /lib64/libc.so.6(on_exit+0) [0x14a297da6c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14a2a596bc20]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0xb1c) [0x14a295f3435c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14a29609adf0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14a295ef3b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14a295ef8ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14a2a8613af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14a2a88c759f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14a2a887caa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14a2a88728ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14a297d90493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14ce8f935c20]
 /lib64/libc.so.6(+0x39ad7) [0x14ce81d70ad7]
 /lib64/libc.so.6(on_exit+0) [0x14ce81d70c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14ce8f935c20]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0xcc6) [0x14ce7fefe506]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14ce80064df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14ce7febdb84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14ce7fec2ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14ce925ddaf7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14ce9289159f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14ce92846aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14ce9283c8ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14ce81d5a493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12ce0) [0x14a715ff0ce0]
 /lib64/libc.so.6(+0x511b7) [0x14a7084431b7]
 /lib64/libc.so.6(on_exit+0) [0x14a708443320]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12ce0) [0x14a715ff0ce0]
 /lib64/libc.so.6(__sched_yield+0xb) [0x14a70842b03b]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x14a7065b9923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14a70671fdf0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14a706578b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14a70657dca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14a718c98af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14a718f4c59f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14a718f01aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14a718ef78ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14a70842cca3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14ab473a1c20]
 /lib64/libc.so.6(+0x39ad7) [0x14ab397dcad7]
 /lib64/libc.so.6(on_exit+0) [0x14ab397dcc40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14ab473a1c20]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x1168) [0x14ab3796a9a8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14ab37ad0df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14ab37929b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14ab3792eca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14ab4a049af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14ab4a2fd59f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14ab4a2b2aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14ab4a2a88ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14ab397c6493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x146fd9b10c20]
 /lib64/libc.so.6(+0x39ad7) [0x146fcbf4bad7]
 /lib64/libc.so.6(on_exit+0) [0x146fcbf4bc40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x146fd9b10c20]
 /lib64/libc.so.6(__sched_yield+0xb) [0x146fcbff56bb]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x146fca0d9923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x146fca23fdf0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x146fca098b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x146fca09dca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x146fdc7b8af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x146fdca6c59f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x146fdca21aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x146fdca178ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x146fcbf35493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x148663117c20]
 /lib64/libc.so.6(+0x39ad7) [0x148655552ad7]
 /lib64/libc.so.6(on_exit+0) [0x148655552c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x148663117c20]
 /lib64/libc.so.6(__sched_yield+0xb) [0x1486555fc6bb]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x1486536e0923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x148653846df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14865369fb84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x1486536a4ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x148665dbfaf7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14866607359f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x148666028aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14866601e8ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14865553c493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14ae70ab9c20]
 /lib64/libc.so.6(+0x39ad7) [0x14ae62ef4ad7]
 /lib64/libc.so.6(on_exit+0) [0x14ae62ef4c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14ae70ab9c20]
 /lib64/libc.so.6(__sched_yield+0xb) [0x14ae62f9e6bb]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0x10e3) [0x14ae61082923]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14ae611e8df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14ae61041b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14ae61046ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x14ae73761af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14ae73a1559f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x14ae739caaa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x14ae739c08ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x14ae62ede493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]
 myid (0): Fatal signal raised sig = SIGSEGV 
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x28913dc]
 /lib64/libpthread.so.0(+0x12c20) [0x14564e5f8c20]
 /lib64/libc.so.6(+0x39ad7) [0x145640a33ad7]
 /lib64/libc.so.6(on_exit+0) [0x145640a33c40]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x288e50e]
 /lib64/libpthread.so.0(+0x12c20) [0x14564e5f8c20]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPIDI_CH3I_Progress+0xdc9) [0x14563ebc1609]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0x29ddf0) [0x14563ed27df0]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(+0xf6b84) [0x14563eb80b84]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi/lnamd64/intel/lib/libmpi.so.12(PMPI_Allreduce+0x686) [0x14563eb85ca6]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/mpi_wrapper/lnamd64/intel/libmpi_wrapper.so(WMPI_Allreduce+0xa3) [0x1456512a0af7]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(mpt_mpi_reduce+0x1a8) [0x14565155459f]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum+0x11d) [0x145651509aa8]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(MPT_gdsum1+0xa0) [0x1456514ff8ba]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_Boundary_Velocity+0x6c93) [0xd1dd13]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(Update_After_Data_Read+0x673) [0xc610f3]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5c1dbd]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(eval+0x4b5) [0x28ab0f5]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Command_Start+0x133c) [0x6b730c]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(PRF_Node_repl+0x2e) [0x6b85ce]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(fluent_main+0x3f9) [0x6cf6f9]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0(main+0x59) [0x6e1589]
 /lib64/libc.so.6(__libc_start_main+0xf3) [0x145640a1d493]
 /shared/apps/ansys/21.2/v212/fluent/fluent21.2.0/lnamd64/3ddp_node/fluent_mpi.21.2.0() [0x5bb251]

[Rob]

Are you writing .cas.gz and .dat.gz or .cas.h5 and .dat.h5 ?  The log's not much use in this case, other than the error may be node related. 

[Florian Coors]

I am writing .dat.h5 and .cas.h5.

I am running the simulation on a hpc-computer. I just figured out,when using a login node it is surprisingly always node 7 that gives the error (using 30 threads). But in batch mode, I am running on 48 cores with 40 threads and it is always node 10. Is there anything i could do about it?

[Rob]

Try writing legacy (cas.gz & dat.gz), it's possible there's a data packet loss from one of the nodes. Not sure about the cores as Fluent will use what it's told to (but can't not use a specific thread on a core). Maybe IT need to reboot it?