Learning Forum

[jms26]

Hi, I am having an issue running a steady 3ddp conjugate HT simulation in parallel on Linux where everything runs smoothly until I activate /solve/set/advanced/retain-cell-residuals. Once I do this and restart calculation, a SIGSEGV error is thrown on numerous nodes; for example: "Node 111: Process 150786: Received signal SIGSEGV."

This is followed by some lines with the error: "double free or corruption (!prev): 0x000000000e9ee2c0 ***" and then tons of lines following underneath a section called "Backtrace".

Fluent properly developed a shadow BC for the connection between the fluid and solid volume meshes after I imported it, with them set to "Coupled" as desired. I have triple checked my BCs, and they all appear perfectly fine.

A few more things:

1) I have verified that it is not a memory issue, and I have tried starting Fluent with the -cflush option with no luck.

2) I have seen that UDFs may cause this issue, but I am not using any UDFs.

3) I am able run the simulation and export data with no issues - this issue ONLY occurs when I activate retain-cell-residuals.

Any advice would be greatly appreciated.

[KR]

Hello:
This seems weird. What happens when you start the simulation from scratch and use the TUI for retaining the residuals?
Karthik

[Rob]

cflush just clears the system, if you're adding too much load it'll not do anything. If you use "top" how much free RAM is there? Is it Stack backtrace you're seeing?

[jms26]

Hi Karthik Thanks for your reply - the same issue occurs when I start from scratch using standard initialization and run it in the TUI. I had completed this same simulation with only the fluid domain and had no issues retaining cell residuals. It may be something to do with now having a solid AND fluid domain. Please let me know if any additional info would be helpful.

[jms26]

Hi Rob Thanks for your reply. I've tried both with using excessive amounts of RAM (>3TB) and also watching htop while trying to run; it definitely doesn't come close to the maximum amount of memory available. I don't see the word "Stack" anywhere if that's what you're referring to (my knowledge of this lower level computational stuff doesn't really go beyond just knowing that SIGSEGV means that something is trying to access memory that doesn't exist). Here are some of the many lines outputted:
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x2ad2e8253329]
/lib64/libc.so.6(+0x39d10)[0x2ad2e820bd10]
/lib64/libc.so.6(+0x39d37)[0x2ad2e820bd37]
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x2ab7fe9b9329]
/lib64/libc.so.6(+0x39d10)[0x2ab7fe971d10]
/lib64/libc.so.6(+0x39d37)[0x2ab7fe971d37]
/home1/apps/ANSYS/2021R2/v212/fluent/fluent21.2.0/multiport/lnamd64/mpi/shared/libmport.so(+0x87d67)[0x2ab7ee0fdd67]

And there are also "Memory Map" sections embedded throughout the terminal output as well. Please let me know if any additional information would be helpful!

[Rob]

What happens if you run with the solid zone and then turn on the residual store?

[jms26]

Hi Rob
Apologies for the late reply - Yes I tried that, it did not work. Very interestingly: if I deactivate the solid zone and the run, this error does not occur and I am able to retain-cell-residuals with no issues. Only when the solid zone is activated is it problematic.
Do you think it may be related at all to issues with the solid-to-fluid cell weighting in the partitions? I am running with 1344 processes.

[Rob]

Shouldn't do. It does seem like a problem with the solid zone only though. I'll check this side, but may not be able to pass much back. What is the cell count as 1344 cores is a fairly large number!