John Doyle
Ansys Employee

Try switching to Level Set Method.  It might be more robust for your convection load.  Refer to the Mechanical USers Guide=>Optimization Analysis Workflow=>Apply Design Objectives and Constraints...Thermal Compliance...

For the Topology Optimization - Level Set Based and Shape Optimization methods, the Thermal Compliance option uses a unique formula regardless of the context. That is, it executes in the presence of Heat Flux, Heat Flow, prescribed Displacement, Convection conditions, and/or Internal Heat Generation.