Failure to converge, or slow convergence, can be caused by many factors.
Certain combinations of objective and response constraints might be creating a mathematical over constraint (i.e. not all the criteria can be satisfied simultaneously).
Or perhaps it can be solved, but you need more unconstrained DOFs (finer mesh) in the design domain to capture the optimal solution.
I would recommend when your model fails to converge, start by reviewing the solver output for any relevant feedback as to what the code is struggling with.
Since you mention that this is a vessel, I assume that one of the criteria is that it needs to envelop a "watertight cavity."
Ansys supports this feature as a manufacturing constraint. Are you including that? If not, this might help with convergence and with quality of result.
For more details, please refer to Mechanical User's Guide =>Analysis Type=>Structural Optimization=>Opt Workflow=>Obj & Constraints=>Manufacturing Constraints…