Case involving thin structures (sheet metal, membranes, etc), structural materials with low youngs modulus are typically unstable. To identify the instability you can monitor pressure/force convergence for FSI surfaces within each time step, which will help you to identify if the case is about to 'blow up' (Oscillation within time step) or you have achieved a good and stable convergence. Solution may be instable due to poor initialization.
Keep 'Solution Stabilization' enabled, it will slow down the pressure response in ANSYS Fluent at the deforming wall. Here higher sacle factor results in more stabilization (Start with low value and increase gradually)
Ensure that individual participant solution (e.g. Fluent alone) converges well. Look at the residuals, which equation is causing convergence issue. Tune in the parameters related to that equation.