Hello I am not aware how does ANSYS calculates buckling. However in analytical , to determine the buckling modes of a certain composite material you only need the Force being applied and the respective ABD matrixes. The matrixes are dependent on elastic properties of the material such as E1, E2, G12, v12. As such, it is possible to understand the buckling modes of a plate, without considering yield strength or a failure criterion. Buckling does not mean failure of the material. When buckling occurs the plate or beam cannot handle anymore shear loads. However it does not mean that it has suffer a catastrophic failure.