It could be a mesh problem. And keep in mind that we are also discretizing in the time space, so a smaller time step may help. What are you using for the time advancement? Allowing multiple iterations per time step is usually necessary.

If you are mesh size limited, I would try to reduce it to a 2d planar or an axisymmetric case. It may not give you the final results you want, but it will give you a way to quickly test the necessary mesh and time step sizes. From there you can estimate what the 3D mesh size will be, and the time step will be similar to what you used in the 2D case. From there you can judge if it is even possible to solve with the license.