The numerical beach will start to effect the waves at 10m into the domain, so you may want to reduce that. The other possible issue is cell shape. VOF can be sensitive to aspect ratio, so 100 cells/m length ways and 20cells/5cm height gives an aspect ratio of 1cm: 0.25cm. That should be OK, but try putting a more-or-less uniform square mesh on the domain and then adapt the wave region to improve the resolution.
Note, 5cm waves aren't very big so you are going to need a good mesh and time resolution to get good results.
You are navigating away from the AIS Discovery experience