Yes, the number of contact points changing status should go down to a small number or zero for that increment to be considered to have converged.
You can reduce the number of iterations needed to achieve this by softening the contact. In the Details for a Frictional Contact, set the Normal Stiffness to a Factor and set the Factor to 0.1 instead of the default 1.0