I think you have duplicated your question in other post:
S parameters Please do not duplicate it since this is a waste of resources.
From your result, it is clear that the frequency range is too wide that the phase change of S parameters is larger than 2pi. I believe that it is this issue that causes problem. This is the math issue not simulation issue. One simple way is to simulate different frequency range with phase change less than 2Pi. However this may create other issues such as the result is not continuous. I still suggest that you plot the two intermediate quantities and see if they are continuous, and then try to improve the result by all means.
From my experience, sometimes even the two intermediate quantities are continuous, how to calculate the square-root also may lead to different result. This may be script-dependent, eg, different program language could lead two different results but mathematically they are correct. This is mainly due to the branches of the complex square-root operation.
You are navigating away from the AIS Discovery experience