Guilin Sun
Ansys Employee

"How can know which number of trial modes is physically correct? "

As long as you can find the desired modes.

"when I run a frequency sweep, the n_eff plotted in the sweep window does not correspond with the n_eff I get when solving Mode Analysis for the same wavelength."

If it is for the same mode, it is unlikely. The most possible explanation is, it tracks the wrong mode in sweep or independently. Please make sure the modes are the same.

if you use metal BCs, there are many no physical modes since the metals form a cavity, leading to cavity modes, or "box modes".