Dear Guilin Sun
thanks for your advises to my problem. i changed my mesh setting and also extended the simulation region. so i got acceptable response for my waveguide.
now i have another problem. when i simulate my waveguide in mode solution my answers is very sensitive to waveguide radius. my waveguide is a multilayer cylinderical structure and each layer has about 200-300 nanometer radius. when i sweep for 1 nanometer in radius, i get very different and irregular neff that it is not seems to be reasonable. my dielectric layer is a magneto optic material and i havnt teoritical formula for solving neff analytically.