Guilin Sun
Ansys Employee

First, this method neff_total=neff_cell1*(fill_factor_cell1)+neff_cell2*(fill_factor_cell2) is an estimation and sometimes people do use it. 

I do not find any guideline in our official webaite, and nor find any publications on how to simulate the band structure of higher order modes.  However, there might be different ways doing this. For example, you can either use TE1 excitation and get the phase of the mode though Fourier transform. Please refer to this example  Once you have the phase function as frequency, you can get neff and ng based on defintion.  You can have a DFT monitor and check if the final field profile is the same as the 2nd order mode.