I think you said it worked in transient mode? If that's the case there's something off in the UDF or mass transfer macros. Common one to check is TP & PP for particle states (covered in the UDF manual).
Do transient particles on steady flow give sensible results?