Should I drop the aromatase inhibitor

My gyno flares up and progresses whenever my estrogen is not crashed from aromasin. I'm afraid that if I drop it while I'm seemingly not responding to androgens my body will become feminine, that my hips will widen, I will develop man tits that can't be hidden by a shirt, and I will lose any skeletal growth in my torso that I have left. I am not content with my current frame and I should have had years left for it to fill out properly. On the other hand, estradiol is important for the immune system and I fear I may not recover to a point where I actually do respond to androgens if I keep it gone.

What would you do in my position? Am I being paranoid about the stunted growth and the feminisation?