I would just drop them all if you feel they are making you worse. Although, your biggest side effect is skin issues so hormones can moderately help with that due to the reduction in cortisol. But again, skin issues can be a million reasons in itself.
Other than that I'm not sure the duration you been taking hormones but I wouldn't suggest anything else at this point, it seems you are not a good responder to hormones so I would just use the herbs if you felt fine on that.