Because I've always had this issue of brain fog/ general fatigue after masturbating, I wouldn't be able to tell you whether masturbating every 4/5 days will set you back.
What I can tell you, however, is that masturbation does increase prolactin, which subsequently decreases dopamine. It's likely that, by decreasing your dopamine/ increasing your prolactin, you are screwing yourself over.