You do what's comfortable for you. There are ALOT of people who do Wim Hof and I'm pretty sure none of them are brain damaged from it....lol.
What problems are you referring to? Most people who practice this have said nothing but how it transformed their lives for the better, myself included. The thing is, people, for the most part, are notoriously horrible at following directions properly. It's just the human condition. Of course you're going to have some people do it wrong and cause their own problems because of it.
But as you mentioned, it is not a breath holding contest. You do what is comfortable, never forced on the inhales or retentions, holds, etc.