Definitely don't worry about it. My reaction to weed is strange. If I smoke like once every 3 months, then it actually has an amazing effect on my progress. My libido sky rockets, more blood flows down there, and I have stronger erections. But if I then start smoking more frequently, it has the opposite effect and it kills my libido and gives me a numb feeling.
I used to be a huge pot head but now I try to save it for just a handful of times a year on special occasions (typically concerts, which unfortunately/fortunately won't be happening this year).