Don't worry, that topic of yours is in my modelling Valhalla. Turned out it was the normal mapping as I thought, I think I used a GIMP generated file in photoshop and that messed up the transparency of the bumps layer, but I'm still not sure. Kudos to Louis Lux for helping me. Now it's working like a charm:
Spoiler Alert, click show to read: