I believe if values are repetitively added, I suppose you should do an inverse operation at the end of what you are doing..? So maybe the inverse matrix or something would help (Matrix.invert()).
What you could also do is not apply the initial rotation at first but only at the end maybe. So after your vertex controller things..?
I'm not sure if I entirely understand your problem. It's hard to understand if you don't see what's happening