I've no idea how you are using the controller and apply the heightmap, so it's hard to find out what's wrong here. I'm not aware of any problem with using the controller, so my first guess would be that your code does something that it shouldn't when applying the actual height. A test case would be helpful...