build() does nothing to the objects geometry. However, it does something to the rotation pivot, i.e. it calculates a new one. Without build(), the pivot is at (0,0,0). Try to build the objects (you are not supposed to omit this call, because otherwise, no normals and no bounding box are present) and reset the rotation pivor to (0,0,0) afterwards to see if that helps.