No, there isn't. You are right that the matrix interpolation fails, if the input matrices are too different. Imagine one matrix that rotates a vertex to 0,0,1 and another that rotates it to 0,0,-1. It's impossible to interpolate between these matrices by using a linear interpolation of the matrices' values and create proper inbetween states that way. jPCT ensures that the interpolated matrix is a valid rotation matrix, but it doesn't make any assurances if it really fits the purpose.

You have to use your own quaternion implementation for this. Cyberkilla's old Skeletal-API from the download page should have one IIRC. Maybe that helps.