One thing I tried was to multiply the mouse coordinates by 2. I think I need to do this, but it didn't solve my problem.
For whatever reason, the project3D2D works fine. Given a point in 3D, I can figure out where it is on the 2D screen.
I was thinking I could do the mapping from 2D to 3D myself (knowing the mapping of the origin, a point at 1,0,0 and a point at 0,1,0), but the perspective rendering make it difficult.
Let me try to better explain what I'm trying to do:
Suppose I have a rectangle in the z=0 plane with the 4 points: (0,0,0),(1,0,0),(1,1,0),(0,1,0). (in world space)
I could be viewing this rectangle from any direction. (example, put the camera at position (1,1,1) pointing to the origin)
When the mouse moves over this rectangle, I want to know the coordinates within the rectangle. (ie. map a point in on the 2D screen to a point in the 3D world space at z=0).