Author Topic: Problem with object picking and modified camera backbuffer matrix  (Read 14021 times)

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Hi,

I'm currently developing an AR-App and use jPCT-ae for rendering. To affix the model on a marker I receive a pose matrix and apply her to the camera with camera.setBack(). This works for putting my model on the marker, but I can't get picking to work with it. If I just render my model without changing the camera matrix it works fine, but as soon as I change her, it doesn't work anymore (calcMinDistanceAndObject3D() returns everywhere on my object COLLISION_NONE). Since the picking needs the camera position, I assume that something with the position vector is wrong.

My picking code:
Code: [Select]

private void doPicking(int touchX, int touchY){
SimpleVector dir = Interact2D.reproject2D3DWS(camera, frameBuffer, touchX, touchY).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
  if (res[1] != null) {
    //do something...
 
  }
}


Object properties:

Code: [Select]
Object3D[] serializedObject = Loader.loadSerializedObjectArray(serializedInputStream);

for (int i = 0; i < serializedObject.length; i++) {
serializedObject[i].setCollisionMode(Object3D.COLLISION_CHECK_OTHERS);
if (i > 0) serializedObject[i].addParent(serializedObject[0]);
serializedObject[i].strip();

}


Camera pose setting:

Code: [Select]

public void onDrawFrame(GL10 arg0) {

//...

float[] matrix = getMatrixForJPCT(new float[16]);

Matrix mat = new Matrix();
mat.setDump(matrix);

//If I comment this out, picking works flawless:
camera.setBack(mat);

//...
}


Any idea why that is?

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #1 on: December 15, 2011, 04:24:07 pm »
Does the tweaked matrix include a translation? If so, try to remove it from the matrix and set it as the camera's position instead.

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #2 on: December 19, 2011, 11:30:56 am »
Thanks, that brought me on the right track. The AR-engine converted the pose to a OpenGL-compatible matrix beforehand, which additionally screwed my calculations. But now I have some trouble doing the correct conversions between the coordinate systems now.
The AR-engine uses a right handed system (Picture). According to your wiki, I need to rotate the matrix by 180 around x to convert it to the jPCT system. But do I need to do it before or after setting the camera matrix? I currently extract the rotation/translation from the float array, dump it into a Matrix/SimpleVector, rotate the Matrix/SimpleVector by 180 around x and set them as the backbuffer/position. The car is fixed to the marker, but behaves totally unpredictable (it seems as the model sticks to the camera, and not to the marker) as I move around my device, so I guess my conversion is wrong.
« Last Edit: December 19, 2011, 12:50:16 pm by redfalcon »

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #3 on: December 19, 2011, 07:45:12 pm »
Apart from the picking, was it working before? If so, just do what you did before separate the rotation from the translation afterwards.

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #4 on: December 20, 2011, 09:23:01 am »
Yes, besides some unwanted minor movement of the model when I move my device, everything but picking works. But the separating doesn't really work. As soon as I set the position of the camera, the model doesn't stay on the marker.

The GL-like matrix that does the placement correctly (but not the picking) looks like this:
Code: [Select]
-0.996   0.069  -0.063   0.000
0.092   0.808  -0.582   0.000
0.011  -0.586  -0.811   0.000
-75.993 -110.593 1902.010   1.000

I set this directly as the backbuffer and leave the position alone.

The unmodified pose itself is just transposed:

Code: [Select]
-0.996   0.092   0.011 -75.993
0.069   0.808  -0.586 -110.593
-0.063  -0.582  -0.811 1902.010

Looks like a 3x3 rotation matrix and a 3x1 translation vector to me.

If I set this matrix as the cameras backbuffer and this vector as the position, the picking works but the model is somewhere far off and doesn't stay on the marker if I move the device.

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #5 on: December 20, 2011, 09:43:05 am »
Strange...i don't see why it shouldn't work that way assuming that the rotation matrix doesn't contain the translation anymore when setting it.
If nothing else helps, you can try to do the projection into world space on your own based on the combined matrix. The reason why it doesn't work with the combined matrix is most likely that reproject2D3DWS treats the matrix as a 3x3 matrix for performance reasons. You can try

Code: [Select]
public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = reproject2D3D(camera, buffer, x, y, VIEWPLANE_Z_VALUE, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}

instead. However, the clean solution would be to separate the two transformations...

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #6 on: December 20, 2011, 10:12:00 am »
Just that I get it right:
What should I use for VIEWPLANE_Z_VALUE, the far clipping plane?
Code: [Select]

//...
public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = Interact2D.reproject2D3D(camera, buffer, x, y, 2000, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}

public void findNearestObject(int touchX, int touchY) {

SimpleVector dir = reproject2D3DWS(camera, frameBuffer, touchX, touchY, new SimpleVector()).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
              //...
}


Is that correct?

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #7 on: December 20, 2011, 10:35:29 am »
No, just use 1f.

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #8 on: December 20, 2011, 12:10:59 pm »
No Luck, that doesn't work either :/

I tried a different approach, and set the GL-Matrix as the rotation matrix of my model. This works fine for placement and picking, but now the rotations (With object.rotateX()) rotate around an axis which seems to be far above the object (so it's pretty unusable). Which is normal I guess, since I modified the rotation matrix. Is there a way to get a proper axis that I can use with rotateAxis() or the like?

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #9 on: December 20, 2011, 12:20:55 pm »
If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #10 on: December 20, 2011, 12:22:16 pm »

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #11 on: December 20, 2011, 01:41:16 pm »
If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.

If I set the negative pivot as pivot, the model will instantly snap back to its previous position when trying to rotate it.

I've tried the other solution, but since it uses a different (geographic) coordinate system again, it doesn't work either. The model gets attached to the camera and doesn't stay where it should.

Offline EgonOlsen

  • Administrator
  • quad
  • *****
  • Posts: 11769
    • View Profile
    • http://www.jpct.net
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #12 on: December 20, 2011, 01:57:07 pm »
Then go back to this: http://www.jpct.net/forum2/index.php/topic,2461.msg18064.html#msg18064

The problem here seems to be that camera.getPosition() doesn't provide the proper position now that you don't set it correctly but hide it in the matrix. Try to multiply the camera's position with the matrix and use that as the new position.

Offline redfalcon

  • byte
  • *
  • Posts: 14
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #13 on: December 23, 2011, 08:13:55 pm »
I scrapped my approach and now pass the rotation values (from the touchscreen) directly to the pose calculation method. The AR engine provides a method for rotating the pose matrix, so I just pass the values I usually would use for object.rotateX/Y/Z() to the JNI function and rotate the matrix itself.
Thanks anyway!

Offline Vi_O

  • byte
  • *
  • Posts: 17
    • View Profile
Re: Problem with object picking and modified camera backbuffer matrix
« Reply #14 on: January 09, 2012, 05:04:11 pm »
Hello,

I am having the same issue than redfalcon and indeed, separating doesn't work.

I made a bit of code to see the difference between normal and rotation/translation separated :

Code: [Select]

_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);

float x = matrix[12]; float y = matrix[13]; float z = matrix[14];
matrix[12] = matrix[13] = matrix[14] = 0;
_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);
_camera.setPosition(x, y, z);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);

It is strange because getPosition and get*Axis give the same results for both but the separated one doesn't place the camera well... And still no picking...

From what you were writing there, it is a problem with the getPosition that returns {0.0,0.0,0.0}.

Would it be possible then to extends Camera and Overriding setPosition/getPosition to make the reproject2D3DWS and calcMinDistanceAndObject3D works (assuming it uses getPosition to work) ?

I can't test it now as I don't have my source right now, but i'll try tomorrow if you don't tell me it is useless before :)