www.jpct.net

jPCT-AE - a 3d engine for Android => Support => Topic started by: redfalcon on December 15, 2011, 03:40:51 pm

Title: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 15, 2011, 03:40:51 pm
Hi,

I'm currently developing an AR-App and use jPCT-ae for rendering. To affix the model on a marker I receive a pose matrix and apply her to the camera with camera.setBack(). This works for putting my model on the marker, but I can't get picking to work with it. If I just render my model without changing the camera matrix it works fine, but as soon as I change her, it doesn't work anymore (calcMinDistanceAndObject3D() returns everywhere on my object COLLISION_NONE). Since the picking needs the camera position, I assume that something with the position vector is wrong.

My picking code:
Code: [Select]

private void doPicking(int touchX, int touchY){
SimpleVector dir = Interact2D.reproject2D3DWS(camera, frameBuffer, touchX, touchY).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
  if (res[1] != null) {
    //do something...
 
  }
}


Object properties:

Code: [Select]
Object3D[] serializedObject = Loader.loadSerializedObjectArray(serializedInputStream);

for (int i = 0; i < serializedObject.length; i++) {
serializedObject[i].setCollisionMode(Object3D.COLLISION_CHECK_OTHERS);
if (i > 0) serializedObject[i].addParent(serializedObject[0]);
serializedObject[i].strip();

}


Camera pose setting:

Code: [Select]

public void onDrawFrame(GL10 arg0) {

//...

float[] matrix = getMatrixForJPCT(new float[16]);

Matrix mat = new Matrix();
mat.setDump(matrix);

//If I comment this out, picking works flawless:
camera.setBack(mat);

//...
}


Any idea why that is?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 15, 2011, 04:24:07 pm
Does the tweaked matrix include a translation? If so, try to remove it from the matrix and set it as the camera's position instead.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 19, 2011, 11:30:56 am
Thanks, that brought me on the right track. The AR-engine converted the pose to a OpenGL-compatible matrix beforehand, which additionally screwed my calculations. But now I have some trouble doing the correct conversions between the coordinate systems now.
The AR-engine uses a right handed system (Picture (https://ar.qualcomm.at/resources/images/coordinateSystems.jpg)). According to your wiki, I need to rotate the matrix by 180° around x to convert it to the jPCT system. But do I need to do it before or after setting the camera matrix? I currently extract the rotation/translation from the float array, dump it into a Matrix/SimpleVector, rotate the Matrix/SimpleVector by 180° around x and set them as the backbuffer/position. The car is fixed to the marker, but behaves totally unpredictable (it seems as the model sticks to the camera, and not to the marker) as I move around my device, so I guess my conversion is wrong.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 19, 2011, 07:45:12 pm
Apart from the picking, was it working before? If so, just do what you did before separate the rotation from the translation afterwards.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 20, 2011, 09:23:01 am
Yes, besides some unwanted minor movement of the model when I move my device, everything but picking works. But the separating doesn't really work. As soon as I set the position of the camera, the model doesn't stay on the marker.

The GL-like matrix that does the placement correctly (but not the picking) looks like this:
Code: [Select]
-0.996   0.069  -0.063   0.000
0.092   0.808  -0.582   0.000
0.011  -0.586  -0.811   0.000
-75.993 -110.593 1902.010   1.000

I set this directly as the backbuffer and leave the position alone.

The unmodified pose itself is just transposed:

Code: [Select]
-0.996   0.092   0.011 -75.993
0.069   0.808  -0.586 -110.593
-0.063  -0.582  -0.811 1902.010

Looks like a 3x3 rotation matrix and a 3x1 translation vector to me.

If I set this matrix as the cameras backbuffer and this vector as the position, the picking works but the model is somewhere far off and doesn't stay on the marker if I move the device.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 20, 2011, 09:43:05 am
Strange...i don't see why it shouldn't work that way assuming that the rotation matrix doesn't contain the translation anymore when setting it.
If nothing else helps, you can try to do the projection into world space on your own based on the combined matrix. The reason why it doesn't work with the combined matrix is most likely that reproject2D3DWS treats the matrix as a 3x3 matrix for performance reasons. You can try

Code: [Select]
public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = reproject2D3D(camera, buffer, x, y, VIEWPLANE_Z_VALUE, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}

instead. However, the clean solution would be to separate the two transformations...
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 20, 2011, 10:12:00 am
Just that I get it right:
What should I use for VIEWPLANE_Z_VALUE, the far clipping plane?
Code: [Select]

//...
public SimpleVector reproject2D3DWS(Camera camera, FrameBuffer buffer, int x, int y, SimpleVector toFill) {
SimpleVector ray = Interact2D.reproject2D3D(camera, buffer, x, y, 2000, toFill);
ray.matMul(camera.getBack().invert());
return ray;
}

public void findNearestObject(int touchX, int touchY) {

SimpleVector dir = reproject2D3DWS(camera, frameBuffer, touchX, touchY, new SimpleVector()).normalize();
Object[] res = world.calcMinDistanceAndObject3D(camera.getPosition(), dir, 10000);
              //...
}


Is that correct?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 20, 2011, 10:35:29 am
No, just use 1f.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 20, 2011, 12:10:59 pm
No Luck, that doesn't work either :/

I tried a different approach, and set the GL-Matrix as the rotation matrix of my model. This works fine for placement and picking, but now the rotations (With object.rotateX()) rotate around an axis which seems to be far above the object (so it's pretty unusable). Which is normal I guess, since I modified the rotation matrix. Is there a way to get a proper axis that I can use with rotateAxis() or the like?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 20, 2011, 12:20:55 pm
If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 20, 2011, 12:22:16 pm
BTW: Is this solution somehow related: http://www.jpct.net/forum2/index.php/topic,1586.msg11938.html#msg11938 (http://www.jpct.net/forum2/index.php/topic,1586.msg11938.html#msg11938)?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 20, 2011, 01:41:16 pm
If you use a combined matrix as rotation matrix, you'll change the rotation pivot by this translation. The new rotation pivot should be the negative translation.

If I set the negative pivot as pivot, the model will instantly snap back to its previous position when trying to rotate it.

I've tried the other solution, but since it uses a different (geographic) coordinate system again, it doesn't work either. The model gets attached to the camera and doesn't stay where it should.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on December 20, 2011, 01:57:07 pm
Then go back to this: http://www.jpct.net/forum2/index.php/topic,2461.msg18064.html#msg18064 (http://www.jpct.net/forum2/index.php/topic,2461.msg18064.html#msg18064)

The problem here seems to be that camera.getPosition() doesn't provide the proper position now that you don't set it correctly but hide it in the matrix. Try to multiply the camera's position with the matrix and use that as the new position.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: redfalcon on December 23, 2011, 08:13:55 pm
I scrapped my approach and now pass the rotation values (from the touchscreen) directly to the pose calculation method. The AR engine provides a method for rotating the pose matrix, so I just pass the values I usually would use for object.rotateX/Y/Z() to the JNI function and rotate the matrix itself.
Thanks anyway!
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: Vi_O on January 09, 2012, 05:04:11 pm
Hello,

I am having the same issue than redfalcon and indeed, separating doesn't work.

I made a bit of code to see the difference between normal and rotation/translation separated :

Code: [Select]

_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);

float x = matrix[12]; float y = matrix[13]; float z = matrix[14];
matrix[12] = matrix[13] = matrix[14] = 0;
_cameraMat.setDump(matrix);
_camera.setBack(_cameraMat);
_camera.setPosition(x, y, z);

Log.d("JpctEngine", "Valeur camera : " + _camera.getPosition().x + ";" + _camera.getPosition().y +
";" + _camera.getPosition().z);
Log.d("JpctEngine", "Valeur camera : " + _camera.getXAxis().x + ";" + _camera.getYAxis().x +
";" + _camera.getZAxis().x);

It is strange because getPosition and get*Axis give the same results for both but the separated one doesn't place the camera well... And still no picking...

From what you were writing there, it is a problem with the getPosition that returns {0.0,0.0,0.0}.

Would it be possible then to extends Camera and Overriding setPosition/getPosition to make the reproject2D3DWS and calcMinDistanceAndObject3D works (assuming it uses getPosition to work) ?

I can't test it now as I don't have my source right now, but i'll try tomorrow if you don't tell me it is useless before :)

Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on January 09, 2012, 05:34:57 pm
That won't work, because Camera accesses the position directly and not via some getter. Are you sure that the position is in 12, 13, 14 and not in 3, 7, 11.... why else should it be the origin in the second case?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: Vi_O on January 10, 2012, 05:43:28 pm
I have check my code, and after adjusting some tranformation matrices, it works ! Thanks !

One problem still remains : I don't know why, it doesn't pick planes... Is there a special settings to apply in order to pick planes ?
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on January 10, 2012, 08:16:49 pm
No. A plane is an object just like any other. Make sure that you have the collision modes set correctly and that you are not trying to pick the backside of a plane with culling disabled, because you can't pick backsides even if they are visible.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: Vi_O on January 11, 2012, 11:08:33 am
That was a backside picking problem, I have flipped the plan and now it works !

Thanks a lot !
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: EgonOlsen on January 11, 2012, 11:33:20 am
Can you post your final solution to this problem? It seems to be a common problem and i haven't seen a solution that anybody was able to use so far.
Title: Re: Problem with object picking and modified camera backbuffer matrix
Post by: Vi_O on January 11, 2012, 12:16:17 pm
Ok, I'll try to synthetize my code and make it as clear as I can :

object part :

Code: [Select]
                ...
_object.strip();
_object.build();

_object.setCollisionMode(Object3D.COLLISION_CHECK_OTHERS|Object3D.COLLISION_CHECK_SELF);
_object.setCollisionOptimization(true);

I'm not sure collision optimization and CHECK_SELF are necessary but It works that way so... I keep it.

Camera matrix part :

Code: [Select]

public boolean setCameraMatrix(float[] matrix){

if (_camera != null){

_cameraPosition.set(matrix[12],matrix[13],matrix[14]);
matrix[12] = matrix[13] = matrix[14] = 0;

_cameraMatrix.setDump(matrix);
_cameraMatrix = _cameraMatrix.invert();

_camera.setBack(_cameraMat);
_camera.setPosition(_cameraPosition);

return true;

} else {
return false;
}

}

Here, I guess you could invert the matrix before calling setCameraMatrix ( and you would have to change 12/13/14 to 3/7/11) but again, it works that way... So I keep it.

Picking part :

Code: [Select]
public Object3D pick(int x,int y){

Log.d("jpctEngine", "Picking...");

SimpleVector dir=Interact2D.reproject2D3DWS(_camera, _frameBuffer, x, y).normalize();
Object[] res=_world.calcMinDistanceAndObject3D(_camera.getPosition(), dir, 10000 /*or whatever*/);

if (res[1] != null){
Log.d("Jpct-Engine","Picking réussi.");
return res[1];
}

return null;

}

In fact, the advices you gave on picking work well.
But somehow, dumping a false(maybe inverted) matrix still gives a nice graphical result and craps the picking. Seeing graphics working, I thought the matrix was OK and looked for the problem elsewhere... Matrices are very tricky...

Well, now that's fixed ^^ thanks again, I hope this comment will help !

Bye.