Integrating JPCT-AE with Vuforia

Revision as of 00:11, 4 February 2013 by Kelmer (Talk | contribs)

Jump to: navigation, search

Qualcomm's Vuforia engine is one of the most powerfull Augmented Reality engines out there. It's integration with jPCT-AE is a wonderful combination for easily creating spectacular AR scenes with your Android device.

The integration process, though, might get a little messy and confusing for someone without much experience with matrices or scene graphs. This guide will try to detail how to achieve a quick integration in a step-by-step fashion.

First of all, you should follow Qualcomm’s Getting Started guide to set up your environment to use Vuforia.

You should get to the point where you have run the ImageTargets demo app, since we will start from that sample.

Setting up the environment

Open the ImageTargets demo project on Eclipse. Right click on the project, then select properties. Go to Java Build Path, then Libraries and add the jpct-ae.jar library using the Add External JARs… option.

Now we can start getting jpct-ae code into our app.

jPCT-AE and Vuforia working together

Now we can start getting jpct-ae code into our app.

First, we will make jpct-ae and vuforia’s native code share the same GLsurface. For that, first open under the ImageTargets sample app (package com.qualcomm.QCARSamples.ImageTargets).

This is the OpenGL renderer, and thus it’s where our jPCT code should be injected in. We will start by taking JPCT-AE Hello World sample app into this renderer. First, create a constructor for ImageTargetsRenderer. This is for the Activity reference to be passed on to our renderer, instead of explicitly setting the attribute, as Qualcomm’s demo does. We will also init our scene here.

public ImageTargetsRenderer(ImageTargets activity){
        this.mActivity = activity;
        world = new World();
	world.setAmbientLight(20, 20, 20);

	sun = new Light(world);
	sun.setIntensity(250, 250, 250);

	// Create a texture out of the icon...:-)
	Texture texture = new Texture(BitmapHelper.rescale(BitmapHelper.convert(mActivity.getResources().getDrawable(R.drawable.ic_launcher)), 64, 64));
	TextureManager.getInstance().addTexture("texture", texture);

	cube = Primitives.getCube(10);


	Camera cam = world.getCamera();
	cam.moveCamera(Camera.CAMERA_MOVEOUT, 50);

	SimpleVector sv = new SimpleVector();
	sv.y -= 100;
	sv.z -= 100;


As you can see, I just copied and pasted the code in the Hello World demo (and changed the icon reference). Now go to, then change the initialization of ImageTargetsRenderer to include the activity in the constructor, and remove the line setting this immediately after.

Then go back to ImageTargetsRenderer, into the onSurfaceChanged method. This method is called whenever our surface changes size. We should put jpct-ae framebuffer initialization code here, right from the Hello World demo.

if (fb != null) {
fb = new FrameBuffer(width, height);

NOTE: You can use OpenGL 1.0 or 2.0. If you have a phone that supports 2.0, Vuforia will fire a 2.0 surface and thus creating a framebuffer for 1.0 will cause the app to crash. For simplicity I am posting here the code that corresponds to OpenGLES 2.0 framebuffer initialization. You can add a OGL version check here and create the FB in consequence. If you want to use 1.0 only, you can force Vuforia to do so by going into file under jni folder and setting the USE_OPENGL_ES_1_1 directive to true.

Okay, we have initialized our scene and framebuffer, but we have yet to tell jpct to render it. This is done in the onDrawFrame method. From the HelloWorld sample, paste the following code directly after the renderFrame() native call :


As you can see, I have omitted the fb.clear() line. This is because QCAR’s native openGL code already clears the framebuffer for us. If we did include this line, we would be clearing the video information from the camera, that the renderFrame() puts there for us.

Now fire the app, and you will see a cube over the camera scene.

Passing on the marker transformations

Now for the fun part, we need to modify the native codes. Open the file ImageTargets.cpp under the jni directory within your project (you can open it with Eclipse, it even has syntax highlighting). You should go directly to the frame rendering function, which due to JNI's naming convention has the awkward name of JNIEXPORT void JNICALL Java_com_qualcomm_QCARSamples_ImageTargets_ImageTargetsRenderer_renderFrame(JNIEnv *, jobject)

If you’re used to regular OpenGL code, the contents of this function will sound familiar to you. This is basically the render loop, where the framebuffer is cleant, the projection, model and view matrixes are created and the objects are displayed.

I am assuming that you won’t be needing Vuforia to render anything, since we’ll be leaving this job to jPCT-AE. So the first thing we’ll do is stripping this function to a few lines. This is what my renderFrame function looked like after removing the unnecessary code.

    // Clear color and depth buffer 
    // Get the state from QCAR and mark the beginning of a rendering section
    QCAR::State state = QCAR::Renderer::getInstance().begin();
    // Explicitly render the Video Background
    // Did we find any trackables this frame?
    for(int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++)
        // Get the trackable:
        const QCAR::TrackableResult* result = state.getTrackableResult(tIdx);
        const QCAR::Trackable& trackable = result->getTrackable();
        QCAR::Matrix44F modelViewMatrix = QCAR::Tool::convertPose2GLMatrix(result->getPose());        

If you run the app now, you will see that you no longer get a teapot if you face the marker with the camera.

What we need to do now, is pass through the modelview matrix representing the marker position and orientation from the native code, into our java code.

For that matter, we will create a method in ImageTargetsRenderer,