The mosaic texture idea didn't solve the problem, because there is still that noticible .2 second blocking every time a texture is swapped in the TextureManager (happens less often, of course, but it doesn't go away). I also tried performing the swap on a seperate thread, but as expected, that just caused thread synchronization issues and random crashes.
However, I was able to use Egon's suggestion of ITextureEffect:
Demo Applet (LINK TO SOURCE REMOVED)
I am aware of the random blocking problems with the above applet - they are due to an ugly hack of a JMF streamer class that I threw together for aquiring video frames as instances of awt.Image. I think there is also a synchronization issue between changing the frame Image and updating the texture (causes light-blue artifacts showing up randomly on the texture - usually coincides with the blocking). I am sure there are much better ways to acomplish this if I wanted to take the time to learn the JMF API (which I don't) and further debug my code. However, I think it works well enough to demonstrate that video textures are definitely possible using ITextureEffect. You can look at the above source code if you like, but it is rather ugly. Here are the relevant parts (not in any particular order):
textureWidth = nextPowerOf2( movie.getWidth() );
textureHeight = nextPowerOf2( movie.getHeight() );
float uw = (float) movie.getWidth() / (float) textureWidth;
float uh = (float) movie.getHeight() / (float) textureHeight;
screenTexture = new Texture( textureWidth, textureHeight, Color.BLACK );
TextureManager.getInstance().addTexture( "TV Screen", screenTexture );
screenTexture.setEffect( new VideoTextureEffect() );
SimpleVector screenOffset = new SimpleVector( 80 * tvScale,
-14 * tvScale,
12 * tvScale );
SimpleVector br = new SimpleVector( 0, 0, 0 );
SimpleVector bl = new SimpleVector( 105 * tvScale, 0, 0 );
SimpleVector tr = new SimpleVector( 0, -77 * tvScale, 0 );
SimpleVector tl = new SimpleVector( 105 * tvScale, -77 * tvScale, 0 );
tvScreen = new Object3D( 2 );
tvScreen.addTriangle( tl, 0, 0, bl, 0, uh, br, uw, uh,
TextureManager.getInstance().getTextureID( "TV Screen" ) );
tvScreen.addTriangle( br, uw, uh, tr, uw, 0, tl, 0, 0,
TextureManager.getInstance().getTextureID( "TV Screen" ) );
tvScreen.build();
tvScreen.translate( screenOffset );
world.addObject( tvScreen );
tvScreen.setVisibility( true );
tvPivot.addChild( tvScreen );
...
currentFrameImage = movie.getCurrentFrame();
...
synchronized( jPCTSync )
{
screenTexture.applyEffect();
}
...
/**
* The VideoTextureEffect class changes the video texture to the current frame.
*/
private class VideoTextureEffect implements ITextureEffect
{
/**
* Ignored by VideoTextureEffect.
* @param tex The video texture.
*/
@Override
public void init( Texture tex)
{}
/**
* Changes the contents of the video texture to the current frame.
* @param dest Pixels to change (change to current frame).
* @param dest Current pixels (previous frame).
*/
@Override
public void apply(int[] dest, int[] source)
{
if( lastEffectFrame != currentFrame )
{
lastEffectFrame = currentFrame;
if( currentFrameImage != null )
{
PixelGrabber pg = new PixelGrabber( currentFrameImage, 0, 0,
movie.getWidth(),
movie.getHeight(),
dest, 0, textureWidth );
try
{
pg.grabPixels();
}
catch( InterruptedException e )
{
System.err.println( "interrupted waiting for pixels!" );
return;
}
if( ( pg.getStatus() & ImageObserver.ABORT ) != 0 )
{
System.err.println( "image fetch aborted or errored" );
return;
}
}
}
}
...
/**
* Returns the next power of two higher than x, or x itself it it is already a
* power of two.
* @param x A positive integer value.
* @return The next power of two.
*/
private static int nextPowerOf2( int x )
{
--x;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
return ++x;
}On a related topic, I added an audio track to the above applet just to see how it looks. You will notice that the sound and video are definitely not synced in the above applet. This is due to the following factors:
1) The audio track is an ogg created from an mp3 I downloaded - it wasn't split from the actual video.
2) The random blocking delays (most likely due to a thread synchronization problem) affect both the video frame rate and the audio stream over time.
3) Frames are processed in the main game-loop so moving through the world changes the FPS, which affects the video frame rate over time.
I added in some tweaking with frame offset, framerate, and pitch to make the audio and video match up fairly close on my computer (but these settings won't be the same for every computer). A much better way to synchronize the audio and video would be to count the frames and milliseconds and plug that into some mathematical formula to calculate tiny pitch changes and/or framerate changes to compensate. However, that was beyond the scope of the above example.
