www.jpct.net

jPCT-AE - a 3d engine for Android => Support => Topic started by: thiamant on September 10, 2010, 01:10:13 pm

Title: Loading big models (memory issues)
Post by: thiamant on September 10, 2010, 01:10:13 pm
I'm trying to load a model with a reasonable size and it's working more or less properly (maybe at a low frame rate which is around 10 atm) in my orange boston. The app crashes in HTC Desire though. I think this is happening due to some memory issues (maybe a limited heap size). So, how exactly is the model being loaded (is jpct loading the .obj completely in memory and uncompressed? the obj size is just around 3-4MB and it's uncompressed but clearly the heap size is larger than 4 MB).

Can I load the model in several parts so only the parts needed are stored in memory?
Do you think I could have to optimize the model to make it work properly? What are my chances here?
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 10, 2010, 01:22:54 pm
Don't load these models directly in AE. Load them in desktop jPCT and serialize them in reduced mode: (http://www.jpct.net/wiki/index.php/Differences_between_jPCT_and_jPCT-AE#Performance_and_memory_issues.2C_serialized_objects (http://www.jpct.net/wiki/index.php/Differences_between_jPCT_and_jPCT-AE#Performance_and_memory_issues.2C_serialized_objects)).

Or are you already doing that and it's still hitting the memory barrier?

Title: Re: Loading big models (memory issues)
Post by: thiamant on September 10, 2010, 02:08:23 pm
Yes, that's the point. I'm already serializing/deserializing the model... (And enabling the reduced mode)
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 10, 2010, 02:20:46 pm
Are you sure then that the crash is related to low memory? Can you provide me with the model to see for myself (preferably the "normal" and the serialized version is possible)?
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 10, 2010, 02:27:06 pm
I'm sorry but im under NDA and can't show these models. But in the emulator it crashes due to an OutOfMemoryException. I have to launch the emulator with 256 MB RAM and max heap size of around 128 MB to make it work, and I think that htc desire has lower heap size allowed.
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 10, 2010, 09:47:44 pm
When comparing the specs of the orange boston and the htc desire, the desire is actually the more advanced phone, isn't it? Older Android version like 1.5/1.6 have a usable heap size of around 12mb, never versions like 2.2 should have something around 20mb. I don't see why something that loads fine on the boston should fail on the desire!?

How are you loading the textures? As drawables or as raws? How big are they? Have you tried what happens if you omit loading them?
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 11, 2010, 12:28:07 pm
I've already tried to load the model without textures and it still crashes on htc desire.

I will try to perform a debug on this one and see what happens.
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 13, 2010, 05:19:39 pm
I've found out that this is a problem with the android version. Using the same hardware specifications in the emulator and running both 1.6 and 2.1 versions the program crashes only on 2.1 due to OutOfMemoryException, which seems kind of weird. Any ideas?
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 13, 2010, 10:41:35 pm
Do you have a log output to see what exactly happens before this crash?
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 14, 2010, 03:37:34 pm
This is the exception that is causing the crash:

09-14 19:31:48.118: ERROR/mm-camera(57): +++ DELETING LEAKED MEMORY AT 0x2a980 (2 REMAINING)
09-14 19:31:48.118: ERROR/mm-camera(57):      6 0xa942ba44
09-14 19:31:48.118: ERROR/mm-camera(57):      5 0xa940bfc2
09-14 19:31:48.118: ERROR/mm-camera(57):      4 0xa940d768
09-14 19:31:48.118: ERROR/mm-camera(57):      3 0xa940d27e
09-14 19:31:48.118: ERROR/mm-camera(57):      2 0xafe10074
09-14 19:31:48.118: ERROR/mm-camera(57):      1 0xafe0fb48
09-14 19:31:48.118: ERROR/mm-camera(57): +++ DELETING LEAKED MEMORY AT 0x2a580 (1 REMAINING)
09-14 19:31:48.118: ERROR/mm-camera(57):      6 0xa942ba44
09-14 19:31:48.118: ERROR/mm-camera(57):      5 0xa940bfc2
09-14 19:31:48.118: ERROR/mm-camera(57):      4 0xa940d768
09-14 19:31:48.118: ERROR/mm-camera(57):      3 0xa940d27e
09-14 19:31:48.118: ERROR/mm-camera(57):      2 0xafe10074
09-14 19:31:48.118: ERROR/mm-camera(57):      1 0xafe0fb48
09-14 19:38:20.528: ERROR/AndroidRuntime(10181): ERROR: thread attach failed
09-14 19:38:21.968: ERROR/AndroidRuntime(10196): ERROR: thread attach failed
09-14 19:38:22.958: ERROR/PackageInstallationReceiver(8167): Remove /data/local/tmp/com.jpct.apk Fail!
09-14 19:38:36.628: ERROR/dalvikvm-heap(10208): 288000-byte external allocation too large for this process.
09-14 19:38:36.638: ERROR/dalvikvm(10208): Out of memory: Heap Size=12487KB, Allocated=9857KB, Bitmap Size=3894KB
09-14 19:38:36.668: ERROR/AndroidRuntime(10208): Uncaught handler: thread GLThread 7 exiting due to uncaught exception
09-14 19:38:36.718: ERROR/AndroidRuntime(10208): java.lang.OutOfMemoryError
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at org.apache.harmony.luni.platform.OSMemory.malloc(Native Method)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at org.apache.harmony.luni.platform.PlatformAddressFactory.alloc(PlatformAddressFactory.java:145)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:67)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at java.nio.ReadWriteDirectByteBuffer.<init>(ReadWriteDirectByteBuffer.java:51)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at java.nio.BufferFactory.newDirectByteBuffer(BufferFactory.java:95)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:71)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at com.threed.jpct.CompiledInstance.fill(CompiledInstance.java:662)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at com.threed.jpct.Object3DCompiler.compile(Object3DCompiler.java:134)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at com.threed.jpct.World.compile(World.java:2021)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at com.threed.jpct.World.renderScene(World.java:1057)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at com.jpct.Main$MyRenderer.onDrawFrame(Main.java:410)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1136)
09-14 19:38:36.718: ERROR/AndroidRuntime(10208):     at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:977)
09-14 19:38:42.038: ERROR/ActivityManager(83): fail to set top app changed!

This happens when calling world.renderScene(fb) for the first time.
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 15, 2010, 09:01:59 am
The complete log (not just the exception) might help too.
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 15, 2010, 08:26:11 pm
Edited the post with additional information
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 16, 2010, 08:16:18 am
What's mm-camera and why is it giving all these strange messages? Can't you post the complete log wrapped in code-tags. I'm still not seeing a single jPCT related line... ???
Title: Re: Loading big models (memory issues)
Post by: thiamant on September 20, 2010, 12:50:28 pm
There's no jPCT lines that show any error at all. Anyway, we've reduced the model and it's working ok now.
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 25, 2010, 10:24:06 pm
However, i tried to load the model you provided with 1.5 on the actual phone and 2.1 and 2.2 in the emulator (Both with default setting, i.e. no heap adjustments made). All loaded fine, so i can't reproduce this problem. If native memory size (and not java heap size) is the actual problem (and it looks like it), it may help to do a <yourObject>.forceGeometryIndicies(true);...but that will make the compilation process much slower in return. It will use less native memory though.
Title: Re: Loading big models (memory issues)
Post by: iagofg on September 11, 2011, 01:03:36 am
I'm also getting that problem relating this, appears that the apk access allows only to objects bellow 1MB in size... here is my exception:

09-10 22:17:48.435: INFO/jPCT-AE(373): Loading Texture...
09-10 22:17:48.465: INFO/jPCT-AE(373): Loading Texture...
09-10 22:17:48.465: INFO/jPCT-AE(373): onSurfaceChanged
09-10 22:17:48.508: INFO/jPCT-AE(373): OpenGL vendor:     Android
09-10 22:17:48.508: INFO/jPCT-AE(373): OpenGL renderer:   Android PixelFlinger 1.3
09-10 22:17:48.508: INFO/jPCT-AE(373): OpenGL version:    OpenGL ES-CM 1.0
09-10 22:17:48.508: INFO/jPCT-AE(373): OpenGL renderer initialized (using 2 texture stages)
09-10 22:17:48.948: INFO/ARMAssembler(373): generated scanline__00000077:03545444_00009501_00000000 [142 ipp] (195 ins) at [0x284ad8:0x284de4] in 49500022 ns
09-10 22:17:49.015: INFO/ActivityManager(59): Displayed activity com.gargore.zpunch/.ZPunchActivity: 5059 ms (total 5059 ms)
09-10 22:17:49.025: WARN/KeyCharacterMap(373): No keyboard for id 0
09-10 22:17:49.106: WARN/KeyCharacterMap(373): Using default keymap: /system/usr/keychars/qwerty.kcm.bin
09-10 22:17:52.855: WARN/InputManagerService(59): Window already focused, ignoring focus gain of: com.android.internal.view.IInputMethodClient$Stub$Proxy@450e4058
09-10 22:17:57.666: WARN/InputManagerService(59): Window already focused, ignoring focus gain of: com.android.internal.view.IInputMethodClient$Stub$Proxy@44fe6750
09-10 22:17:57.676: DEBUG/asset(373): Data exceeds UNCOMPRESS_DATA_MAX (4071780 vs 1048576)
09-10 22:17:57.676: WARN/System.err(373): java.io.IOException
09-10 22:17:57.925: WARN/System.err(373):     at android.content.res.AssetManager.readAsset(Native Method)
09-10 22:17:57.995: WARN/System.err(373):     at android.content.res.AssetManager.access$700(AssetManager.java:36)
09-10 22:17:57.995: WARN/System.err(373):     at android.content.res.AssetManager$AssetInputStream.read(AssetManager.java:574)
09-10 22:17:57.995: WARN/System.err(373):     at com.threed.jpct.DeSerializer.read(DeSerializer.java:439)
09-10 22:17:57.995: WARN/System.err(373):     at com.threed.jpct.DeSerializer.readInt(DeSerializer.java:423)
09-10 22:17:57.995: WARN/System.err(373):     at com.threed.jpct.DeSerializer.deserialize(DeSerializer.java:29)
09-10 22:17:57.995: WARN/System.err(373):     at com.threed.jpct.Loader.loadSerializedObject(Loader.java:97)
09-10 22:17:57.995: WARN/System.err(373):     at com.gargore.zpunch.ZPunchActivity$2.run(ZPunchActivity.java:412)
09-10 22:17:57.995: INFO/jPCT-AE(373): [ 1315693078000 ] - ERROR: Can't deserialize object: null
09-10 22:17:57.995: WARN/System.err(373): java.lang.RuntimeException: [ 1315693078000 ] - ERROR: Can't deserialize object: null
09-10 22:17:58.025: WARN/System.err(373):     at com.threed.jpct.Logger.log(Logger.java:189)
09-10 22:17:58.025: WARN/System.err(373):     at com.threed.jpct.DeSerializer.deserialize(DeSerializer.java:201)
09-10 22:17:58.025: WARN/System.err(373):     at com.threed.jpct.Loader.loadSerializedObject(Loader.java:97)
09-10 22:17:58.025: WARN/System.err(373):     at com.gargore.zpunch.ZPunchActivity$2.run(ZPunchActivity.java:412)
09-10 22:17:58.035: WARN/dalvikvm(373): threadid=8: thread exiting with uncaught exception (group=0x4001d800)
09-10 22:17:58.085: ERROR/AndroidRuntime(373): FATAL EXCEPTION: Thread-9
09-10 22:17:58.085: ERROR/AndroidRuntime(373): java.lang.RuntimeException: java.lang.RuntimeException: [ 1315693078000 ] - ERROR: Can't deserialize object: null
09-10 22:17:58.085: ERROR/AndroidRuntime(373):     at com.gargore.zpunch.ZPunchActivity$2.run(ZPunchActivity.java:418)
09-10 22:17:58.085: ERROR/AndroidRuntime(373): Caused by: java.lang.RuntimeException: [ 1315693078000 ] - ERROR: Can't deserialize object: null
09-10 22:17:58.085: ERROR/AndroidRuntime(373):     at com.threed.jpct.Logger.log(Logger.java:189)
09-10 22:17:58.085: ERROR/AndroidRuntime(373):     at com.threed.jpct.DeSerializer.deserialize(DeSerializer.java:201)
09-10 22:17:58.085: ERROR/AndroidRuntime(373):     at com.threed.jpct.Loader.loadSerializedObject(Loader.java:97)
09-10 22:17:58.085: ERROR/AndroidRuntime(373):     at com.gargore.zpunch.ZPunchActivity$2.run(ZPunchActivity.java:412)
09-10 22:17:58.266: WARN/ActivityManager(59):   Force finishing activity com.gargore.zpunch/.ZPunchActivity
09-10 22:17:58.326: INFO/jPCT-AE(373): onPause
09-10 22:18:00.135: INFO/jPCT-AE(373): onStop
09-10 22:18:01.295: DEBUG/dalvikvm(302): GC_EXPLICIT freed 905 objects / 60472 bytes in 4548ms
09-10 22:18:01.845: INFO/Process(373): Sending signal. PID: 373 SIG: 9
09-10 22:18:02.782: INFO/ActivityManager(59): Process com.gargore.zpunch (pid 373) has died.

Seems the cause is on:
09-10 22:17:57.676: DEBUG/asset(373): Data exceeds UNCOMPRESS_DATA_MAX (4071780 vs 1048576)

What I don't understand is why the serialized file is so big (I yet put it on reduced mode and the source .obj files total size are less than 2mb, with all the geometry and wavefront files waste a lot of space, I also merged the geometry for generate animation like in http://www.jpct.net/wiki/index.php/Loading_3ds_Keyframes_from_Blender but with obj files.

Maybe there is something else to do? (the model is yet reduced).

I'm thinking on implement my own stream filter to store in parts the big serialized file...
Title: Re: Loading big models (memory issues)
Post by: iagofg on September 11, 2011, 04:14:51 am
I've workaround yet my own InputStream child class that loads model form splitted file successfully.

It is a minefield test-only yet, but works and perhaps helps someone (I tried with BufferedInputStream but Loader seems not to be compatible):

   //zp = Loader.loadSerializedObject(res.openRawResource(R.raw.zpmodel));
   zp = Loader.loadSerializedObject(new MyModelStream());
   
   [...]
   
   class MyModelStream extends java.io.InputStream {
      private int i = 0;
      private java.io.InputStream s = null;
      private void changestream() {
         ++i;
         //Log.v(TAG, "MyModelStream reading resource file no. " + i);
         if (i == 1) s = getResources().openRawResource(R.raw.zpmodel_aa);
         if (i == 2) s = getResources().openRawResource(R.raw.zpmodel_ab);
         if (i == 3) s = getResources().openRawResource(R.raw.zpmodel_ac);
         if (i == 4) s = getResources().openRawResource(R.raw.zpmodel_ad);
      }
      public int available() throws IOException {
         if (s == null) changestream();
         if (s == null) return 0;
         int v = s.available();
         if (v <= 0) { // needs to change stream here also for Loader compat
            //Log.v(TAG, "MyModelStream.available returns <= 0, next block and requesting again...");
            changestream();
            v = s.available();
         }
         //Log.v(TAG, "MyModelStream.available returns " + v);
         return v;
      }
      public void close() throws IOException {
         if (s == null) changestream();
         if (s == null) return;
         //Log.v(TAG, "MyModelStream.close has been called");
         s.close();
      }
      public void mark(int readlimit) {
         //Log.v(TAG, "MyModelStream.mark has been called");
         return;
      }
      public boolean markSupported() {
         boolean v = false;
         //Log.v(TAG, "MyModelStream.markSupported returns " + v);
         return v;
      }
      public int read() throws IOException {
         if (s == null) changestream();
         if (s == null) return -1;
         int v = s.read();
         if (v < 0) {
            //Log.v(TAG, "MyModelStream.read returns < 0, next block and reading again...");
            changestream();
            v = s.read();
         }
         //Log.v(TAG, "MyModelStream.read returns " + v + " bytes");
         return v;
      }
      public int read(byte[] b) throws IOException, NullPointerException {
         if (s == null) changestream();
         if (s == null) return -1;
         int v = s.read(b);
         if (v < 0) {
            //Log.v(TAG, "MyModelStream.read returns < 0, next block and reading again...");
            changestream();
            v = s.read(b);
         }
         //Log.v(TAG, "MyModelStream.read returns " + v + " bytes");
         return v;
      }
      public int read(byte[] b, int off, int len) throws IOException, NullPointerException {
         if (s == null) changestream();
         if (s == null) return -1;
         int v = s.read(b, off, len);
         if (v < 0) {
            //Log.v(TAG, "MyModelStream.read returns < 0, next block and reading again...");
            changestream();
            v = s.read(b, off, len);
         }
         //Log.v(TAG, "MyModelStream.read returns " + v + " bytes");
         return v;
      }
      public void reset() throws IOException {
         //Log.v(TAG, "MyModelStream.reset has been called");
         throw new IOException();
      }
      public long skip(long n) throws IOException {
         if (s == null) changestream();
         if (s == null) return 0;
         long v = s.skip(n);
         //Log.v(TAG, "MyModelStream.skip returns " + v);
         return v;
      }
   }
Title: Re: Loading big models (memory issues)
Post by: EgonOlsen on September 11, 2011, 10:27:27 am
Just rename the file to .mp3 and these restrictions will go away.