Webstart is the better solution anyway. Maybe the applet can work with lower detail textures and webstart can be used as a higher quality option. It doesn't seem to be possible to increase the memory from the page that starts the applet, which is a real shame. Here's what i found regarding this topic (directly from sun):
As for your second question, heap/stack/gc memory management for your JRE can be sent by setting the JRE's supported arguments. You can do this via the Java Control Panel or by setting the deployment.javapi.jre.<version>.args property in deployment.properties. W3C's OBJECT and APPLET tags currently do not have any attribute for JRE memory management. We may consider revisiting the tags' attribute to investigate the possibility of supporting the setting of the JRE's arguments. Remember, because only one instance of the JVM exists in a browser session, once the JRE arguments are set, the setting arguments will be in effect for the entire browser process's life.