Try increasing heap size with -Xmx / -Xms params to jvm. If it works fine then:
- leave as is, update memory requirements
- review you code for possible optimizations that will reduce memory usage
If not, then you may say you've got a memory leak. You will have to review the code / launch application under profiler to discover the problem. From the past experience the problem is most likely that there are references to "obsolete" objects stored inside your "active" objects in a running threads:
- in object's properties
- in hashmaps/collections (caches, "listeners", etc.)
- in another thread, that is still alive (daemons, launching actions in new thread)
- non-static inner classes - they still carry a reference to outer class instance
- AWT/Swing elements. They might have your objects in model/listeners
- double check static objects - 90% of problems go there
If i were you, I'd use JProfiler (it's commercial but you may find a workarond at astalavista.box.sk if you need it badly) and it's memory monitor to find out more about memory usage in your application.