Customer states that their portal ( relay / loader / server ) is running out of memory.
There are two separate memory pools that can run out.
The main java memory allocated to webNetwork and the permgen memory.
When it is the permgen running out of memory, the message : " java.lang.OutOfMemoryError: PermGen space " Is typically displayed on the webNetwork console or possibly the VIEW LOG.
To fix this, modify the stoneware webnetwork.lax file, locate the line that starts with : lax.nl.java.option.additional=
Look for the parameters : -XX:PermSize=64m -XX:MaxPermSize=512m
If those do not exist, add them to the end. Save and shut down webNetwork and start it back up. Check this on all of your servers.
If they do exist then there are other things that should be checked before increasing the MaxPermSize.
The first thing to check is the relay object and verify if Request Logging is enabled and if that is going to an HSQL database.
The HSQL database can not handle large amounts of information and can get corrupt because it is a file based database, and is not as resilient as other databases like MySQL or MSSQL.
It is a sure sign that if you see the errors around midnight that it is a Relay Request Logging database issue because that is when it tries to purge old data.
Check on the webNetwork server (the database files may be on each stoneware install, but are only utilized on the main webNetwork server a.k.a : Loaders) for the size of stonewarehsqllogging.data
If this is over 500 meg there is a chance that it is corrupt.
Go to webadmin, find the relay object, click on it, go to the "relay logging" panel and either disable logging or make sure it is set to rotate.
Shut down webNetwork, go to stonewarehsql delete logging.* then copy stonewarehsqloriglogging.* to stonewarehsql
This will replace the logging database files with a fresh non-corrupt copy.
Start webNetwork back up.
If you see errors like : An exception was thrown: java.rmi.ServerError: Error occurred in server thread; nested exception is: java.lang.OutOfMemoryError: unable to create new native thread
Then you may not have enough memory allocated to Java.
Edit stoneware inwebnetwork.lax
Locate the line that starts with : lax.nl.java.option.additional=
Example of what the memory parameters may look like : -Xms64m -Xmx512m
You will want to change the -Xmx???m one to something appropriate.
If you are on a 32 bit server OS and have more than 1.5 gig of ram. Set this number to -Xmx8192m. (1024 is the max for a 32 bit OS)
If you have less than 1 GB of RAM in server, then set the memory to approximately 256 MB less than your amount of ram. For Example if you have 1 GB of RAM, set the -Xmx to 768
You may also see out of memory errors if you have done a restart server or restart relay many times. There is a known bug that does not free up all memory during a restart.