There are a few things to check and these should be done on the server that is having the issue. If your webNetwork Server (loader) is having the issue then your webNetwork Relay won't start up because it would not be able to obtain a connection with the webNetwork Server.
A little background on this, this issue is dependent on the version / patch level / flavor of linux you have installed. Some customers never have to change this, others may have to tune their linux after rolling it out to many more users. Tuning in this case involves increasing the number of open files that the system will allow.
The following URL helps explain this issue further and is from another 3rd party webserver running on linux.
The first thing you want to do is do a ulimit –a (full information) or ulimit –n (just number of open files allowed) to see what value your system is using.
Next you would want to do some investigation on what type of open files you have on your system. Keep in mind that every TCP connection, every file open and running on the system is considered an "open file" so if you are experiencing a denial of service attack you may have thousands of tcp connections opened, which in turn count as an open file.
To find how many open files are related to webNetwork, you can do the following:
Find the process id that java is running as. ps -ef | grep stoneware
This will show something like : root 6546 1 3 Mar26 ? 00:42:57 /usr/stoneware/bin/../jre/bin/java
The process id is 6546
Next perform the following command :
lsof -p 6546 > /tmp/listoffiles.txt
then
lsof -p 6546 | wc -l
and note the number. This is the number of open files that java has opened.
The file /tmp/listoffiles.txt will show what files are currently open.
Or you can do a lsof | grep java | wc -l
Next you may need to increase the number of open files your system allows. There are few ways to increase the number of open files.
To increase the limit, edit /etc/security/limits.conf and add
* soft nofile 4096
* hard nofile 65535
OR
To increase the limit edit your /etc/init.d/boot.local and add the following to the bottom.
ulimit -n 4096
Replace 4096 with an appropriate number for your site.
Reboot box when done. Once back up, check again with the ulimit -a
If the issue is not enough file handles, use the following commands to increase:
sysctl –a
The fs.file-max should be above 200000. If it is not, edit /etc/sysctl.conf and add that line.
type sysctl -p to apply changes.
The numbers here are examples, yours may vary based on the version / patch level / flavor of linux and the number of users accessing your system at the same time.
Some other uses of "lsof"
Show TCP connections : lsof -i TCP
Show TCP connections on port 80 : lsof -i TCP:80
Show files opened by a program : lsof -c mysq
Show connections on specific ip port : lsof -i TCP@192.168.0.2:636
Another article about tuning Linux can be found
here