Ya he encontrado la solucion si se pone
echo 1 > /proc/sys/vm/overcommit_memory
Funciona la instalacion sin dar ningun problema... ni da el error que me daba
IO error :java.io.IOException: Cannot allocate memory
No sabia yo que Lotus Domino es una aplicacion cientifica...
Controls overcommit of system memory, possibly allowing processes to allocate (but not use) more memory than is actually available.
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slighly more memory in this mode. This is the default.
1 - Always overcommit. Appropriate for some scientific applications.
2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap plus a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while attempting to use already-allocated memory but will receive errors on memory allocation as appropriate.