Am 24.02.2015 um 17:14 schrieb Simon Andrews <[email protected]>: > On 24/02/2015 15:40, "Reuti" <[email protected]> wrote: > >> >>> Despite this we're getting jobs which are dying due to not being able to >>> allocate memory. The nodes on which these failures happen still have >>> plenty of free memory and the jobs are dying from internal malloc >>> errors, >>> rather than being killed due to the limit which was imposed by grid >>> engine. >>> I suspect that what is happening is that we're getting memory >>> fragmentation, so that even though there is plenty of memory available, >>> the programs aren't able to allocate a large enough contiguous block of >>> memory and are therefore dying. >> >> Yes. But the fragmentation seems to be created on an application level, >> not on an OS level by other jobs. >> >> >>> Does this seem like a likely explanation? If so, is there anything >>> which >>> can be done in the configuration of either the queues or the nodes to >>> try >>> to minimise the chances of these kinds of errors occurring? >> >> AFAICT this is nothing SGE can handle, but has to be taken care of by the >> programmer of the application to use the granted memory in a better way. > > Thanks for the reply. I kind of guessed this might be the answer. I > think the issue is that we're assuming that the allocated memory will be > available as a single block, whereas this isn't actually the case. A lot > of our applications rely on being able to allocate very large arrays which > absolutely have to be contiguous, and when you get a lot of the same job > type running on the same node then the effective memory can be exhausted > long before the actual physical memory has gone.
I never ran into this with our applications. But I wouldn't say that one job tampers an other one. Each process gets its own virtual address space. Where this is in real memory (and contiguous or not) should be transparent to the application. Does the application allocate the memory once for the runtime of the job or are there several malloc() and free() inside each application's run? -- Reuti > The only mitigation I > can think of is under-comiting some nodes to reduce the chance of this > happening, but that doesn't make anywhere near best use of the resources > we have. > > Simon. > > The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT > Registered Charity No. 1053902. > The information transmitted in this email is directed only to the addressee. > If you received this in error, please contact the sender and delete this > email from your system. The contents of this e-mail are the views of the > sender and do not necessarily represent the views of the Babraham Institute. > Full conditions at: www.babraham.ac.uk<http://www.babraham.ac.uk/terms> > _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
