AFAIK, glibc allocator uses arenas for minimize contention between threads when they trying to access or free preallocated bit of memory. But seems that we use -XX:+AlwaysPreTouch, so heap is allocated and committed at start time. We allocate memory for durable memory in one thread. So I think there will be not so much contention between threads for native memory pools.
Also, there is another approach -- try to use jemalloc. This allocator shows better result than default glibc malloc in our scenarios. (memory consumption) [1] [1] -- http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/ чт, 23 июл. 2020 г. в 14:19, Ivan Bessonov <bessonov...@gmail.com>: > Hello Ivan, > > It feels like the problem is more about new starting threads rather than > the > allocation of offheap regions. Plus I'd like to see results soon, your > proposal is > a major change for Ignite that can't be implemented fast enough. > > Anyway, I think this makes sense, considering that one day Unsafe will be > removed. But I wouldn't think about it right now, maybe as a separate > proposal... > > > > чт, 23 июл. 2020 г. в 13:40, Ivan Daschinsky <ivanda...@gmail.com>: > > > Ivan, I think that we should use mmap/munmap to allocate huge chunks of > > memory. > > > > I've experimented with JNA and invoke mmap/munmap with it and it works > > fine. > > May be we can create module (similar to direct-io) that use mmap/munap on > > platforms, that support them > > and fallback to Unsafe if not? > > > > чт, 23 июл. 2020 г. в 13:31, Ivan Bessonov <bessonov...@gmail.com>: > > > > > Hello Igniters, > > > > > > I'd like to discuss the current issue with "out of memory" fails on > > > TeamCity. Particularly suites [1] > > > and [2], they have quite a lot of "Exit code 137" failures. > > > > > > I investigated the "PDS (Indexing)" suite under [3]. There's another > > > similar issue as well: [4]. > > > I came to the conclusion that the main problem is inside the default > > memory > > > allocator (malloc). > > > Let me explain the way I see it right now: > > > > > > "malloc" is allowed to allocate (for internal usages) up to 8 * (number > > of > > > cores) blocks called > > > ARENA, 64 mb each. This may happen when a program creates/stops threads > > > frequently and > > > allocates a lot of memory all the time, which is exactly what our tests > > do. > > > Given that TC agents > > > have 32 cores, 8 * 32 * 64 mb gives 16 gigabytes, that's like the whole > > > amount of RAM on the > > > single agent. > > > > > > The total amount of arenas can be manually lowered by setting > > > the MALLOC_ARENA_MAX > > > environment variable to 4 (or other small value). I tried it locally > and > > in > > > PDS (Indexing) suite > > > settings on TC, results look very promising: [5] > > > > > > It is said that changing this variable may lead to some performance > > > degradation, but it's hard to tell whether we have it or not, because > the > > > suite usually failed before it was completed. > > > > > > So, I have two questions right now: > > > > > > - can those of you, who are into hardcore Linux and C, confirm that the > > > solution can help us? Experiments show that it completely solves the > > > problem. > > > - can you please point me to a person who usually does TC maintenance? > > I'm > > > not entirely sure > > > that I can propagate this environment variable to all suites by myself, > > > which is necessary to > > > avoid occasional error 137 (resulted from the same problem) in future. > I > > > just don't know all the > > > details about suites structure. > > > > > > Thank you! > > > > > > [1] > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing&tab=buildTypeHistoryList&state=failed&branch_IgniteTests24Java8=%3Cdefault%3E > > > [2] > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Pds4&tab=buildTypeHistoryList&branch_IgniteTests24Java8=%3Cdefault%3E&state=failed > > > [3] https://issues.apache.org/jira/browse/IGNITE-13266 > > > [4] https://issues.apache.org/jira/browse/IGNITE-13263 > > > [5] > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing&tab=buildTypeHistoryList&branch_IgniteTests24Java8=pull%2F8051%2Fhead > > > > > > -- > > > Sincerely yours, > > > Ivan Bessonov > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > > > -- > Sincerely yours, > Ivan Bessonov > -- Sincerely yours, Ivan Daschinskiy
