Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
On 14-07-2014 12:20, shhuiw wrote: > Will you please show us your application code, if possible? Or at > least the mmap part. The application code is proprietary, so I cannot disclose that. For the mmap part, we are not doing mmap explicitly in our code, rather the binary is linked to some dynamic library (here in the strace that I took, it was oracle client). Application crashes while doing mmap for the oracle library, here is the trace: - Call Stack Trace - calling call entryargument values in hex location type point(? means dubious value) mmap(offset=33230848, len=2068480) failed with errno=12 for the file ./libclntsh.so.11.1 mmap(offset=33230848, len=2068480) failed with errno=12 for the file ./libclntsh.so.11.1 kpedbg_dmp_stack()+ call 556FC244 FFE3D814 ? 0 ? 219 kpeDbgCrash()+72 call 556F06E4 0 ? 5 ? FFE3D8E4 ? A42C90 ? 4 ? FFE3D890 ? 56AA4FDF call 55701184 0 ? 5 ? 56E2AEFC ? 2 ? 4 ? 50 ? 4 ? FFE3E8FD ? 56785C74 call FFE3D8F4 ? 56F27E60 ? signal B ? FFE3EDAC ? FFE3EE2C ? PKc()+255 call _ZN11CThreadDataC1E FFE3F2C8 ? FFE3F32E ? 13 ? PKc()5720C14A ? 57144297 ? 0 ? )+127 start_thread()+201 call 92ECDD8 ? FFE3FB70 ? FFE3FB70 ? FFE3FB70 ? clone()+94 call FFE3FB70 ? 0 ? 0 ? 0 ? 0 ? 0 ? We have tried by reducing the footprint for our application by decreasing the application caching of some data and see that the application can come up properly until the VIRT in top command output is about 2.8GB after which application starts crashing. > So that we can figure out the testcase to reproduce on our boxes. -- Thanks, -aka http://blog.amit-agarwal.co.in ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Hi Dave, On 11-07-2014 12:29, Dave Tian wrote: Compare the .config and ‘sysctl -a’ of the two running system may give a hint from the kernel configuration and running parameters. ‘sysctl -a’ may be much more useful in this case, I guess. I compared the config files and they are verbatim same. For sysctl, there are some differences which I felt should not matter in this case. However I am attaching the diff, in case I missed something. -- Thanks, -aka http://blog.amit-agarwal.co.in Left file: System having issue Right file: Working system 21 kernel.core_pattern = core.%p <> 20 kernel.core_pattern = |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e 22 kernel.core_pipe_limit = 0 21 kernel.core_pipe_limit = 4 37 kernel.threads-max = 254259 <> 36 kernel.threads-max = 192117 39 kernel.random.entropy_avail = 129 <> 38 kernel.random.entropy_avail = 3468 48 kernel.pid_max = 65536<> 47 kernel.pid_max = 32768 83 kernel.slow-work.max-threads = 64 <> 82 kernel.slow-work.max-threads = 32 381 kernel.hostname = <> 668 kernel.hostname = 388 kernel.msgmni = 256 <> 675 kernel.msgmni = 24044 389 kernel.msgmnb = 78 676 kernel.msgmnb = 65536 391 kernel.auto_msgmni = 0<> 678 kernel.auto_msgmni = 1 393 kernel.pty.nr = 859 <> 680 kernel.pty.nr = 13 416 vm.drop_caches = 1<> 703 vm.drop_caches = 0 417 vm.min_free_kbytes = 128 704 vm.min_free_kbytes = 65535 434 fs.inode-nr = 2855250 <> 721 fs.inode-nr = 13704 2128 435 fs.inode-state = 285525 0 0 0 0 0 0722 fs.inode-state = 13704 21280 0 0 0 0 436 fs.file-nr = 1760 0 1606964 723 fs.file-nr = 1216 0 1214127 437 fs.file-max = 1606964724 fs.file-max = 1214127 439 fs.dentry-state = 1498730 1490314 45 0 0 0 <> 726 fs.dentry-state = 11667 375645 0 0 0 445 fs.aio-nr = 896 <> 732 fs.aio-nr = 0 450 fs.epoll.max_user_watches = 3332628 <> 737 fs.epoll.max_user_watches = 2518118 451 fs.suid_dumpable = 1 738 fs.suid_dumpable = 0 467 fs.nfs.nlm_grace_period = 0 +- 468 fs.nfs.nlm_timeout = 10 469 fs.nfs.nlm_udpport = 0 470 fs.nfs.nlm_tcpport = 0 471 fs.nfs.nsm_use_hostnames = 0 472 fs.nfs.nsm_local_state = 0 549 net.netfilter.nf_conntrack_count = 23 <> 830 net.netfilter.nf_conntrack_count = 164 560 net.core.rmem_max = 16777216 <> 841 net.core.rmem_max = 131071 797 net.ipv4.tcp_mem = 1525536 2034048 3051072 <> 1036 net.ipv4.tcp_mem = 1152672 1536896 2305344 798 net.ipv4.tcp_wmem = 409665535 16777216 1037 net.ipv4.tcp_wmem = 409616384 4194304 799 net.ipv4.tcp_rmem = 409665535 16777216 1038 net.ipv4.tcp_rmem = 409687380 4194304 814 net.ipv4.tcp_dma_copybreak = 262144 <> 1053 net.ipv4.tcp_dma_copybreak = 4096 825 net.ipv4.udp_mem = 1525536 2034048 3051072 <> 1064 net.ipv4.udp_mem = 1152672 1536896 2305344 ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Hi, On 10-07-2014 11:12, shhuiw wrote: > Hi, > > How about setting /proc/sys/vm/min_free_kbytes to small values, e.g 128? > And what's the output on other workable system? > Tried changing min_free_kbytes, with this change, application comes up but dies a little later while allocating memory later. Output on the work-able system is as follows: Node 0, zone Normal 2974 3405 2898 2237 1671 1021 462 164 23 2 0 Node 0, zone Normal 2878 3384 2898 2237 1671 1021 462 164 23 2 0 Node 0, zone Normal 2827 3365 2898 2237 1671 1021 462 164 23 2 0 Node 0, zone Normal 2899 3358 2899 2237 1671 1021 462 164 23 2 0 Node 0, zone Normal 2930 3363 2899 2237 1671 1021 462 164 23 2 0 Node 0, zone Normal 2897 3362 2898 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2860 3329 2898 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2831 3327 2883 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2869 3330 2872 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2869 3331 2874 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2844 3328 2872 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2896 3332 2872 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 2859 3336 2872 2236 1672 1021 462 164 23 2 0 Node 0, zone Normal 283 208 132 77 46 816 462 164 23 0 0 Node 0, zone Normal 295 204 132 77 46 21 31 163 23 0 0 Node 0, zone Normal 280 204 132 77 45 21 31 163 23 0 0 Node 0, zone Normal 372 213 135 79 46 20 31 163 23 0 0 Node 0, zone Normal 273 205 133 78 46 21 32 164 22 0 0 Node 0, zone Normal 336 210 133 79 47 21 30 164 22 0 0 Here we clearly see that once the number's for power-0 comes down, the numbers for power-1 and so on, decreases but that does not happen on the other system. -- Thanks, -aka http://blog.amit-agarwal.co.in ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Compare the .config and ‘sysctl -a’ of the two running system may give a hint from the kernel configuration and running parameters. ‘sysctl -a’ may be much more useful in this case, I guess. Dave Tian dave.jing.t...@gmail.com On Jul 10, 2014, at 10:46 PM, Amit Agarwal wrote: > Hi Shhuiw, > > On 11-07-2014 11:12, shhuiw wrote: >> What's your workable OS and kernel on the same system, please? >> > We can't change the OS and kernel on the system having issues as it has > other services running which are live and we cannot disturb them. So, we > tried with same OS and kernel in another system where we found that the > application was able to come up. Am trying to get the buddyinfo on that > system and will provide the same on the working system. > > -- > Thanks, > -aka > http://blog.amit-agarwal.co.in > > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Hi Shhuiw, On 11-07-2014 11:12, shhuiw wrote: > What's your workable OS and kernel on the same system, please? > We can't change the OS and kernel on the system having issues as it has other services running which are live and we cannot disturb them. So, we tried with same OS and kernel in another system where we found that the application was able to come up. Am trying to get the buddyinfo on that system and will provide the same on the working system. -- Thanks, -aka http://blog.amit-agarwal.co.in ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Hi Shhuiw, On 10-07-2014 12:36, shhuiw wrote: > . > Confused. > The words "When we are running the same application on other system, > it comes up. " > How can the program run on other systems? By another system, I meant to say with same OS and kernel but on another host. -- Thanks, -aka http://blog.amit-agarwal.co.in ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
Hi Rik, On 10-07-2014 11:38, Rik van Riel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 07/09/2014 08:23 AM, Amit Agarwal wrote: >> Hi All, >> >> We are running a 32 bit application on RHEL6.3-64 bit OS with >> kernel version 2.6.32-279.el6.x86_64. >> >> While running this application we see the following when running >> under strace: mmap(offset=33230848, len=2068480) failed with >> errno=12 for the file >> >> The system has enough RAM, total 16GB and about 12 GB free. > The system may have enough memory, but your 32 bit application > is limited to slightly less than 4GB of virtual memory. I know that. On another system with same operating system and kernel version, the application comes up with about 3.5GB, so the application should be able to come up on this system as well. But on this particular system, the application is crashing in "mmap" for a dynamic library at about 2.8GB memory usage, which seems un-usual to me. -- Thanks, -aka ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
On 2014-07-10 10:33, Greg KH wrote: > On Wed, Jul 09, 2014 at 05:53:29PM +0530, Amit Agarwal wrote: >> Hi All, >> >> We are running a 32 bit application on RHEL6.3-64 bit OS with kernel >> version 2.6.32-279.el6.x86_64. > > Great. So ask them for support for this as you are paying for it, > nothing that we can do about it here, in a community forum, sorry. I understand that but my question was not related to the OS. I wanted to understand when there is enough memory, why kernel is not able to allocate memory to the process. Is there some kernel parameter that is set differently. I provided the information for the OS, just in case, it is useful. -- Thanks, -aka http://blog.amit-agarwal.co.in ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/09/2014 08:23 AM, Amit Agarwal wrote: > Hi All, > > We are running a 32 bit application on RHEL6.3-64 bit OS with > kernel version 2.6.32-279.el6.x86_64. > > While running this application we see the following when running > under strace: mmap(offset=33230848, len=2068480) failed with > errno=12 for the file > > The system has enough RAM, total 16GB and about 12 GB free. The system may have enough memory, but your 32 bit application is limited to slightly less than 4GB of virtual memory. Errno 12 corresponds to -ENOMEM. The process running out of its slightly-less-than-4GB of memory corresponds nicely with your system still having 12GB of free memory. This suggests you have run out of virtual memory space in the process. If your program needs more than 4GB of memory, eg. because you have a large data set, you need to use a 64 bit version of the program. This is not a kernel problem. - -- All rights reversed. -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTvi3EAAoJEM553pKExN6D1/QIAJ26mjn0UHz6MK1CPGywlvaZ JuE+2JvHrImHALnQ8jxQOutNLWp9IPzB/zByoyMkqUiJkrQ15aj60pyzZ9PvzuQo CX1ve3xMY/sf/617mRqRvbfCAbB3UArlXG4tU8OmzBdH4Qy/V6jFLY69hmFUqiXk hMIkG7NhuptYSwPmHV7vw4qTFDVUhh/p+etPYDeuAtDcFlScZ8CjdX2pzwR8TxPr M16UAOuhw2ONkZCO25XuM9AteKeLC85uV63gTgcm4h+W3Hqyeyebfsu7FmPkLq7v JavXFNL9gCMeVxgvjU6A7o+NjT8HNgkh/IPdE2LrqFysE6SeU77T61AcJEMJulM= =fmS4 -END PGP SIGNATURE- ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Memory allocation problems on RHEL 6.3 kernel version 2.6.32-279.el6.x86_64
On Wed, Jul 09, 2014 at 05:53:29PM +0530, Amit Agarwal wrote: > Hi All, > > We are running a 32 bit application on RHEL6.3-64 bit OS with kernel > version 2.6.32-279.el6.x86_64. Great. So ask them for support for this as you are paying for it, nothing that we can do about it here, in a community forum, sorry. Best of luck, greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
