PFA of error log hs_err_pid9656.log <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj < rahul.bhard...@indiamart.com> wrote: > free RAM: > > > free -m > total used free shared buffers cached > Mem: 64398 23753 40644 0 108 8324 > -/+ buffers/cache: 15319 49078 > Swap: 2925 15 2909 > > > ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 515041 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 515041 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > Also attaching complete error file > > > On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <anis...@gmail.com> > wrote: > >> the memory cassandra is trying to allocate is pretty small. you sure >> there is no hardware failure on the machine. what is the free ram on the >> box ? >> >> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj < >> rahul.bhard...@indiamart.com> wrote: >> >>> Hi All, >>> >>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running >>> in healthy state. Suddenly one machine's cassandra daemon stops working and >>> shut down. >>> >>> On restarting it after 2 minutes it again stops and is getting stop >>> after returning below error in cassandra.log >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot >>> allocate memory' (errno=12) >>> # >>> # There is insufficient memory for the Java Runtime Environment to >>> continue. >>> # Native memory allocation (malloc) failed to allocate 12288 bytes for >>> committing reserved memory. >>> # An error report file with more information is saved as: >>> # /tmp/hs_err_pid23215.log >>> INFO 09:50:41 Loading settings from >>> file:/etc/cassandra/default.conf/cassandra.yaml >>> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator; >>> authorizer=AllowAllAuthorizer; auto_snapshot=true; >>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; >>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; >>> cluster_name=Test Cluster; column_index_size_in_kb=64; >>> commit_failure_policy=stop; >>> commitlog_directory=/var/lib/cassandra/commitlog; >>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic; >>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; >>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32; >>> concurrent_writes=32; counter_cache_save_period=7200; >>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; >>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; >>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; >>> dynamic_snitch_reset_interval_in_ms=600000; >>> dynamic_snitch_update_interval_in_ms=100; >>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true; >>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false; >>> index_summary_capacity_in_mb=null; >>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; >>> internode_compression=all; key_cache_save_period=14400; >>> key_cache_size_in_mb=null; listen_address=null; >>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; >>> memtable_allocation_type=heap_buffers; native_transport_port=9042; >>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; >>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000; >>> read_request_timeout_in_ms=999999999; >>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler; >>> request_timeout_in_ms=999999999; row_cache_save_period=0; >>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true; >>> rpc_port=9160; rpc_server_type=sync; >>> saved_caches_directory=/var/lib/cassandra/saved_caches; >>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, >>> parameters=[{seeds=206.191.151.199}]}]; >>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false; >>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; >>> start_native_transport=true; start_rpc=true; storage_port=7000; >>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; >>> tombstone_warn_threshold=1000; trickle_fsync=false; >>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; >>> write_request_timeout_in_ms=999999999] >>> ERROR 09:50:41 Exception encountered during startup >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60] >>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) >>> ~[na:1.7.0_60] >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:714) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) >>> at >>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>> at >>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448) >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537) >>> Exception encountered during startup: unable to create new native thread >>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot >>> allocate memory' (errno=12 >>> >>> This is our production cluster and rest 2 nodes are still in healthy >>> state. >>> We have also changed heap size from 10G to 20 G but problem still >>> persists. Please help in finding the reason behind this. >>> >>> >>> Regards: >>> Rahul Bhardwaj >>> >>> >>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on >>> this and more: <https://plus.google.com/+indiamart> >>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>> Mobile Channel: >>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>> <http://m.indiamart.com/> >>> >>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>> Check out some smart tips to generate more business - *IndiaMART >>> Learning Centre* <http://corporate.indiamart.com/quick-learn/> >> >> >> > -- Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> Check out some smart tips to generate more business - *IndiaMART Learning Centre* <http://corporate.indiamart.com/quick-learn/>