free RAM:
free -m
total used free shared buffers cached
Mem: 64398 23753 40644 0 108 8324
-/+ buffers/cache: 15319 49078
Swap: 2925 15 2909
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515041
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 515041
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Also attaching complete error file
On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <anis...@gmail.com> wrote:
> the memory cassandra is trying to allocate is pretty small. you sure there
> is no hardware failure on the machine. what is the free ram on the box ?
>
> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
> rahul.bhard...@indiamart.com> wrote:
>
>> Hi All,
>>
>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running in
>> healthy state. Suddenly one machine's cassandra daemon stops working and
>> shut down.
>>
>> On restarting it after 2 minutes it again stops and is getting stop after
>> returning below error in cassandra.log
>>
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>> allocate memory' (errno=12)
>> #
>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>> committing reserved memory.
>> # An error report file with more information is saved as:
>> # /tmp/hs_err_pid23215.log
>> INFO 09:50:41 Loading settings from
>> file:/etc/cassandra/default.conf/cassandra.yaml
>> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>> commit_failure_policy=stop;
>> commitlog_directory=/var/lib/cassandra/commitlog;
>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>> concurrent_writes=32; counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>> dynamic_snitch_reset_interval_in_ms=600000;
>> dynamic_snitch_update_interval_in_ms=100;
>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>> index_summary_capacity_in_mb=null;
>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>> internode_compression=all; key_cache_save_period=14400;
>> key_cache_size_in_mb=null; listen_address=null;
>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>> read_request_timeout_in_ms=999999999;
>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>> rpc_port=9160; rpc_server_type=sync;
>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>> parameters=[{seeds=206.191.151.199}]}];
>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>> start_native_transport=true; start_rpc=true; storage_port=7000;
>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>> tombstone_warn_threshold=1000; trickle_fsync=false;
>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>> write_request_timeout_in_ms=999999999]
>> ERROR 09:50:41 Exception encountered during startup
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>> ~[na:1.7.0_60]
>> at
>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:714)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at
>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>> at
>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>> Exception encountered during startup: unable to create new native thread
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>> allocate memory' (errno=12
>>
>> This is our production cluster and rest 2 nodes are still in healthy
>> state.
>> We have also changed heap size from 10G to 20 G but problem still
>> persists. Please help in finding the reason behind this.
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Check out some smart tips to generate more business - *IndiaMART
>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
