There was some issue with the i3 instances and Cassandra. Did you had this
cluster running always on i3?
On Apr 6, 2017 13:06, "Cogumelos Maravilha" <cogumelosmaravi...@sapo.pt>
wrote:
> Limit Soft Limit Hard Limit
> Units
> Max cpu time unlimited unlimited
> seconds
> Max file size unlimited unlimited
> bytes
> Max data size unlimited unlimited
> bytes
> Max stack size 8388608 unlimited
> bytes
> Max core file size 0 unlimited
> bytes
> Max resident set unlimited unlimited
> bytes
> Max processes 122575 122575
> processes
> Max open files 100000 100000
> files
> Max locked memory unlimited unlimited
> bytes
> Max address space unlimited unlimited
> bytes
> Max file locks unlimited unlimited
> locks
> Max pending signals 122575 122575
> signals
> Max msgqueue size 819200 819200
> bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
> Please find something wrong there!
>
> Thanks.
>
> On 04/06/2017 11:50 AM, benjamin roth wrote:
>
> Limits: You should check them in /proc/$pid/limits
>
> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha <cogumelosmaravi...@sapo.pt
> >:
>
>> Yes C* is running as cassandra:
>>
>> cassand+ 2267 1 99 10:18 ? 00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>> -XX:Threa...
>>
>> INFO [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>> configuration:[allocate_tokens_for_keyspace=null;
>> authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer;
>> auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false;
>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>> broadcast_address=null; broadcast_rpc_address=null;
>> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=6000000;
>> cdc_enabled=false; cdc_free_space_check_interval_ms=250;
>> cdc_raw_directory=null; cdc_total_space_in_mb=0;
>> client_encryption_options=<REDACTED>; cluster_name=company;
>> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>> commit_failure_policy=ignore; commitlog_compression=null;
>> commitlog_directory=/mnt/cassandra/commitlog;
>> commitlog_max_compression_buffers_in_pool=3;
>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
>> commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null;
>> compaction_large_partition_warning_threshold_mb=100;
>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=6000000;
>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>> data_file_directories=[Ljava.lang.String;@223f3642;
>> disk_access_mode=auto; disk_failure_policy=ignore;
>> disk_optimization_estimate_percentile=0.95;
>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
>> dynamic_snitch_reset_interval_in_ms=600000;
>> dynamic_snitch_update_interval_in_ms=100;
>> enable_scripted_user_defined_functions=false;
>> enable_user_defined_functions=false;
>> enable_user_defined_functions_threads=true;
>> encryption_options=null; endpoint_snitch=SimpleSnitch;
>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
>> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
>> hints_compression=null; hints_directory=/mnt/cassandra/hints;
>> hints_flush_period_in_ms=10000; incremental_backups=false;
>> index_interval=null; index_summary_capacity_in_mb=null;
>> index_summary_resize_interval_in_minutes=60; initial_token=null;
>> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>> inter_dc_tcp_nodelay=false; internode_authenticator=null;
>> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
>> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
>> key_cache_save_period=14400; key_cache_size_in_mb=null;
>> listen_address=10.100.100.213; listen_interface=null;
>> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
>> max_streaming_retries=3; max_value_size_in_mb=256;
>> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
>> memtable_flush_writers=0; memtable_heap_space_in_mb=null;
>> memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
>> native_transport_max_concurrent_connections=-1;
>> native_transport_max_concurrent_connections_per_ip=-1;
>> native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128;
>> native_transport_port=9042; native_transport_port_ssl=null;
>> num_tokens=256; otc_coalescing_strategy=TIMEHORIZON;
>> otc_coalescing_window_us=200;
>> partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>> permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1;
>> permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
>> prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=6000000;
>> read_request_timeout_in_ms=6000000; request_scheduler=org.apache.c
>> assandra.scheduler.NoScheduler; request_scheduler_id=null;
>> request_scheduler_options=null; request_timeout_in_ms=6000000;
>> role_manager=CassandraRoleManager; roles_cache_max_entries=1000;
>> roles_update_interval_in_ms=-1; roles_validity_in_ms=2000;
>> row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
>> row_cache_keys_to_save=2147483647; row_cache_save_period=0;
>> row_cache_size_in_mb=0; rpc_address=10.100.100.213; rpc_interface=null;
>> rpc_interface_prefer_ipv6=false; rpc_keepalive=true;
>> rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16;
>> rpc_port=9160; rpc_recv_buff_size_in_bytes=null;
>> rpc_send_buff_size_in_bytes=null; rpc_server_type=sync;
>> saved_caches_directory=/mnt/cassandra/saved_caches;
>> seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19,
>> 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52,
>> 10.100.1000.213}; server_encryption_options=<REDACTED>;
>> slow_query_log_timeout_in_ms=6000000; snapshot_before_compaction=false;
>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>> start_native_transport=true; start_rpc=false; storage_port=7000;
>> stream_throughput_outbound_megabits_per_sec=200;
>> streaming_keep_alive_period_in_secs=300;
>> streaming_socket_timeout_in_ms=86400000;
>> thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16;
>> thrift_prepared_statements_cache_size_mb=null;
>> tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
>> tracetype_query_ttl=86400; tracetype_repair_ttl=604800;
>> transparent_data_encryption_options=org.apache.cassandra.con
>> fig.TransparentDataEncryptionOptions@38c5cc4c; trickle_fsync=false;
>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=6000000;
>> unlogged_batch_across_partitions_warn_threshold=10;
>> user_defined_function_fail_timeout=1500;
>> user_defined_function_warn_timeout=500;
>> user_function_timeout_policy=die; windows_timer_interval=1;
>> write_request_timeout_in_ms=6000000]
>> Thanks
>>
>>
>> On 04/06/2017 11:30 AM, benjamin roth wrote:
>>
>> Have you checked the effective limits of a running CS process?
>> Is CS run as Cassandra? Just to rule out missing file perms.
>>
>>
>> Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" <
>> cogumelosmaravi...@sapo.pt>:
>>
>> From cassandra.yaml:
>> hints_directory: /mnt/cassandra/hints
>> data_file_directories:
>> - /mnt/cassandra/data
>> commitlog_directory: /mnt/cassandra/commitlog
>> saved_caches_directory: /mnt/cassandra/saved_caches
>>
>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 mnt/
>>
>> drwxr-xr-x 6 cassandra cassandra 68 Apr 5 16:17 ./
>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 ../
>> drwxr-xr-x 2 cassandra cassandra 80 Apr 6 10:07 commitlog/
>> drwxr-xr-x 8 cassandra cassandra 124 Apr 5 16:17 data/
>> drwxr-xr-x 2 cassandra cassandra 72 Apr 5 16:20 hints/
>> drwxr-xr-x 2 cassandra cassandra 49 Apr 5 20:17 saved_caches/
>>
>> cassand+ 2267 1 99 10:18 ? 00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>> -XX:Threa...
>>
>> /dev/mapper/um_vg-xfs_lv 885G 27G 858G 4% /mnt
>>
>> On /etc/security/limits.conf
>>
>> * - memlock unlimited
>> * - nofile 100000
>> * - nproc 32768
>> * - as unlimited
>>
>> On /etc/security/limits.d/cassandra.conf
>>
>> cassandra - memlock unlimited
>> cassandra - nofile 100000
>> cassandra - as unlimited
>> cassandra - nproc 32768
>>
>> On /etc/sysctl.conf
>>
>> vm.max_map_count = 1048575
>>
>> On /etc/systcl.d/cassanda.conf
>>
>> vm.max_map_count = 1048575
>> net.ipv4.tcp_keepalive_time=600
>> On /etc/pam.d/su
>> ...
>> session required pam_limits.so
>> ...
>>
>> Distro is the currently Ubuntu LTS.
>> Thanks
>>
>>
>>
>> On 04/06/2017 10:39 AM, benjamin roth wrote:
>>
>> Cassandra cannot write an SSTable to disk. Are you sure the disk/volume
>> where SSTables reside (normally /var/lib/cassandra/data) is writeable for
>> the CS user and has enough free space?
>> The CDC warning also implies that.
>> The other warnings indicate you are probably not running CS as root and
>> you did not set an appropriate limit for max open files. Running out of
>> open files can also be a reason for the IO error.
>>
>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha <
>> cogumelosmaravi...@sapo.pt>:
>>
>>> Hi list,
>>>
>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have
>>> one node that is always dieing and I don't understand why. Can anyone
>>> give me some hints please. All nodes using the same configuration.
>>>
>>> Thanks in advance.
>>>
>>> INFO [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:22,5,main]
>>> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
>>> error
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn
>>> ternal(SequentialWriter.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(S
>>> equentialWriter.java:185)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.compress.CompressedSequentialWriter.
>>> access$100(CompressedSequentialWriter.java:38)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.compress.CompressedSequentialWriter$
>>> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.utils.concurrent.Transactional$Abstract
>>> Transactional.prepareToCommit(Transactional.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommi
>>> t(SequentialWriter.java:358)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.format.big.BigTableWriter$Tr
>>> ansactionalProxy.doPrepare(BigTableWriter.java:367)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.utils.concurrent.Transactional$Abstract
>>> Transactional.prepareToCommit(Transactional.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableWriter.prepare
>>> ToCommit(SSTableWriter.java:281)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.pre
>>> pareToCommit(SimpleSSTableMultiWriter.java:101)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtabl
>>> e(ColumnFamilyStore.java:1153)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFa
>>> milyStore.java:1086)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>> ~[na:1.8.0_121]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>> [na:1.8.0_121]
>>> at
>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$th
>>> readLocalDeallocator$0(NamedThreadFactory.java:79)
>>> [apache-cassandra-3.10.jar:3.10]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
>>> Caused by: java.io.IOException: Input/output error
>>> at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
>>> ~[na:1.8.0_121]
>>> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>> ~[na:1.8.0_121]
>>> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>> ~[na:1.8.0_121]
>>> at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn
>>> ternal(SequentialWriter.java:169)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> ... 15 common frames omitted
>>> INFO [IndexSummaryManager:1] 2017-04-06 06:22:18,366
>>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:31,5,main]
>>> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
>>> error
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn
>>> ternal(SequentialWriter.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(S
>>> equentialWriter.java:185)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.compress.CompressedSequentialWriter.
>>> access$100(CompressedSequentialWriter.java:38)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.compress.CompressedSequentialWriter$
>>> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.utils.concurrent.Transactional$Abstract
>>> Transactional.prepareToCommit(Transactional.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommi
>>> t(SequentialWriter.java:358)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.format.big.BigTableWriter$Tr
>>> ansactionalProxy.doPrepare(BigTableWriter.java:367)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.utils.concurrent.Transactional$Abstract
>>> Transactional.prepareToCommit(Transactional.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.format.SSTableWriter.prepare
>>> ToCommit(SSTableWriter.java:281)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.pre
>>> pareToCommit(SimpleSSTableMultiWriter.java:101)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtabl
>>> e(ColumnFamilyStore.java:1153)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFa
>>> milyStore.java:1086)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>> ~[na:1.8.0_121]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>> [na:1.8.0_121]
>>> at
>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$th
>>> readLocalDeallocator$0(NamedThreadFactory.java:79)
>>> [apache-cassandra-3.10.jar:3.10]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
>>> Caused by: java.io.IOException: Input/output error
>>> at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
>>> ~[na:1.8.0_121]
>>> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>> ~[na:1.8.0_121]
>>> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>> ~[na:1.8.0_121]
>>> at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn
>>> ternal(SequentialWriter.java:169)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> ... 15 common frames omitted
>>> INFO [main] 2017-04-06 07:11:57,289 YamlConfigurationLoader.java:89 -
>>> Configuration location: file:/etc/cassandra/cassandra.yaml
>>>
>>>
>>> Some ERRORs messages:
>>>
>>> ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:2,5,main]
>>> ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:3,5,main]
>>> ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:4,5,main]
>>> ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:19:13,453
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MessagingService-Incoming-/10.0.120.52,5,main]
>>> ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[epollEventLoopGroup-2-6,10,main]
>>> ERROR [Native-Transport-Requests-36] 2017-04-06 03:25:45,915
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-49] 2017-04-06 03:25:45,915
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-69] 2017-04-06 03:25:45,916
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-46] 2017-04-06 03:26:18,465
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-156] 2017-04-06 03:26:18,465
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-48] 2017-04-06 03:26:24,696 ?:? - JVM
>>> state determined to be unstable. Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-66] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-77] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [GossipTasks:1] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-133] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-135] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-70] 2017-04-06 03:27:11,569
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[IndexSummaryManager:1,1,main]
>>> ERROR [Native-Transport-Requests-103] 2017-04-06 03:27:24,049
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-69] 2017-04-06 03:27:24,049
>>> SEPWorker.java:145 - Failed to execute task, unexpected exception killed
>>> worker: {}
>>> ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:27:55,079
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079
>>> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> ERROR [Native-Transport-Requests-64] 2017-04-06 03:28:43,285
>>> SEPWorker.java:145 - Failed to execute task, unexpected exception killed
>>> worker: {}
>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:22,5,main]
>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:31,5,main]
>>>
>>> Also some WARNs:
>>>
>>> WARN [main] 2017-04-06 09:26:49,725 CLibrary.java:178 - Unable to lock
>>> JVM memory (ENOMEM). This can result in part of the JVM being swapped
>>> out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run
>>> Cassandra as root.
>>>
>>> WARN [main] 2017-04-06 09:25:07,355 StartupChecks.java:157 - JMX is not
>>> enabled to receive remote connections. Please see cassandra-env.sh for
>>> more info.
>>>
>>> WARN [main] 2017-04-06 09:25:07,369 SigarLibrary.java:174 - Cassandra
>>> server running in degraded mode. Is swap disabled? : true, Address
>>> space adequate? : true, nofile limit adequate? : false, nproc limit
>>> adequate? : true
>>>
>>> WARN [main] 2017-04-06 09:25:07,091 DatabaseDescriptor.java:493 - Small
>>> cdc volume detected at /var/lib/cassandra/cdc_raw; setting
>>> cdc_total_space_in_mb to 2502. You can override this in cassandra.yaml
>>>
>>>
>>>
>>
>>
>>
>>
>
>
--
--
