[jira] [Updated] (CASSANDRA-8660) Cassandra makes Java 1.7+ SEGFAULT
[ https://issues.apache.org/jira/browse/CASSANDRA-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Belanger updated CASSANDRA-8660: --- Attachment: hs_err_pid-patch1.log Here is the hs_err_pid file with the first proposed patch. Cassandra makes Java 1.7+ SEGFAULT -- Key: CASSANDRA-8660 URL: https://issues.apache.org/jira/browse/CASSANDRA-8660 Project: Cassandra Issue Type: Bug Components: Core Environment: Linux Redhat 6.5 Reporter: Pierre Belanger Priority: Critical Attachments: 8660, hs_err_pid-patch1.log, hs_err_pid12004.log We upgraded a 3 nodes cluster from Cassandra 2.0.11-5f54285e9e to 2.1-7e6d9eb842. With Cassandra 2.1 , within ~10 min. cassandra manages to makes Java 1.7.0_72-b14 segfault. This is also tested and same behavior w/ Java 1.7.0_60-b19 and 1.8.0_25-b17. We were able to use same data, downgrade back to 2.0 and things ran for ~3-4 days w/ no hick up. Upgrading back to Cassandra-2.1, Java does a segfault again. See attached hs_err file. Pierre -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8660) Cassandra makes Java 1.7+ SEGFAULT
[ https://issues.apache.org/jira/browse/CASSANDRA-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285929#comment-14285929 ] Pierre Belanger commented on CASSANDRA-8660: About 600G / node Cassandra makes Java 1.7+ SEGFAULT -- Key: CASSANDRA-8660 URL: https://issues.apache.org/jira/browse/CASSANDRA-8660 Project: Cassandra Issue Type: Bug Components: Core Environment: Linux Redhat 6.5 Reporter: Pierre Belanger Priority: Critical Attachments: 8660, hs_err_pid12004.log We upgraded a 3 nodes cluster from Cassandra 2.0.11-5f54285e9e to 2.1-7e6d9eb842. With Cassandra 2.1 , within ~10 min. cassandra manages to makes Java 1.7.0_72-b14 segfault. This is also tested and same behavior w/ Java 1.7.0_60-b19 and 1.8.0_25-b17. We were able to use same data, downgrade back to 2.0 and things ran for ~3-4 days w/ no hick up. Upgrading back to Cassandra-2.1, Java does a segfault again. See attached hs_err file. Pierre -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8660) Cassandra makes Java 1.7+ SEGFAULT
Pierre Belanger created CASSANDRA-8660: -- Summary: Cassandra makes Java 1.7+ SEGFAULT Key: CASSANDRA-8660 URL: https://issues.apache.org/jira/browse/CASSANDRA-8660 Project: Cassandra Issue Type: Bug Components: Core Environment: Linux Redhat 6.5 Reporter: Pierre Belanger Priority: Critical Attachments: hs_err_pid12004.log We upgraded a 3 nodes cluster from Cassandra 2.0.11-5f54285e9e to 2.1-7e6d9eb842. With Cassandra 2.1 , within ~10 min. cassandra manages to makes Java 1.7.0_72-b14 segfault. This is also tested and same behavior w/ Java 1.7.0_60-b19 and 1.8.0_25-b17. We were able to use same data, downgrade back to 2.0 and things ran for ~3-4 days w/ no hick up. Upgrading back to Cassandra-2.1, Java does a segfault again. See attached hs_err file. Pierre -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8660) Cassandra makes Java 1.7+ SEGFAULT
[ https://issues.apache.org/jira/browse/CASSANDRA-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284991#comment-14284991 ] Pierre Belanger commented on CASSANDRA-8660: FYI - Out of 525 cassandra crashes , all hs_err file reports current thread was CompactionExecutor. # grep ^= hs_err_pid* | grep CompactionExecutor 525 - We have the following for our CF : compaction_strategy = 'LeveledCompactionStrategy' sstable_size_in_mb = 200 sstable_compression = 'SnappyCompressor' tombstone_compaction_interval = 300 tombstone_threshold = 0.1; - cassandra.yaml stripped : cluster_name: 'Default Cluster' hinted_handoff_enabled: true max_hint_window_in_ms: 1080 # 3 hours hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 batchlog_replay_throttle_in_kb: 1024 authenticator: AllowAllAuthenticator authorizer: AllowAllAuthorizer permissions_validity_in_ms: 2000 partitioner: org.apache.cassandra.dht.Murmur3Partitioner data_file_directories: - /home/var/cassandra/data commitlog_directory: /home/var/cassandra/commitlog disk_failure_policy: stop commit_failure_policy: stop key_cache_size_in_mb: 600 key_cache_save_period: 14400 row_cache_size_in_mb: 2 row_cache_save_period: 14400 counter_cache_size_in_mb: 0 counter_cache_save_period: 7200 memory_allocator: JEMallocAllocator saved_caches_directory: /home/var/cassandra/saved_caches commitlog_sync: periodic commitlog_sync_period_in_ms: 1 commitlog_segment_size_in_mb: 32 seed_provider: - class_name: com.company.cassandra.SmartrAutoSeedProvider parameters: - numHosts: 3 concurrent_reads: 160 concurrent_writes: 160 concurrent_counter_writes: 32 memtable_heap_space_in_mb: 6000 memtable_offheap_space_in_mb: 6000 memtable_allocation_type: heap_buffers commitlog_total_space_in_mb: 8192 index_summary_capacity_in_mb: index_summary_resize_interval_in_minutes: 60 trickle_fsync: true trickle_fsync_interval_in_kb: 10240 storage_port: 7000 ssl_storage_port: 7001 start_native_transport: true native_transport_port: 9042 native_transport_max_threads: 1024 start_rpc: true rpc_port: 9160 rpc_keepalive: true rpc_server_type: sync thrift_framed_transport_size_in_mb: 15 incremental_backups: false snapshot_before_compaction: false auto_snapshot: false tombstone_warn_threshold: 1000 tombstone_failure_threshold: 100 column_index_size_in_kb: 128 batch_size_warn_threshold_in_kb: 5 concurrent_compactors: 40 compaction_throughput_mb_per_sec: 0 sstable_preemptive_open_interval_in_mb: 50 read_request_timeout_in_ms: 6 range_request_timeout_in_ms: 6 write_request_timeout_in_ms: 6 counter_write_request_timeout_in_ms: 5000 cas_contention_timeout_in_ms: 1000 truncate_request_timeout_in_ms: 6 request_timeout_in_ms: 3 cross_node_timeout: false endpoint_snitch: SimpleSnitch dynamic_snitch_update_interval_in_ms: 100 dynamic_snitch_reset_interval_in_ms: 60 dynamic_snitch_badness_threshold: 0.1 request_scheduler: org.apache.cassandra.scheduler.NoScheduler internode_compression: dc inter_dc_tcp_nodelay: false disk_access_mode: mmap_index_only thrift_max_message_length_in_mb: 16 index_interval: 128 Cassandra makes Java 1.7+ SEGFAULT -- Key: CASSANDRA-8660 URL: https://issues.apache.org/jira/browse/CASSANDRA-8660 Project: Cassandra Issue Type: Bug Components: Core Environment: Linux Redhat 6.5 Reporter: Pierre Belanger Priority: Critical Attachments: hs_err_pid12004.log We upgraded a 3 nodes cluster from Cassandra 2.0.11-5f54285e9e to 2.1-7e6d9eb842. With Cassandra 2.1 , within ~10 min. cassandra manages to makes Java 1.7.0_72-b14 segfault. This is also tested and same behavior w/ Java 1.7.0_60-b19 and 1.8.0_25-b17. We were able to use same data, downgrade back to 2.0 and things ran for ~3-4 days w/ no hick up. Upgrading back to Cassandra-2.1, Java does a segfault again. See attached hs_err file. Pierre -- This message was sent by Atlassian JIRA (v6.3.4#6332)
