Hi, I want to add something: 5. do you know on which table are you getting these reads timeout ? 6. if yes, can you see if you don't have Excessive tombstone activity 7. how often do you run repair ? 8. can you send a system.log and also report of nodetool tpstats 9. Swap is enabled or not ?
Best regards _____________________________________________________________ [image: https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour] <https://www.facebook.com/DMN-BigData-371074727032197/?modal=admin_todo_tour> <https://twitter.com/dmnbigdata> <https://www.instagram.com/> <https://www.linkedin.com/in/dngaya/> *Dieudonne Madishon NGAYA* Datastax, Cassandra Architect *P: *7048580065 *w: *www.dmnbigdata.com *E: *dmng...@dmnbigdata.com *Private E: *dmng...@gmail.com *A: *Charlotte,NC,28273, USA On Fri, Mar 15, 2019 at 11:32 PM Jon Haddad <j...@jonhaddad.com> wrote: > > 1. What was the read request? Are you fetching a single row, a million, > something else? > 2. What are your GC settings? > 3. What's the hardware in use? What resources have been allocated to each > instance? > 4. Did you see this issue after a single request or is the cluster under > heavy load? > > If you're going to share a config it's much easier to read as an actual > text file rather than a double spaced paste into the ML. In the future if > you could share a link to the yaml you might get more eyes on it. > > Jon > > On Sat, Mar 16, 2019 at 3:57 PM Sundaramoorthy, Natarajan < > natarajan_sundaramoor...@optum.com> wrote: > >> 3 pod deployed in openshift. Read request timed out due to GC collection. >> Can you please look at below parameters and value to see if anything is out >> of place? Thanks >> >> >> >> >> >> cat cassandra.yaml >> >> >> >> num_tokens: 256 >> >> >> >> >> >> >> >> hinted_handoff_enabled: true >> >> >> >> hinted_handoff_throttle_in_kb: 1024 >> >> >> >> max_hints_delivery_threads: 2 >> >> >> >> hints_directory: /cassandra_data/hints >> >> >> >> hints_flush_period_in_ms: 10000 >> >> >> >> max_hints_file_size_in_mb: 128 >> >> >> >> >> >> batchlog_replay_throttle_in_kb: 1024 >> >> >> >> authenticator: PasswordAuthenticator >> >> >> >> authorizer: AllowAllAuthorizer >> >> >> >> role_manager: CassandraRoleManager >> >> >> >> roles_validity_in_ms: 2000 >> >> >> >> >> >> permissions_validity_in_ms: 2000 >> >> >> >> >> >> >> >> >> >> partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> >> >> >> data_file_directories: >> >> - /cassandra_data/data >> >> >> >> commitlog_directory: /cassandra_data/commitlog >> >> >> >> disk_failure_policy: stop >> >> >> >> commit_failure_policy: stop >> >> >> >> key_cache_size_in_mb: >> >> >> >> key_cache_save_period: 14400 >> >> >> >> >> >> >> >> row_cache_size_in_mb: 0 >> >> >> >> row_cache_save_period: 0 >> >> >> >> >> >> counter_cache_size_in_mb: >> >> >> >> counter_cache_save_period: 7200 >> >> >> >> >> >> saved_caches_directory: /cassandra_data/saved_caches >> >> >> >> commitlog_sync: periodic >> >> commitlog_sync_period_in_ms: 10000 >> >> >> >> commitlog_segment_size_in_mb: 32 >> >> >> >> >> >> seed_provider: >> >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider >> >> parameters: >> >> - seeds: >> "cassandra-0.cassandra.ihr-ei.svc.cluster.local,cassandra-1.cassandra.ihr-ei.svc.cluster.local" >> >> >> >> concurrent_reads: 32 >> >> concurrent_writes: 32 >> >> concurrent_counter_writes: 32 >> >> >> >> concurrent_materialized_view_writes: 32 >> >> >> >> >> >> >> >> >> >> disk_optimization_strategy: ssd >> >> >> >> >> >> >> >> memtable_allocation_type: heap_buffers >> >> >> >> commitlog_total_space_in_mb: 2048 >> >> >> >> >> >> index_summary_capacity_in_mb: >> >> >> >> index_summary_resize_interval_in_minutes: 60 >> >> >> >> trickle_fsync: false >> >> trickle_fsync_interval_in_kb: 10240 >> >> >> >> storage_port: 7000 >> >> >> >> ssl_storage_port: 7001 >> >> >> >> listen_address: 10.130.7.245 >> >> >> >> broadcast_address: 10.130.7.245 >> >> >> >> >> >> >> >> start_native_transport: true >> >> native_transport_port: 9042 >> >> >> >> >> >> >> >> start_rpc: true >> >> >> >> rpc_address: 0.0.0.0 >> >> >> >> rpc_port: 9160 >> >> >> >> broadcast_rpc_address: 10.130.7.245 >> >> >> >> rpc_keepalive: true >> >> >> >> rpc_server_type: sync >> >> >> >> >> >> >> >> >> >> thrift_framed_transport_size_in_mb: 15 >> >> >> >> incremental_backups: false >> >> >> >> snapshot_before_compaction: false >> >> >> >> auto_snapshot: true >> >> >> >> tombstone_warn_threshold: 1000 >> >> tombstone_failure_threshold: 100000 >> >> >> >> column_index_size_in_kb: 64 >> >> >> >> >> >> batch_size_warn_threshold_in_kb: 5 >> >> >> >> batch_size_fail_threshold_in_kb: 50 >> >> >> >> >> >> compaction_throughput_mb_per_sec: 16 >> >> >> >> compaction_large_partition_warning_threshold_mb: 100 >> >> >> >> sstable_preemptive_open_interval_in_mb: 50 >> >> >> >> >> >> >> >> read_request_timeout_in_ms: 50000 >> >> range_request_timeout_in_ms: 100000 >> >> write_request_timeout_in_ms: 20000 >> >> counter_write_request_timeout_in_ms: 5000 >> >> cas_contention_timeout_in_ms: 1000 >> >> truncate_request_timeout_in_ms: 60000 >> >> request_timeout_in_ms: 100000 >> >> >> >> cross_node_timeout: false >> >> >> >> >> >> phi_convict_threshold: 12 >> >> >> >> endpoint_snitch: GossipingPropertyFileSnitch >> >> >> >> dynamic_snitch_update_interval_in_ms: 100 >> >> dynamic_snitch_reset_interval_in_ms: 600000 >> >> dynamic_snitch_badness_threshold: 0.1 >> >> >> >> request_scheduler: org.apache.cassandra.scheduler.NoScheduler >> >> >> >> >> >> >> >> server_encryption_options: >> >> internode_encryption: none >> >> keystore: conf/.keystore >> >> truststore: conf/.truststore >> >> >> >> client_encryption_options: >> >> enabled: false >> >> optional: false >> >> keystore: conf/.keystore >> >> >> >> internode_compression: all >> >> >> >> inter_dc_tcp_nodelay: false >> >> >> >> tracetype_query_ttl: 86400 >> >> tracetype_repair_ttl: 604800 >> >> >> >> gc_warn_threshold_in_ms: 1000 >> >> >> >> enable_user_defined_functions: false >> >> >> >> enable_scripted_user_defined_functions: false >> >> >> >> windows_timer_interval: 1 >> >> >> >> >> >> auto_bootstrap: false >> >> >> This e-mail, including attachments, may include confidential and/or >> proprietary information, and may be used only by the person or entity >> to which it is addressed. If the reader of this e-mail is not the intended >> recipient or his or her authorized agent, the reader is hereby notified >> that any dissemination, distribution or copying of this e-mail is >> prohibited. If you have received this e-mail in error, please notify the >> sender by replying to this message and delete this e-mail immediately. >> >
