[ https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bing Wu updated CASSANDRA-12886: -------------------------------- Description: While running "nodetool repair", I see many instances of "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in system.logs on some nodes in the cluster. Timestamps correspond to streaming source/initiator's error messages of "sync failed between ..." Setup: - Cassandra 3.7.01 - CentOS 6.7 in AWS (multi-region) - JDK version: {noformat} java version "1.8.0_102" Java(TM) SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) {noformat} - cassandra.yaml: {noformat} server_encryption_options: internode_encryption: all keystore: [path] keystore_password: [password] truststore: [path] truststore_password: [password] # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] require_client_auth: false {noformat} Error messages in system.log on the target host: {noformat} ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475 StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e] Streaming error occurred on session with peer 54.247.111.232 javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_102] at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_102] at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_102] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_102] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_102] at org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66) ~[apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371) [apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342) [apache-cassandra-3.7.0.jar:3.7.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset {noformat} was: While running "nodetool repair", I see many instances of "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in system.logs on some nodes in the cluster. Timestamp corresponds to streaming source/initiator's error messages of "sync failed between ..." Setup: - Cassandra 3.7.01 - CentOS 6.7 in AWS (multi-region) - JDK version: {noformat} java version "1.8.0_102" Java(TM) SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) {noformat} - cassandra.yaml: {noformat} server_encryption_options: internode_encryption: all keystore: [path] keystore_password: [password] truststore: [path] truststore_password: [password] # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] require_client_auth: false {noformat} Error messages in system.log on the target host: {noformat} ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475 StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e] Streaming error occurred on session with peer 54.247.111.232 javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_102] at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_102] at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_102] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_102] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_102] at org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66) ~[apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371) [apache-cassandra-3.7.0.jar:3.7.0] at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342) [apache-cassandra-3.7.0.jar:3.7.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset {noformat} > Streaming failed due to SSL Socket connection reset > --------------------------------------------------- > > Key: CASSANDRA-12886 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12886 > Project: Cassandra > Issue Type: Bug > Reporter: Bing Wu > > While running "nodetool repair", I see many instances of > "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in > system.logs on some nodes in the cluster. Timestamps correspond to streaming > source/initiator's error messages of "sync failed between ..." > Setup: > - Cassandra 3.7.01 > - CentOS 6.7 in AWS (multi-region) > - JDK version: {noformat} > java version "1.8.0_102" > Java(TM) SE Runtime Environment (build 1.8.0_102-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) > {noformat} > - cassandra.yaml: > {noformat} > server_encryption_options: > internode_encryption: all > keystore: [path] > keystore_password: [password] > truststore: [path] > truststore_password: [password] > # More advanced defaults below: > # protocol: TLS > # algorithm: SunX509 > # store_type: JKS > # cipher_suites: > [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] > require_client_auth: false > {noformat} > Error messages in system.log on the target host: > {noformat} > ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475 > StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e] > Streaming error occurred on session with peer 54.247.111.232 > javax.net.ssl.SSLException: Connection has been shutdown: > javax.net.ssl.SSLException: java.net.SocketException: Connection reset > at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) > ~[na:1.8.0_102] > at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) > ~[na:1.8.0_102] > at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) > ~[na:1.8.0_102] > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > ~[na:1.8.0_102] > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > ~[na:1.8.0_102] > at > org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66) > ~[apache-cassandra-3.7.0.jar:3.7.0] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371) > [apache-cassandra-3.7.0.jar:3.7.0] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342) > [apache-cassandra-3.7.0.jar:3.7.0] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] > Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection > reset > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)