[ https://issues.apache.org/jira/browse/CASSANDRA-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812720#comment-17812720 ]
Stefan Miklosovic edited comment on CASSANDRA-16999 at 1/31/24 2:31 PM: ------------------------------------------------------------------------ [~absurdfarce] would you try again with this branch [https://github.com/apache/cassandra/pull/3076] ? I think I fixed that, I see different ports for each native_port_ssl column and 3 nodes cluster just started fine. Even though I see 9042 and 4567 as native_port and native_port_ssl in system.peers_vs, when I use this as contact points, the driver just seems to use 9042 {noformat} contact-points = ["172.19.0.5:4567", "172.19.0.8:4567", "172.19.0.9:4567"] {noformat} Notice this: {code:java} You specified dc1 as the local DC, but some contact points are from a different DC: Node(endPoint=/172.19.0.9:4567, hostId=null, hashCode=23897cc3)=null, Node(endPoint=/172.19.0.8:4567, hostId=null, hashCode=1e687984)=null; please provide the correct local DC, or check your contact points {code} whole log: {code:java} [main] org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext: Root WebApplicationContext: initialization completed in 758 ms [main] com.datastax.oss.driver.internal.core.DefaultMavenCoordinates: DataStax Java driver for Apache Cassandra(R) (com.datastax.oss:java-driver-core) version 4.14.0 [s0-admin-0] com.datastax.oss.driver.internal.core.time.Clock: Using native clock for microsecond precision [s0-admin-0] com.datastax.oss.driver.internal.core.loadbalancing.helper.OptionalLocalDcHelper: [s0|default] You specified dc1 as the local DC, but some contact points are from a different DC: Node(endPoint=/172.19.0.9:4567, hostId=null, hashCode=23897cc3)=null, Node(endPoint=/172.19.0.8:4567, hostId=null, hashCode=1e687984)=null; please provide the correct local DC, or check your contact points [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xba55eb50, L:/172.19.0.1:33148 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) 2[s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xc0106973, L:/172.19.0.1:50954 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [main] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport: Initializing ExecutorService 'applicationTaskExecutor' [main] org.apache.juli.logging.DirectJDKLog: Starting ProtocolHandler ["http-nio-127.0.0.1-8080"] [main] org.springframework.boot.web.embedded.tomcat.TomcatWebServer: Tomcat started on port(s): 8080 (http) with context path '' [main] org.springframework.boot.StartupInfoLogger: Started Application in 2.546 seconds (JVM running for 2.892) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x89395b9d, L:/172.19.0.1:50962 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xdbb2febd, L:/172.19.0.1:33164 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x8a96329c, L:/172.19.0.1:33168 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xb1258035, L:/172.19.0.1:50970 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xef0a8273, L:/172.19.0.1:33170 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x4c1e1b09, L:/172.19.0.1:50980 - R:cassandra-node-1/172.19.0.8:9042] Protoc {code} It seems to ignore 9042 and still tries to connect to 9042 anyway ... My application.conf: {code:java} datastax-java-driver { basic { contact-points = ["172.19.0.5:4567", "172.19.0.8:4567", "172.19.0.9:4567"] load-balancing-policy { local-datacenter = "dc1" } } advanced.auth-provider { class = PlainTextAuthProvider username = admin password = admin } advanced.ssl-engine-factory { class = DefaultSslEngineFactory cipher-suites = ["TLS_RSA_WITH_AES_256_CBC_SHA"] hostname-validation = false truststore-path = server-truststore.jks truststore-password = cassandra keystore-path = spark-master-1-server-keystore.p12 keystore-password = cassandra } } {code} I do not have any problem to connect to each node on 9042 or 4567 port via cqlsh. I see this on each node: *from 172.19.0.5* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.9 peer_port | 7000 native_address | 172.19.0.9 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.8 peer_port | 7000 native_address | 172.19.0.8 native_port | 9042 native_port_ssl | 4567 data_center | dc1 {noformat} *from 172.19.0.8* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.9 peer_port | 7000 native_address | 172.19.0.9 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.5 peer_port | 7000 native_address | 172.19.0.5 native_port | 9042 native_port_ssl | 4567 data_center | dc1 {noformat} *from 172.19.0.9:* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.8 peer_port | 7000 native_address | 172.19.0.8 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.5 peer_port | 7000 native_address | 172.19.0.5 native_port | 9042 native_port_ssl | 4567 data_center | dc1{noformat} was (Author: smiklosovic): [~absurdfarce] would you try again with this branch [https://github.com/apache/cassandra/pull/3076] ? I think I fixed that, I see different ports for each native_port_ssl column and 3 nodes cluster just started fine. Even though I see 9042 and 4567 as native_port and native_port_ssl in system.peers_vs, when I use this as contact points, the driver just seems to use 9042 {noformat} contact-points = ["172.19.0.5:4567", "172.19.0.8:4567", "172.19.0.9:4567"] {noformat} Notice this: {code:java} You specified dc1 as the local DC, but some contact points are from a different DC: Node(endPoint=/172.19.0.9:4567, hostId=null, hashCode=23897cc3)=null, Node(endPoint=/172.19.0.8:4567, hostId=null, hashCode=1e687984)=null; please provide the correct local DC, or check your contact points {code} whole log: {code:java} [main] org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext: Root WebApplicationContext: initialization completed in 758 ms [main] com.datastax.oss.driver.internal.core.DefaultMavenCoordinates: DataStax Java driver for Apache Cassandra(R) (com.datastax.oss:java-driver-core) version 4.14.0 [s0-admin-0] com.datastax.oss.driver.internal.core.time.Clock: Using native clock for microsecond precision [s0-admin-0] com.datastax.oss.driver.internal.core.loadbalancing.helper.OptionalLocalDcHelper: [s0|default] You specified dc1 as the local DC, but some contact points are from a different DC: Node(endPoint=/172.19.0.9:4567, hostId=null, hashCode=23897cc3)=null, Node(endPoint=/172.19.0.8:4567, hostId=null, hashCode=1e687984)=null; please provide the correct local DC, or check your contact points [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xba55eb50, L:/172.19.0.1:33148 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) 2[s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xc0106973, L:/172.19.0.1:50954 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [main] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport: Initializing ExecutorService 'applicationTaskExecutor' [main] org.apache.juli.logging.DirectJDKLog: Starting ProtocolHandler ["http-nio-127.0.0.1-8080"] [main] org.springframework.boot.web.embedded.tomcat.TomcatWebServer: Tomcat started on port(s): 8080 (http) with context path '' [main] org.springframework.boot.StartupInfoLogger: Started Application in 2.546 seconds (JVM running for 2.892) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x89395b9d, L:/172.19.0.1:50962 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xdbb2febd, L:/172.19.0.1:33164 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x8a96329c, L:/172.19.0.1:33168 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xb1258035, L:/172.19.0.1:50970 - R:cassandra-node-1/172.19.0.8:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.9:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0xef0a8273, L:/172.19.0.1:33170 - R:cassandra-node-2/172.19.0.9:9042] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, CLIENT_ID=42d95edb-9504-40ec-a2b2-cee22dd66fea}): failed to send request (io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 8500000000000000680000000a0062496e76616c6964206f7220756e737570706f727465642070726f746f636f6c2076657273696f6e20283232293b20737570706f727465642076657273696f6e73206172652028332f76332c20342f76342c20352f76352c20362f76362d6265746129)) [s0-admin-0] com.datastax.oss.driver.internal.core.util.Loggers: [s0|/172.19.0.8:9042] Error while opening new channel (ConnectionInitException: [s0|id: 0x4c1e1b09, L:/172.19.0.1:50980 - R:cassandra-node-1/172.19.0.8:9042] Protoc {code} It seems to ignore 9042 and still tries to connect to 9042 anyway ... My application.conf: {code:java} datastax-java-driver { basic { contact-points = ["172.19.0.5:4567", "172.19.0.8:4567", "172.19.0.9:4567"] load-balancing-policy { local-datacenter = "dc1" } } advanced.auth-provider { class = PlainTextAuthProvider username = admin password = admin } advanced.ssl-engine-factory { class = DefaultSslEngineFactory cipher-suites = ["TLS_RSA_WITH_AES_256_CBC_SHA"] hostname-validation = false truststore-path = server-truststore.jks truststore-password = cassandra keystore-path = spark-master-1-server-keystore.p12 keystore-password = cassandra } } {code} I see this on each node: *from 172.19.0.5* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.9 peer_port | 7000 native_address | 172.19.0.9 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.8 peer_port | 7000 native_address | 172.19.0.8 native_port | 9042 native_port_ssl | 4567 data_center | dc1 {noformat} *from 172.19.0.8* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.9 peer_port | 7000 native_address | 172.19.0.9 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.5 peer_port | 7000 native_address | 172.19.0.5 native_port | 9042 native_port_ssl | 4567 data_center | dc1 {noformat} *from 172.19.0.9:* {noformat} admin@cqlsh> select peer, peer_port, native_address, native_port, native_port_ssl, data_center from system.peers_v2; @ Row 1 -----------------+------------ peer | 172.19.0.8 peer_port | 7000 native_address | 172.19.0.8 native_port | 9042 native_port_ssl | 4567 data_center | dc1 @ Row 2 -----------------+------------ peer | 172.19.0.5 peer_port | 7000 native_address | 172.19.0.5 native_port | 9042 native_port_ssl | 4567 data_center | dc1{noformat} > system.peers and system.peers_v2 do not contain the native_transport and/or > native_transport_port_ssl > ----------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-16999 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16999 > Project: Cassandra > Issue Type: Bug > Components: Local/Other > Reporter: Steve Lacerda > Assignee: Brandon Williams > Priority: Normal > Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x > > Time Spent: 10m > Remaining Estimate: 0h > > system.peers_v2 includes a “native_port” but has no notion of > native_transport_port vs. native_transport_port_ssl. Given this limited > information, there’s no clear way for the driver to know that different ports > are being used for SSL vs. non-SSL or which of those two ports is identified > by “native_port”. > > The issue we ran into is that the java driver, since it has no notion of the > transport port SSL, the driver was only using the contact points and was not > load balancing. > > The customer had both set: > native_transport_port: 9042 > native_transport_port_ssl: 9142 > > They were attempting to connect to 9142, but that was failing. They could > only use 9042, and so their applications load balancing was failing. We found > that any node that was a contact point was connecting, but the other nodes > were never acting as coordinators. > > There are still issues in the driver, for which I have created JAVA-2967, > which also refers to JAVA-2638, but the system.peers and system.peers_v2 > tables should both contain native_transport_port and > native_transport_port_ssl. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org