[ https://issues.apache.org/jira/browse/CASSANDRA-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550493#comment-14550493 ]
Sam Tunnicliffe commented on CASSANDRA-9419: -------------------------------------------- I'm still seeing failures locally, where the log output looks like this (with some of the WARNs elided): {noformat} [junit] Testsuite: org.apache.cassandra.pig.CqlTableTest [junit] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 47.007 sec [junit] [junit] ------------- Standard Output --------------- ... [junit] WARN 13:27:58 job_local_0003 [junit] java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320) ~[main/:na] [junit] WARN 13:27:58 job_local_0003 [junit] java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320) ~[main/:na] [junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. [junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. [junit] ERROR 13:27:58 1 map reduce job(s) failed! [junit] ERROR 13:27:58 1 map reduce job(s) failed! [junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:27:59 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:27:59 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004 [junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004 [junit] WARN 13:28:02 SchemaTupleBackend has already been initialized [junit] WARN 13:28:02 SchemaTupleBackend has already been initialized [junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:28:03 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:28:03 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:28:03 Output path is null in cleanup [junit] WARN 13:28:03 Output path is null in cleanup [junit] WARN 13:28:03 job_local_0005 [junit] java.io.IOException: java.lang.RuntimeException: failed to prepare cql query update cql3ks.collectiontable set n = ? WHERE "m" = ? [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:368) ~[main/:na] [junit] Caused by: java.lang.RuntimeException: failed to prepare cql query update cql3ks.collectiontable set n = ? WHERE "m" = ? [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:392) ~[main/:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:346) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [junit] at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:79) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:388) ~[main/:na] [junit] ... 1 common frames omitted [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [junit] at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:107) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.SessionManager.execute(SessionManager.java:538) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.SessionManager.prepareAsync(SessionManager.java:124) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:77) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] ... 2 common frames omitted [junit] WARN 13:28:03 job_local_0005 [junit] java.io.IOException: java.lang.RuntimeException: failed to prepare cql query update cql3ks.collectiontable set n = ? WHERE "m" = ? [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:368) ~[main/:na] [junit] Caused by: java.lang.RuntimeException: failed to prepare cql query update cql3ks.collectiontable set n = ? WHERE "m" = ? [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:392) ~[main/:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:346) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [junit] at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:289) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:79) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.preparedStatement(CqlRecordWriter.java:388) ~[main/:na] [junit] ... 1 common frames omitted [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) [junit] at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:107) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.SessionManager.execute(SessionManager.java:538) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.SessionManager.prepareAsync(SessionManager.java:124) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:77) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] ... 2 common frames omitted {noformat} I think you want to add a {{continue}} at the end of the catch block where the connection is opened. Otherwise when the connection attempt fails, unless we meet the conditions to {{break:outer}}, it goes on to try and prepare the statement, which results in a {{RuntimeException}}. > CqlRecordWriter does not adequately determine if driver failed to connect due > to interrupt > ------------------------------------------------------------------------------------------ > > Key: CASSANDRA-9419 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9419 > Project: Cassandra > Issue Type: Bug > Reporter: Philip Thompson > Assignee: Philip Thompson > Fix For: 2.2.0 rc1 > > Attachments: 9419.txt > > > Until https://datastax-oss.atlassian.net/browse/JAVA-769 is implemented, I > have had to implement a workaround. > Java's thread interrupt status flag can be cleared for any number of reasons. > After discussion, I've changed to parsing the driver's DriverException to > tell if we are safe to retry the failed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)