[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513111#comment-16513111
]
Cyril Scetbon edited comment on CASSANDRA-10751 at 6/14/18 11:07 PM:
-
[~michaelsembwever], It's a problem on 2.1.14. I haven't checked on another
version
was (Author: cscetbon):
[~michaelsembwever], It's a problem on 2.1.14. I haven't check on another
version
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
> host(s) tried for query failed (tried:
> node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511528#comment-16511528
]
Andy Tolbert edited comment on CASSANDRA-10751 at 6/13/18 6:51 PM:
---
(y) the driver does not quote the keyspace value given to {{Cluster.connect}}
so this change will cause problems for mixed-case keyspaces. To further
expound on [~jjordan]'s comment, it's misleading but the [original
code|https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477]
referenced is quoting the value returned from a {{set_keyspace}} response sent
by C*. In that case, C* sends the internal form of the keyspace, which is
never quoted, so we have to quote it there.
was (Author: andrew.tolbert):
(y) the driver does not quote the keyspace value given to {{Cluster.connect
}}so this change will cause problems for mixed-case keyspaces. To further
expound on [~jjordan]'s comment, it's misleading but the [original
code|https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477]
referenced is quoting the value returned from a {{set_keyspace}} response sent
by C*. In that case, C* sends the internal form of the keyspace, which is
never quoted, so we have to quote it there.
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511528#comment-16511528
]
Andy Tolbert edited comment on CASSANDRA-10751 at 6/13/18 6:37 PM:
---
(y) the driver does not quote the keyspace value given to {{Cluster.connect
}}so this change will cause problems for mixed-case keyspaces. To further
expound on [~jjordan]'s comment, it's misleading but the [original
code|https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477]
referenced is quoting the value returned from a {{set_keyspace}} response sent
by C*. In that case, C* sends the internal form of the keyspace, which is
never quoted, so we have to quote it there.
was (Author: andrew.tolbert):
(y), to further expound on [~jjordan]'s comment, it's misleading but the
[original
code|https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477]
referenced is quoting the value returned from a {{set_keyspace}} response sent
by C*. In that case, C* sends the internal form of the keyspace, which is
never quoted, so we have to quote it there.
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511413#comment-16511413
]
Jeremiah Jordan edited comment on CASSANDRA-10751 at 6/13/18 4:47 PM:
--
[~michaelsembwever] the C* 3.0+ java drivers to not quote that. This change is
bad on versions 3.0+ and breaks stuff.
https://github.com/datastax/java-driver/blob/3.0.8/driver-core/src/main/java/com/datastax/driver/core/Cluster.java#L336
was (Author: jjordan):
[~michaelsembwever] the C* 3.0+ java drivers to not quote that. This change is
bad on versions 3.0+ and breaks stuff.
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
> host(s) tried for query failed (tried:
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511413#comment-16511413
]
Jeremiah Jordan edited comment on CASSANDRA-10751 at 6/13/18 4:47 PM:
--
[~michaelsembwever] the C* 3.0+ java drivers do not quote that. This change is
bad on versions 3.0+ and breaks stuff.
https://github.com/datastax/java-driver/blob/3.0.8/driver-core/src/main/java/com/datastax/driver/core/Cluster.java#L336
was (Author: jjordan):
[~michaelsembwever] the C* 3.0+ java drivers to not quote that. This change is
bad on versions 3.0+ and breaks stuff.
https://github.com/datastax/java-driver/blob/3.0.8/driver-core/src/main/java/com/datastax/driver/core/Cluster.java#L336
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by:
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511413#comment-16511413
]
Jeremiah Jordan edited comment on CASSANDRA-10751 at 6/13/18 4:44 PM:
--
[~michaelsembwever] the C* 3.0+ java drivers to not quote that. This change is
bad on versions 3.0+ and breaks stuff.
was (Author: jjordan):
[~michaelsembwever] the C* 3.0+ java drivers to not quote that. This change is
bad.
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Fix For: 4.0, 2.2.13, 3.0.17, 3.11.3
>
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
> host(s) tried for query failed (tried:
> node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459637#comment-16459637
]
mck edited comment on CASSANDRA-10751 at 5/2/18 2:35 AM:
-
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/539/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/538/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
was (Author: michaelsembwever):
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/538/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459637#comment-16459637
]
mck edited comment on CASSANDRA-10751 at 5/1/18 11:51 PM:
--
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/538/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
was (Author: michaelsembwever):
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/535/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459637#comment-16459637
]
mck edited comment on CASSANDRA-10751 at 5/1/18 11:44 AM:
--
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/535/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
was (Author: michaelsembwever):
Sorry [~cscetbon] that this got completely forgotten. The patch makes sense.
I've put your patch into relevant branches, and will commit once they go green.
Here's the patch for trunk, 3.0 and 3.11:
|| Branch || uTest || dTest ||
|[cassandra-2.2_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-2.2_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-2.2_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/534/
|
|[cassandra-3.0_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.0_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.0_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/535/
|
|[cassandra-3.11_10751|https://github.com/thelastpickle/cassandra/tree/mck/cassandra-3.11_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Fcassandra-3.11_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/536/
|
|[trunk_10751|https://github.com/thelastpickle/cassandra/tree/mck/trunk_10751]|[!https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751.svg?style=svg!|https://circleci.com/gh/thelastpickle/cassandra/tree/mck%2Ftrunk_10751]|
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/537/
|
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Cyril Scetbon
>Priority: Major
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025776#comment-15025776
]
Cyril Scetbon edited comment on CASSANDRA-10751 at 11/25/15 12:13 AM:
--
[~alexliu68], so the issue is that CqlRecord calls cluster.connect using a
[quoted keyspace
string|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordReader.java#L138]
and that the java driver quotes it too when it accesses the keyspace using
[CQL|https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477].
If I remove the first quote my job runs.
was (Author: cscetbon):
[~alexliu68], so the issue is that CqlRecord calls cluster.connect using a
[quoted keyspace
string|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/hadoop/cql3/CqlRecordReader.java#L138]
and that the java driver quotes it too
https://github.com/datastax/java-driver/blob/2.1.8/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L477.
If I remove the first quote my job runs.
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Alex Liu
> Attachments: output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026002#comment-15026002
]
Cyril Scetbon edited comment on CASSANDRA-10751 at 11/25/15 2:10 AM:
-
[~alexliu68] Here are the patches for each version
was (Author: cscetbon):
[~alexliu68] Here are the patches foreach versions
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>Reporter: Cyril Scetbon
>Assignee: Alex Liu
> Attachments: CASSANDRA-10751-2.2.patch, CASSANDRA-10751-3.0.patch,
> output.log
>
>
> Trying to execute an Hadoop job on Yarn, I get errors from Cassandra's
> internal code. It seems that connections are shutdown but we can't understand
> why ...
> Here is a subtract of the errors. I also add a file with the complete debug
> logs.
> {code}
> 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying
> node006.internal.net/192.168.12.22:9042, trying next host (error is:
> com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown)
> Failed with exception java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception
> java.io.IOException:java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> java.io.IOException: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException:
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried: node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 15 more
> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All
> host(s) tried for query failed (tried:
> node006.internal.net/192.168.12.22:9042
> (com.datastax.driver.core.ConnectionException:
> [node006.internal.net/192.168.12.22:9042] Pool is shutdown))
> at
> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024784#comment-15024784
]
Cyril Scetbon edited comment on CASSANDRA-10751 at 11/24/15 4:34 PM:
-
[~alexliu68] The number of splits has not changed and is the same as with
2.0.12. The strange errors are :
{code}
15/11/24 15:26:32 [cluster2-blocking-task-worker-1]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 1
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 2
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 1
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 0
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG Host.STATES:
Defuncting Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] because:
[/10.234.62.20:9042] Error while setting keyspace
{code}
I can see that the error "Error while setting keyspace" is displayed when there
is a ExecutionException. It just weird that it happens so quickly ...
was (Author: cscetbon):
[~alexliu68] The number of splits has not changed and is the same as with
2.0.12. The strange errors are :
{code}
15/11/24 15:26:32 [cluster2-blocking-task-worker-1]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 1
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 2
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 1
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 0
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG Host.STATES:
Defuncting Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] because:
[/10.234.62.20:9042] Error while setting keyspace
{code}
I can see that the error "Error while setting keyspace" is displayed when there
is a BusyConnectionException. It just weird that it happens so quickly ...
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>
[jira] [Comment Edited] (CASSANDRA-10751) "Pool is shutdown" error when running Hadoop jobs on Yarn
[
https://issues.apache.org/jira/browse/CASSANDRA-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024784#comment-15024784
]
Cyril Scetbon edited comment on CASSANDRA-10751 at 11/24/15 4:34 PM:
-
[~alexliu68] The number of splits has not changed and is the same as with
2.0.12. The strange errors are :
{code}
15/11/24 15:26:32 [cluster2-blocking-task-worker-1]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 1
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 2
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 1
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 0
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG Host.STATES:
Defuncting Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] because:
[/10.234.62.20:9042] Error while setting keyspace
{code}
I can see that the error "Error while setting keyspace" is displayed when there
is a *ExecutionException*. It just weird that it happens so quickly ...
was (Author: cscetbon):
[~alexliu68] The number of splits has not changed and is the same as with
2.0.12. The strange errors are :
{code}
15/11/24 15:26:32 [cluster2-blocking-task-worker-1]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG
core.HostConnectionPool: Creating new connection on busy pool to
/10.234.62.20:9042
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=false] Connection opened
successfully
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 1
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] new connection created, total = 2
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] closing connection
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 1
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG Host.STATES:
[/10.234.62.20:9042] connection closed, remaining = 0
15/11/24 15:26:32 [cluster2-nio-worker-35]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-nio-worker-34]: DEBUG core.Connection:
Connection[/10.234.62.20:9042-5, inFlight=0, closed=true] has already terminated
15/11/24 15:26:32 [cluster2-blocking-task-worker-0]: DEBUG Host.STATES:
Defuncting Connection[/10.234.62.20:9042-6, inFlight=0, closed=true] because:
[/10.234.62.20:9042] Error while setting keyspace
{code}
I can see that the error "Error while setting keyspace" is displayed when there
is a ExecutionException. It just weird that it happens so quickly ...
> "Pool is shutdown" error when running Hadoop jobs on Yarn
> -
>
> Key: CASSANDRA-10751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10751
> Project: Cassandra
> Issue Type: Bug
> Environment: Hadoop 2.7.1 (HDP 2.3.2)
> Cassandra 2.1.11
>
