[jira] [Updated] (CASSANDRA-8471) mapred/hive queries fail when there is just 1 node down RF is > 1
[ https://issues.apache.org/jira/browse/CASSANDRA-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Kołaczkowski updated CASSANDRA-8471: -- Reviewer: Piotr Kołaczkowski > mapred/hive queries fail when there is just 1 node down RF is > 1 > - > > Key: CASSANDRA-8471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8471 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Artem Aliev > Labels: easyfix, hadoop, patch > Fix For: 2.0.12, 2.1.3 > > Attachments: cassandra-2.0-8471.txt > > > The hive and map reduce queries fail when just 1 node is down, even with RF=3 > (in a 6 node cluster) and default consistency levels for Read and Write. > The simpliest way to reproduce it is to use DataStax integrated hadoop > environment with hive. > {quote} > alter keyspace "HiveMetaStore" WITH replication = > {'class':'NetworkTopologyStrategy', 'DC1':3} ; > alter keyspace cfs WITH replication = {'class':'NetworkTopologyStrategy', > 'DC1':3} ; > alter keyspace cfs_archive WITH replication = > {'class':'NetworkTopologyStrategy', 'DC1':3} ; > CREATE KEYSPACE datamart WITH replication = { > 'class': 'NetworkTopologyStrategy', > 'DC1': '3' > }; > CREATE TABLE users1 ( > id int, > name text, > PRIMARY KEY ((id)) > ) > {quote} > Insert data. > Shutdown one cassandra node. > Run map reduce task. Hive in this case > {quote} > $ dse hive > hive> use datamart; > hive> select count(*) from users1; > {quote} > {quote} > ... > ... > 2014-12-10 18:33:53,090 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:54,093 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:55,096 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:56,099 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:57,102 Stage-1 map = 100%, reduce = 100%, Cumulative CPU > 6.39 sec > MapReduce Total cumulative CPU time: 6 seconds 390 msec > Ended Job = job_201412100017_0006 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://i-9d0306706.c.eng-gce-support.internal:50030/jobdetails.jsp?jobid=job_201412100017_0006 > Examining task ID: task_201412100017_0006_m_05 (and more) from job > job_201412100017_0006 > Task with the most failures(4): > - > Task ID: > task_201412100017_0006_m_01 > URL: > > http://i-9d0306706.c.eng-gce-support.internal:50030/taskdetails.jsp?jobid=job_201412100017_0006&tipid=task_201412100017_0006_m_01 > - > Diagnostic Messages for this Task: > java.io.IOException: java.io.IOException: > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 > (com.datastax.driver.core.TransportException: > [i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042] Cannot connect)) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:244) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:538) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:260) > Caused by: java.io.IOException: > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 > (com.datastax.driver.core.TransportException: > [i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042] Cannot connect)) > at > org.apache.hadoop.hive.cassandra.cql3.input.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:206) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:241) > ... 9 more > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All > host(s) tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 >
[jira] [Updated] (CASSANDRA-8471) mapred/hive queries fail when there is just 1 node down RF is > 1
[ https://issues.apache.org/jira/browse/CASSANDRA-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Aliev updated CASSANDRA-8471: --- Attachment: cassandra-2.0-8471.txt > mapred/hive queries fail when there is just 1 node down RF is > 1 > - > > Key: CASSANDRA-8471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8471 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Artem Aliev > Attachments: cassandra-2.0-8471.txt > > > The hive and map reduce queries fail when just 1 node is down, even with RF=3 > (in a 6 node cluster) and default consistency levels for Read and Write. > The simpliest way to reproduce it is to use DataStax integrated hadoop > environment with hive. > {quote} > alter keyspace "HiveMetaStore" WITH replication = > {'class':'NetworkTopologyStrategy', 'DC1':3} ; > alter keyspace cfs WITH replication = {'class':'NetworkTopologyStrategy', > 'DC1':3} ; > alter keyspace cfs_archive WITH replication = > {'class':'NetworkTopologyStrategy', 'DC1':3} ; > CREATE KEYSPACE datamart WITH replication = { > 'class': 'NetworkTopologyStrategy', > 'DC1': '3' > }; > CREATE TABLE users1 ( > id int, > name text, > PRIMARY KEY ((id)) > ) > {quote} > Insert data. > Shutdown one cassandra node. > Run map reduce task. Hive in this case > {quote} > $ dse hive > hive> use datamart; > hive> select count(*) from users1; > {quote} > {quote} > ... > ... > 2014-12-10 18:33:53,090 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:54,093 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:55,096 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:56,099 Stage-1 map = 75%, reduce = 25%, Cumulative CPU 6.39 > sec > 2014-12-10 18:33:57,102 Stage-1 map = 100%, reduce = 100%, Cumulative CPU > 6.39 sec > MapReduce Total cumulative CPU time: 6 seconds 390 msec > Ended Job = job_201412100017_0006 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://i-9d0306706.c.eng-gce-support.internal:50030/jobdetails.jsp?jobid=job_201412100017_0006 > Examining task ID: task_201412100017_0006_m_05 (and more) from job > job_201412100017_0006 > Task with the most failures(4): > - > Task ID: > task_201412100017_0006_m_01 > URL: > > http://i-9d0306706.c.eng-gce-support.internal:50030/taskdetails.jsp?jobid=job_201412100017_0006&tipid=task_201412100017_0006_m_01 > - > Diagnostic Messages for this Task: > java.io.IOException: java.io.IOException: > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 > (com.datastax.driver.core.TransportException: > [i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042] Cannot connect)) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:244) > at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:538) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:197) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:260) > Caused by: java.io.IOException: > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 > (com.datastax.driver.core.TransportException: > [i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042] Cannot connect)) > at > org.apache.hadoop.hive.cassandra.cql3.input.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:206) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:241) > ... 9 more > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All > host(s) tried for query failed (tried: > i-6ac985f7d.c.eng-gce-support.internal/10.240.124.16:9042 > (com.datastax.driver.core.TransportException: > [i-6ac985f7d.c.eng-gce-support.internal/10.