[jira] [Updated] (HIVE-14028) stats is not updated
[ https://issues.apache.org/jira/browse/HIVE-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14028: -- Fix Version/s: 2.1.1 > stats is not updated > > > Key: HIVE-14028 > URL: https://issues.apache.org/jira/browse/HIVE-14028 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14028.01.patch, HIVE-14028.02.patch > > > {code} > DROP TABLE users; > CREATE TABLE users(key string, state string, country string, country_id int) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "info:state,info:country,info:country_id" > ); > INSERT OVERWRITE TABLE users SELECT 'user1', 'IA', 'USA', 0 FROM src; > desc formatted users; > {code} > the result is > {code} > A masked pattern was here > Table Type: MANAGED_TABLE > Table Parameters: > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} > numFiles0 > numRows 0 > rawDataSize 0 > storage_handler > org.apache.hadoop.hive.hbase.HBaseStorageHandler > totalSize 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14028) stats is not updated
[ https://issues.apache.org/jira/browse/HIVE-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354691#comment-15354691 ] Lefty Leverenz commented on HIVE-14028: --- This was also committed to branch-2.1 on June 21 after 2.1.0 was released, so I'm adding fix version 2.1.1. (See commit cf127f7afe6a04e00a4ff188021cfbd6ae692a1e.) > stats is not updated > > > Key: HIVE-14028 > URL: https://issues.apache.org/jira/browse/HIVE-14028 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14028.01.patch, HIVE-14028.02.patch > > > {code} > DROP TABLE users; > CREATE TABLE users(key string, state string, country string, country_id int) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "info:state,info:country,info:country_id" > ); > INSERT OVERWRITE TABLE users SELECT 'user1', 'IA', 'USA', 0 FROM src; > desc formatted users; > {code} > the result is > {code} > A masked pattern was here > Table Type: MANAGED_TABLE > Table Parameters: > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} > numFiles0 > numRows 0 > rawDataSize 0 > storage_handler > org.apache.hadoop.hive.hbase.HBaseStorageHandler > totalSize 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354628#comment-15354628 ] Hive QA commented on HIVE-13945: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814566/HIVE-13945.03.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 10288 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_divide org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_9 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_expressions org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_9 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_9 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testByteDivideShort org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal2 org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimalSameParams org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivisionResultType org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testLongDivideDecimal org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testReturnTypeAnsiSql org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testReturnTypeBackwardCompat {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/307/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/307/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-307/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 34 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814566 - PreCommit-HIVE-MASTER-Build > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.029297'. Then select with that value displays its rounded > value, which is 4327269606205.029300 > {noformat} > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test; > No rows affected (0.229 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hi
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354507#comment-15354507 ] Hive QA commented on HIVE-13945: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814462/HIVE-13945.03.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10288 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/306/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/306/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-306/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814462 - PreCommit-HIVE-MASTER-Build > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.029297'. Then select with that value displays its rounded > value, which is 4327269606205.029300 > {noformat} > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test; > No rows affected (0.229 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc > decimal(38,18)); > No rows affected (0.125 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values > (4327269606205.029297); > No rows affected (2.372 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029297 | > +---+--+ > 1 row selected (0.123 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc = > 4327269606205.029297; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029300 | > +---+--+ > 1 row selected (0.109 seconds) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13945: Attachment: HIVE-13945.03.patch Changed the default decimal precision in division, not sure why it was so low by default. May break more tests than it fixes in terms of result changes... we'll see > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.029297'. Then select with that value displays its rounded > value, which is 4327269606205.029300 > {noformat} > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test; > No rows affected (0.229 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc > decimal(38,18)); > No rows affected (0.125 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values > (4327269606205.029297); > No rows affected (2.372 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029297 | > +---+--+ > 1 row selected (0.123 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc = > 4327269606205.029297; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029300 | > +---+--+ > 1 row selected (0.109 seconds) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13945: Attachment: HIVE-13945.03.patch Another round of updates and fixes. I added double literals and used them in some places where it makes sense.. had to make some changes to the parser to get around "code too large" errors. Also adjusted the behavior of some fns to support decimal properly, and fixed an unrelated bug in Lead/Lag fn exposed by this patch. Still need to look at precision for operations, and incorrect results for struct-in (probably related to decimal vs double in the struct), and ORC ppd test (didn't look yet). > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.03.patch, HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.029297'. Then select with that value displays its rounded > value, which is 4327269606205.029300 > {noformat} > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test; > No rows affected (0.229 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc > decimal(38,18)); > No rows affected (0.125 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values > (4327269606205.029297); > No rows affected (2.372 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029297 | > +---+--+ > 1 row selected (0.123 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc = > 4327269606205.029297; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029300 | > +---+--+ > 1 row selected (0.109 seconds) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[ https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353973#comment-15353973 ] Ferdinand Xu commented on HIVE-14037: - LGTM +1 > java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path > in mapreduce > --- > > Key: HIVE-14037 > URL: https://issues.apache.org/jira/browse/HIVE-14037 > Project: Hive > Issue Type: Bug >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch, > HIVE-14037.3.patch, HIVE-14037.4.patch > > > The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process > while they are not available in the Mapper or Reducer nodes which will throw > the following exception. > {noformat} > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469) > at > org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at > org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126) > ... 22 more > Caused by: java.lang.ClassNotFoundException: test.UDF > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11653) Beeline asks for password even when connecting with Kerberos
[ https://issues.apache.org/jira/browse/HIVE-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353937#comment-15353937 ] Vihang Karajgaonkar commented on HIVE-11653: HIVE-14062 already fixes this issue. > Beeline asks for password even when connecting with Kerberos > > > Key: HIVE-11653 > URL: https://issues.apache.org/jira/browse/HIVE-11653 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 0.14.0, 1.0.1, 1.2.1, 2.0.0 > Environment: Kerberos and Hive >Reporter: Loïc C. Chanel >Assignee: Oleksiy Sayankin > Attachments: HIVE-11653.1.patch > > > When connecting to HiveServer via Beeline, Beeline asks for a password even > if Kerberos is enabled and there is a ticket in cache (kinit have been > successfully executed, as klist shows the ticket is in cache). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Issue Type: Sub-task (was: Bug) Parent: HIVE-12991 > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353912#comment-15353912 ] Hive QA commented on HIVE-14035: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814052/HIVE-14035.04.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 10284 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_orig_table org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_dynamic_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_non_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_after_multiple_inserts org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_acid3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_part_col_char org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join_acid_non_acid org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.hcatalog.streaming.TestStreaming.testFileDump org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptDataFiles org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptSideFiles org.apache.hive.hcatalog.streaming.TestStreaming.testInterleavedTransactionBatchCommits org.apache.hive.hcatalog.streaming.TestStreaming.testMultipleTransactionBatchCommits org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json {noformat} Test results: https://builds.apache.org/job/Pre
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: HIVE-14111.01.patch Updated based on comments and added some TODOs > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.patch, > sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353821#comment-15353821 ] Sergey Shelukhin commented on HIVE-14119: - +1 pending tests > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353807#comment-15353807 ] Sergey Shelukhin commented on HIVE-14111: - Wrt close(), I think it should be ok. Right now the session object is often being reused after close; it should work with that. I added another TODO > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Status: Patch Available (was: Open) > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Attachment: HIVE-14119.1.patch deserializer was created with escapes disabled > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353780#comment-15353780 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO Lots of exceptions can come out of this one invocation. If not returned in the finally block - things blow up. {noformat} It's inside the try...block. What do you mean? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353787#comment-15353787 ] Sergey Shelukhin commented on HIVE-14111: - What is session.destroy? Rather what object is this; do you mean TezClient::stop? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353774#comment-15353774 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO What happens in this case - no new sessions launched ? {noformat} Yes. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353770#comment-15353770 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} if (session == null || conf == null) { // TODO Will the session being passed in here be null if this is being used from the set of default sessions. // Really don't see why - since SessionState can be re-used - in which case the session will be re-used. return false; } {noformat} Because it's supposed to be returned to the pool and unset. Same for some other comments > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13884: --- Attachment: HIVE-13884.9.patch Here's another patch that includes some tests on TestHiveMetaStore. There was an issue found by those tests when requesting a MAX number of partitions. It is fixed now. > Disallow queries in HMS fetching more than a configured number of partitions > > > Key: HIVE-13884 > URL: https://issues.apache.org/jira/browse/HIVE-13884 > Project: Hive > Issue Type: Improvement >Reporter: Mohit Sabharwal >Assignee: Sergio Peña > Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, > HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch, > HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch, HIVE-13884.9.patch > > > Currently the PartitionPruner requests either all partitions or partitions > based on filter expression. In either scenarios, if the number of partitions > accessed is large there can be significant memory pressure at the HMS server > end. > We already have a config {{hive.limit.query.max.table.partition}} that > enforces limits on number of partitions that may be scanned per operator. But > this check happens after the PartitionPruner has already fetched all > partitions. > We should add an option at PartitionPruner level to disallow queries that > attempt to access number of partitions beyond a configurable limit. > Note that {{hive.mapred.mode=strict}} disallow queries without a partition > filter in PartitionPruner, but this check accepts any query with a pruning > condition, even if partitions fetched are large. In multi-tenant > environments, admins could use more control w.r.t. number of partitions > allowed based on HMS memory capacity. > One option is to have PartitionPruner first fetch the partition names > (instead of partition specs) and throw an exception if number of partitions > exceeds the configured value. Otherwise, fetch the partition specs. > Looks like the existing {{listPartitionNames}} call could be used if extended > to take partition filter expressions like {{getPartitionsByExpr}} call does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353766#comment-15353766 ] Sergey Shelukhin commented on HIVE-14111: - Actually I think there are many tangentially related TODOs that can be addressed separately. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353762#comment-15353762 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO Define a helper class which can accept and validate potential values HIVE_EXECUTION_MODE("hive.execution.mode", "container", new StringSet("container", "llap"), "Chooses whether query fragments will run in container or in llap"), {noformat} This is completely unrelated. I'll just ignore other unrelated comments here ;) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14118) Make the alter partition exception more meaningful
[ https://issues.apache.org/jira/browse/HIVE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14118: Status: Patch Available (was: Open) Patch-1: include the underline exception when throwing the exception. > Make the alter partition exception more meaningful > -- > > Key: HIVE-14118 > URL: https://issues.apache.org/jira/browse/HIVE-14118 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14118.1.patch > > > Right now when the alter partitions fails, "alter is not possible" is shown > in the log while the real exception/failure is hidden. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14118) Make the alter partition exception more meaningful
[ https://issues.apache.org/jira/browse/HIVE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14118: Attachment: HIVE-14118.1.patch > Make the alter partition exception more meaningful > -- > > Key: HIVE-14118 > URL: https://issues.apache.org/jira/browse/HIVE-14118 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14118.1.patch > > > Right now when the alter partitions fails, "alter is not possible" is shown > in the log while the real exception/failure is hidden. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353738#comment-15353738 ] Hive QA commented on HIVE-14093: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814051/HIVE-14093.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10286 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/292/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/292/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-292/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814051 - PreCommit-HIVE-MASTER-Build > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch, HIVE-14093.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14027: --- Attachment: HIVE-14027.patch > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14027.patch > > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14027: --- Attachment: (was: HIVE-14027.patch) > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13945: Target Version/s: 1.3.0, 2.2.0 (was: 1.3.0, 2.2.0, 2.1.1, 2.0.2) > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.029297'. Then select with that value displays its rounded > value, which is 4327269606205.029300 > {noformat} > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test; > No rows affected (0.229 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc > decimal(38,18)); > No rows affected (0.125 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values > (4327269606205.029297); > No rows affected (2.372 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029297 | > +---+--+ > 1 row selected (0.123 seconds) > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> > 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc = > 4327269606205.029297; > +---+--+ > | test.dc | > +---+--+ > | 4327269606205.029300 | > +---+--+ > 1 row selected (0.109 seconds) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14109) query execuction throws NPE when hive.exec.submitviachild is set to true
[ https://issues.apache.org/jira/browse/HIVE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14109: Status: Patch Available (was: Open) Patch-1: queryState was null when we call ExecDriver() constructor. Since we only need QueryId in HadoopJobExecHelper, so pass in QueryId only in this patch. > query execuction throws NPE when hive.exec.submitviachild is set to true > > > Key: HIVE-14109 > URL: https://issues.apache.org/jira/browse/HIVE-14109 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14109.1.patch > > > If we set hive.exec.submitviachild to true and execute select count(*) from > src, the following exception is thrown. > Seems queryState is not initialized when ExecDriver is called from main() in > ExecDriver. > {noformat} > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14109) query execuction throws NPE when hive.exec.submitviachild is set to true
[ https://issues.apache.org/jira/browse/HIVE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14109: Attachment: HIVE-14109.1.patch > query execuction throws NPE when hive.exec.submitviachild is set to true > > > Key: HIVE-14109 > URL: https://issues.apache.org/jira/browse/HIVE-14109 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14109.1.patch > > > If we set hive.exec.submitviachild to true and execute select count(*) from > src, the following exception is thrown. > Seems queryState is not initialized when ExecDriver is called from main() in > ExecDriver. > {noformat} > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11986) describe extended command is missing tableType, while describe formatted outputs it
[ https://issues.apache.org/jira/browse/HIVE-11986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353655#comment-15353655 ] Andy Kruth edited comment on HIVE-11986 at 6/28/16 8:31 PM: I see the same behavior. I am trying to find the partitionKey information which comes after the truncation point. I am on: Hive 0.13.0-mapr-1510. You can recreate the issue with this simple table definition: ``` create table simple_table (f1 string) row format delimited fields terminated by '\t'; ``` and see that the describe output is truncated: ``` # col_name data_type comment f1 string # Detailed Table Information Database: default Owner: andy.kruth CreateTime: Tue Jun 28 16:19:04 EDT 2016 LastAccessTime: UNKNOWN Protect Mode: None Retention: 0 Location: maprfs:/user/hive/warehouse/simple_table Table Type: MANAGED_TABLE Table Parameters: transient_lastDdlTime 1467145144 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat:org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets:-1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \t serialization.format\t Time taken: 0.124 seconds, Fetched: 27 row(s) ``` Simply changing the field delimiter specified to something like a comma fixes the issue. Is there a way around this behavior if my table is tab delimited? Addendum: after changing the delimiter to a comma I could see the full describe output including partitionKeys and tableType with 'describe extended'. 'describe formatted' still didn't work as expected. was (Author: kruthar): I see the same behavior. I am trying to find the partitionKey information which comes after the truncation point. I am on: Hive 0.13.0-mapr-1510. You can recreate the issue with this simple table definition: ``` create table simple_table (f1 string) row format delimited fields terminated by '\t'; ``` and see that the describe output is truncated: ``` # col_name data_type comment f1 string # Detailed Table Information Database: default Owner: andy.kruth CreateTime: Tue Jun 28 16:19:04 EDT 2016 LastAccessTime: UNKNOWN Protect Mode: None Retention: 0 Location: maprfs:/user/hive/warehouse/simple_table Table Type: MANAGED_TABLE Table Parameters: transient_lastDdlTime 1467145144 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat:org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets:-1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \t serialization.format\t Time taken: 0.124 seconds, Fetched: 27 row(s) ``` Simply changing the field delimiter specified to something like a comma fixes the issue. Is there a way around this behavior if my table is tab delimited? > describe extended command is missing tableType, while describe formatted > outputs it > --- > > Key: HIVE-11986 > URL: https://issues.apache.org/jira/browse/HIVE-11986 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, Thrift API >Affects Versions: 0.13.0 >Reporter: Aviv >Priority: Minor > > The output of this command "describe extended" is missing the tableType: > property for some tables. it is being truncated after "serialization.format= > , field.delim=" > The output for: describe extended hivesampletable is: > "Detailed Table Information Table(tableName:hivesampletable, > dbName:default, owner:hdpinternaluser, createTime:1440859101, > lastAccessTime:0, retention:0, > sd:StorageDescriptor(cols:[FieldSchema(name:clientid, type:string, > comment:null), FieldSchema(name:querytime, type:string, comment:null), > FieldSchema(name:market, type:string, comment:null), > FieldSchema(name:deviceplatform, type:string, comment:null), > FieldSchema(name:devicemake, type:string, comment:null), > FieldSchema(name:devicemodel, type:string, comment:null), > FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, > type:string, comment:null), FieldSchema(name:querydwelltime, type:double, > comment:null), FieldSchema(name:sessionid, type:
[jira] [Commented] (HIVE-11986) describe extended command is missing tableType, while describe formatted outputs it
[ https://issues.apache.org/jira/browse/HIVE-11986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353655#comment-15353655 ] Andy Kruth commented on HIVE-11986: --- I see the same behavior. I am trying to find the partitionKey information which comes after the truncation point. I am on: Hive 0.13.0-mapr-1510. You can recreate the issue with this simple table definition: ``` create table simple_table (f1 string) row format delimited fields terminated by '\t'; ``` and see that the describe output is truncated: ``` # col_name data_type comment f1 string # Detailed Table Information Database: default Owner: andy.kruth CreateTime: Tue Jun 28 16:19:04 EDT 2016 LastAccessTime: UNKNOWN Protect Mode: None Retention: 0 Location: maprfs:/user/hive/warehouse/simple_table Table Type: MANAGED_TABLE Table Parameters: transient_lastDdlTime 1467145144 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat:org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets:-1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \t serialization.format\t Time taken: 0.124 seconds, Fetched: 27 row(s) ``` Simply changing the field delimiter specified to something like a comma fixes the issue. Is there a way around this behavior if my table is tab delimited? > describe extended command is missing tableType, while describe formatted > outputs it > --- > > Key: HIVE-11986 > URL: https://issues.apache.org/jira/browse/HIVE-11986 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, Thrift API >Affects Versions: 0.13.0 >Reporter: Aviv >Priority: Minor > > The output of this command "describe extended" is missing the tableType: > property for some tables. it is being truncated after "serialization.format= > , field.delim=" > The output for: describe extended hivesampletable is: > "Detailed Table Information Table(tableName:hivesampletable, > dbName:default, owner:hdpinternaluser, createTime:1440859101, > lastAccessTime:0, retention:0, > sd:StorageDescriptor(cols:[FieldSchema(name:clientid, type:string, > comment:null), FieldSchema(name:querytime, type:string, comment:null), > FieldSchema(name:market, type:string, comment:null), > FieldSchema(name:deviceplatform, type:string, comment:null), > FieldSchema(name:devicemake, type:string, comment:null), > FieldSchema(name:devicemodel, type:string, comment:null), > FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, > type:string, comment:null), FieldSchema(name:querydwelltime, type:double, > comment:null), FieldSchema(name:sessionid, type:bigint, comment:null), > FieldSchema(name:sessionpagevieworder, type:bigint, comment:null)], > location:wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable, > inputFormat:org.apache.hadoop.mapred.TextInputFormat, > outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, > compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, > serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, > parameters:{serialization.format= , field.delim=" > while the output for describe formatted hivesampletable is: > "# Detailed Table Information > Database: default > Owner:hdpinternaluser > CreateTime: Sat Aug 29 14:38:21 GMT 2015 > LastAccessTime: UNKNOWN > Protect Mode: None > Retention:0 > Location: > wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable > > Table Type: MANAGED_TABLE > Table Parameters: > COLUMN_STATS_ACCURATE true > numFiles1 > numRows 59793 > rawDataSize 4895922 > totalSize 5015508 > transient_lastDdlTime 1441320153 > > # Storage Information > SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > > InputFormat: org.apache.hadoop.mapred.TextInputFormat > OutputFormat: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > Compressed:
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353605#comment-15353605 ] Sergey Shelukhin commented on HIVE-14111: - No, all changes are part of this patch... TODO comment removal needs to be done in some patch, that's the only unrelated one > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14091: Attachment: HIVE-14091.01.patch ping [~jdere]. Can you take a look wrt the addition of taskFailed? When running the end to end test, the error is not propagated without it. Sid says it should come via heartbeat. Is heartbeat not coming test specific? > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.01.patch, HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353599#comment-15353599 ] Jesus Camacho Rodriguez edited comment on HIVE-14027 at 6/28/16 7:42 PM: - Problem was in the initialization of MapJoin operator. We use the object inspector from the key of the BigTable in the MapJoin for the key of the broadcast table, but this is not correct for outer joins (if you have not match for a left record, the value for the key column in the right side will be null). I have uploaded a fix and the test case to validate the fix. was (Author: jcamachorodriguez): Problem was in the initialization of operator. We use the object inspector from the key of the BigTable in the MapJoin for the key of the broadcast table, but this is not correct for outer joins (if you have not match for a left record, the value for the key column in the right side will be null). I have uploaded a fix and the test case to validate the fix. > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14027.patch > > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14027: --- Attachment: HIVE-14027.patch Problem was in the initialization of operator. We use the object inspector from the key of the BigTable in the MapJoin for the key of the broadcast table, but this is not correct for outer joins (if you have not match for a left record, the value for the key column in the right side will be null). I have uploaded a fix and the test case to validate the fix. > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14027.patch > > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353598#comment-15353598 ] Sergey Shelukhin commented on HIVE-14091: - It's handled via IOException. taskFailed is apparently needed. There isn't a call to the heartbeat with error that I can see; if I remove taskFailed, it never propagates properly. InputStream cleans up the socket (added a comment) > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14027: --- Status: Patch Available (was: In Progress) > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14027 started by Jesus Camacho Rodriguez. -- > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[ https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-14027: -- Assignee: Jesus Camacho Rodriguez (was: Vaibhav Gumashta) > NULL values produced by left outer join do not behave as NULL > - > > Key: HIVE-14027 > URL: https://issues.apache.org/jira/browse/HIVE-14027 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1, 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Jesus Camacho Rodriguez > > Consider the following setup: > {code} > create table tbl (n bigint, t string); > insert into tbl values (1, 'one'); > insert into tbl values(2, 'two'); > select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n = > 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n; > 1onefalsetrue > {code} > The query should return true for isnull(b.n). > I've tested by inserting a row with null value for the bigint column into > tbl, and isnull returns true in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13886) HiveSplitGenerator does not read dfs.blocksize correctly
[ https://issues.apache.org/jira/browse/HIVE-13886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353559#comment-15353559 ] shanyu zhao commented on HIVE-13886: The test failure seems irrelevant. > HiveSplitGenerator does not read dfs.blocksize correctly > > > Key: HIVE-13886 > URL: https://issues.apache.org/jira/browse/HIVE-13886 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 1.2.1 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HIVE-13886.patch > > > In src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java, it > uses conf.getLong() to read "dfs.blocksize". This results in Exception if one > configures it to something like "256m". It should use conf.getLongBytes() to > read it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14115) Custom FetchFormatter is not supported
[ https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353551#comment-15353551 ] Hive QA commented on HIVE-14115: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814026/HIVE-14115.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10285 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/291/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/291/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-291/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814026 - PreCommit-HIVE-MASTER-Build > Custom FetchFormatter is not supported > -- > > Key: HIVE-14115 > URL: https://issues.apache.org/jira/browse/HIVE-14115 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Attachments: HIVE-14115.01.patch > > > The following code is supported only FetchFormatter of ThriftFormatter and > DefaultFetchFormatter. It can not be used Custom FetchFormatter. > {code} > if (SessionState.get().isHiveServerQuery()) { > > conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName()); > } else { > conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER, > DefaultFetchFormatter.class.getName()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13884: --- Summary: Disallow queries in HMS fetching more than a configured number of partitions (was: Disallow queries fetching more than a configured number of partitions in PartitionPruner) > Disallow queries in HMS fetching more than a configured number of partitions > > > Key: HIVE-13884 > URL: https://issues.apache.org/jira/browse/HIVE-13884 > Project: Hive > Issue Type: Improvement >Reporter: Mohit Sabharwal >Assignee: Sergio Peña > Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, > HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch, > HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch > > > Currently the PartitionPruner requests either all partitions or partitions > based on filter expression. In either scenarios, if the number of partitions > accessed is large there can be significant memory pressure at the HMS server > end. > We already have a config {{hive.limit.query.max.table.partition}} that > enforces limits on number of partitions that may be scanned per operator. But > this check happens after the PartitionPruner has already fetched all > partitions. > We should add an option at PartitionPruner level to disallow queries that > attempt to access number of partitions beyond a configurable limit. > Note that {{hive.mapred.mode=strict}} disallow queries without a partition > filter in PartitionPruner, but this check accepts any query with a pruning > condition, even if partitions fetched are large. In multi-tenant > environments, admins could use more control w.r.t. number of partitions > allowed based on HMS memory capacity. > One option is to have PartitionPruner first fetch the partition names > (instead of partition specs) and throw an exception if number of partitions > exceeds the configured value. Otherwise, fetch the partition specs. > Looks like the existing {{listPartitionNames}} call could be used if extended > to take partition filter expressions like {{getPartitionsByExpr}} call does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353450#comment-15353450 ] Szehon Ho commented on HIVE-14117: -- nice idea, +1 > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Attachment: HIVE-14117.1.patch > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Status: Patch Available (was: Open) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[ https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353410#comment-15353410 ] Aihua Xu commented on HIVE-14013: - The tests are not related. Pushed to master. Thanks Yongzhi for reviewing. > Describe table doesn't show unicode properly > > > Key: HIVE-14013 > URL: https://issues.apache.org/jira/browse/HIVE-14013 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.2.0 > > Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch, > HIVE-14013.3.patch, HIVE-14013.4.patch > > > Describe table output will show comments incorrectly rather than the unicode > itself. > {noformat} > hive> desc formatted t1; > # Detailed Table Information > Table Type: MANAGED_TABLE > Table Parameters: > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} > comment \u8868\u4E2D\u6587\u6D4B\u8BD5 > numFiles0 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14013) Describe table doesn't show unicode properly
[ https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14013: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > Describe table doesn't show unicode properly > > > Key: HIVE-14013 > URL: https://issues.apache.org/jira/browse/HIVE-14013 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.2.0 > > Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch, > HIVE-14013.3.patch, HIVE-14013.4.patch > > > Describe table output will show comments incorrectly rather than the unicode > itself. > {noformat} > hive> desc formatted t1; > # Detailed Table Information > Table Type: MANAGED_TABLE > Table Parameters: > COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} > comment \u8868\u4E2D\u6587\u6D4B\u8BD5 > numFiles0 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[ https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14037: Attachment: HIVE-14037.4.patch Patch-4: address comments. > java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path > in mapreduce > --- > > Key: HIVE-14037 > URL: https://issues.apache.org/jira/browse/HIVE-14037 > Project: Hive > Issue Type: Bug >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch, > HIVE-14037.3.patch, HIVE-14037.4.patch > > > The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process > while they are not available in the Mapper or Reducer nodes which will throw > the following exception. > {noformat} > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469) > at > org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at > org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431) > at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126) > ... 22 more > Caused by: java.lang.ClassNotFoundException: test.UDF > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14113) Create function failed but function in show function list
[ https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353399#comment-15353399 ] Hive QA commented on HIVE-14113: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814010/HIVE-14113.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10258 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-vectorization_13.q-tez_bmj_schema_evolution.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/289/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/289/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-289/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814010 - PreCommit-HIVE-MASTER-Build > Create function failed but function in show function list > - > > Key: HIVE-14113 > URL: https://issues.apache.org/jira/browse/HIVE-14113 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.2.0 >Reporter: niklaus xiao >Assignee: Navis > Fix For: 1.3.0 > > Attachments: HIVE-14113.1.patch > > > 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists > {quote} > create function my_lower as 'com.tang.UDFLower' using jar > 'hdfs:///udf/udf-test.jar'; > {quote} > Failed with following exception: > {quote} > 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as > 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar'; > INFO : converting to local hdfs:///udf/udf-test.jar > ERROR : Failed to read external resource hdfs:///udf/udf-test.jar > java.lang.RuntimeException: Failed to read external resource > hdfs:///udf/udf-test.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250) > at > org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306) > at > org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466) > at > org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206) > at > org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551) > at > org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136) > at > org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167) > at > org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > a
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353371#comment-15353371 ] Gunther Hagleitner commented on HIVE-10100: --- [~sseth] we can ask, but the warning has been around for a long time. My guess eventually the idea is to get rid of it. So might as well change these places. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353289#comment-15353289 ] Mohit Sabharwal commented on HIVE-13884: LGTM. A unit test would be great in TestHiveMetaStore#testListPartitions - we can do that as a follow-up item. > Disallow queries fetching more than a configured number of partitions in > PartitionPruner > > > Key: HIVE-13884 > URL: https://issues.apache.org/jira/browse/HIVE-13884 > Project: Hive > Issue Type: Improvement >Reporter: Mohit Sabharwal >Assignee: Sergio Peña > Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, > HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch, > HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch > > > Currently the PartitionPruner requests either all partitions or partitions > based on filter expression. In either scenarios, if the number of partitions > accessed is large there can be significant memory pressure at the HMS server > end. > We already have a config {{hive.limit.query.max.table.partition}} that > enforces limits on number of partitions that may be scanned per operator. But > this check happens after the PartitionPruner has already fetched all > partitions. > We should add an option at PartitionPruner level to disallow queries that > attempt to access number of partitions beyond a configurable limit. > Note that {{hive.mapred.mode=strict}} disallow queries without a partition > filter in PartitionPruner, but this check accepts any query with a pruning > condition, even if partitions fetched are large. In multi-tenant > environments, admins could use more control w.r.t. number of partitions > allowed based on HMS memory capacity. > One option is to have PartitionPruner first fetch the partition names > (instead of partition specs) and throw an exception if number of partitions > exceeds the configured value. Otherwise, fetch the partition specs. > Looks like the existing {{listPartitionNames}} call could be used if extended > to take partition filter expressions like {{getPartitionsByExpr}} call does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353207#comment-15353207 ] Hive QA commented on HIVE-13945: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814004/HIVE-13945.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 10272 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-order_null.q-orc_merge10.q-insert_values_tmp_table.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_lazyserde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_gby2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_printf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_example_format org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compare_double_bigint org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes org.apache.hive.spark.client.TestSparkClient.testJobSubmission {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/288/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/288/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-288/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 40 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814004 - PreCommit-HIVE-MASTER-Build > Decimal value is displayed as rounded when selecting where clause with that > decimal value. > -- > > Key: HIVE-13945 > URL: https://issues.apache.org/jira/browse/HIVE-13945 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch, > HIVE-13945.patch > > > Create a table withe a column of decimal type(38,18) and insert > '4327269606205.0
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353057#comment-15353057 ] Sergio Peña commented on HIVE-13884: [~mohitsabharwal] [~szehon] The patch is ready, could you let me know if there are other comments or if I can commit this? > Disallow queries fetching more than a configured number of partitions in > PartitionPruner > > > Key: HIVE-13884 > URL: https://issues.apache.org/jira/browse/HIVE-13884 > Project: Hive > Issue Type: Improvement >Reporter: Mohit Sabharwal >Assignee: Sergio Peña > Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch, > HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch, > HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch > > > Currently the PartitionPruner requests either all partitions or partitions > based on filter expression. In either scenarios, if the number of partitions > accessed is large there can be significant memory pressure at the HMS server > end. > We already have a config {{hive.limit.query.max.table.partition}} that > enforces limits on number of partitions that may be scanned per operator. But > this check happens after the PartitionPruner has already fetched all > partitions. > We should add an option at PartitionPruner level to disallow queries that > attempt to access number of partitions beyond a configurable limit. > Note that {{hive.mapred.mode=strict}} disallow queries without a partition > filter in PartitionPruner, but this check accepts any query with a pruning > condition, even if partitions fetched are large. In multi-tenant > environments, admins could use more control w.r.t. number of partitions > allowed based on HMS memory capacity. > One option is to have PartitionPruner first fetch the partition names > (instead of partition specs) and throw an exception if number of partitions > exceeds the configured value. Otherwise, fetch the partition specs. > Looks like the existing {{listPartitionNames}} call could be used if extended > to take partition filter expressions like {{getPartitionsByExpr}} call does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-14114: - Assignee: Eugene Koifman (was: Abhishek) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352977#comment-15352977 ] Hive QA commented on HIVE-13901: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12814005/HIVE-13901.7.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10285 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityDelayedAllocation org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-287/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12814005 - PreCommit-HIVE-MASTER-Build > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14078) LLAP input split should get task attempt number from conf if available
[ https://issues.apache.org/jira/browse/HIVE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352822#comment-15352822 ] Hive QA commented on HIVE-14078: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12813985/HIVE-14078.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10285 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/286/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/286/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-286/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12813985 - PreCommit-HIVE-MASTER-Build > LLAP input split should get task attempt number from conf if available > -- > > Key: HIVE-14078 > URL: https://issues.apache.org/jira/browse/HIVE-14078 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch, HIVE-14078.2.patch > > > Currently the attempt number is hard-coded to 0. If the split is being > fetched as part of a hadoop job we can get the task attempt ID from the conf > if it has been set, and use the attempt number from that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352677#comment-15352677 ] Hive QA commented on HIVE-11402: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12813993/HIVE-11402.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10285 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-285/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12813993 - PreCommit-HIVE-MASTER-Build > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352673#comment-15352673 ] Peter Vary commented on HIVE-14100: --- The jira for the original function > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Priority: Minor > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateAuthenticator the current_user returns the username which > is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[ https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352653#comment-15352653 ] Alexandre Linte commented on HIVE-13830: Hi, I upgraded Hive to Hive 2.1.0. Now I have the following errors: - HQL commands {noformat} hive (shfs3453)> SELECT COUNT(year) FROM chicagocrimes GROUP BY year; SELECT COUNT(year) FROM chicagocrimes GROUP BY year FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. {noformat} - Client logs {noformat} Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.conf.HiveConfUsing the default value passed in for log id: c10f51a3-a72d-40c7-9ff6-26e5fb3732da Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.session.SessionStateUpdating thread name to c10f51a3-a72d-40c7-9ff6-26e5fb3732da main Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.DriverCompiling command(queryId=shfs3453_20160628110208_f0b51237-d391-472d-abe8-f2dd2457a9ed): SELECT COUNT(year) FROM chicagocrimes GROUP BY year Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerStarting Semantic Analysis Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted phase 1 of Semantic Analysis Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination tables Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1 Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted getting MetaData in Semantic Analysis Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination tables Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1 Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.common.FileUtilsCreating directory if it doesn't exist: hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1/-mr-10001/.hive-staging_hive_2016-06-28_11-02-08_399_7245611464735028300-1 Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.parse.CalcitePlannerCBO Succeeded; optimized logical plan. Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for FS(6) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(5) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(4) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for RS(3) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(2) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(1) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for TS(0) Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 oldColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]} Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 newColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]} Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryloading spark properties from:spark-defaults.conf Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property from spark-defaults.conf (spark.default.parallelism -> 10). Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property from spark-defaults.conf (spark.kryoserializer.buffer -> 100m). Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property from spark-defaults.conf (spark.executor.memory -> 4g). Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property from spark-defaults.conf (spark.driver.memory -> 2g). Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property from spark-defaults.conf (spark.kryo.classesToRegister -> org.ap
[jira] [Updated] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[ https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Linte updated HIVE-13830: --- Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos (was: Hadoop 2.7.2, Hive 2.0.0, Spark 1.6.1, Kerberos) > Hive on spark driver crash with Spark 1.6.1 > --- > > Key: HIVE-13830 > URL: https://issues.apache.org/jira/browse/HIVE-13830 > Project: Hive > Issue Type: Bug > Components: Spark, spark-branch >Affects Versions: 2.0.0, 2.1.0 > Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos >Reporter: Alexandre Linte > > With Hive 1.2.1 I was able to use Hive on Spark successfully with the use of > the spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar". > Today with Hive 2.0.0, I'm unable to use Hive on Spark whether it be with the > spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar" or the spark-assembly > "spark-assembly-1.6.1-hadoop2.7.2.jar". > My configuration is the following: > * spark-default.conf available in HIVE_DIR/conf > * spark assembly available in HIVE_DIR/lib > I gathered several logs below: > - HQL commands > {noformat} > $ hive -v --database shfs3453 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Spark/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > Logging initialized using configuration in > file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties > use shfs3453 > OK > Time taken: 1.425 seconds > Hive-on-MR is deprecated in Hive 2 and may not be available in the future > versions. Consider using a different execution engine (i.e. tez, spark) or > using Hive 1.X releases. > hive (shfs3453)> set hive.execution.engine=spark; > set hive.execution.engine=spark > hive (shfs3453)> set spark.master=yarn-client; > set spark.master=yarn-client > hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, > Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING, > LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, > District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT, > YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude > FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED > AS TEXTFILE; > CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block > STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription > STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT, > CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year > INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW > FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE > OK > Time taken: 0.408 seconds > hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM > chicagocrimes WHERE Description = 'FIRST DEGREE MURDER'; > INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE > Description = 'FIRST DEGREE MURDER' > Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73 > Total jobs = 1 > Launching Job 1 out of 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer= > In order to limit the maximum number of reducers: > set hive.exec.reducers.max= > In order to set a constant number of reducers: > set mapreduce.job.reduces= > Starting Spark Job = 79484279-8e75-4b13-8e71-7de463f4d51e > Status: SENT > Failed to execute spark task, with exception > 'java.lang.IllegalStateException(RPC channel is closed.)' > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.spark.SparkTask > {noformat} > - Client logs > {noformat} > May 24 09:32:19 hive-cli WARN - > org.apache.hive.spark.client.rpc.RpcDispatcherReceived error > message:io.netty.handler.codec.DecoderException: > java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.jav
[jira] [Updated] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[ https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Linte updated HIVE-13830: --- Affects Version/s: 2.1.0 > Hive on spark driver crash with Spark 1.6.1 > --- > > Key: HIVE-13830 > URL: https://issues.apache.org/jira/browse/HIVE-13830 > Project: Hive > Issue Type: Bug > Components: Spark, spark-branch >Affects Versions: 2.0.0, 2.1.0 > Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos >Reporter: Alexandre Linte > > With Hive 1.2.1 I was able to use Hive on Spark successfully with the use of > the spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar". > Today with Hive 2.0.0, I'm unable to use Hive on Spark whether it be with the > spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar" or the spark-assembly > "spark-assembly-1.6.1-hadoop2.7.2.jar". > My configuration is the following: > * spark-default.conf available in HIVE_DIR/conf > * spark assembly available in HIVE_DIR/lib > I gathered several logs below: > - HQL commands > {noformat} > $ hive -v --database shfs3453 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Spark/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > Logging initialized using configuration in > file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties > use shfs3453 > OK > Time taken: 1.425 seconds > Hive-on-MR is deprecated in Hive 2 and may not be available in the future > versions. Consider using a different execution engine (i.e. tez, spark) or > using Hive 1.X releases. > hive (shfs3453)> set hive.execution.engine=spark; > set hive.execution.engine=spark > hive (shfs3453)> set spark.master=yarn-client; > set spark.master=yarn-client > hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, > Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING, > LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, > District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT, > YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude > FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED > AS TEXTFILE; > CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block > STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription > STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT, > CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year > INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW > FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE > OK > Time taken: 0.408 seconds > hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM > chicagocrimes WHERE Description = 'FIRST DEGREE MURDER'; > INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE > Description = 'FIRST DEGREE MURDER' > Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73 > Total jobs = 1 > Launching Job 1 out of 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer= > In order to limit the maximum number of reducers: > set hive.exec.reducers.max= > In order to set a constant number of reducers: > set mapreduce.job.reduces= > Starting Spark Job = 79484279-8e75-4b13-8e71-7de463f4d51e > Status: SENT > Failed to execute spark task, with exception > 'java.lang.IllegalStateException(RPC channel is closed.)' > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.spark.SparkTask > {noformat} > - Client logs > {noformat} > May 24 09:32:19 hive-cli WARN - > org.apache.hive.spark.client.rpc.RpcDispatcherReceived error > message:io.netty.handler.codec.DecoderException: > java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageD
[jira] [Assigned] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek reassigned HIVE-14114: --- Assignee: Abhishek (was: Eugene Koifman) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Abhishek > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek updated HIVE-14114: Status: Open (was: Patch Available) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Abhishek > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352535#comment-15352535 ] Hive QA commented on HIVE-13369: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12813981/HIVE-13369.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10273 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/284/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/284/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-284/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12813981 - PreCommit-HIVE-MASTER-Build > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13369.1.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to aborted the txn. This can't > (reasonably) happen with auto commit mode, but with multi statement txns it's > possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352512#comment-15352512 ] Thejas M Nair commented on HIVE-10815: -- reattaching file to kick off tests before commit > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10815: - Assignee: Nemon Lou (was: Thejas M Nair) > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10815: - Attachment: HIVE-10815.2.patch > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Thejas M Nair > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-10815: Assignee: Thejas M Nair (was: Nemon Lou) > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Thejas M Nair > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352506#comment-15352506 ] Thejas M Nair commented on HIVE-14080: -- [~sushanth] Can you please review this patch ? The tests in TestMetaStoreSchemaInfo are actually ones I missed in earlier commit. > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Attachment: HIVE-14035.04.patch Pivot split-update feature to split update events into different insert & delete delta files, instead of writing to columns in the same file > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Issue Type: Improvement (was: Bug) > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Status: Patch Available (was: Open) > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: In Progress) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: In Progress (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Attachment: HIVE-14080.1.patch > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Attachment: HIVE-14093.2.patch Add config setting for max pending writes value. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch, HIVE-14093.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352488#comment-15352488 ] Carl Steinbach commented on HIVE-11402: --- HiveSessionImpl is starting to look a lot like SessionState. I suppose that was inevitable :( > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)