[jira] [Commented] (HIVE-7660) Hive to support qualify analytic filtering
[ https://issues.apache.org/jira/browse/HIVE-7660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400494#comment-15400494 ] Dudu Markovitz commented on HIVE-7660: -- This is a syntactic sugar supported by Teradata. e.g. In order to get the last 3 records of each buyer: select * from purchase qualify row_number () over (partition by buyer order by ts desc) <= 3; without the "qualify" it would look something like: select * from (select *,row_number () over (partition by buyer order by ts desc) as rn from purchase) as t where t.rn <= 3; > Hive to support qualify analytic filtering > -- > > Key: HIVE-7660 > URL: https://issues.apache.org/jira/browse/HIVE-7660 > Project: Hive > Issue Type: New Feature >Reporter: Viji >Priority: Trivial > > Currently, Hive does not support qualify analytic filtering. It would be > useful fi this feature were added in the future. > As a workaround, since it is just a filter, we can replace it with a subquery > and filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10142) Calculating formula based on difference between each row's value and current row's in Windowing function
[ https://issues.apache.org/jira/browse/HIVE-10142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400487#comment-15400487 ] Dudu Markovitz commented on HIVE-10142: --- Although I can relate to the request, I've never seen it implemented before, probably because it is an O(N^2) operation. Take this for example - For every event I would like to count the number of events with higher higher values that occurred before it. Assuming we have a new keyword "CURRENT_ROW", the analytic function would look something like this: count (case when val > CURRENT_ROW.val then 1 end) over (order by ts rows between unbounded preceding and current row) The thing is that in order to implement this we would probably sort the data set by ts (so far so good) and then compare each record against its preceding records which is a O(N^2) operation. That mean that for a table of 1M (1,000,000) record we are at the scale of 1T (1,000,000,000,000) operations. I'm not sure want to go there. > Calculating formula based on difference between each row's value and current > row's in Windowing function > > > Key: HIVE-10142 > URL: https://issues.apache.org/jira/browse/HIVE-10142 > Project: Hive > Issue Type: New Feature > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Yi Zhang >Assignee: Aihua Xu > > For analytics with windowing function, the calculation formula sometimes > needs to perform over each row's value against current tow's value. The decay > value is a good example, such as sums of value with a decay function based on > difference of timestamp between each row and current row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[ https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400463#comment-15400463 ] Hive QA commented on HIVE-14355: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820897/HIVE-14355.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10418 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/692/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/692/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-692/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12820897 - PreCommit-HIVE-MASTER-Build > Schema evolution for ORC in llap is broken for int to string conversion > --- > > Key: HIVE-14355 > URL: https://issues.apache.org/jira/browse/HIVE-14355 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch, > HIVE-14355.2.java-only.patch, HIVE-14355.2.patch, > HIVE-14355.3.java-only.patch, HIVE-14355.3.patch > > > When schema is evolved from any integer type to string then following > exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen > even for other conversions. > {code} > hive> create table orc_integer(b bigint) stored as orc; > hive> insert into orc_integer values(100); > hive> select count(*) from orc_integer where b=100; > OK > 1 > hive> alter table orc_integer change column b b string; > hive> select count(*) from orc_integer where b=100; > // FAIL with following exception > {code} > {code:title=When vectorization is enabled} > 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()] > vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0, > 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor: > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: > Hive Runtime Error while processing row > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) > at java.util.concurrent.FutureTask.run(Futur
[jira] [Commented] (HIVE-14363) bucketmap inner join query fails due to NullPointerException in some cases
[ https://issues.apache.org/jira/browse/HIVE-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400436#comment-15400436 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-14363: -- None of the errors look related. > bucketmap inner join query fails due to NullPointerException in some cases > -- > > Key: HIVE-14363 > URL: https://issues.apache.org/jira/browse/HIVE-14363 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jagruti Varia >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14363.1.patch > > > Bucketmap inner join query between bucketed tables throws following exception > when one table contains all the empty buckets while other has all the > non-empty buckets. > {noformat} > Vertex failed, vertexName=Map 2, vertexId=vertex_1466710232033_0432_4_01, > diagnostics=[Task failed, taskId=task_1466710232033_0432_4_01_00, > diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( > failure ) : > attempt_1466710232033_0432_4_01_00_0:java.lang.RuntimeException: > java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:330) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184) > ... 14 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getKeyValueReader(MapRecordProcessor.java:372) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.initializeMapRecordSources(MapRecordProcessor.java:344) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:292) > ... 15 more > ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : > attempt_1466710232033_0432_4_01_00_1:java.lang.RuntimeException: > java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.Runtime
[jira] [Commented] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[ https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400404#comment-15400404 ] Hive QA commented on HIVE-14366: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820850/HIVE-14366.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/691/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/691/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-691/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]] + export JAVA_HOME=/usr/java/jdk1.8.0_25 + JAVA_HOME=/usr/java/jdk1.8.0_25 + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-691/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive c8648aa..6b0131b master -> origin/master + git reset --hard HEAD HEAD is now at c8648aa HIVE-14348: Add tests for alter table exchange partition (Vaibhav Gumashta reviewed by Thejas Nair) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at 6b0131b Revert "HIVE-14303: CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice. (Zhihai Xu, reviewed by Xuefu Zhang)" + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12820850 - PreCommit-HIVE-MASTER-Build > Conversion of a Non-ACID table to an ACID table produces non-unique primary > keys > > > Key: HIVE-14366 > URL: https://issues.apache.org/jira/browse/HIVE-14366 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Saket Saurabh >Assignee: Saket Saurabh >Priority: Blocker > Attachments: HIVE-14366.01.patch, HIVE-14366.02.patch > > > When a Non-ACID table is converted to an ACID table, the primary key > consisting of (original transaction id, bucket_id, row_id) is not generated > uniquely. Currently, the row_id is always set to 0 for most rows. This leads > to correctness issue for such tables. > Quickest way to reproduce is to add the following unit test to > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java > {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid} > @Test > public void testOriginalReader() throws Exception { > FileSystem fs = FileSystem.get(hiveConf); > FileStatus[] status; > // 1. Insert five rows to Non-ACID table. > runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) > values(1,2),(3,4),(5,6),(7,8),(9,10)"); > // 2. Convert NONACIDORCTBL to ACID table. > runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET > TBLPROPERTIES ('transactional'='true')"); > // 3. Perform a major compaction. > runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact > 'MAJOR'"); > runWorker(hiveConf); > // 4. Perform a delete. > runStatementOnDriver("delete from " + Table.N
[jira] [Commented] (HIVE-14363) bucketmap inner join query fails due to NullPointerException in some cases
[ https://issues.apache.org/jira/browse/HIVE-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400403#comment-15400403 ] Hive QA commented on HIVE-14363: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820610/HIVE-14363.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 186 failed/errored test(s), 10386 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_join_partition_key org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_bucketmapjoin1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliD
[jira] [Reopened] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reopened HIVE-14303: - > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.secur
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400355#comment-15400355 ] Chao Sun commented on HIVE-14303: - Yes, it seems the test failures are related to this patch. [~zxu] can you investigate and resubmit? meanwhile I'll revert it. thanks. > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTas
[jira] [Commented] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400350#comment-15400350 ] Akash Sethi commented on HIVE-14382: Sir i go through the URL [http://hplsql.org/for-range] you give This URL also says the syntax is like :- FOR index IN [REVERSE] lower_bound..upper_bound [BY | STEP increment] LOOP statements END LOOP; first lower_bound then upper_bound > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14303: Comment: was deleted (was: [~ashutoshc] I chose one failed test (auto_smb_mapjoin_14.q) and run it with the git commit before this one, and it failed as well - the q.out file is different.) > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.Reduc
[jira] [Updated] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Sethi updated HIVE-14382: --- Status: Patch Available (was: Open) > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14382 stopped by Akash Sethi. -- > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Sethi updated HIVE-14382: --- Attachment: HIVE-14382.1-branch-2.1.patch Please Review my patch for branch branch-2.1 > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400346#comment-15400346 ] Chao Sun commented on HIVE-14303: - [~ashutoshc] I chose one failed test (auto_smb_mapjoin_14.q) and run it with the git commit before this one, and it failed as well - the q.out file is different. > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.a
[jira] [Commented] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400344#comment-15400344 ] Akash Sethi commented on HIVE-14382: i do know it sir but as new user start using the hive they expect sql like syntax. don't apply my patch now but probably on the next release this can be done. That's why i added 2 patch one for master and another for branch2.1 > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash Sethi updated HIVE-14382: --- Attachment: HIVE-14382.1.patch Please Review my Patch regarding reverse for loop functionality > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14382 started by Akash Sethi. -- > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14357: -- Affects Version/s: 2.1.1 Target Version/s: 1.3.0, 2.1.1 > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Rajat Khandelwal >Assignee: Sergey Shelukhin > Attachments: HIVE-14357.patch > > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400332#comment-15400332 ] Eugene Koifman commented on HIVE-14357: --- +1 pending tests > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Sergey Shelukhin > Attachments: HIVE-14357.patch > > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[ https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400330#comment-15400330 ] Hive QA commented on HIVE-13813: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820606/HIVE-13183.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 187 failed/errored test(s), 10382 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestPrivilegesV2 - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_join_partition_key org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_bucketmapjoin1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain org.
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400297#comment-15400297 ] Ashutosh Chauhan commented on HIVE-14303: - Seems like these reported failures were indeed related. See failures in subsequent run at https://issues.apache.org/jira/browse/HIVE-14365 Please verify and revert this patch. > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(Reduce
[jira] [Commented] (HIVE-14348) Add tests for alter table exchange partition
[ https://issues.apache.org/jira/browse/HIVE-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400294#comment-15400294 ] Vaibhav Gumashta commented on HIVE-14348: - Committed to 2.1 and master. Will attach a patch for branch-1 > Add tests for alter table exchange partition > > > Key: HIVE-14348 > URL: https://issues.apache.org/jira/browse/HIVE-14348 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14348.1.patch, HIVE-14348.2.patch, > HIVE-14348.3.patch, HIVE-14348.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14378) Data size may be estimated as 0 if no columns are being projected after an operator
[ https://issues.apache.org/jira/browse/HIVE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400290#comment-15400290 ] Ashutosh Chauhan commented on HIVE-14378: - [~jcamachorodriguez] Can you please take a look? > Data size may be estimated as 0 if no columns are being projected after an > operator > --- > > Key: HIVE-14378 > URL: https://issues.apache.org/jira/browse/HIVE-14378 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14378.patch > > > in those cases we still emit rows.. but they may not have any columns within > it. We shouldn't estimate 0 data size in such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14357: Status: Patch Available (was: Open) [~ekoifman] [~wzheng] can you take a look? > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Sergey Shelukhin > Attachments: HIVE-14357.patch > > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14357: Attachment: HIVE-14357.patch Removed the order dependency from test. When it fails, the error looks like this: {noformat} java.lang.AssertionError: Could't find {SHARED_READ, ACQUIRED, default, XYZY, null} in [ShowLocksResponseElement(lockid:2, dbname:default, tablename:xyz, state:ACQUIRED, type:SHARED_READ, txnid:0, lastheartbeat:1469837191676, acquiredat:1469837191676, user:XYZ, hostname:reznor-mbp-2.local, agentInfo:sergey_20160729170623_e1153b39-54d3-4cab-9766-c1349c8efa4a, lockIdInternal:1)]{noformat} > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Sergey Shelukhin > Attachments: HIVE-14357.patch > > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Status: Patch Available (was: Open) > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch, HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Attachment: HIVE-14367.1.patch > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch, HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14367: Status: Open (was: Patch Available) > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14381) Handle null value in WindowingTableFunction.WindowingIterator.next()
[ https://issues.apache.org/jira/browse/HIVE-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14381: - Attachment: HIVE-14381.2.patch Thanks [~jcamachorodriguez] for the review. Patch 2 takes {{output.set(j, out);}} out from the if block > Handle null value in WindowingTableFunction.WindowingIterator.next() > > > Key: HIVE-14381 > URL: https://issues.apache.org/jira/browse/HIVE-14381 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-14381.1.patch, HIVE-14381.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13257) GroupBy with column alias does not support AVG
[ https://issues.apache.org/jira/browse/HIVE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400173#comment-15400173 ] stephen sprague edited comment on HIVE-13257 at 7/29/16 11:25 PM: -- use "order by 1" as your test case.you'll get the same error. I should mention this is hive v2.1.0 {code} [dwrdevnn1:10001] hive> select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc; Fri Jul 29 15:47:06 2016 - Thrift::API::HiveClient2::execute: Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 1:7 Not yet supported place for UDAF 'count'; HQL was: "select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc" {code} and yes "hive.groupby.orderby.position.alias=true" thoughts? was (Author: spragues): use "order by 1" as your test case.you'll get the same error. {code} [dwrdevnn1:10001] hive> select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc; Fri Jul 29 15:47:06 2016 - Thrift::API::HiveClient2::execute: Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 1:7 Not yet supported place for UDAF 'count'; HQL was: "select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc" {code} and yes "hive.groupby.orderby.position.alias=true" thoughts? > GroupBy with column alias does not support AVG > -- > > Key: HIVE-13257 > URL: https://issues.apache.org/jira/browse/HIVE-13257 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran > > For the following query, with hive.groupby.orderby.position.alias set to true > {code:title=Query} > SELECT Avg(`t0`.`x_measure__0`) AS `avg_calculation_270497503505567749_ok` > FROM (SELECT `store_sales`.`ss_ticket_number` AS `ss_ticket_number`, >Sum(`store_sales`.`ss_net_paid`) AS `x_measure__0` > FROM `store_sales` `store_sales` >JOIN `item` `item` > ON ( `store_sales`.`ss_item_sk` = `item`.`i_item_sk` ) > GROUP BY `store_sales`.`ss_ticket_number`) `t0` > GROUP BY 1 > HAVING ( Count(1) > 0 ); > {code} > it throws the following exception > {code:title=Exception} > FAILED: SemanticException [Error 10128]: Line 2:7 Not yet supported place for > UDAF 'Avg’ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14348) Add tests for alter table exchange partition
[ https://issues.apache.org/jira/browse/HIVE-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400192#comment-15400192 ] Vaibhav Gumashta commented on HIVE-14348: - Test failures unrelated. I'll commit shortly and create a patch for branch-1. > Add tests for alter table exchange partition > > > Key: HIVE-14348 > URL: https://issues.apache.org/jira/browse/HIVE-14348 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-14348.1.patch, HIVE-14348.2.patch, > HIVE-14348.3.patch, HIVE-14348.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13257) GroupBy with column alias does not support AVG
[ https://issues.apache.org/jira/browse/HIVE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400173#comment-15400173 ] stephen sprague commented on HIVE-13257: use "order by 1" as your test case.you'll get the same error. {code} [dwrdevnn1:10001] hive> select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc; Fri Jul 29 15:47:06 2016 - Thrift::API::HiveClient2::execute: Error while compiling statement: FAILED: SemanticException [Error 10128]: Line 1:7 Not yet supported place for UDAF 'count'; HQL was: "select count(*), date_key FROM Raw_logs.fact_tracking_datekey_by_hour group by date_key order by 1 desc" {code} and yes "hive.groupby.orderby.position.alias=true" thoughts? > GroupBy with column alias does not support AVG > -- > > Key: HIVE-13257 > URL: https://issues.apache.org/jira/browse/HIVE-13257 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran > > For the following query, with hive.groupby.orderby.position.alias set to true > {code:title=Query} > SELECT Avg(`t0`.`x_measure__0`) AS `avg_calculation_270497503505567749_ok` > FROM (SELECT `store_sales`.`ss_ticket_number` AS `ss_ticket_number`, >Sum(`store_sales`.`ss_net_paid`) AS `x_measure__0` > FROM `store_sales` `store_sales` >JOIN `item` `item` > ON ( `store_sales`.`ss_item_sk` = `item`.`i_item_sk` ) > GROUP BY `store_sales`.`ss_ticket_number`) `t0` > GROUP BY 1 > HAVING ( Count(1) > 0 ); > {code} > it throws the following exception > {code:title=Exception} > FAILED: SemanticException [Error 10128]: Line 2:7 Not yet supported place for > UDAF 'Avg’ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14365) Simplify logic for check introduced in HIVE-10022
[ https://issues.apache.org/jira/browse/HIVE-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400154#comment-15400154 ] Hive QA commented on HIVE-14365: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820589/HIVE-14365.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 183 failed/errored test(s), 10377 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_join_partition_key org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_bucketmapjoin1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCli
[jira] [Resolved] (HIVE-8513) Vectorize multiple children operators
[ https://issues.apache.org/jira/browse/HIVE-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline resolved HIVE-8513. Resolution: Won't Fix > Vectorize multiple children operators > -- > > Key: HIVE-8513 > URL: https://issues.apache.org/jira/browse/HIVE-8513 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Matt McCline >Assignee: Matt McCline > > For HIVE-8498 we turned off vectorization if any operator is encountered with > multiple children. > Thus, this query will not be vectorized: > from orc1 a > insert overwrite table orc_rn1 select a.* where a.rn < 100 > insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000 > insert overwrite table orc_rn3 select a.* where a.rn >= 1000; > But Prashanth [~prasanthj] also points that correlation operators Mux/Demux > will also not be vectorized, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-14357: --- Assignee: Sergey Shelukhin (was: Wei Zheng) > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Sergey Shelukhin > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-14357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-14357: Assignee: Wei Zheng > TestDbTxnManager2#testLocksInSubquery failing in branch-2.1 > --- > > Key: HIVE-14357 > URL: https://issues.apache.org/jira/browse/HIVE-14357 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Wei Zheng > > {noformat} > checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S > where a in (select a from T where b = 1)")); > txnMgr.openTxn("three"); > txnMgr.acquireLocks(driver.getPlan(), ctx, "three"); > locks = getLocks(); > Assert.assertEquals("Unexpected lock count", 3, locks.size()); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, > locks.get(0)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, > locks.get(1)); > checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, > locks.get(2)); > {noformat} > This test case is failing. The expected order of locks is supposed to be T, > S, R. But upon closer inspection, it seems to be R,S,T. > I'm not much familiar with what these locks are and why the order is > important. Raising this jira so while I try to understand it all. Meanwhile, > if somebody can explain here, would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-12826) Vectorization: fix VectorUDAF* suspect isNull checks
[ https://issues.apache.org/jira/browse/HIVE-12826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400095#comment-15400095 ] Matt McCline edited comment on HIVE-12826 at 7/29/16 10:11 PM: --- This fix is present in branch-2.1 before tags/release-2.1.0-rc3 was (Author: mmccline): This fix is present in branch-2.1 > Vectorization: fix VectorUDAF* suspect isNull checks > > > Key: HIVE-12826 > URL: https://issues.apache.org/jira/browse/HIVE-12826 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 1.3.0, 2.0.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.0.0 > > Attachments: HIVE-12826.1.patch > > > for isRepeating=true, checking isNull[selected[i]] might return incorrect > results (without a heavy array fill of isNull). > VectorUDAFSum/Min/Max/Avg and SumDecimal impls need to be reviewed for this > pattern. > {code} > private void iterateHasNullsRepeatingSelectionWithAggregationSelection( > VectorAggregationBufferRow[] aggregationBufferSets, > int aggregateIndex, >value, > int batchSize, > int[] selection, > boolean[] isNull) { > > for (int i=0; i < batchSize; ++i) { > if (!isNull[selection[i]]) { > Aggregation myagg = getCurrentAggregationBuffer( > aggregationBufferSets, > aggregateIndex, > i); > myagg.sumValue(value); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14374) BeeLine argument, and configuration handling cleanup
[ https://issues.apache.org/jira/browse/HIVE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400101#comment-15400101 ] Peter Vary commented on HIVE-14374: --- [~stakiar] Currently only a handful of parameters are documented as a command line option, but almost twice as many could be set from the configuration file. I don't know how much of it is by design, and how much of this is by error. I hope someone with more experience in Hive could help here. +1 for the commons-cli solution and then we could forget every annotation magic. I wasn't brave enough to suggest it, although it is already used in beeline compatibility mode. > BeeLine argument, and configuration handling cleanup > > > Key: HIVE-14374 > URL: https://issues.apache.org/jira/browse/HIVE-14374 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > BeeLine uses reflection, to set the BeeLineOpts attributes when parsing > command line arguments, and when loading the configuration file. > This means, that creating a setXXX, getXXX method in BeeLineOpts is a > potential risk of exposing an attribute for the user unintentionally. There > is a possibility to exclude an attribute from saving the value in the > configuration file with the Ignore annotation. This does not restrict the > loading or command line setting of these parameters which means there are > many undocumented "features" as-is, like setting the lastConnectedUrl, > allowMultilineCommand, maxHeight, trimScripts, etc. from command line. > This part of the code needs a little cleanup. > I think we should make this exposure more explicit, and be able to > differentiate the configurable options depending on the source (command line, > and configuration file), so I propose to create a mechanism to tell > explicitly which BeeLineOpts attributes are settable by command line, and > configuration file, and every other attribute should be inaccessible by the > user of the beeline cli. > One possible solution could be two annotations like these: > - CommandLineOption - there could be a mandatory text parameter here, so the > developer had to provide the help text for it which could be displayed to the > user > - ConfigurationFileOption - no text is required here > Something like this: > - This attribute could be provided by command line, and from a configuration > file too: > {noformat} > @CommandLineOption("automatically save preferences") > @ConfigurationFileOption > public void setAutosave(boolean autosave) { > this.autosave = autosave; > } > public void getAutosave() { > return this.autosave; > } > {noformat} > - This attribute could be set through the configuration only > {noformat} > @ConfigurationFileOption > public void setLastConnectedUrl(String lastConnectedUrl) { > this.lastConnectedUrl = lastConnectedUrl; > } > > public String getLastConnectedUrl() > { > return lastConnectedUrl; > } > {noformat} > - Attribute could be set through command line only - I think this is not too > relevant, but possible > {noformat} > @CommandLineOption("specific command line option") > public void setSpecificCommandLineOption(String specificCommandLineOption) { > this.specificCommandLineOption = specificCommandLineOption; > } > > public String getSpecificCommandLineOption() { > return specificCommandLineOption; > } > {noformat} > - Attribute could not be set > {noformat} > public static Env getEnv() { > return env; > } > public static void setEnv(Env envToUse) { > env = envToUse; > } > {noformat} > Accouring to our previous conversations, I think you might be interested in: > [~spena], [~vihangk1], [~aihuaxu], [~ngangam], [~ychena], [~xuefuz] > but anyone is welcome to discuss this. > What do you think about the proposed solution? > Any better ideas, or extensions? > Thanks, > Peter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12826) Vectorization: fix VectorUDAF* suspect isNull checks
[ https://issues.apache.org/jira/browse/HIVE-12826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400095#comment-15400095 ] Matt McCline commented on HIVE-12826: - This fix is present in branch-2.1 > Vectorization: fix VectorUDAF* suspect isNull checks > > > Key: HIVE-12826 > URL: https://issues.apache.org/jira/browse/HIVE-12826 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 1.3.0, 2.0.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.0.0 > > Attachments: HIVE-12826.1.patch > > > for isRepeating=true, checking isNull[selected[i]] might return incorrect > results (without a heavy array fill of isNull). > VectorUDAFSum/Min/Max/Avg and SumDecimal impls need to be reviewed for this > pattern. > {code} > private void iterateHasNullsRepeatingSelectionWithAggregationSelection( > VectorAggregationBufferRow[] aggregationBufferSets, > int aggregateIndex, >value, > int batchSize, > int[] selection, > boolean[] isNull) { > > for (int i=0; i < batchSize; ++i) { > if (!isNull[selection[i]]) { > Aggregation myagg = getCurrentAggregationBuffer( > aggregationBufferSets, > aggregateIndex, > i); > myagg.sumValue(value); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400036#comment-15400036 ] Peter Vary commented on HIVE-14146: --- [~aihuaxu] Checked, and found the following: - Beeline - when using "describe pretty" the \n is padded and displayed - this is as intended - the original behavior is not changed - Cli - when using "describe pretty" the \n is padded and displayed - this is as intended - the original behavior is not changed What I have missed, that in Cli when using "describe formatted" the following code sets the output to padded: {noformat} package org.apache.hadoop.hive.ql.exec; [..] public class DDLTask extends Task implements Serializable { [..] private int describeTable(Hive db, DescTableDesc descTbl) throws HiveException { [..] // In case the query is served by HiveServer2, don't pad it with spaces, // as HiveServer2 output is consumed by JDBC/ODBC clients. boolean isOutputPadded = !SessionState.get().isHiveServerQuery(); {noformat} So the column comments are printed as a formatted strings with new lines - but not the table comment. (same as in pretty mode) What do you think we should do here? I personally would prefer to remove this code below (pretty comment formatting), so every comment should be handled in the same way (printing \n instead of newline): {noformat} package org.apache.hadoop.hive.ql.metadata.formatting; [..] public final class MetaDataFormatUtils { [..] private static void formatWithIndentation(String colName, String colType, String colComment, [..] // comment indent processing for multi-line comments // comments should be indented the same amount on each line // if the first line comment starts indented by k, // the following line comments should also be indented by k String[] commentSegments = colComment.split("\n|\r|\r\n"); tableInfo.append(String.format("%-" + ALIGNMENT + "s", commentSegments[0])).append(LINE_DELIM); int colNameLength = ALIGNMENT > colName.length() ? ALIGNMENT : colName.length(); int colTypeLength = ALIGNMENT > colType.length() ? ALIGNMENT : colType.length(); for (int i = 1; i < commentSegments.length; i++) { tableInfo.append(String.format("%" + colNameLength + "s" + FIELD_DELIM + "%" + colTypeLength + "s" + FIELD_DELIM + "%s", "", "", commentSegments[i])).append(LINE_DELIM); } {noformat} > Column comments with "\n" character "corrupts" table metadata > - > > Key: HIVE-14146 > URL: https://issues.apache.org/jira/browse/HIVE-14146 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch, > HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch, HIVE-14146.patch > > > Create a table with the following(noting the \n in the COMMENT): > {noformat} > CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an > individual’); > {noformat} > Describe shows that now the metadata is messed up: > {noformat} > beeline> describe commtest; > +---++---+--+ > | col_name | data_type |comment| > +---++---+--+ > | first_nm | string | Indicates First name | > | of an individual | NULL | NULL | > +---++---+--+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14376) Schema evolution tests takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14376: - Assignee: (was: Prasanth Jayachandran) > Schema evolution tests takes a long time > > > Key: HIVE-14376 > URL: https://issues.apache.org/jira/browse/HIVE-14376 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0, 2.1.1 >Reporter: Prasanth Jayachandran >Priority: Minor > > schema_evol_* tests (14 tests) takes 47 mins in tez, 40mins in TestCliDriver. > Same set of tests are added to llap as well in HIVE-14355 which will some > more time. Most tests uses INSERT into table which can be slow and is not > needed. Even some individual tests takes about 5 mins to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14123) Add beeline configuration option to show database in the prompt
[ https://issues.apache.org/jira/browse/HIVE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1546#comment-1546 ] Peter Vary commented on HIVE-14123: --- [~leftylev] Is this the standard method of taking note of the required documentation changes? - Label - Url of the doc which should be changed > Add beeline configuration option to show database in the prompt > --- > > Key: HIVE-14123 > URL: https://issues.apache.org/jira/browse/HIVE-14123 > Project: Hive > Issue Type: Improvement > Components: Beeline, CLI >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14123.10.patch, HIVE-14123.2.patch, > HIVE-14123.3.patch, HIVE-14123.4.patch, HIVE-14123.5.patch, > HIVE-14123.6.patch, HIVE-14123.7.patch, HIVE-14123.8.patch, > HIVE-14123.9.patch, HIVE-14123.patch > > > There are several jira issues complaining that, the Beeline does not respect > hive.cli.print.current.db. > This is partially true, since in embedded mode, it uses the > hive.cli.print.current.db to change the prompt, since HIVE-10511. > In beeline mode, I think this function should use a beeline command line > option instead, like for the showHeader option emphasizing, that this is a > client side option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14386) UGI clone shim also needs to clone credentials
[ https://issues.apache.org/jira/browse/HIVE-14386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14386: Attachment: HIVE-14386.patch [~sseth] can you take a look? > UGI clone shim also needs to clone credentials > -- > > Key: HIVE-14386 > URL: https://issues.apache.org/jira/browse/HIVE-14386 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14386.patch > > > Discovered while testing HADOOP-13081 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14386) UGI clone shim also needs to clone credentials
[ https://issues.apache.org/jira/browse/HIVE-14386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14386: Status: Patch Available (was: Open) > UGI clone shim also needs to clone credentials > -- > > Key: HIVE-14386 > URL: https://issues.apache.org/jira/browse/HIVE-14386 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14386.patch > > > Discovered while testing HADOOP-13081 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399979#comment-15399979 ] Jimmy Xiang commented on HIVE-14340: Multiple configurations/interfaces should be fine. > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14377) LLAP IO: issue with how estimate cache removes unneeded buffers
[ https://issues.apache.org/jira/browse/HIVE-14377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399964#comment-15399964 ] Prasanth Jayachandran commented on HIVE-14377: -- +1 > LLAP IO: issue with how estimate cache removes unneeded buffers > --- > > Key: HIVE-14377 > URL: https://issues.apache.org/jira/browse/HIVE-14377 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-14377.01.patch, HIVE-14377.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399950#comment-15399950 ] Hive QA commented on HIVE-13560: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820568/HIVE-13560.10.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10379 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hive.service.cli.session.TestSessionManagerMetrics.org.apache.hive.service.cli.session.TestSessionManagerMetrics {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/687/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/687/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-687/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12820568 - PreCommit-HIVE-MASTER-Build > Adding Omid as connection manager for HBase Metastore > - > > Key: HIVE-13560 > URL: https://issues.apache.org/jira/browse/HIVE-13560 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13560.1.patch, HIVE-13560.10.patch, > HIVE-13560.11.patch, HIVE-13560.2.patch, HIVE-13560.3.patch, > HIVE-13560.4.patch, HIVE-13560.5.patch, HIVE-13560.6.patch, > HIVE-13560.7.patch, HIVE-13560.8.patch, HIVE-13560.9.patch > > > Adding Omid as a transaction manager to HBase Metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14303: Resolution: Fixed Fix Version/s: (was: 2.1.0) 2.2.0 Status: Resolved (was: Patch Available) > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399943#comment-15399943 ] Chao Sun commented on HIVE-14303: - Committed to master. Thanks [~zxu] for the patch and [~xuefuz] for the review! > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.2.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.
[jira] [Updated] (HIVE-14322) Postgres db issues after Datanucleus 4.x upgrade
[ https://issues.apache.org/jira/browse/HIVE-14322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14322: Attachment: HIVE-14322.04.patch And again. Looks like it never got picked up. > Postgres db issues after Datanucleus 4.x upgrade > > > Key: HIVE-14322 > URL: https://issues.apache.org/jira/browse/HIVE-14322 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0, 2.0.1 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-14322.02.patch, HIVE-14322.03.patch, > HIVE-14322.04.patch, HIVE-14322.1.patch > > > With the upgrade to datanucleus 4.x versions in HIVE-6113, hive does not > work properly with postgres. > The nullable fields in the database have string "NULL::character varying" > instead of real NULL values. This causes various issues. > One example is - > {code} > hive> create table t(i int); > OK > Time taken: 1.9 seconds > hive> create view v as select * from t; > OK > Time taken: 0.542 seconds > hive> select * from v; > FAILED: SemanticException Unable to fetch table v. > java.net.URISyntaxException: Relative path in absolute URI: > NULL::character%20varying > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[ https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399874#comment-15399874 ] Sergey Shelukhin commented on HIVE-14355: - +1 > Schema evolution for ORC in llap is broken for int to string conversion > --- > > Key: HIVE-14355 > URL: https://issues.apache.org/jira/browse/HIVE-14355 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch, > HIVE-14355.2.java-only.patch, HIVE-14355.2.patch, > HIVE-14355.3.java-only.patch, HIVE-14355.3.patch > > > When schema is evolved from any integer type to string then following > exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen > even for other conversions. > {code} > hive> create table orc_integer(b bigint) stored as orc; > hive> insert into orc_integer values(100); > hive> select count(*) from orc_integer where b=100; > OK > 1 > hive> alter table orc_integer change column b b string; > hive> select count(*) from orc_integer where b=100; > // FAIL with following exception > {code} > {code:title=When vectorization is enabled} > 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()] > vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0, > 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor: > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: > Hive Runtime Error while processing row > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row > at > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86) > ... 18 more > Caused by: java.lang.ClassCastException: > org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector > at > org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42) > at > org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) > at > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774) > ... 19 more > {code} > {code:title=When vectorization is disabled} > 2016-07-27T01:52:43,328 INFO [TezTaskRunner > (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1 > 2016-07-27T01:52:43,328 INFO [TezTaskRunner > (1469608604787_0002_26_00_00_0)] exec.OperatorUt
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399859#comment-15399859 ] zhihai xu commented on HIVE-14303: -- Thanks for the review [~csun]! All these test failures are not related to my patch! > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.1.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399857#comment-15399857 ] Chao Sun commented on HIVE-14340: - Another reason is that user may need to use the same hook object for compilation and execution, to maintain some internal state. Neither semantic analysis hook nor exec hook satisfy this requirement. Actually my patch also failed on this since it triggers in different functions.. I'll upload a new patch. > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14380) Queries on tables with remote HDFS paths fail in "encryption" checks.
[ https://issues.apache.org/jira/browse/HIVE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399853#comment-15399853 ] Mithun Radhakrishnan commented on HIVE-14380: - Thanks for reviewing, [~spena]. :] Also, yikes. I'm not sure how 2 JIRAs got raised. :/ > Queries on tables with remote HDFS paths fail in "encryption" checks. > - > > Key: HIVE-14380 > URL: https://issues.apache.org/jira/browse/HIVE-14380 > Project: Hive > Issue Type: Bug > Components: Encryption >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-14380.1.patch > > > If a table has table/partition locations set to remote HDFS paths, querying > them will cause the following IAException: > {noformat} > 2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner > (SemanticAnalyzer.java:getMetaData(1867)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if > hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: > java.lang.IllegalArgumentException: Wrong FS: > hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: > hdfs://bar.ygrid.yahoo.com:8020 > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274) > ... > {noformat} > This is because of the following code in {{SessionState}}: > {code:title=SessionState.java|borderStyle=solid} > public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws > HiveException { > if (hdfsEncryptionShim == null) { > try { > FileSystem fs = FileSystem.get(sessionConf); > if ("hdfs".equals(fs.getUri().getScheme())) { > hdfsEncryptionShim = > ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf); > } else { > LOG.debug("Could not get hdfsEncryptionShim, it is only applicable > to hdfs filesystem."); > } > } catch (Exception e) { > throw new HiveException(e); > } > } > return hdfsEncryptionShim; > } > {code} > When the {{FileSystem}} instance is created, using the {{sessionConf}} > implies that the current HDFS is going to be used. This call should instead > fetch the {{FileSystem}} instance corresponding to the path being checked. > A fix is forthcoming... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399847#comment-15399847 ] Chao Sun commented on HIVE-14340: - {quote} Query hook is a little ambiguous. {quote} Hmm... how about {{hive.query.lifetime.hooks}}? I'm open to suggestions. :) {quote} Since post execution hooks are already supported, does this mean we just need to add pre compilation hook? Isn't pre semantic analysis good for this purpose? {quote} Yeah, I've hesitated on this - not sure if its a good idea to add this new hook.. One reason is that I'm not sure how to get the query command for the post-exec case, and another is that the user needs to implement two interfaces and add multiple configurations. > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to avoid NPE if ExecReducer.close is called twice.
[ https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399844#comment-15399844 ] Chao Sun commented on HIVE-14303: - Going to commit this. [~zxu] are the test failures related? > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. > - > > Key: HIVE-14303 > URL: https://issues.apache.org/jira/browse/HIVE-14303 > Project: Hive > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.1.0 > > Attachments: HIVE-14303.0.patch > > > CommonJoinOperator.checkAndGenObject should return directly at CLOSE state to > avoid NPE if ExecReducer.close is called twice. ExecReducer.close implements > Closeable interface and ExecReducer.close can be called multiple time. We saw > the following NPE which hide the real exception due to this bug. > {code} > Error: java.lang.RuntimeException: Hive Runtime Error while closing > operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284) > ... 8 more > {code} > The code from ReduceTask.runOldReducer: > {code} > reducer.close(); //line 453 > reducer = null; > > out.close(reporter); > out = null; > } finally { > IOUtils.cleanup(LOG, reducer);// line 459 > closeQuietly(out, reporter); > } > {code} > Based on the above stack trace and code, reducer.close() is called twice > because the exception happened when reducer.close() is called for the first > time at line 453, the code exit before reducer was set to null. > NullPointerException is triggered when reducer.close() is called for the > second time in IOUtils.cleanup at line 459. NullPointerException hide the > real exception which happened when reducer.close() is called for the first > time at line 453. > The reason for NPE is: > The first reducer.close called CommonJoinOperator.closeOp which clear > {{storage}} > {code} > Arrays.fill(storage, null); > {code} > the second reduce.close generated NPE due to null {{storage[alias]}} which is > set to null by first reducer.close. > The following reducer log can give more proof: > {code} > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0 > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing... > 2016-07-14 22:24:51,016 INFO [main] > org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written - > 53466 > 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing > operators - failing tree > 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: Hive Runtime Error > while closing operators: null > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296) > at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.
[jira] [Commented] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion
[ https://issues.apache.org/jira/browse/HIVE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399840#comment-15399840 ] Prasanth Jayachandran commented on HIVE-14355: -- [~sershe] Can you take a look again plz? > Schema evolution for ORC in llap is broken for int to string conversion > --- > > Key: HIVE-14355 > URL: https://issues.apache.org/jira/browse/HIVE-14355 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-14355-java-only.patch, HIVE-14355.1.patch, > HIVE-14355.2.java-only.patch, HIVE-14355.2.patch, > HIVE-14355.3.java-only.patch, HIVE-14355.3.patch > > > When schema is evolved from any integer type to string then following > exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen > even for other conversions. > {code} > hive> create table orc_integer(b bigint) stored as orc; > hive> insert into orc_integer values(100); > hive> select count(*) from orc_integer where b=100; > OK > 1 > hive> alter table orc_integer change column b b string; > hive> select count(*) from orc_integer where b=100; > // FAIL with following exception > {code} > {code:title=When vectorization is enabled} > 2016-07-27T01:48:05,611 INFO [TezTaskRunner ()] > vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0, > 2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor: > java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: > Hive Runtime Error while processing row > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) > at > org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at > org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row > at > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86) > ... 18 more > Caused by: java.lang.ClassCastException: > org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector > at > org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42) > at > org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) > at > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774) > ... 19 more > {code} > {code:title=When vectorization is disabled} > 2016-07-27T01:52:43,328 INFO [TezTaskRunner > (1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1 > 2016-07-27T01:52:43,328 INFO [TezTaskRunner > (146
[jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[ https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14366: -- Priority: Blocker (was: Critical) > Conversion of a Non-ACID table to an ACID table produces non-unique primary > keys > > > Key: HIVE-14366 > URL: https://issues.apache.org/jira/browse/HIVE-14366 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Saket Saurabh >Assignee: Saket Saurabh >Priority: Blocker > Attachments: HIVE-14366.01.patch, HIVE-14366.02.patch > > > When a Non-ACID table is converted to an ACID table, the primary key > consisting of (original transaction id, bucket_id, row_id) is not generated > uniquely. Currently, the row_id is always set to 0 for most rows. This leads > to correctness issue for such tables. > Quickest way to reproduce is to add the following unit test to > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java > {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid} > @Test > public void testOriginalReader() throws Exception { > FileSystem fs = FileSystem.get(hiveConf); > FileStatus[] status; > // 1. Insert five rows to Non-ACID table. > runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) > values(1,2),(3,4),(5,6),(7,8),(9,10)"); > // 2. Convert NONACIDORCTBL to ACID table. > runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET > TBLPROPERTIES ('transactional'='true')"); > // 3. Perform a major compaction. > runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact > 'MAJOR'"); > runWorker(hiveConf); > // 4. Perform a delete. > runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = > 1"); > // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since > (1,2) has been deleted. > List rs = runStatementOnDriver("select a,b from " + > Table.NONACIDORCTBL + " order by a,b"); > int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}}; > Assert.assertEquals(stringifyValues(resultData), rs); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[ https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399830#comment-15399830 ] Eugene Koifman commented on HIVE-13991: --- back ported to branch-1 https://github.com/apache/hive/commit/123f088a3a029c4b01db3e7cc0769a5836e1f6c6 > Union All on view fail with no valid permission on underneath table > --- > > Key: HIVE-13991 > URL: https://issues.apache.org/jira/browse/HIVE-13991 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13991.1.patch, HIVE-13991.2.patch > > > When sentry is enabled. > create view V as select * from T; > When the user has read permission on view V, but does not have read > permission on table T, > select * from V union all select * from V > failed with: > {noformat} > 0: jdbc:hive2://> select * from s07view union all select * from > s07view limit 1; > Error: Error while compiling statement: FAILED: SemanticException No valid > privileges > Required privileges for this query: > Server=server1->Db=default->Table=sample_07->action=select; > (state=42000,code=4) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[ https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13991: -- Fix Version/s: 1.3.0 > Union All on view fail with no valid permission on underneath table > --- > > Key: HIVE-13991 > URL: https://issues.apache.org/jira/browse/HIVE-13991 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13991.1.patch, HIVE-13991.2.patch > > > When sentry is enabled. > create view V as select * from T; > When the user has read permission on view V, but does not have read > permission on table T, > select * from V union all select * from V > failed with: > {noformat} > 0: jdbc:hive2://> select * from s07view union all select * from > s07view limit 1; > Error: Error while compiling statement: FAILED: SemanticException No valid > privileges > Required privileges for this query: > Server=server1->Db=default->Table=sample_07->action=select; > (state=42000,code=4) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14383) SparkClientImpl should pass principal and keytab to spark-submit instead of calling kinit explicitely
[ https://issues.apache.org/jira/browse/HIVE-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399826#comment-15399826 ] Xuefu Zhang commented on HIVE-14383: I think the old version of Spark doesn't do kinit for the client, and that's why kinit is called explicitly. If the bug is fixed in new version of Spark, this is certainly a welcome fix for Hive on Spark. Other than that, code looks good to me. +1 > SparkClientImpl should pass principal and keytab to spark-submit instead of > calling kinit explicitely > - > > Key: HIVE-14383 > URL: https://issues.apache.org/jira/browse/HIVE-14383 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 2.1.0 >Reporter: Mubashir Kazia >Assignee: Chaoyu Tang > Attachments: HIVE-14383.patch > > > In > https://github.com/apache/hive/blob/master/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java#L331, > kinit is manually called before spark-submit is called. This is problematic > if the spark session outlives the delegation token expiry period or the > kerberos ticket expiry period. A better way to handle this would be to pass > the keytab file location and principal name as parameters to the spark-submit > command when kerberos auth is in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14383) SparkClientImpl should pass principal and keytab to spark-submit instead of calling kinit explicitely
[ https://issues.apache.org/jira/browse/HIVE-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14383: --- Status: Patch Available (was: Open) Did some manual tests in kerberized cluster and the patch works fine. [~xuefuz], [~mkazia], [~csun] Could you review the patch? Thanks > SparkClientImpl should pass principal and keytab to spark-submit instead of > calling kinit explicitely > - > > Key: HIVE-14383 > URL: https://issues.apache.org/jira/browse/HIVE-14383 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 2.1.0 >Reporter: Mubashir Kazia >Assignee: Chaoyu Tang > Attachments: HIVE-14383.patch > > > In > https://github.com/apache/hive/blob/master/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java#L331, > kinit is manually called before spark-submit is called. This is problematic > if the spark session outlives the delegation token expiry period or the > kerberos ticket expiry period. A better way to handle this would be to pass > the keytab file location and principal name as parameters to the spark-submit > command when kerberos auth is in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14376) Schema evolution tests takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399812#comment-15399812 ] Prasanth Jayachandran commented on HIVE-14376: -- Actually it gets worse there are 13 more tests for schema evolution on text. Replacing INSERT INTO queries is extremely laborious task. Most q files are >800 LOC. All of the select queries also have order by. > Schema evolution tests takes a long time > > > Key: HIVE-14376 > URL: https://issues.apache.org/jira/browse/HIVE-14376 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0, 2.1.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > > schema_evol_* tests (14 tests) takes 47 mins in tez, 40mins in TestCliDriver. > Same set of tests are added to llap as well in HIVE-14355 which will some > more time. Most tests uses INSERT into table which can be slow and is not > needed. Even some individual tests takes about 5 mins to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13560: -- Attachment: HIVE-13560.11.patch > Adding Omid as connection manager for HBase Metastore > - > > Key: HIVE-13560 > URL: https://issues.apache.org/jira/browse/HIVE-13560 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13560.1.patch, HIVE-13560.10.patch, > HIVE-13560.11.patch, HIVE-13560.2.patch, HIVE-13560.3.patch, > HIVE-13560.4.patch, HIVE-13560.5.patch, HIVE-13560.6.patch, > HIVE-13560.7.patch, HIVE-13560.8.patch, HIVE-13560.9.patch > > > Adding Omid as a transaction manager to HBase Metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14383) SparkClientImpl should pass principal and keytab to spark-submit instead of calling kinit explicitely
[ https://issues.apache.org/jira/browse/HIVE-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14383: --- Attachment: HIVE-14383.patch > SparkClientImpl should pass principal and keytab to spark-submit instead of > calling kinit explicitely > - > > Key: HIVE-14383 > URL: https://issues.apache.org/jira/browse/HIVE-14383 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 2.1.0 >Reporter: Mubashir Kazia >Assignee: Chaoyu Tang > Attachments: HIVE-14383.patch > > > In > https://github.com/apache/hive/blob/master/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java#L331, > kinit is manually called before spark-submit is called. This is problematic > if the spark session outlives the delegation token expiry period or the > kerberos ticket expiry period. A better way to handle this would be to pass > the keytab file location and principal name as parameters to the spark-submit > command when kerberos auth is in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14383) SparkClientImpl should pass principal and keytab to spark-submit instead of calling kinit explicitely
[ https://issues.apache.org/jira/browse/HIVE-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang reassigned HIVE-14383: -- Assignee: Chaoyu Tang > SparkClientImpl should pass principal and keytab to spark-submit instead of > calling kinit explicitely > - > > Key: HIVE-14383 > URL: https://issues.apache.org/jira/browse/HIVE-14383 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 2.1.0 >Reporter: Mubashir Kazia >Assignee: Chaoyu Tang > > In > https://github.com/apache/hive/blob/master/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java#L331, > kinit is manually called before spark-submit is called. This is problematic > if the spark session outlives the delegation token expiry period or the > kerberos ticket expiry period. A better way to handle this would be to pass > the keytab file location and principal name as parameters to the spark-submit > command when kerberos auth is in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14361) Empty method in TestClientCommandHookFactory
[ https://issues.apache.org/jira/browse/HIVE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399794#comment-15399794 ] Hive QA commented on HIVE-14361: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820545/HIVE-14361.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10380 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union org.apache.hive.spark.client.TestSparkClient.testJobSubmission {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/686/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/686/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-686/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12820545 - PreCommit-HIVE-MASTER-Build > Empty method in TestClientCommandHookFactory > > > Key: HIVE-14361 > URL: https://issues.apache.org/jira/browse/HIVE-14361 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Trivial > Attachments: HIVE-14361.patch > > > Remove the empty method left in TestClientCommandHookFactory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399753#comment-15399753 ] Jimmy Xiang commented on HIVE-14340: Query hook is a little ambiguous. Since post execution hooks are already supported, does this mean we just need to add pre compilation hook? Isn't pre semantic analysis good for this purpose? > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14374) BeeLine argument, and configuration handling cleanup
[ https://issues.apache.org/jira/browse/HIVE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399734#comment-15399734 ] Sahil Takiar commented on HIVE-14374: - A lot of the reflection code is also in {{org.apache.hive.beeline.Reflector}}. I agree this code requires some cleanup. I believe [~vihangk1] has brought up the issue that it makes the code a lot less readable. [~pvary] are there a lot of scenarios where a configuration parameter should only be set via the command, but not in the configuration file (or vice versa)? If not I would say it may be simpler to just have one annotation {{ConfigurationKey}} that indicates this setter method should be use as a configuration key. The advantage here is that users don't need to track which configuration keys can be configured from which location (e.g. command line or properties file). If we are going to invest time in cleaning up this code, it may be worth considering whether we should just use a standard Java Command Line Options Parser (commons-cli, jopt, jcommander, etc.). A simple pattern could be to just parse in the BeeLine configuration file, and then parse in the command line options. If a configuration key is set in both places, the command line option takes precedence. > BeeLine argument, and configuration handling cleanup > > > Key: HIVE-14374 > URL: https://issues.apache.org/jira/browse/HIVE-14374 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > BeeLine uses reflection, to set the BeeLineOpts attributes when parsing > command line arguments, and when loading the configuration file. > This means, that creating a setXXX, getXXX method in BeeLineOpts is a > potential risk of exposing an attribute for the user unintentionally. There > is a possibility to exclude an attribute from saving the value in the > configuration file with the Ignore annotation. This does not restrict the > loading or command line setting of these parameters which means there are > many undocumented "features" as-is, like setting the lastConnectedUrl, > allowMultilineCommand, maxHeight, trimScripts, etc. from command line. > This part of the code needs a little cleanup. > I think we should make this exposure more explicit, and be able to > differentiate the configurable options depending on the source (command line, > and configuration file), so I propose to create a mechanism to tell > explicitly which BeeLineOpts attributes are settable by command line, and > configuration file, and every other attribute should be inaccessible by the > user of the beeline cli. > One possible solution could be two annotations like these: > - CommandLineOption - there could be a mandatory text parameter here, so the > developer had to provide the help text for it which could be displayed to the > user > - ConfigurationFileOption - no text is required here > Something like this: > - This attribute could be provided by command line, and from a configuration > file too: > {noformat} > @CommandLineOption("automatically save preferences") > @ConfigurationFileOption > public void setAutosave(boolean autosave) { > this.autosave = autosave; > } > public void getAutosave() { > return this.autosave; > } > {noformat} > - This attribute could be set through the configuration only > {noformat} > @ConfigurationFileOption > public void setLastConnectedUrl(String lastConnectedUrl) { > this.lastConnectedUrl = lastConnectedUrl; > } > > public String getLastConnectedUrl() > { > return lastConnectedUrl; > } > {noformat} > - Attribute could be set through command line only - I think this is not too > relevant, but possible > {noformat} > @CommandLineOption("specific command line option") > public void setSpecificCommandLineOption(String specificCommandLineOption) { > this.specificCommandLineOption = specificCommandLineOption; > } > > public String getSpecificCommandLineOption() { > return specificCommandLineOption; > } > {noformat} > - Attribute could not be set > {noformat} > public static Env getEnv() { > return env; > } > public static void setEnv(Env envToUse) { > env = envToUse; > } > {noformat} > Accouring to our previous conversations, I think you might be interested in: > [~spena], [~vihangk1], [~aihuaxu], [~ngangam], [~ychena], [~xuefuz] > but anyone is welcome to discuss this. > What do you think about the proposed solution? > Any better ideas, or extensions? > Thanks, > Peter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14381) Handle null value in WindowingTableFunction.WindowingIterator.next()
[ https://issues.apache.org/jira/browse/HIVE-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399732#comment-15399732 ] Jesus Camacho Rodriguez commented on HIVE-14381: +1, I was checking the code, and the change makes sense, although we should still call {{output.set(j, out);}} in every case, as stated above. We make a similar conversion to NULL value in two other places; it seems like it was forgotten to do it in the {{next()}} method too. > Handle null value in WindowingTableFunction.WindowingIterator.next() > > > Key: HIVE-14381 > URL: https://issues.apache.org/jira/browse/HIVE-14381 > Project: Hive > Issue Type: Bug > Components: PTF-Windowing >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-14381.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[ https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-14343: Attachment: HIVE-14343.1.patch > HiveDriverRunHookContext's command is null in HS2 mode > -- > > Key: HIVE-14343 > URL: https://issues.apache.org/jira/browse/HIVE-14343 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14343.0.patch, HIVE-14343.1.patch > > > Looking at the {{Driver#runInternal(String command, boolean > alreadyCompiled)}}: > {code} > HiveDriverRunHookContext hookContext = new > HiveDriverRunHookContextImpl(conf, command); > // Get all the driver run hooks and pre-execute them. > List driverRunHooks; > {code} > The context is initialized with the {{command}} passed in to the method. > However, this command is always null if {{alreadyCompiled}} is true, which is > the case for HS2 mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399700#comment-15399700 ] Sahil Takiar commented on HIVE-14170: - Hey [~taoli-hwx], thanks for taking a look. I plan to set incremental to true by default, but in another JIRA HIVE-7224. Once the changes in this JIRA have been changed, I will merge HIVE-7224. > Beeline IncrementalRows should buffer rows and incrementally re-calculate > width if TableOutputFormat is used > > > Key: HIVE-14170 > URL: https://issues.apache.org/jira/browse/HIVE-14170 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14170.1.patch, HIVE-14170.2.patch, > HIVE-14170.3.patch, HIVE-14170.4.patch > > > If {{--incremental}} is specified in Beeline, rows are meant to be printed > out immediately. However, if {{TableOutputFormat}} is used with this option > the formatting can look really off. > The reason is that {{IncrementalRows}} does not do a global calculation of > the optimal width size for {{TableOutputFormat}} (it can't because it only > sees one row at a time). The output of {{BufferedRows}} looks much better > because it can do this global calculation. > If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width > should be re-calculated every "x" rows ("x" can be configurable and by > default it can be 1000). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Attachment: HIVE-14117.1.patch Re-upload for QA run. > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch, HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Status: Patch Available (was: Open) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch, HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Status: Open (was: Patch Available) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
[ https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14366: -- Assignee: Saket Saurabh (was: Eugene Koifman) > Conversion of a Non-ACID table to an ACID table produces non-unique primary > keys > > > Key: HIVE-14366 > URL: https://issues.apache.org/jira/browse/HIVE-14366 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Saket Saurabh >Assignee: Saket Saurabh >Priority: Critical > Attachments: HIVE-14366.01.patch, HIVE-14366.02.patch > > > When a Non-ACID table is converted to an ACID table, the primary key > consisting of (original transaction id, bucket_id, row_id) is not generated > uniquely. Currently, the row_id is always set to 0 for most rows. This leads > to correctness issue for such tables. > Quickest way to reproduce is to add the following unit test to > ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java > {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid} > @Test > public void testOriginalReader() throws Exception { > FileSystem fs = FileSystem.get(hiveConf); > FileStatus[] status; > // 1. Insert five rows to Non-ACID table. > runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) > values(1,2),(3,4),(5,6),(7,8),(9,10)"); > // 2. Convert NONACIDORCTBL to ACID table. > runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET > TBLPROPERTIES ('transactional'='true')"); > // 3. Perform a major compaction. > runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact > 'MAJOR'"); > runWorker(hiveConf); > // 4. Perform a delete. > runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = > 1"); > // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since > (1,2) has been deleted. > List rs = runStatementOnDriver("select a,b from " + > Table.NONACIDORCTBL + " order by a,b"); > int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}}; > Assert.assertEquals(stringifyValues(resultData), rs); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399630#comment-15399630 ] Chao Sun commented on HIVE-14340: - I don't think the test failures are related. [~xuefuz], [~jxiang]: can you give a review when you have time? thanks. > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399610#comment-15399610 ] Aihua Xu commented on HIVE-14146: - >From the test case result. seems we still have some comments not escaped. >[~pvary] Can you check? The one in code review seems to be correct. > Column comments with "\n" character "corrupts" table metadata > - > > Key: HIVE-14146 > URL: https://issues.apache.org/jira/browse/HIVE-14146 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch, > HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch, HIVE-14146.patch > > > Create a table with the following(noting the \n in the COMMENT): > {noformat} > CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an > individual’); > {noformat} > Describe shows that now the metadata is messed up: > {noformat} > beeline> describe commtest; > +---++---+--+ > | col_name | data_type |comment| > +---++---+--+ > | first_nm | string | Indicates First name | > | of an individual | NULL | NULL | > +---++---+--+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14380) Queries on tables with remote HDFS paths fail in "encryption" checks.
[ https://issues.apache.org/jira/browse/HIVE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399600#comment-15399600 ] Sergio Peña commented on HIVE-14380: Cool [~mithun], thanks for taking a look at this. I reviewed the patch and it looks good. +1 Let's see what HiveQA says. > Queries on tables with remote HDFS paths fail in "encryption" checks. > - > > Key: HIVE-14380 > URL: https://issues.apache.org/jira/browse/HIVE-14380 > Project: Hive > Issue Type: Bug > Components: Encryption >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-14380.1.patch > > > If a table has table/partition locations set to remote HDFS paths, querying > them will cause the following IAException: > {noformat} > 2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner > (SemanticAnalyzer.java:getMetaData(1867)) - > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if > hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: > java.lang.IllegalArgumentException: Wrong FS: > hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: > hdfs://bar.ygrid.yahoo.com:8020 > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274) > ... > {noformat} > This is because of the following code in {{SessionState}}: > {code:title=SessionState.java|borderStyle=solid} > public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws > HiveException { > if (hdfsEncryptionShim == null) { > try { > FileSystem fs = FileSystem.get(sessionConf); > if ("hdfs".equals(fs.getUri().getScheme())) { > hdfsEncryptionShim = > ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf); > } else { > LOG.debug("Could not get hdfsEncryptionShim, it is only applicable > to hdfs filesystem."); > } > } catch (Exception e) { > throw new HiveException(e); > } > } > return hdfsEncryptionShim; > } > {code} > When the {{FileSystem}} instance is created, using the {{sessionConf}} > implies that the current HDFS is going to be used. This call should instead > fetch the {{FileSystem}} instance corresponding to the path being checked. > A fix is forthcoming... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14150) Hive does not compile against Hadoop-2.9.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HIVE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HIVE-14150: - Status: Patch Available (was: Open) > Hive does not compile against Hadoop-2.9.0-SNAPSHOT > --- > > Key: HIVE-14150 > URL: https://issues.apache.org/jira/browse/HIVE-14150 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HIVE-14150.01.patch > > > JvmPauseMonitor, JvmMetrics used in LLAP are private classes in Hadoop - and > have changed in 2.9. > Hive has it's own version of JvmPauseMonitor. Need to see what can be done > with JvmMetrics -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14150) Hive does not compile against Hadoop-2.9.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HIVE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HIVE-14150: - Attachment: HIVE-14150.01.patch Attaching a patch to use the JvmPauseMonitor of Hive. In addition, I've created Hive's JvmMetrics and other related classes. The JvmMetrics of Hadoop is {{@Private}}, so I'm thinking it's better to implement Hive's own JvmMetrics. > Hive does not compile against Hadoop-2.9.0-SNAPSHOT > --- > > Key: HIVE-14150 > URL: https://issues.apache.org/jira/browse/HIVE-14150 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HIVE-14150.01.patch > > > JvmPauseMonitor, JvmMetrics used in LLAP are private classes in Hadoop - and > have changed in 2.9. > Hive has it's own version of JvmPauseMonitor. Need to see what can be done > with JvmMetrics -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14329) fix flapping qtests - because of output string ordering
[ https://issues.apache.org/jira/browse/HIVE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399565#comment-15399565 ] Hive QA commented on HIVE-14329: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820719/HIVE-14329.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 10366 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-tez_self_join.q-filter_join_breaktask.q-vector_decimal_precision.q-and-12-more - did not produce a TEST-*.xml file TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_alter_list_bucketing_table1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_skewed_table1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_grouping_operators org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lb_fs_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_column_names_with_leading_and_trailing_spaces org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_deleteAnalyze org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_stats org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testC
[jira] [Assigned] (HIVE-14150) Hive does not compile against Hadoop-2.9.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HIVE-14150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HIVE-14150: Assignee: Akira Ajisaka > Hive does not compile against Hadoop-2.9.0-SNAPSHOT > --- > > Key: HIVE-14150 > URL: https://issues.apache.org/jira/browse/HIVE-14150 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Akira Ajisaka >Priority: Critical > > JvmPauseMonitor, JvmMetrics used in LLAP are private classes in Hadoop - and > have changed in 2.9. > Hive has it's own version of JvmPauseMonitor. Need to see what can be done > with JvmMetrics -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14374) BeeLine argument, and configuration handling cleanup
[ https://issues.apache.org/jira/browse/HIVE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399546#comment-15399546 ] Peter Vary commented on HIVE-14374: --- {noformat} public boolean set(String key, String value, boolean quiet) { try { beeLine.getReflector().invoke(this, "set" + key, new Object[] {value}); return true; } catch (Exception e) { if (!quiet) { beeLine.error(beeLine.loc("error-setting", new Object[] {key, e})); } return false; } } {noformat} > BeeLine argument, and configuration handling cleanup > > > Key: HIVE-14374 > URL: https://issues.apache.org/jira/browse/HIVE-14374 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > BeeLine uses reflection, to set the BeeLineOpts attributes when parsing > command line arguments, and when loading the configuration file. > This means, that creating a setXXX, getXXX method in BeeLineOpts is a > potential risk of exposing an attribute for the user unintentionally. There > is a possibility to exclude an attribute from saving the value in the > configuration file with the Ignore annotation. This does not restrict the > loading or command line setting of these parameters which means there are > many undocumented "features" as-is, like setting the lastConnectedUrl, > allowMultilineCommand, maxHeight, trimScripts, etc. from command line. > This part of the code needs a little cleanup. > I think we should make this exposure more explicit, and be able to > differentiate the configurable options depending on the source (command line, > and configuration file), so I propose to create a mechanism to tell > explicitly which BeeLineOpts attributes are settable by command line, and > configuration file, and every other attribute should be inaccessible by the > user of the beeline cli. > One possible solution could be two annotations like these: > - CommandLineOption - there could be a mandatory text parameter here, so the > developer had to provide the help text for it which could be displayed to the > user > - ConfigurationFileOption - no text is required here > Something like this: > - This attribute could be provided by command line, and from a configuration > file too: > {noformat} > @CommandLineOption("automatically save preferences") > @ConfigurationFileOption > public void setAutosave(boolean autosave) { > this.autosave = autosave; > } > public void getAutosave() { > return this.autosave; > } > {noformat} > - This attribute could be set through the configuration only > {noformat} > @ConfigurationFileOption > public void setLastConnectedUrl(String lastConnectedUrl) { > this.lastConnectedUrl = lastConnectedUrl; > } > > public String getLastConnectedUrl() > { > return lastConnectedUrl; > } > {noformat} > - Attribute could be set through command line only - I think this is not too > relevant, but possible > {noformat} > @CommandLineOption("specific command line option") > public void setSpecificCommandLineOption(String specificCommandLineOption) { > this.specificCommandLineOption = specificCommandLineOption; > } > > public String getSpecificCommandLineOption() { > return specificCommandLineOption; > } > {noformat} > - Attribute could not be set > {noformat} > public static Env getEnv() { > return env; > } > public static void setEnv(Env envToUse) { > env = envToUse; > } > {noformat} > Accouring to our previous conversations, I think you might be interested in: > [~spena], [~vihangk1], [~aihuaxu], [~ngangam], [~ychena], [~xuefuz] > but anyone is welcome to discuss this. > What do you think about the proposed solution? > Any better ideas, or extensions? > Thanks, > Peter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14367) Estimated size for constant nulls is 0
[ https://issues.apache.org/jira/browse/HIVE-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399548#comment-15399548 ] Jesus Camacho Rodriguez commented on HIVE-14367: +1 pending the change in RB and passing QA > Estimated size for constant nulls is 0 > -- > > Key: HIVE-14367 > URL: https://issues.apache.org/jira/browse/HIVE-14367 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14367.1.patch > > > since type is incorrectly assumed as void. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14374) BeeLine argument, and configuration handling cleanup
[ https://issues.apache.org/jira/browse/HIVE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399545#comment-15399545 ] Sergio Peña commented on HIVE-14374: How is reflection used on BeeLineOpts? I was looking in the code, but I did not find it. > BeeLine argument, and configuration handling cleanup > > > Key: HIVE-14374 > URL: https://issues.apache.org/jira/browse/HIVE-14374 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > BeeLine uses reflection, to set the BeeLineOpts attributes when parsing > command line arguments, and when loading the configuration file. > This means, that creating a setXXX, getXXX method in BeeLineOpts is a > potential risk of exposing an attribute for the user unintentionally. There > is a possibility to exclude an attribute from saving the value in the > configuration file with the Ignore annotation. This does not restrict the > loading or command line setting of these parameters which means there are > many undocumented "features" as-is, like setting the lastConnectedUrl, > allowMultilineCommand, maxHeight, trimScripts, etc. from command line. > This part of the code needs a little cleanup. > I think we should make this exposure more explicit, and be able to > differentiate the configurable options depending on the source (command line, > and configuration file), so I propose to create a mechanism to tell > explicitly which BeeLineOpts attributes are settable by command line, and > configuration file, and every other attribute should be inaccessible by the > user of the beeline cli. > One possible solution could be two annotations like these: > - CommandLineOption - there could be a mandatory text parameter here, so the > developer had to provide the help text for it which could be displayed to the > user > - ConfigurationFileOption - no text is required here > Something like this: > - This attribute could be provided by command line, and from a configuration > file too: > {noformat} > @CommandLineOption("automatically save preferences") > @ConfigurationFileOption > public void setAutosave(boolean autosave) { > this.autosave = autosave; > } > public void getAutosave() { > return this.autosave; > } > {noformat} > - This attribute could be set through the configuration only > {noformat} > @ConfigurationFileOption > public void setLastConnectedUrl(String lastConnectedUrl) { > this.lastConnectedUrl = lastConnectedUrl; > } > > public String getLastConnectedUrl() > { > return lastConnectedUrl; > } > {noformat} > - Attribute could be set through command line only - I think this is not too > relevant, but possible > {noformat} > @CommandLineOption("specific command line option") > public void setSpecificCommandLineOption(String specificCommandLineOption) { > this.specificCommandLineOption = specificCommandLineOption; > } > > public String getSpecificCommandLineOption() { > return specificCommandLineOption; > } > {noformat} > - Attribute could not be set > {noformat} > public static Env getEnv() { > return env; > } > public static void setEnv(Env envToUse) { > env = envToUse; > } > {noformat} > Accouring to our previous conversations, I think you might be interested in: > [~spena], [~vihangk1], [~aihuaxu], [~ngangam], [~ychena], [~xuefuz] > but anyone is welcome to discuss this. > What do you think about the proposed solution? > Any better ideas, or extensions? > Thanks, > Peter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399532#comment-15399532 ] Sergio Peña commented on HIVE-14146: Latest patch looks good. +1 Wait for HiveQa before commit. > Column comments with "\n" character "corrupts" table metadata > - > > Key: HIVE-14146 > URL: https://issues.apache.org/jira/browse/HIVE-14146 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch, > HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch, HIVE-14146.patch > > > Create a table with the following(noting the \n in the COMMENT): > {noformat} > CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an > individual’); > {noformat} > Describe shows that now the metadata is messed up: > {noformat} > beeline> describe commtest; > +---++---+--+ > | col_name | data_type |comment| > +---++---+--+ > | first_nm | string | Indicates First name | > | of an individual | NULL | NULL | > +---++---+--+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14146: -- Attachment: HIVE-14146.6.patch Latest patch, see review > Column comments with "\n" character "corrupts" table metadata > - > > Key: HIVE-14146 > URL: https://issues.apache.org/jira/browse/HIVE-14146 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch, > HIVE-14146.4.patch, HIVE-14146.5.patch, HIVE-14146.6.patch, HIVE-14146.patch > > > Create a table with the following(noting the \n in the COMMENT): > {noformat} > CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an > individual’); > {noformat} > Describe shows that now the metadata is messed up: > {noformat} > beeline> describe commtest; > +---++---+--+ > | col_name | data_type |comment| > +---++---+--+ > | first_nm | string | Indicates First name | > | of an individual | NULL | NULL | > +---++---+--+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14294) HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types
[ https://issues.apache.org/jira/browse/HIVE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14294: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~gszadovszky] for your contribution. I committed this to 2.2 > HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into > proper Parquet types > > > Key: HIVE-14294 > URL: https://issues.apache.org/jira/browse/HIVE-14294 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Cheng Lian >Assignee: Gabor Szadovszky > Fix For: 2.2.0 > > Attachments: HIVE-14294.patch > > > To reproduce this issue, run the following DDL: > {code:sql} > CREATE TABLE foo STORED AS PARQUET AS SELECT CAST(1 AS TINYINT); > {code} > And then check the schema of the written Parquet file: > {noformat} > $ parquet-schema $WAREHOUSE_PATH/foo/00_0 > message hive_schema { > optional int32 _c0; > } > {noformat} > When translating Hive types into Parquet types, {{TINYINT}} and {{SMALLINT}} > should be translated into the {{int32 (INT_8)}} and {{int32 (INT_16)}} > respectively. However, {{HiveSchemaConverter}} converts all of {{TINYINT}}, > {{SMALLINT}}, and {{INT}} into Parquet {{int32}}. This causes problem when > accessing Parquet files generated by Hive in other systems since type > information gets wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12646) beeline and HIVE CLI do not parse ; in quote properly
[ https://issues.apache.org/jira/browse/HIVE-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399411#comment-15399411 ] Sergio Peña commented on HIVE-12646: It does not need documentation. This fixes a bug with ; only. Thanks for asking. > beeline and HIVE CLI do not parse ; in quote properly > - > > Key: HIVE-12646 > URL: https://issues.apache.org/jira/browse/HIVE-12646 > Project: Hive > Issue Type: Bug > Components: CLI, Clients >Reporter: Yongzhi Chen >Assignee: Sahil Takiar > Fix For: 2.2.0 > > Attachments: HIVE-12646.2.patch, HIVE-12646.3.patch, > HIVE-12646.4.patch, HIVE-12646.5.patch, HIVE-12646.patch > > > Beeline and Cli have to escape ; in the quote while most other shell scripts > need not. For example: > in Beeline: > {noformat} > 0: jdbc:hive2://localhost:1> select ';' from tlb1; > select ';' from tlb1; > 15/12/10 10:45:26 DEBUG TSaslTransport: writing data length: 115 > 15/12/10 10:45:26 DEBUG TSaslTransport: CLIENT: reading data length: 3403 > Error: Error while compiling statement: FAILED: ParseException line 1:8 > cannot recognize input near '' ' > {noformat} > while in mysql shell: > {noformat} > mysql> SELECT CONCAT(';', 'foo') FROM test limit 3; > ++ > | ;foo | > | ;foo | > | ;foo | > ++ > 3 rows in set (0.00 sec) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14359) Hive on Spark might fail in HS2 with LDAP authentication in a kerberized cluster
[ https://issues.apache.org/jira/browse/HIVE-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14359: --- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Committed to 2.2.0 and 2.1.1. Thanks [~xuefuz] for review. > Hive on Spark might fail in HS2 with LDAP authentication in a kerberized > cluster > > > Key: HIVE-14359 > URL: https://issues.apache.org/jira/browse/HIVE-14359 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14359.patch > > > When HS2 is used as a gateway for the LDAP users to access and run the > queries in kerberized cluster, it's authentication mode is configured as LDAP > and at this time, HoS might fail by the same reason as HIVE-10594. > hive.server2.authentication is not a proper property to determine if a > cluster is kerberized, instead hadoop.security.authentication should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14359) Hive on Spark might fail in HS2 with LDAP authentication in a kerberized cluster
[ https://issues.apache.org/jira/browse/HIVE-14359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399317#comment-15399317 ] Chaoyu Tang commented on HIVE-14359: The failed tests are aged and not related to this patch. > Hive on Spark might fail in HS2 with LDAP authentication in a kerberized > cluster > > > Key: HIVE-14359 > URL: https://issues.apache.org/jira/browse/HIVE-14359 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14359.patch > > > When HS2 is used as a gateway for the LDAP users to access and run the > queries in kerberized cluster, it's authentication mode is configured as LDAP > and at this time, HoS might fail by the same reason as HIVE-10594. > hive.server2.authentication is not a proper property to determine if a > cluster is kerberized, instead hadoop.security.authentication should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14340) Add a new hook triggers before query compilation and after query execution
[ https://issues.apache.org/jira/browse/HIVE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399295#comment-15399295 ] Hive QA commented on HIVE-14340: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12820523/HIVE-14340.0.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10381 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/683/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/683/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-683/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12820523 - PreCommit-HIVE-MASTER-Build > Add a new hook triggers before query compilation and after query execution > -- > > Key: HIVE-14340 > URL: https://issues.apache.org/jira/browse/HIVE-14340 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-14340.0.patch > > > In some cases we may need to have a hook that activates before a query > compilation and after its execution. For instance, dynamically generate a UDF > specifically for the running query and clean up the resource after the query > is done. The current hooks only covers pre & post semantic analysis, pre & > post query execution, which doesn't fit the requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399292#comment-15399292 ] Dmitry Tolpeko commented on HIVE-14382: --- Akash, right now this functionality is part of Hive HPL/SQL component, see docs: http://hplsql.org/for-range > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14382 started by Akash Sethi. -- > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14382 stopped by Akash Sethi. -- > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399185#comment-15399185 ] Amareshwari Sriramadasu commented on HIVE-14335: If there are no objections, I would like to commit to branch-2.1 cc [~szehon] [~jcamachorodriguez] > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.2.0 > > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12954) NPE with str_to_map on null strings
[ https://issues.apache.org/jira/browse/HIVE-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-12954: - Attachment: HIVE-12954.2.patch Fixed the review findings in the second patch: - removed additional spaces - added the @Test annotations to the unit tests > NPE with str_to_map on null strings > --- > > Key: HIVE-12954 > URL: https://issues.apache.org/jira/browse/HIVE-12954 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0, 1.0.0 >Reporter: Charles Pritchard >Assignee: Marta Kuczora > Attachments: HIVE-12954.2.patch, HIVE-12954.patch > > > Running str_to_map on a null string will return a NullPointerException. > Workaround is to use coalesce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12954) NPE with str_to_map on null strings
[ https://issues.apache.org/jira/browse/HIVE-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-12954: - Status: Patch Available (was: In Progress) > NPE with str_to_map on null strings > --- > > Key: HIVE-12954 > URL: https://issues.apache.org/jira/browse/HIVE-12954 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0, 0.14.0 >Reporter: Charles Pritchard >Assignee: Marta Kuczora > Attachments: HIVE-12954.2.patch, HIVE-12954.patch > > > Running str_to_map on a null string will return a NullPointerException. > Workaround is to use coalesce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12954) NPE with str_to_map on null strings
[ https://issues.apache.org/jira/browse/HIVE-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-12954: - Status: In Progress (was: Patch Available) > NPE with str_to_map on null strings > --- > > Key: HIVE-12954 > URL: https://issues.apache.org/jira/browse/HIVE-12954 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0, 0.14.0 >Reporter: Charles Pritchard >Assignee: Marta Kuczora > Attachments: HIVE-12954.patch > > > Running str_to_map on a null string will return a NullPointerException. > Workaround is to use coalesce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)