[jira] [Commented] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565872#comment-14565872
]
Hive QA commented on HIVE-7193:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736168/HIVE-7193.2.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4100/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4100/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4100/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736168 - PreCommit-HIVE-TRUNK-Build
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Attachments: HIVE-7193.2.patch, HIVE-7193.patch,
> LDAPAuthentication_Design_Doc.docx, LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2.
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10853) Create ExplainTask in ATS hook through ExplainWork
[
https://issues.apache.org/jira/browse/HIVE-10853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565906#comment-14565906
]
Hive QA commented on HIVE-10853:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736209/HIVE-10853.02.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4101/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4101/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4101/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736209 - PreCommit-HIVE-TRUNK-Build
> Create ExplainTask in ATS hook through ExplainWork
> --
>
> Key: HIVE-10853
> URL: https://issues.apache.org/jira/browse/HIVE-10853
> Project: Hive
> Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Pengcheng Xiong
> Attachments: HIVE-10853.01.patch, HIVE-10853.02.patch
>
>
> Right now ExplainTask is created directly. That's fragile and can lead to
> stuff like: HIVE-10829
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6867) Bucketized Table feature fails in some cases
[
https://issues.apache.org/jira/browse/HIVE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565947#comment-14565947
]
Hive QA commented on HIVE-6867:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736207/HIVE-6867.05.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4102/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4102/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4102/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736207 - PreCommit-HIVE-TRUNK-Build
> Bucketized Table feature fails in some cases
>
>
> Key: HIVE-6867
> URL: https://issues.apache.org/jira/browse/HIVE-6867
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Laljo John Pullokkaran
>Assignee: Pengcheng Xiong
> Attachments: HIVE-6867.01.patch, HIVE-6867.02.patch,
> HIVE-6867.03.patch, HIVE-6867.04.patch, HIVE-6867.05.patch
>
>
> Bucketized Table feature fails in some cases. if src & destination is
> bucketed on same key, and if actual data in the src is not bucketed (because
> data got loaded using LOAD DATA LOCAL INPATH ) then the data won't be
> bucketed while writing to destination.
> Example
> --
> CREATE TABLE P1(key STRING, val STRING)
> CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH '/Users/jp/apache-hive1/data/files/P1.txt' INTO TABLE
> P1;
> – perform an insert to make sure there are 2 files
> INSERT OVERWRITE TABLE P1 select key, val from P1;
> --
> This is not a regression. This has never worked.
> This got only discovered due to Hadoop2 changes.
> In Hadoop1, in local mode, number of reducers will always be 1, regardless of
> what is requested by app. Hadoop2 now honors the number of reducer setting in
> local mode (by spawning threads).
> Long term solution seems to be to prevent load data for bucketed table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10807) Invalidate basic stats for insert queries if autogather=false
[
https://issues.apache.org/jira/browse/HIVE-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565986#comment-14565986
]
Hive QA commented on HIVE-10807:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736216/HIVE-10807.5.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 8983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4103/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4103/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4103/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736216 - PreCommit-HIVE-TRUNK-Build
> Invalidate basic stats for insert queries if autogather=false
> -
>
> Key: HIVE-10807
> URL: https://issues.apache.org/jira/browse/HIVE-10807
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Gopal V
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-10807.2.patch, HIVE-10807.3.patch,
> HIVE-10807.4.patch, HIVE-10807.5.patch, HIVE-10807.patch
>
>
> if stats.autogather=false leads to incorrect basic stats in case of insert
> statements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566028#comment-14566028 ] Chaoyu Tang commented on HIVE-10835: The 3 failed tests seems not related to this patch. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10869) fold_case.q failing on trunk
[
https://issues.apache.org/jira/browse/HIVE-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566033#comment-14566033
]
Hive QA commented on HIVE-10869:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736220/HIVE-10869.patch
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 8983 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_empty_dir_in_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_external_table_with_space_in_location_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_leftsemijoin_mr
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_quotedid_smb
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_remote_script
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_schemeAuthority2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_smb_mapjoin_8
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_stats_counter
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_stats_counter_partitioned
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_truncate_column_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_uber_reduce
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4104/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4104/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4104/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736220 - PreCommit-HIVE-TRUNK-Build
> fold_case.q failing on trunk
>
>
> Key: HIVE-10869
> URL: https://issues.apache.org/jira/browse/HIVE-10869
> Project: Hive
> Issue Type: Test
> Components: Tests
>Affects Versions: 1.3.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-10869.patch
>
>
> Race condition of commits between HIVE-10716 & HIVE-10812
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8627) Compute stats on a table from impala caused the table to be corrupted
[
https://issues.apache.org/jira/browse/HIVE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566062#comment-14566062
]
Sergey Tryuber commented on HIVE-8627:
--
For those who encounter similar issue (the patch has been backported by
Cloudera to CDH5.2.1, but the bug is still present in HDP2.2), the only way
which I've found to recover Impala without dropping tables is:
# Connect to Hive metastore database (in my case it was "mysql -h localhost
-Dhive")
# Truncate column statistics ("_truncate table TAB_COL_STATS;_" command)
# Do not execute "COMPUTE STATS" until you're sure that your Hive has this
patch applied. Use Hive's native equivalents instead.
> Compute stats on a table from impala caused the table to be corrupted
> -
>
> Key: HIVE-8627
> URL: https://issues.apache.org/jira/browse/HIVE-8627
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.0, 0.13.1
>Reporter: Na Yang
>Assignee: Na Yang
> Fix For: 1.1.0
>
> Attachments: HIVE-8627.2.patch, HIVE-8627.patch
>
>
> Use impala 2.0 to connect to hive-0.13 Metastore.
> From impala, run the following queries:
> {noformat}
> create table voter1(voter_id int,name string,age tinyint, registration
> string,contributions decimal(5,2),voterzone smallint,create_time timestamp)
> row
> format delimited fields terminated by '\t';
> load data inpath '/tmp/votertab' into table voter1;
> {noformat}
> After this, can successfully select from table voter 1.
> Execute the following from impala shell:
> {noformat}
> > compute stats voter1;
> {noformat}
> After this, got the following error selecting from table voter1:
> {noformat}
> > select * from voter1 limit 5;
> Query: select * from voter1 limit 5
> ERROR: AnalysisException: Failed to load metadata for table: default.voter1
> CAUSED BY: TableLoadingException: Failed to load metadata for table: voter1
> CAUSED BY: TTransportException: java.net.SocketException: Broken pipe
> CAUSED BY: SocketException: Broken pipe
> {noformat}
> Below is the exception found in Hive log:
> {noformat}
> org.apache.thrift.protocol.TProtocolException: Cannot write a TUnion with no
> set value!
> at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:240)
> at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:213)
> at org.apache.thrift.TUnion.write(TUnion.java:152)
> at
> org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj$ColumnStatisticsObjStandardScheme.write(ColumnStatisticsObj.java:550)
> at
> org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj$ColumnStatisticsObjStandardScheme.write(ColumnStatisticsObj.java:488)
> at
> org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj.write(ColumnStatisticsObj.java:414)
> at
> org.apache.hadoop.hive.metastore.api.TableStatsResult$TableStatsResultStandardScheme.write(TableStatsResult.java:388)
> at
> org.apache.hadoop.hive.metastore.api.TableStatsResult$TableStatsResultStandardScheme.write(TableStatsResult.java:338)
> at
> org.apache.hadoop.hive.metastore.api.TableStatsResult.write(TableStatsResult.java:288)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result.write(ThriftHiveMetastore.java)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10870) Merge Spark branch to trunk 5/29/2015
[
https://issues.apache.org/jira/browse/HIVE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566098#comment-14566098
]
Hive QA commented on HIVE-10870:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736229/HIVE-10870.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8983 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4105/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4105/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4105/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736229 - PreCommit-HIVE-TRUNK-Build
> Merge Spark branch to trunk 5/29/2015
> -
>
> Key: HIVE-10870
> URL: https://issues.apache.org/jira/browse/HIVE-10870
> Project: Hive
> Issue Type: Task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-10870.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4239) Remove lock on compilation stage
[
https://issues.apache.org/jira/browse/HIVE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566100#comment-14566100
]
Hive QA commented on HIVE-4239:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736232/HIVE-4239.02.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4106/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4106/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4106/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4106/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
83cc691..ae4790e master -> origin/master
+ git reset --hard HEAD
HEAD is now at 83cc691 HIVE-10722 : external table creation with msck in Hive
can create unusable partition (Sergey Shelukhin, reviewed by Sushanth Sowmyan)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
Removing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java
Removing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SmallTableCache.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at ae4790e HIVE-10788 - Change sort_array to support non-primitive
types (Chao Sun, reviewed by Alexander Pivovarov)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736232 - PreCommit-HIVE-TRUNK-Build
> Remove lock on compilation stage
>
>
> Key: HIVE-4239
> URL: https://issues.apache.org/jira/browse/HIVE-4239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Processor
>Reporter: Carl Steinbach
>Assignee: Sergey Shelukhin
> Attachments: HIVE-4239.01.patch, HIVE-4239.02.patch, HIVE-4239.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9069) Simplify filter predicates for CBO
[
https://issues.apache.org/jira/browse/HIVE-9069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-9069:
--
Attachment: HIVE-9069.18.patch
> Simplify filter predicates for CBO
> --
>
> Key: HIVE-9069
> URL: https://issues.apache.org/jira/browse/HIVE-9069
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Jesus Camacho Rodriguez
> Fix For: 0.14.1
>
> Attachments: HIVE-9069.01.patch, HIVE-9069.02.patch,
> HIVE-9069.03.patch, HIVE-9069.04.patch, HIVE-9069.05.patch,
> HIVE-9069.06.patch, HIVE-9069.07.patch, HIVE-9069.08.patch,
> HIVE-9069.08.patch, HIVE-9069.09.patch, HIVE-9069.10.patch,
> HIVE-9069.11.patch, HIVE-9069.12.patch, HIVE-9069.13.patch,
> HIVE-9069.14.patch, HIVE-9069.14.patch, HIVE-9069.15.patch,
> HIVE-9069.16.patch, HIVE-9069.17.patch, HIVE-9069.17.patch,
> HIVE-9069.18.patch, HIVE-9069.18.patch, HIVE-9069.patch
>
>
> Simplify predicates for disjunctive predicates so that can get pushed down to
> the scan.
> Looks like this is still an issue, some of the filters can be pushed down to
> the scan.
> {code}
> set hive.cbo.enable=true
> set hive.stats.fetch.column.stats=true
> set hive.exec.dynamic.partition.mode=nonstrict
> set hive.tez.auto.reducer.parallelism=true
> set hive.auto.convert.join.noconditionaltask.size=32000
> set hive.exec.reducers.bytes.per.reducer=1
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
> set hive.support.concurrency=false
> set hive.tez.exec.print.summary=true
> explain
> select substr(r_reason_desc,1,20) as r
>,avg(ws_quantity) wq
>,avg(wr_refunded_cash) ref
>,avg(wr_fee) fee
> from web_sales, web_returns, web_page, customer_demographics cd1,
> customer_demographics cd2, customer_address, date_dim, reason
> where web_sales.ws_web_page_sk = web_page.wp_web_page_sk
>and web_sales.ws_item_sk = web_returns.wr_item_sk
>and web_sales.ws_order_number = web_returns.wr_order_number
>and web_sales.ws_sold_date_sk = date_dim.d_date_sk and d_year = 1998
>and cd1.cd_demo_sk = web_returns.wr_refunded_cdemo_sk
>and cd2.cd_demo_sk = web_returns.wr_returning_cdemo_sk
>and customer_address.ca_address_sk = web_returns.wr_refunded_addr_sk
>and reason.r_reason_sk = web_returns.wr_reason_sk
>and
>(
> (
> cd1.cd_marital_status = 'M'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = '4 yr Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 100.00 and 150.00
> )
>or
> (
> cd1.cd_marital_status = 'D'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Primary'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 50.00 and 100.00
> )
>or
> (
> cd1.cd_marital_status = 'U'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Advanced Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 150.00 and 200.00
> )
>)
>and
>(
> (
> ca_country = 'United States'
> and
> ca_state in ('KY', 'GA', 'NM')
> and ws_net_profit between 100 and 200
> )
> or
> (
> ca_country = 'United States'
> and
> ca_state in ('MT', 'OR', 'IN')
> and ws_net_profit between 150 and 300
> )
> or
> (
> ca_country = 'United States'
> and
> ca_state in ('WI', 'MO', 'WV')
> and ws_net_profit between 50 and 250
> )
>)
> group by r_reason_desc
> order by r, wq, ref, fee
> limit 100
> OK
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> Edges:
> Map 9 <- Map 1 (BROADCAST_EDGE)
> Reducer 3 <- Map 13 (SIMPLE_EDGE), Map 2 (SIMPLE_EDGE)
> Reducer 4 <- Map 9 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Map 14 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Map 10 (SIMPLE_EDGE), Map 11 (BROADCAST_EDGE), Map 12
> (BROADCAST_EDGE), Reducer 5 (SIMPLE_EDGE)
> Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
> Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
> DagName: mmokhtar_2014161818_f5fd23ba-d783-4b13-8507-7faa65851798:1
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: web_page
> filterExpr: wp_web_page_sk is not null (type: boolean)
> Statistics: Num row
[jira] [Commented] (HIVE-10864) CBO (Calcite Return Path): auto_join2.q returning wrong results
[ https://issues.apache.org/jira/browse/HIVE-10864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566105#comment-14566105 ] Jesus Camacho Rodriguez commented on HIVE-10864: The two test fails are unrelated. > CBO (Calcite Return Path): auto_join2.q returning wrong results > --- > > Key: HIVE-10864 > URL: https://issues.apache.org/jira/browse/HIVE-10864 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-10864.patch > > > auto_join2.q returns wrong results when return path is on. The problem is > that we create the same join expression once per input reference when we are > translating. Thus, we incorrectly end up with a key composed by multiple > expressions in those cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566126#comment-14566126 ] Vaibhav Gumashta commented on HIVE-10835: - I'll commit this shortly. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566141#comment-14566141
]
Hive QA commented on HIVE-10802:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736233/HIVE-10802.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8988 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_in_file
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4107/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4107/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4107/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736233 - PreCommit-HIVE-TRUNK-Build
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566165#comment-14566165
]
Ashutosh Chauhan commented on HIVE-10802:
-
[~aihuaxu] Can you create a RB entry for this ?
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10528) Hiveserver2 in HTTP mode is not applying auth_to_local rules
[
https://issues.apache.org/jira/browse/HIVE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-10528:
Fix Version/s: 1.2.1
> Hiveserver2 in HTTP mode is not applying auth_to_local rules
>
>
> Key: HIVE-10528
> URL: https://issues.apache.org/jira/browse/HIVE-10528
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.0.0, 1.2.0, 1.1.0, 1.3.0
> Environment: Centos 6
>Reporter: Abdelrahman Shettia
>Assignee: Abdelrahman Shettia
> Fix For: 1.2.1
>
> Attachments: HIVE-10528.1.patch, HIVE-10528.1.patch,
> HIVE-10528.2.patch, HIVE-10528.3.patch, REPRO-10528.txt
>
>
> PROBLEM: Authenticating to HS2 in HTTP mode with Kerberos, auth_to_local
> mappings do not get applied. Because of this various permissions checks
> which rely on the local cluster name for a user are going to fail.
> STEPS TO REPRODUCE:
> 1. Create kerberos cluster and HS2 in HTTP mode
> 2. Create a new user, test, along with a kerberos principal for this user
> 3. Create a separate principal, mapped-test
> 4. Create an auth_to_local rule to make sure that mapped-test is mapped to
> test
> 5. As the test user, connect to HS2 with beeline and create a simple table:
> {code}
> CREATE TABLE permtest (field1 int);
> {code}
> There is no need to load anything into this table.
> 6. Establish that it works as the test user:
> {code}
> show create table permtest;
> {code}
> 7. Drop the test identity and become mapped-test
> 8. Re-connect to HS2 with beeline, re-run the above command:
> {code}
> show create table permtest;
> {code}
> You will find that when this is done in HTTP mode, you will get an HDFS error
> (because of StorageBasedAuthorization doing a HDFS permissions check) and the
> user will be mapped-test and NOT test as it should be.
> ANALYSIS: This appears to be HTTP specific and the problem seems to come in
> {{ThriftHttpServlet$HttpKerberosServerAction.getPrincipalWithoutRealmAndHost()}}:
> {code}
> try {
> fullKerberosName =
> ShimLoader.getHadoopShims().getKerberosNameShim(fullPrincipal);
> } catch (IOException e) {
> throw new HttpAuthenticationException(e);
> }
> return fullKerberosName.getServiceName();
> {code}
> getServiceName applies no auth_to_local rules. Seems like maybe this should
> be getShortName()?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10835: Fix Version/s: 1.3.0 > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 1.3.0 > > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10835: Affects Version/s: 1.1.1 1.0.1 1.1.0 0.14.1 0.15.0 0.13.0 0.14.0 0.13.1 1.0.0 0.13 > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1, 1.0.0, 1.2.0, > 1.1.0, 1.0.1, 1.1.1, 0.13 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 1.3.0 > > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10834) Support First_value()/last_value() over x preceding and y preceding windowing
[
https://issues.apache.org/jira/browse/HIVE-10834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566191#comment-14566191
]
Hive QA commented on HIVE-10834:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736237/HIVE-10834.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8987 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4108/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4108/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4108/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736237 - PreCommit-HIVE-TRUNK-Build
> Support First_value()/last_value() over x preceding and y preceding windowing
> -
>
> Key: HIVE-10834
> URL: https://issues.apache.org/jira/browse/HIVE-10834
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10834.patch
>
>
> Currently the following query
> {noformat}
> select ts, f, first_value(f) over (partition by ts order by t rows between 2
> preceding and 1 preceding) from over10k limit 100;
> {noformat}
> throws exception:
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"2013-03-01
> 09:11:58.703071","reducesinkkey1":-3},"value":{"_col3":0.83}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0) {"key":{"reducesinkkey0":"2013-03-01
> 09:11:58.703071","reducesinkkey1":-3},"value":{"_col3":0.83}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error:
> cannot generate all output rows for a Partition
> at
> org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:519)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:114)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7810) Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-7810:
-
Labels: TODOC-SPARK TODOC15 (was: TODOC-SPARK)
> Insert overwrite table query has strange behavior when set
> hive.optimize.union.remove=true [Spark Branch]
> -
>
> Key: HIVE-7810
> URL: https://issues.apache.org/jira/browse/HIVE-7810
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
> Labels: TODOC-SPARK, TODOC15
> Fix For: 1.1.0
>
> Attachments: HIVE-7810.1-spark.patch
>
>
> Insert overwrite table query has strange behavior when
> set hive.optimize.union.remove=true
> set hive.mapred.supports.subdirectories=true;
> set hive.merge.mapfiles=true;
> set hive.merge.mapredfiles=true;
> We expect the following two sets of queries return the same set of data
> result, but they do not.
> 1)
> {noformat}
> insert overwrite table outputTbl1
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b;
> select * from outputTbl1 order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 2
> 2 1
> 2 2
> 3 1
> 3 2
> 7 1
> 7 2
> 8 2
> 8 2
> 8 2
> {noformat}
> 2)
> {noformat}
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 1
> 1 2
> 2 1
> 2 1
> 2 2
> 3 1
> 3 1
> 3 2
> 7 1
> 7 1
> 7 2
> 8 1
> 8 1
> 8 2
> 8 2
> 8 2
> {noformat}
> Some data is missing in the first set of query result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566205#comment-14566205 ] Chaoyu Tang commented on HIVE-10835: Thanks [~vgumashta] for reviewing and committing the patch. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1, 1.0.0, 1.2.0, > 1.1.0, 1.0.1, 1.1.1, 0.13 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 1.3.0 > > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7810) Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566207#comment-14566207
]
Lefty Leverenz commented on HIVE-7810:
--
Adding TODOC15 (which means TODOC1.1.0).
Besides documenting *hive.merge.sparkfiles* in Configuration Properties, usage
notes should be included in the HoS doc. Also see HIVE-8043, Support merging
small files.
* [Hive on Spark: Getting Started |
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started]
* [Configuration Properties -- Spark |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark]
with crossreferences to & from:
** [hive.merge.mapfiles |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.merge.mapfiles]
** [hive.merge.mapredfiles |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.merge.mapredfiles]
** and maybe [hive.optimize.union.remove |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.union.remove]
(see following question)
Does *hive.merge.sparkfiles* affect *hive.optimize.union.remove* like
*hive.merge.mapfiles* and *hive.merge.mapredfiles*?
bq. The merge is triggered if either of hive.merge.mapfiles or
hive.merge.mapredfiles is set to true. If the user has set hive.merge.mapfiles
to true and hive.merge.mapredfiles to false, the idea was that the number of
reducers are few, so the number of files anyway is small. However, with this
optimization, we are increasing the number of files possibly by a big margin.
So, we merge aggresively.
> Insert overwrite table query has strange behavior when set
> hive.optimize.union.remove=true [Spark Branch]
> -
>
> Key: HIVE-7810
> URL: https://issues.apache.org/jira/browse/HIVE-7810
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
> Labels: TODOC-SPARK, TODOC15
> Fix For: 1.1.0
>
> Attachments: HIVE-7810.1-spark.patch
>
>
> Insert overwrite table query has strange behavior when
> set hive.optimize.union.remove=true
> set hive.mapred.supports.subdirectories=true;
> set hive.merge.mapfiles=true;
> set hive.merge.mapredfiles=true;
> We expect the following two sets of queries return the same set of data
> result, but they do not.
> 1)
> {noformat}
> insert overwrite table outputTbl1
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b;
> select * from outputTbl1 order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 2
> 2 1
> 2 2
> 3 1
> 3 2
> 7 1
> 7 2
> 8 2
> 8 2
> 8 2
> {noformat}
> 2)
> {noformat}
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 1
> 1 2
> 2 1
> 2 1
> 2 2
> 3 1
> 3 1
> 3 2
> 7 1
> 7 1
> 7 2
> 8 1
> 8 1
> 8 2
> 8 2
> 8 2
> {noformat}
> Some data is missing in the first set of query result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8043) Support merging small files [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566210#comment-14566210
]
Lefty Leverenz commented on HIVE-8043:
--
Thanks [~lirui], I added a doc comment on HIVE-7810.
> Support merging small files [Spark Branch]
> --
>
> Key: HIVE-8043
> URL: https://issues.apache.org/jira/browse/HIVE-8043
> Project: Hive
> Issue Type: Task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Rui Li
> Labels: Spark-M1
> Fix For: 1.1.0
>
> Attachments: HIVE-8043.1-spark.patch, HIVE-8043.2-spark.patch,
> HIVE-8043.3-spark.patch
>
>
> Hive currently supports merging small files with MR as the execution engine.
> There are options available for this, such as
> {code}
> hive.merge.mapfiles
> hive.merge.mapredfiles
> {code}
> Hive.merge.sparkfiles is already introduced in HIVE-7810. To make it work, we
> might need a little more research and design on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566212#comment-14566212
]
Aihua Xu commented on HIVE-10802:
-
Thanks Ashutosh. I created RB above. I'm checking if the test failures are
related.
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7810) Insert overwrite table query has strange behavior when set hive.optimize.union.remove=true [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566213#comment-14566213
]
Lefty Leverenz commented on HIVE-7810:
--
ALTER TABLE ... CONCATENATE might need additional documentation too -- see
[~lirui]'s comment on HIVE-8043.
* [DDL -- Alter Table/Partition Concatenate |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate]
* [Rui Li's comment |
https://issues.apache.org/jira/browse/HIVE-8043?focusedCommentId=14138824&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14138824]
> Insert overwrite table query has strange behavior when set
> hive.optimize.union.remove=true [Spark Branch]
> -
>
> Key: HIVE-7810
> URL: https://issues.apache.org/jira/browse/HIVE-7810
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
> Labels: TODOC-SPARK, TODOC15
> Fix For: 1.1.0
>
> Attachments: HIVE-7810.1-spark.patch
>
>
> Insert overwrite table query has strange behavior when
> set hive.optimize.union.remove=true
> set hive.mapred.supports.subdirectories=true;
> set hive.merge.mapfiles=true;
> set hive.merge.mapredfiles=true;
> We expect the following two sets of queries return the same set of data
> result, but they do not.
> 1)
> {noformat}
> insert overwrite table outputTbl1
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b;
> select * from outputTbl1 order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 2
> 2 1
> 2 2
> 3 1
> 3 2
> 7 1
> 7 2
> 8 2
> 8 2
> 8 2
> {noformat}
> 2)
> {noformat}
> SELECT * FROM
> (
> select key, 1 as values from inputTbl1
> union all
> select * FROM (
> SELECT key, count(1) as values from inputTbl1 group by key
> UNION ALL
> SELECT key, 2 as values from inputTbl1
> ) a
> )b order by key, values;
> {noformat}
> Below is the query result:
> {noformat}
> 1 1
> 1 1
> 1 2
> 2 1
> 2 1
> 2 2
> 3 1
> 3 1
> 3 2
> 7 1
> 7 1
> 7 2
> 8 1
> 8 1
> 8 2
> 8 2
> 8 2
> {noformat}
> Some data is missing in the first set of query result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566214#comment-14566214
]
Aihua Xu commented on HIVE-10802:
-
3 tests are related to my change and my patch will make the query more
efficient, such as evaluating "UDFToString(null)" to be "null" in compile time.
I will make the change to the unit test baseline.
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10834) Support First_value()/last_value() over x preceding and y preceding windowing
[
https://issues.apache.org/jira/browse/HIVE-10834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566215#comment-14566215
]
Aihua Xu commented on HIVE-10834:
-
The test failures seem unrelated to my change.
> Support First_value()/last_value() over x preceding and y preceding windowing
> -
>
> Key: HIVE-10834
> URL: https://issues.apache.org/jira/browse/HIVE-10834
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10834.patch
>
>
> Currently the following query
> {noformat}
> select ts, f, first_value(f) over (partition by ts order by t rows between 2
> preceding and 1 preceding) from over10k limit 100;
> {noformat}
> throws exception:
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"2013-03-01
> 09:11:58.703071","reducesinkkey1":-3},"value":{"_col3":0.83}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0) {"key":{"reducesinkkey0":"2013-03-01
> 09:11:58.703071","reducesinkkey1":-3},"value":{"_col3":0.83}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error:
> cannot generate all output rows for a Partition
> at
> org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:519)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:114)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10678) update sql standard authorization configuration whitelist - more optimization flags
[ https://issues.apache.org/jira/browse/HIVE-10678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566224#comment-14566224 ] Lefty Leverenz commented on HIVE-10678: --- Doc note: Updated the description of *hive.security.authorization.sqlstd.confwhitelist* in the wiki to include this jira. * [Configuration Properties -- hive.security.authorization.sqlstd.confwhitelist | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.authorization.sqlstd.confwhitelist] > update sql standard authorization configuration whitelist - more optimization > flags > --- > > Key: HIVE-10678 > URL: https://issues.apache.org/jira/browse/HIVE-10678 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.2.1 > > Attachments: HIVE-10678.1.patch, HIVE-10678.2.patch > > > hive.exec.parallel and hive.groupby.orderby.position.alias are optimization > config parameters that should be settable when sql standard authorization is > enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-10802:
Attachment: HIVE-10802.patch
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch, HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-10802:
Attachment: (was: HIVE-10802.patch)
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10788) Change sort_array to support non-primitive types
[
https://issues.apache.org/jira/browse/HIVE-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566226#comment-14566226
]
Lefty Leverenz commented on HIVE-10788:
---
Does this need documentation?
* [Hive Operators and UDFs -- Collection Functions |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-CollectionFunctions]
> Change sort_array to support non-primitive types
>
>
> Key: HIVE-10788
> URL: https://issues.apache.org/jira/browse/HIVE-10788
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 1.3.0
>
> Attachments: HIVE-10788.1.patch, HIVE-10788.2.patch,
> HIVE-10788.3.patch
>
>
> Currently {{sort_array}} only support primitive types. As we already support
> comparison between non-primitive types, it makes sense to remove this
> restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10722) external table creation with msck in Hive can create unusable partition
[ https://issues.apache.org/jira/browse/HIVE-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-10722: -- Labels: TODOC1.3 (was: ) > external table creation with msck in Hive can create unusable partition > --- > > Key: HIVE-10722 > URL: https://issues.apache.org/jira/browse/HIVE-10722 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.1, 1.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Labels: TODOC1.3 > Fix For: 1.3.0 > > Attachments: HIVE-10722.01.patch, HIVE-10722.patch > > > There can be directories in HDFS containing unprintable characters; when > doing hadoop fs -ls, these characters are not even visible, and can only be > seen for example if output is piped thru od. > When these are loaded via msck, they are stored in e.g. mysql as "?" (literal > question mark, findable via LIKE '%?%' in db) and show accordingly in Hive. > However, datanucleus appears to encode it as %3F; this causes the partition > to be unusable - it cannot be dropped, and other operations like drop table > get stuck (didn't investigate in detail why; drop table got unstuck as soon > as the partition was removed from metastore). > We should probably have a 2-way option for such cases - error out on load > (default), or convert to '?'/drop such characters (and have partition that > actually works, too). > We should also check if partitions with '?' inserted explicitly work at all > with datanucleus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10722) external table creation with msck in Hive can create unusable partition
[ https://issues.apache.org/jira/browse/HIVE-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566238#comment-14566238 ] Lefty Leverenz commented on HIVE-10722: --- Doc note: *hive.msck.path.validation* needs to be documented in Configuration Properties. Should it go in the query & DDL execution section or the metastore section? * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] * [Configuration Properties -- MetaStore | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] Where does usage information belong? * [LanguageManual -- DML | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML] * [LanguageManual -- DDL -- Recover Partitions (MSCK REPAIR TABLE) | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)] > external table creation with msck in Hive can create unusable partition > --- > > Key: HIVE-10722 > URL: https://issues.apache.org/jira/browse/HIVE-10722 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.1, 1.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Labels: TODOC1.3 > Fix For: 1.3.0 > > Attachments: HIVE-10722.01.patch, HIVE-10722.patch > > > There can be directories in HDFS containing unprintable characters; when > doing hadoop fs -ls, these characters are not even visible, and can only be > seen for example if output is piped thru od. > When these are loaded via msck, they are stored in e.g. mysql as "?" (literal > question mark, findable via LIKE '%?%' in db) and show accordingly in Hive. > However, datanucleus appears to encode it as %3F; this causes the partition > to be unusable - it cannot be dropped, and other operations like drop table > get stuck (didn't investigate in detail why; drop table got unstuck as soon > as the partition was removed from metastore). > We should probably have a 2-way option for such cases - error out on load > (default), or convert to '?'/drop such characters (and have partition that > actually works, too). > We should also check if partitions with '?' inserted explicitly work at all > with datanucleus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10841) [WHERE col is not null] does not work sometimes for queries with many JOIN statements
[
https://issues.apache.org/jira/browse/HIVE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566239#comment-14566239
]
Alexander Pivovarov commented on HIVE-10841:
disabling hive.ppd.remove.duplicatefilters helps
{code}
set hive.ppd.remove.duplicatefilters=false;
{code}
> [WHERE col is not null] does not work sometimes for queries with many JOIN
> statements
> -
>
> Key: HIVE-10841
> URL: https://issues.apache.org/jira/browse/HIVE-10841
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Query Processor
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.2.0
>Reporter: Alexander Pivovarov
>
> The result from the following SELECT query is 3 rows but it should be 1 row.
> I checked it in MySQL - it returned 1 row.
> To reproduce the issue in Hive
> 1. prepare tables
> {code}
> drop table if exists L;
> drop table if exists LA;
> drop table if exists FR;
> drop table if exists A;
> drop table if exists PI;
> drop table if exists acct;
> create table L as select 4436 id;
> create table LA as select 4436 loan_id, 4748 aid, 4415 pi_id;
> create table FR as select 4436 loan_id;
> create table A as select 4748 id;
> create table PI as select 4415 id;
> create table acct as select 4748 aid, 10 acc_n, 122 brn;
> insert into table acct values(4748, null, null);
> insert into table acct values(4748, null, null);
> {code}
> 2. run SELECT query
> {code}
> select
> acct.ACC_N,
> acct.brn
> FROM L
> JOIN LA ON L.id = LA.loan_id
> JOIN FR ON L.id = FR.loan_id
> JOIN A ON LA.aid = A.id
> JOIN PI ON PI.id = LA.pi_id
> JOIN acct ON A.id = acct.aid
> WHERE
> L.id = 4436
> and acct.brn is not null;
> {code}
> the result is 3 rows
> {code}
> 10122
> NULL NULL
> NULL NULL
> {code}
> but it should be 1 row
> {code}
> 10122
> {code}
> 2.1 "explain select ..." output for hive-1.3.0 MR
> {code}
> STAGE DEPENDENCIES:
> Stage-12 is a root stage
> Stage-9 depends on stages: Stage-12
> Stage-0 depends on stages: Stage-9
> STAGE PLANS:
> Stage: Stage-12
> Map Reduce Local Work
> Alias -> Map Local Tables:
> a
> Fetch Operator
> limit: -1
> acct
> Fetch Operator
> limit: -1
> fr
> Fetch Operator
> limit: -1
> l
> Fetch Operator
> limit: -1
> pi
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> a
> TableScan
> alias: a
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> acct
> TableScan
> alias: acct
> Statistics: Num rows: 3 Data size: 31 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: aid is not null (type: boolean)
> Statistics: Num rows: 2 Data size: 20 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> fr
> TableScan
> alias: fr
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (loan_id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> l
> TableScan
> alias: l
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> pi
> TableScan
> alias: pi
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: b
[jira] [Commented] (HIVE-10826) Support min()/max() functions over x preceding and y preceding windowing
[
https://issues.apache.org/jira/browse/HIVE-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566242#comment-14566242
]
Lefty Leverenz commented on HIVE-10826:
---
Does this need documentation?
* [Windowing and Analytics |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics]
> Support min()/max() functions over x preceding and y preceding windowing
> -
>
> Key: HIVE-10826
> URL: https://issues.apache.org/jira/browse/HIVE-10826
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10826.patch
>
>
> Currently the query
> {noformat}
> select key, value, min(value) over (partition by key order by value rows
> between 1 preceding and 1 preceding) from small;
> {noformat}
> doesn't work. It failed with
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"2"},"value":{"_col0":"500"}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":"2"},"value":{"_col0":"500"}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error:
> cannot generate all output rows for a Partition
> at
> org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:520)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
> at
> org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:114)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10843) desc database and show tables commands don't pass db to HiveAuthorizer check
[
https://issues.apache.org/jira/browse/HIVE-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566265#comment-14566265
]
Hive QA commented on HIVE-10843:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736252/HIVE-10843.2.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8988 tests executed
*Failed tests:*
{noformat}
TestHs2Hooks - did not produce a TEST-*.xml file
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4109/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4109/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4109/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736252 - PreCommit-HIVE-TRUNK-Build
> desc database and show tables commands don't pass db to HiveAuthorizer check
>
>
> Key: HIVE-10843
> URL: https://issues.apache.org/jira/browse/HIVE-10843
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-10843.1.patch, HIVE-10843.2.patch
>
>
> 'show tables' and 'describe database' command should pass the database
> information for the command to HiveAuthorizer . This is needed for any
> auditing the hive authorizer might implement, or any authorization check it
> might decide to do based on the given database name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566274#comment-14566274
]
Ashutosh Chauhan commented on HIVE-10802:
-
+1
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-686) add UDF substring_index
[
https://issues.apache.org/jira/browse/HIVE-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566292#comment-14566292
]
Hive QA commented on HIVE-686:
--
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736253/HIVE-686.1.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8989 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4110/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4110/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4110/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736253 - PreCommit-HIVE-TRUNK-Build
> add UDF substring_index
> ---
>
> Key: HIVE-686
> URL: https://issues.apache.org/jira/browse/HIVE-686
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Reporter: Namit Jain
>Assignee: Alexander Pivovarov
> Attachments: HIVE-686.1.patch, HIVE-686.1.patch, HIVE-686.patch,
> HIVE-686.patch
>
>
> SUBSTRING_INDEX(str,delim,count)
> Returns the substring from string str before count occurrences of the
> delimiter delim. If count is positive, everything to the left of the final
> delimiter (counting from the left) is returned. If count is negative,
> everything to the right of the final delimiter (counting from the right) is
> returned. SUBSTRING_INDEX() performs a case-sensitive match when searching
> for delim.
> Examples:
> {code}
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 3);
> --www.mysql.com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 2);
> --www.mysql
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 1);
> --www
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 0);
> --''
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -1);
> --com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -2);
> --mysql.com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -3);
> --www.mysql.com
> {code}
> {code}
> --#delim does not exist in str
> SELECT SUBSTRING_INDEX('www.mysql.com', 'Q', 1);
> --www.mysql.com
> --#delim is 2 chars
> SELECT SUBSTRING_INDEX('www||mysql||com', '||', 2);
> --www||mysql
> --#delim is empty string
> SELECT SUBSTRING_INDEX('www.mysql.com', '', 2);
> --''
> --#str is empty string
> SELECT SUBSTRING_INDEX('', '.', 2);
> --''
> {code}
> {code}
> --#null params
> SELECT SUBSTRING_INDEX(null, '.', 1);
> --null
> SELECT SUBSTRING_INDEX('www.mysql.com', null, 1);
> --null
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', null);
> --null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10857) Accumulo storage handler fail throwing java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token class: class org.apache.accumulo.core.client.security
[
https://issues.apache.org/jira/browse/HIVE-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HIVE-10857:
--
Attachment: HIVE-10857.patch
Fixes the AccumuloStorageHandler to work with Kerberos. Lots of reflection to
make sure that older versions of Accumulo continue to work with the storage
handler as the Kerberos support is only in the most recent version.
Some basic unit testing, lots of testing on a secured cluster.
> Accumulo storage handler fail throwing java.lang.IllegalArgumentException:
> Cannot determine SASL mechanism for token class: class
> org.apache.accumulo.core.client.security.tokens.PasswordToken
> ---
>
> Key: HIVE-10857
> URL: https://issues.apache.org/jira/browse/HIVE-10857
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>Assignee: Josh Elser
> Fix For: 1.2.1
>
> Attachments: HIVE-10857.patch
>
>
> create table Accumulo storage with Accumulo storage handler fails due to
> ACCUMULO-2815.
> {noformat}
> create table accumulo_1(key string, age int) stored by
> 'org.apache.hadoop.hive.accumulo.AccumuloStorageHandler' with serdeproperties
> ( "accumulo.columns.mapping" = ":rowid,info:age");
> {noformat}
> The error shows:
> {noformat}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:org.apache.accumulo.core.client.AccumuloException:
> java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token
> class: class org.apache.accumulo.core.client.security.tokens.PasswordToken
> at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:67)
> at
> org.apache.accumulo.core.client.impl.ConnectorImpl.(ConnectorImpl.java:67)
> at
> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:248)
> at
> org.apache.hadoop.hive.accumulo.AccumuloConnectionParameters.getConnector(AccumuloConnectionParameters.java:125)
> at
> org.apache.hadoop.hive.accumulo.AccumuloConnectionParameters.getConnector(AccumuloConnectionParameters.java:111)
> at
> org.apache.hadoop.hive.accumulo.AccumuloStorageHandler.preCreateTable(AccumuloStorageHandler.java:245)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:664)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:657)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy5.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:714)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4135)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
[jira] [Updated] (HIVE-9842) Enable session/operation timeout by default in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-9842: --- Attachment: HIVE-9842.4.patch Patch 4 sets session timeout to 7 days to account for scenarios like long weekends, sets operation timeout to a day and doesn't let session timeout kick in till all operation handles are not cleaned (either due to operation completing, or removed due to timeout). [~thejas] do these defaults look good? > Enable session/operation timeout by default in HiveServer2 > -- > > Key: HIVE-9842 > URL: https://issues.apache.org/jira/browse/HIVE-9842 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-9842.1.patch, HIVE-9842.2.patch, HIVE-9842.3.patch, > HIVE-9842.4.patch > > > HIVE-5799 introduced a session/operation timeout which cleans up abandoned > session and op handles. Currently, the default is set to no-op. We should set > it to some reasonable value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10728) deprecate unix_timestamp(void) and make it deterministic
[
https://issues.apache.org/jira/browse/HIVE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566319#comment-14566319
]
Hive QA commented on HIVE-10728:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736290/HIVE-10728.03.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 8986 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4111/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4111/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4111/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736290 - PreCommit-HIVE-TRUNK-Build
> deprecate unix_timestamp(void) and make it deterministic
>
>
> Key: HIVE-10728
> URL: https://issues.apache.org/jira/browse/HIVE-10728
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10728.01.patch, HIVE-10728.02.patch,
> HIVE-10728.03.patch, HIVE-10728.patch
>
>
> We have a proper current_timestamp function that is not evaluated at runtime.
> Behavior of unix_timestamp(void) is both surprising, and is preventing some
> optimizations on the other overload since the function becomes
> non-deterministic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9842) Enable session/operation timeout by default in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-9842: --- Attachment: HIVE-9842.5.patch After some discussion with [~thejas] changing the default op timeout to +5 days. This implementation will need some improvement though as it decouples session/operations wrt timeout and does not update op timeout when checking for op status; will create a followup. > Enable session/operation timeout by default in HiveServer2 > -- > > Key: HIVE-9842 > URL: https://issues.apache.org/jira/browse/HIVE-9842 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-9842.1.patch, HIVE-9842.2.patch, HIVE-9842.3.patch, > HIVE-9842.4.patch, HIVE-9842.5.patch > > > HIVE-5799 introduced a session/operation timeout which cleans up abandoned > session and op handles. Currently, the default is set to no-op. We should set > it to some reasonable value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9842) Enable session/operation timeout by default in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-9842: --- Attachment: HIVE-9842.5.patch > Enable session/operation timeout by default in HiveServer2 > -- > > Key: HIVE-9842 > URL: https://issues.apache.org/jira/browse/HIVE-9842 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-9842.1.patch, HIVE-9842.2.patch, HIVE-9842.3.patch, > HIVE-9842.4.patch, HIVE-9842.5.patch, HIVE-9842.5.patch > > > HIVE-5799 introduced a session/operation timeout which cleans up abandoned > session and op handles. Currently, the default is set to no-op. We should set > it to some reasonable value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9842) Enable session/operation timeout by default in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-9842: --- Attachment: (was: HIVE-9842.5.patch) > Enable session/operation timeout by default in HiveServer2 > -- > > Key: HIVE-9842 > URL: https://issues.apache.org/jira/browse/HIVE-9842 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-9842.1.patch, HIVE-9842.2.patch, HIVE-9842.3.patch, > HIVE-9842.4.patch, HIVE-9842.5.patch > > > HIVE-5799 introduced a session/operation timeout which cleans up abandoned > session and op handles. Currently, the default is set to no-op. We should set > it to some reasonable value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9842) Enable session/operation timeout by default in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566338#comment-14566338 ] Thejas M Nair commented on HIVE-9842: - +1 > Enable session/operation timeout by default in HiveServer2 > -- > > Key: HIVE-9842 > URL: https://issues.apache.org/jira/browse/HIVE-9842 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-9842.1.patch, HIVE-9842.2.patch, HIVE-9842.3.patch, > HIVE-9842.4.patch, HIVE-9842.5.patch > > > HIVE-5799 introduced a session/operation timeout which cleans up abandoned > session and op handles. Currently, the default is set to no-op. We should set > it to some reasonable value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9069) Simplify filter predicates for CBO
[
https://issues.apache.org/jira/browse/HIVE-9069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566350#comment-14566350
]
Hive QA commented on HIVE-9069:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736350/HIVE-9069.18.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 8987 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4112/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4112/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4112/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736350 - PreCommit-HIVE-TRUNK-Build
> Simplify filter predicates for CBO
> --
>
> Key: HIVE-9069
> URL: https://issues.apache.org/jira/browse/HIVE-9069
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Jesus Camacho Rodriguez
> Fix For: 0.14.1
>
> Attachments: HIVE-9069.01.patch, HIVE-9069.02.patch,
> HIVE-9069.03.patch, HIVE-9069.04.patch, HIVE-9069.05.patch,
> HIVE-9069.06.patch, HIVE-9069.07.patch, HIVE-9069.08.patch,
> HIVE-9069.08.patch, HIVE-9069.09.patch, HIVE-9069.10.patch,
> HIVE-9069.11.patch, HIVE-9069.12.patch, HIVE-9069.13.patch,
> HIVE-9069.14.patch, HIVE-9069.14.patch, HIVE-9069.15.patch,
> HIVE-9069.16.patch, HIVE-9069.17.patch, HIVE-9069.17.patch,
> HIVE-9069.18.patch, HIVE-9069.18.patch, HIVE-9069.patch
>
>
> Simplify predicates for disjunctive predicates so that can get pushed down to
> the scan.
> Looks like this is still an issue, some of the filters can be pushed down to
> the scan.
> {code}
> set hive.cbo.enable=true
> set hive.stats.fetch.column.stats=true
> set hive.exec.dynamic.partition.mode=nonstrict
> set hive.tez.auto.reducer.parallelism=true
> set hive.auto.convert.join.noconditionaltask.size=32000
> set hive.exec.reducers.bytes.per.reducer=1
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
> set hive.support.concurrency=false
> set hive.tez.exec.print.summary=true
> explain
> select substr(r_reason_desc,1,20) as r
>,avg(ws_quantity) wq
>,avg(wr_refunded_cash) ref
>,avg(wr_fee) fee
> from web_sales, web_returns, web_page, customer_demographics cd1,
> customer_demographics cd2, customer_address, date_dim, reason
> where web_sales.ws_web_page_sk = web_page.wp_web_page_sk
>and web_sales.ws_item_sk = web_returns.wr_item_sk
>and web_sales.ws_order_number = web_returns.wr_order_number
>and web_sales.ws_sold_date_sk = date_dim.d_date_sk and d_year = 1998
>and cd1.cd_demo_sk = web_returns.wr_refunded_cdemo_sk
>and cd2.cd_demo_sk = web_returns.wr_returning_cdemo_sk
>and customer_address.ca_address_sk = web_returns.wr_refunded_addr_sk
>and reason.r_reason_sk = web_returns.wr_reason_sk
>and
>(
> (
> cd1.cd_marital_status = 'M'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = '4 yr Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 100.00 and 150.00
> )
>or
> (
> cd1.cd_marital_status = 'D'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Primary'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 50.00 and 100.00
> )
>or
> (
> cd1.cd_marital_status = 'U'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Advanced Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 150.00 and 200.00
> )
>)
>and
>(
> (
> ca_country = 'United States'
> and
> ca_state in ('KY', 'GA', 'NM')
> and ws_net_profit between 100 and 200
> )
> or
>
[jira] [Updated] (HIVE-10874) Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to duplicate column name
[
https://issues.apache.org/jira/browse/HIVE-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-10874:
---
Attachment: HIVE-10874.patch
> Fail in TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q due to
> duplicate column name
> -
>
> Key: HIVE-10874
> URL: https://issues.apache.org/jira/browse/HIVE-10874
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10874.patch
>
>
> Aggregate operators may derive row types with duplicate column names. The
> reason is that the column names for grouping sets columns and aggregation
> columns might be generated automatically, but we do not check whether the
> column name already exists in the same row.
> This error can be reproduced by
> TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2.q, which fails
> with the following trace:
> {code}
> junit.framework.AssertionFailedError: Unexpected exception
> java.lang.AssertionError: RecordType(BIGINT $f1, BIGINT $f1)
> at org.apache.calcite.rel.core.Project.isValid(Project.java:200)
> at org.apache.calcite.rel.core.Project.(Project.java:85)
> at org.apache.calcite.rel.core.Project.(Project.java:91)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.(HiveProject.java:70)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.create(HiveProject.java:103)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.introduceDerivedTable(PlanModifierForASTConv.java:211)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.PlanModifierForASTConv.convertOpTree(PlanModifierForASTConv.java:67)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:94)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:617)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:248)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10108)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10761) Create codahale-based metrics system for Hive
[ https://issues.apache.org/jira/browse/HIVE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-10761: - Attachment: HIVE-10761.6.patch Minor change to make file-reporter more resilient against partial-files > Create codahale-based metrics system for Hive > - > > Key: HIVE-10761 > URL: https://issues.apache.org/jira/browse/HIVE-10761 > Project: Hive > Issue Type: New Feature > Components: Diagnosability >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-10761.2.patch, HIVE-10761.3.patch, > HIVE-10761.4.patch, HIVE-10761.5.patch, HIVE-10761.6.patch, HIVE-10761.patch, > hms-metrics.json > > > There is a current Hive metrics system that hooks up to a JMX reporting, but > all its measurements, models are custom. > This is to make another metrics system that will be based on Codahale (ie > yammer, dropwizard), which has the following advantage: > * Well-defined metric model for frequently-needed metrics (ie JVM metrics) > * Well-defined measurements for all metrics (ie max, mean, stddev, mean_rate, > etc), > * Built-in reporting frameworks like JMX, Console, Log, JSON webserver > It is used for many projects, including several Apache projects like Oozie. > Overall, monitoring tools should find it easier to understand these common > metric, measurement, reporting models. > The existing metric subsystem will be kept and can be enabled if backward > compatibility is desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10811) RelFieldTrimmer throws NoSuchElementException in some cases
[
https://issues.apache.org/jira/browse/HIVE-10811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566366#comment-14566366
]
Jesus Camacho Rodriguez commented on HIVE-10811:
Follow-up in HIVE-10874.
> RelFieldTrimmer throws NoSuchElementException in some cases
> ---
>
> Key: HIVE-10811
> URL: https://issues.apache.org/jira/browse/HIVE-10811
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.2.1
>
> Attachments: HIVE-10811.01.patch, HIVE-10811.02.patch,
> HIVE-10811.patch
>
>
> RelFieldTrimmer runs into NoSuchElementException in some cases.
> Stack trace:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Internal error: While
> invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> at org.apache.calcite.util.Util.newInternal(Util.java:743)
> at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> at
> org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> at
> org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:175)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:947)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:820)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:768)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:109)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:730)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:145)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:607)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:244)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10048)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:207)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> ... 32 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method
> 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
> org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> at org.apache.calcite.util.Util.newInternal(Util.java:743)
> at org.apache.calcite.util.ReflectUtil$2
[jira] [Commented] (HIVE-9069) Simplify filter predicates for CBO
[
https://issues.apache.org/jira/browse/HIVE-9069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566367#comment-14566367
]
Jesus Camacho Rodriguez commented on HIVE-9069:
---
[~jpullokkaran], fails are not related to the patch. Thanks
> Simplify filter predicates for CBO
> --
>
> Key: HIVE-9069
> URL: https://issues.apache.org/jira/browse/HIVE-9069
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Jesus Camacho Rodriguez
> Fix For: 0.14.1
>
> Attachments: HIVE-9069.01.patch, HIVE-9069.02.patch,
> HIVE-9069.03.patch, HIVE-9069.04.patch, HIVE-9069.05.patch,
> HIVE-9069.06.patch, HIVE-9069.07.patch, HIVE-9069.08.patch,
> HIVE-9069.08.patch, HIVE-9069.09.patch, HIVE-9069.10.patch,
> HIVE-9069.11.patch, HIVE-9069.12.patch, HIVE-9069.13.patch,
> HIVE-9069.14.patch, HIVE-9069.14.patch, HIVE-9069.15.patch,
> HIVE-9069.16.patch, HIVE-9069.17.patch, HIVE-9069.17.patch,
> HIVE-9069.18.patch, HIVE-9069.18.patch, HIVE-9069.patch
>
>
> Simplify predicates for disjunctive predicates so that can get pushed down to
> the scan.
> Looks like this is still an issue, some of the filters can be pushed down to
> the scan.
> {code}
> set hive.cbo.enable=true
> set hive.stats.fetch.column.stats=true
> set hive.exec.dynamic.partition.mode=nonstrict
> set hive.tez.auto.reducer.parallelism=true
> set hive.auto.convert.join.noconditionaltask.size=32000
> set hive.exec.reducers.bytes.per.reducer=1
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
> set hive.support.concurrency=false
> set hive.tez.exec.print.summary=true
> explain
> select substr(r_reason_desc,1,20) as r
>,avg(ws_quantity) wq
>,avg(wr_refunded_cash) ref
>,avg(wr_fee) fee
> from web_sales, web_returns, web_page, customer_demographics cd1,
> customer_demographics cd2, customer_address, date_dim, reason
> where web_sales.ws_web_page_sk = web_page.wp_web_page_sk
>and web_sales.ws_item_sk = web_returns.wr_item_sk
>and web_sales.ws_order_number = web_returns.wr_order_number
>and web_sales.ws_sold_date_sk = date_dim.d_date_sk and d_year = 1998
>and cd1.cd_demo_sk = web_returns.wr_refunded_cdemo_sk
>and cd2.cd_demo_sk = web_returns.wr_returning_cdemo_sk
>and customer_address.ca_address_sk = web_returns.wr_refunded_addr_sk
>and reason.r_reason_sk = web_returns.wr_reason_sk
>and
>(
> (
> cd1.cd_marital_status = 'M'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = '4 yr Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 100.00 and 150.00
> )
>or
> (
> cd1.cd_marital_status = 'D'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Primary'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 50.00 and 100.00
> )
>or
> (
> cd1.cd_marital_status = 'U'
> and
> cd1.cd_marital_status = cd2.cd_marital_status
> and
> cd1.cd_education_status = 'Advanced Degree'
> and
> cd1.cd_education_status = cd2.cd_education_status
> and
> ws_sales_price between 150.00 and 200.00
> )
>)
>and
>(
> (
> ca_country = 'United States'
> and
> ca_state in ('KY', 'GA', 'NM')
> and ws_net_profit between 100 and 200
> )
> or
> (
> ca_country = 'United States'
> and
> ca_state in ('MT', 'OR', 'IN')
> and ws_net_profit between 150 and 300
> )
> or
> (
> ca_country = 'United States'
> and
> ca_state in ('WI', 'MO', 'WV')
> and ws_net_profit between 50 and 250
> )
>)
> group by r_reason_desc
> order by r, wq, ref, fee
> limit 100
> OK
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> Edges:
> Map 9 <- Map 1 (BROADCAST_EDGE)
> Reducer 3 <- Map 13 (SIMPLE_EDGE), Map 2 (SIMPLE_EDGE)
> Reducer 4 <- Map 9 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
> Reducer 5 <- Map 14 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
> Reducer 6 <- Map 10 (SIMPLE_EDGE), Map 11 (BROADCAST_EDGE), Map 12
> (BROADCAST_EDGE), Reducer 5 (SIMPLE_EDGE)
> Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
> Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
> DagName: mmokhtar_2014161818_f5fd23ba-d783-4b13-8507-7faa65851798:1
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: web_page
> filterExpr:
[jira] [Updated] (HIVE-10843) desc database and show tables commands don't pass db to HiveAuthorizer check
[ https://issues.apache.org/jira/browse/HIVE-10843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10843: - Attachment: HIVE-10843.2.patch attaching file again for another test run, to get TestHs2Hooks tests to run. > desc database and show tables commands don't pass db to HiveAuthorizer check > > > Key: HIVE-10843 > URL: https://issues.apache.org/jira/browse/HIVE-10843 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-10843.1.patch, HIVE-10843.2.patch, > HIVE-10843.2.patch > > > 'show tables' and 'describe database' command should pass the database > information for the command to HiveAuthorizer . This is needed for any > auditing the hive authorizer might implement, or any authorization check it > might decide to do based on the given database name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10869) fold_case.q failing on trunk
[ https://issues.apache.org/jira/browse/HIVE-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566370#comment-14566370 ] Ashutosh Chauhan commented on HIVE-10869: - Test failures are unrelated. Patch is ready for review. > fold_case.q failing on trunk > > > Key: HIVE-10869 > URL: https://issues.apache.org/jira/browse/HIVE-10869 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 1.3.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-10869.patch > > > Race condition of commits between HIVE-10716 & HIVE-10812 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-686) add UDF substring_index
[
https://issues.apache.org/jira/browse/HIVE-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566375#comment-14566375
]
Alexander Pivovarov commented on HIVE-686:
--
Added the function description to the Language Manual
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions
> add UDF substring_index
> ---
>
> Key: HIVE-686
> URL: https://issues.apache.org/jira/browse/HIVE-686
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Reporter: Namit Jain
>Assignee: Alexander Pivovarov
> Fix For: 1.3.0
>
> Attachments: HIVE-686.1.patch, HIVE-686.1.patch, HIVE-686.patch,
> HIVE-686.patch
>
>
> SUBSTRING_INDEX(str,delim,count)
> Returns the substring from string str before count occurrences of the
> delimiter delim. If count is positive, everything to the left of the final
> delimiter (counting from the left) is returned. If count is negative,
> everything to the right of the final delimiter (counting from the right) is
> returned. SUBSTRING_INDEX() performs a case-sensitive match when searching
> for delim.
> Examples:
> {code}
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 3);
> --www.mysql.com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 2);
> --www.mysql
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 1);
> --www
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 0);
> --''
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -1);
> --com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -2);
> --mysql.com
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', -3);
> --www.mysql.com
> {code}
> {code}
> --#delim does not exist in str
> SELECT SUBSTRING_INDEX('www.mysql.com', 'Q', 1);
> --www.mysql.com
> --#delim is 2 chars
> SELECT SUBSTRING_INDEX('www||mysql||com', '||', 2);
> --www||mysql
> --#delim is empty string
> SELECT SUBSTRING_INDEX('www.mysql.com', '', 2);
> --''
> --#str is empty string
> SELECT SUBSTRING_INDEX('', '.', 2);
> --''
> {code}
> {code}
> --#null params
> SELECT SUBSTRING_INDEX(null, '.', 1);
> --null
> SELECT SUBSTRING_INDEX('www.mysql.com', null, 1);
> --null
> SELECT SUBSTRING_INDEX('www.mysql.com', '.', null);
> --null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10802) Table join query with some constant field in select fails
[
https://issues.apache.org/jira/browse/HIVE-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566392#comment-14566392
]
Hive QA commented on HIVE-10802:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12736375/HIVE-10802.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 8987 tests executed
*Failed tests:*
{noformat}
TestUtil - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4113/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4113/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4113/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12736375 - PreCommit-HIVE-TRUNK-Build
> Table join query with some constant field in select fails
> -
>
> Key: HIVE-10802
> URL: https://issues.apache.org/jira/browse/HIVE-10802
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 1.3.0
>
> Attachments: HIVE-10802.patch
>
>
> The following query fails:
> {noformat}
> create table tb1 (year string, month string);
> create table tb2(month string);
> select unix_timestamp(a.year)
> from (select * from tb1 where year='2001') a join tb2 b on (a.month=b.month);
> {noformat}
> with the exception {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:290)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:275)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:175)
> {noformat}
> The issue seems to be: during the query compilation, the field in the select
> should be replaced with the constant when some UDFs are used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-10841) [WHERE col is not null] does not work sometimes for queries with many JOIN statements
[
https://issues.apache.org/jira/browse/HIVE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov reassigned HIVE-10841:
--
Assignee: Alexander Pivovarov
> [WHERE col is not null] does not work sometimes for queries with many JOIN
> statements
> -
>
> Key: HIVE-10841
> URL: https://issues.apache.org/jira/browse/HIVE-10841
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Query Processor
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.2.0
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>
> The result from the following SELECT query is 3 rows but it should be 1 row.
> I checked it in MySQL - it returned 1 row.
> To reproduce the issue in Hive
> 1. prepare tables
> {code}
> drop table if exists L;
> drop table if exists LA;
> drop table if exists FR;
> drop table if exists A;
> drop table if exists PI;
> drop table if exists acct;
> create table L as select 4436 id;
> create table LA as select 4436 loan_id, 4748 aid, 4415 pi_id;
> create table FR as select 4436 loan_id;
> create table A as select 4748 id;
> create table PI as select 4415 id;
> create table acct as select 4748 aid, 10 acc_n, 122 brn;
> insert into table acct values(4748, null, null);
> insert into table acct values(4748, null, null);
> {code}
> 2. run SELECT query
> {code}
> select
> acct.ACC_N,
> acct.brn
> FROM L
> JOIN LA ON L.id = LA.loan_id
> JOIN FR ON L.id = FR.loan_id
> JOIN A ON LA.aid = A.id
> JOIN PI ON PI.id = LA.pi_id
> JOIN acct ON A.id = acct.aid
> WHERE
> L.id = 4436
> and acct.brn is not null;
> {code}
> the result is 3 rows
> {code}
> 10122
> NULL NULL
> NULL NULL
> {code}
> but it should be 1 row
> {code}
> 10122
> {code}
> 2.1 "explain select ..." output for hive-1.3.0 MR
> {code}
> STAGE DEPENDENCIES:
> Stage-12 is a root stage
> Stage-9 depends on stages: Stage-12
> Stage-0 depends on stages: Stage-9
> STAGE PLANS:
> Stage: Stage-12
> Map Reduce Local Work
> Alias -> Map Local Tables:
> a
> Fetch Operator
> limit: -1
> acct
> Fetch Operator
> limit: -1
> fr
> Fetch Operator
> limit: -1
> l
> Fetch Operator
> limit: -1
> pi
> Fetch Operator
> limit: -1
> Alias -> Map Local Operator Tree:
> a
> TableScan
> alias: a
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> acct
> TableScan
> alias: acct
> Statistics: Num rows: 3 Data size: 31 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: aid is not null (type: boolean)
> Statistics: Num rows: 2 Data size: 20 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 _col5 (type: int)
> 1 id (type: int)
> 2 aid (type: int)
> fr
> TableScan
> alias: fr
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (loan_id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> l
> TableScan
> alias: l
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (id = 4436) (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE
> Column stats: NONE
> HashTable Sink Operator
> keys:
> 0 4436 (type: int)
> 1 4436 (type: int)
> 2 4436 (type: int)
> pi
> TableScan
> alias: pi
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: id is not null (type: boolean)
> Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLET
