[jira] [Resolved] (HIVE-28492) Upgrade Janino version to 3.1.12
[ https://issues.apache.org/jira/browse/HIVE-28492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-28492. - Fix Version/s: 4.1.0 Resolution: Fixed > Upgrade Janino version to 3.1.12 > > > Key: HIVE-28492 > URL: https://issues.apache.org/jira/browse/HIVE-28492 > Project: Hive > Issue Type: Improvement >Reporter: shivangi >Assignee: shivangi >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28042) DigestMD5 token expired or does not exist error while opening a new connection to HMS
[
https://issues.apache.org/jira/browse/HIVE-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-28042.
-
Fix Version/s: 4.1.0
Resolution: Fixed
{color:#172b4d}Merged to master !! Thanks for the {color}{color:#172b4d}PR
[~vikramahuja_] {color}
> DigestMD5 token expired or does not exist error while opening a new
> connection to HMS
> -
>
> Key: HIVE-28042
> URL: https://issues.apache.org/jira/browse/HIVE-28042
> Project: Hive
> Issue Type: Bug
>Reporter: Vikram Ahuja
>Assignee: Vikram Ahuja
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Hello,
> In our deployment we are facing the following exception in the HMS logs when
> a HMS connection is opened from the HS2 in cases where a session is open for
> a long time leading to query failures:
> {code:java}
> 2024-01-24T02:11:21,324 ERROR [TThreadPoolServer WorkerProcess-760394]:
> transport.TSaslTransport (TSaslTransport.java:open) - SASL negotiation
> failurejavax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring
> password
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java)
>
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java)
>
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java)
>
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java)
>
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java)
>
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.javA)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java)
>
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java)
>
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
>
> at java.lang.Thread.run(Thread.java)Caused by:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or
> does not exist: HIVE_DELEGATION_TOKEN owner=***, renewer=***,
> realUser=*, issueDate=1705973286139, maxDate=1706578086139,
> sequenceNumber=3294063, masterKeyId=7601
> at
> org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java)
>
> at
> org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java)
>
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java)
>
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java)
>
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java)
> ... 15 more {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27995) Fix inconsistent behavior of LOAD DATA command for partitioned and non-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-27995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-27995.
-
Fix Version/s: 4.1.0
Resolution: Fixed
> Fix inconsistent behavior of LOAD DATA command for partitioned and
> non-partitioned tables
> -
>
> Key: HIVE-27995
> URL: https://issues.apache.org/jira/browse/HIVE-27995
> Project: Hive
> Issue Type: Bug
>Reporter: Shivangi Jha
>Assignee: Shivangi Jha
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> For partitioned tables, while executing LOAD DATA/ LOAD DATA LOCAL commands,
> the check for file existence is not executed on HiveServer2, and this in turn
> throws an error during Runtime once the job is launched.
> {code:java}
> java.io.FileNotFoundException: File file:/ does not exist.{code}
> Non-partitioned tables do not follow this control flow, and the checks are
> run appropriately at compile time.
> Incase the file does not exist, user is presented with the error.
> {code:java}
> Invalid path "file:///: No files matching path
> file:/{code}
> This is inconsistent and error prone behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27995) Fix inconsistent behavior of LOAD DATA command for partitioned and non-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-27995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867172#comment-17867172
]
Chinna Rao Lalam commented on HIVE-27995:
-
{color:#172b4d}Merged to master !! Thanks for the {color}{color:#172b4d}PR
[~shivijha30] {color}
> Fix inconsistent behavior of LOAD DATA command for partitioned and
> non-partitioned tables
> -
>
> Key: HIVE-27995
> URL: https://issues.apache.org/jira/browse/HIVE-27995
> Project: Hive
> Issue Type: Bug
>Reporter: Shivangi Jha
>Assignee: Shivangi Jha
>Priority: Major
> Labels: pull-request-available
>
> For partitioned tables, while executing LOAD DATA/ LOAD DATA LOCAL commands,
> the check for file existence is not executed on HiveServer2, and this in turn
> throws an error during Runtime once the job is launched.
> {code:java}
> java.io.FileNotFoundException: File file:/ does not exist.{code}
> Non-partitioned tables do not follow this control flow, and the checks are
> run appropriately at compile time.
> Incase the file does not exist, user is presented with the error.
> {code:java}
> Invalid path "file:///: No files matching path
> file:/{code}
> This is inconsistent and error prone behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28179) Testcase TestTimestampColumnVector.testMultiThreaded() was broken
Chinna Rao Lalam created HIVE-28179: --- Summary: Testcase TestTimestampColumnVector.testMultiThreaded() was broken Key: HIVE-28179 URL: https://issues.apache.org/jira/browse/HIVE-28179 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: 4.1.0 TestTimestampColumnVector.testMultiThreaded() test was not failing fix is not there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27804) Implement batching in getPartition calls which returns partition list along with auth info
[ https://issues.apache.org/jira/browse/HIVE-27804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-27804. - Fix Version/s: 4.1.0 Resolution: Fixed > Implement batching in getPartition calls which returns partition list along > with auth info > -- > > Key: HIVE-27804 > URL: https://issues.apache.org/jira/browse/HIVE-27804 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Assignee: Vikram Ahuja >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Hive.getPartitions() methods returns partition list along with auth info in > one HMS call. These calls when made on wide tables(> 2000 columns) with very > large number of partitions(100,000+) can cause memory related issues when the > data is being transferred from HMS to HS2 using Thrift calls. These APIs can > be optimised by using PartitionIterable implementation where the partition > list if fetched in batched of a smaller size rather than one huge call. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27804) Implement batching in getPartition calls which returns partition list along with auth info
[ https://issues.apache.org/jira/browse/HIVE-27804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799867#comment-17799867 ] Chinna Rao Lalam commented on HIVE-27804: - Merged to master. Thanks for the patch [~vikramahuja_] > Implement batching in getPartition calls which returns partition list along > with auth info > -- > > Key: HIVE-27804 > URL: https://issues.apache.org/jira/browse/HIVE-27804 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Assignee: Vikram Ahuja >Priority: Major > Labels: pull-request-available > > Hive.getPartitions() methods returns partition list along with auth info in > one HMS call. These calls when made on wide tables(> 2000 columns) with very > large number of partitions(100,000+) can cause memory related issues when the > data is being transferred from HMS to HS2 using Thrift calls. These APIs can > be optimised by using PartitionIterable implementation where the partition > list if fetched in batched of a smaller size rather than one huge call. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27893) Add a range validator in hive.metastore.batch.retrieve.max to only have values greater than 0
[ https://issues.apache.org/jira/browse/HIVE-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-27893. - Fix Version/s: 4.1.0 Resolution: Fixed > Add a range validator in hive.metastore.batch.retrieve.max to only have > values greater than 0 > - > > Key: HIVE-27893 > URL: https://issues.apache.org/jira/browse/HIVE-27893 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Assignee: Vikram Ahuja >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > Currently there is no Validation for the property > hive.metastore.batch.retrieve.max. In case if it is ever set to <= 0, > PartitionIterable will throw up NoSuchElement Exception because the batch > size will be 0 there. Adding a range validator as the part of this Jira to > keep the lower bound value of hive.metastore.batch.retrieve.max to be 1. Also > blocking PartitionIterable call if any user/client ever calls the constructor > directly with batch size <1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27893) Add a range validator in hive.metastore.batch.retrieve.max to only have values greater than 0
[ https://issues.apache.org/jira/browse/HIVE-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794519#comment-17794519 ] Chinna Rao Lalam commented on HIVE-27893: - Merged to master !! Thanks for the patch [~vikramahuja_] > Add a range validator in hive.metastore.batch.retrieve.max to only have > values greater than 0 > - > > Key: HIVE-27893 > URL: https://issues.apache.org/jira/browse/HIVE-27893 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Assignee: Vikram Ahuja >Priority: Major > Labels: pull-request-available > > Currently there is no Validation for the property > hive.metastore.batch.retrieve.max. In case if it is ever set to <= 0, > PartitionIterable will throw up NoSuchElement Exception because the batch > size will be 0 there. Adding a range validator as the part of this Jira to > keep the lower bound value of hive.metastore.batch.retrieve.max to be 1. Also > blocking PartitionIterable call if any user/client ever calls the constructor > directly with batch size <1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27893) Add a range validator in hive.metastore.batch.retrieve.max to only have values greater than 0
[ https://issues.apache.org/jira/browse/HIVE-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-27893: Summary: Add a range validator in hive.metastore.batch.retrieve.max to only have values greater than 0 (was: Do not allow user to set hive.metastore.batch.retrieve.max property to < 1) > Add a range validator in hive.metastore.batch.retrieve.max to only have > values greater than 0 > - > > Key: HIVE-27893 > URL: https://issues.apache.org/jira/browse/HIVE-27893 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Assignee: Vikram Ahuja >Priority: Major > Labels: pull-request-available > > Currently there is no Validation for the property > hive.metastore.batch.retrieve.max. In case if it is ever set to <= 0, > PartitionIterable will throw up NoSuchElement Exception because the batch > size will be 0 there. Adding a range validator as the part of this Jira to > keep the lower bound value of hive.metastore.batch.retrieve.max to be 1. Also > blocking PartitionIterable call if any user/client ever calls the constructor > directly with batch size <1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27894) Enhance HMS Handler Logs for all 'get_partition' functions.
[ https://issues.apache.org/jira/browse/HIVE-27894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793502#comment-17793502 ] Chinna Rao Lalam commented on HIVE-27894: - Merged to master !! Thanks for the patch [~shivijha30] > Enhance HMS Handler Logs for all 'get_partition' functions. > --- > > Key: HIVE-27894 > URL: https://issues.apache.org/jira/browse/HIVE-27894 > Project: Hive > Issue Type: Improvement >Reporter: Shivangi Jha >Assignee: Shivangi Jha >Priority: Major > Labels: pull-request-available > > The HMSHandler > (standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java) > class encompasses various functions pertaining to partition information, yet > its current implementation lacks comprehensive logging of substantial > partition data. Enhancing this aspect would significantly contribute to > improved log readability and facilitate more effective debugging processes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27894) Enhance HMS Handler Logs for all 'get_partition' functions.
[ https://issues.apache.org/jira/browse/HIVE-27894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-27894. - Fix Version/s: 4.1.0 Resolution: Fixed > Enhance HMS Handler Logs for all 'get_partition' functions. > --- > > Key: HIVE-27894 > URL: https://issues.apache.org/jira/browse/HIVE-27894 > Project: Hive > Issue Type: Improvement >Reporter: Shivangi Jha >Assignee: Shivangi Jha >Priority: Major > Labels: pull-request-available > Fix For: 4.1.0 > > > The HMSHandler > (standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java) > class encompasses various functions pertaining to partition information, yet > its current implementation lacks comprehensive logging of substantial > partition data. Enhancing this aspect would significantly contribute to > improved log readability and facilitate more effective debugging processes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27894) Enhance HMS Handler Logs for all 'get_partition' functions.
[ https://issues.apache.org/jira/browse/HIVE-27894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793500#comment-17793500 ] Chinna Rao Lalam commented on HIVE-27894: - +1 LGTM > Enhance HMS Handler Logs for all 'get_partition' functions. > --- > > Key: HIVE-27894 > URL: https://issues.apache.org/jira/browse/HIVE-27894 > Project: Hive > Issue Type: Improvement >Reporter: Shivangi Jha >Assignee: Shivangi Jha >Priority: Major > Labels: pull-request-available > > The HMSHandler > (standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java) > class encompasses various functions pertaining to partition information, yet > its current implementation lacks comprehensive logging of substantial > partition data. Enhancing this aspect would significantly contribute to > improved log readability and facilitate more effective debugging processes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27662) Incorrect parsing of nested complex types containing map during vectorized text processing
[
https://issues.apache.org/jira/browse/HIVE-27662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-27662.
-
Fix Version/s: 4.0.0
Resolution: Fixed
> Incorrect parsing of nested complex types containing map during vectorized
> text processing
> --
>
> Key: HIVE-27662
> URL: https://issues.apache.org/jira/browse/HIVE-27662
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> When reading a text table with vectorization on and
> hive.fetch.task.conversion as none, wrong parsing of delimiter is happening
> in nested complex types containing map. For example, if a columns schema is
> like: map then \u0004 char is coming in
> the output. Here is a example:
>
> Sample q file:
>
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.execution.enabled=true;
> create EXTERNAL table `table4` as
> select
> 'bob' as name,
> map(
> "Map_Key1",
> named_struct(
> 'Id',
> 'Id_Value1',
> 'Name',
> 'Name_Value1'
> ),
> "Map_Key2",
> named_struct(
> 'Id',
> 'Id_Value2',
> 'Name',
> 'Name_Value2'
> )
> ) as testmarks;
> select * from table4;
> set hive.vectorized.execution.enabled=false;
> select * from table4;
> {code}
> Output of 1st select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1\u0004Name_Value1","name":null},"Map_Key2":{"id":"Id_Value2\u0004Name_Value2","name":null}}{code}
> Output of 2nd select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1","name":"Name_Value1"},"Map_Key2":{"id":"Id_Value2","name":"Name_Value2"}}{code}
>
> MAP Complex type is not handling the scenario where it contains a nested
> complex type like STRUCT, ARRAY, UNION.
>
> *To reproduce this issue:*
> *mvn test -Dtest=TestCliDriver -Pitests -Dqfile=`qfile_name`-pl itests/qtest
> -Dtest.output.overwrite*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27662) Incorrect parsing of nested complex types containing map during vectorized text processing
[
https://issues.apache.org/jira/browse/HIVE-27662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792899#comment-17792899
]
Chinna Rao Lalam commented on HIVE-27662:
-
Merged to master !! Thanks for the patch [~Aggarwal_Raghav]
> Incorrect parsing of nested complex types containing map during vectorized
> text processing
> --
>
> Key: HIVE-27662
> URL: https://issues.apache.org/jira/browse/HIVE-27662
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
>
> When reading a text table with vectorization on and
> hive.fetch.task.conversion as none, wrong parsing of delimiter is happening
> in nested complex types containing map. For example, if a columns schema is
> like: map then \u0004 char is coming in
> the output. Here is a example:
>
> Sample q file:
>
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.execution.enabled=true;
> create EXTERNAL table `table4` as
> select
> 'bob' as name,
> map(
> "Map_Key1",
> named_struct(
> 'Id',
> 'Id_Value1',
> 'Name',
> 'Name_Value1'
> ),
> "Map_Key2",
> named_struct(
> 'Id',
> 'Id_Value2',
> 'Name',
> 'Name_Value2'
> )
> ) as testmarks;
> select * from table4;
> set hive.vectorized.execution.enabled=false;
> select * from table4;
> {code}
> Output of 1st select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1\u0004Name_Value1","name":null},"Map_Key2":{"id":"Id_Value2\u0004Name_Value2","name":null}}{code}
> Output of 2nd select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1","name":"Name_Value1"},"Map_Key2":{"id":"Id_Value2","name":"Name_Value2"}}{code}
>
> MAP Complex type is not handling the scenario where it contains a nested
> complex type like STRUCT, ARRAY, UNION.
>
> *To reproduce this issue:*
> *mvn test -Dtest=TestCliDriver -Pitests -Dqfile=`qfile_name`-pl itests/qtest
> -Dtest.output.overwrite*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27662) Incorrect parsing of nested complex types containing map during vectorized text processing
[
https://issues.apache.org/jira/browse/HIVE-27662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787794#comment-17787794
]
Chinna Rao Lalam commented on HIVE-27662:
-
LGTM +1
> Incorrect parsing of nested complex types containing map during vectorized
> text processing
> --
>
> Key: HIVE-27662
> URL: https://issues.apache.org/jira/browse/HIVE-27662
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
>
> When reading a text table with vectorization on and
> hive.fetch.task.conversion as none, wrong parsing of delimiter is happening
> in nested complex types containing map. For example, if a columns schema is
> like: map then \u0004 char is coming in
> the output. Here is a example:
>
> Sample q file:
>
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.execution.enabled=true;
> create EXTERNAL table `table4` as
> select
> 'bob' as name,
> map(
> "Map_Key1",
> named_struct(
> 'Id',
> 'Id_Value1',
> 'Name',
> 'Name_Value1'
> ),
> "Map_Key2",
> named_struct(
> 'Id',
> 'Id_Value2',
> 'Name',
> 'Name_Value2'
> )
> ) as testmarks;
> select * from table4;
> set hive.vectorized.execution.enabled=false;
> select * from table4;
> {code}
> Output of 1st select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1\u0004Name_Value1","name":null},"Map_Key2":{"id":"Id_Value2\u0004Name_Value2","name":null}}{code}
> Output of 2nd select statement:
> {code:java}
> bob·
> {"Map_Key1":{"id":"Id_Value1","name":"Name_Value1"},"Map_Key2":{"id":"Id_Value2","name":"Name_Value2"}}{code}
>
> MAP Complex type is not handling the scenario where it contains a nested
> complex type like STRUCT, ARRAY, UNION.
>
> *To reproduce this issue:*
> *mvn test -Dtest=TestCliDriver -Pitests -Dqfile=`qfile_name`-pl itests/qtest
> -Dtest.output.overwrite*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27584) Backport HIVE-21407 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771785#comment-17771785 ] Chinna Rao Lalam commented on HIVE-27584: - LGTM +1 > Backport HIVE-21407 to branch-3 > --- > > Key: HIVE-27584 > URL: https://issues.apache.org/jira/browse/HIVE-27584 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > > HIVE-21407: Parquet predicate pushdown is not working correctly for char > column types (Marta Kuczora reviewed by Peter Vary) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25671) Hybrid Grace Hash Join NullPointer When query RCFile
[
https://issues.apache.org/jira/browse/HIVE-25671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17439071#comment-17439071
]
Chinna Rao Lalam commented on HIVE-25671:
-
This issue presents in 3.1.2 version, I think this might be fixed in latest
Hive. Please check latest hive once and kyro version.
> Hybrid Grace Hash Join NullPointer When query RCFile
>
>
> Key: HIVE-25671
> URL: https://issues.apache.org/jira/browse/HIVE-25671
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Nemon Lou
>Priority: Major
> Attachments: rcfile_kryo.patch
>
>
> Hive 3.1.0 kryo 3.0.3 tez engine
> the following sql can reproduce this issue
> {code:sql}
> CREATE TABLE `nemon.rt_dm_lpc_customer_sum_tmp3_3`(
>`logo` string,
>`customer_code` string,
>`brand_name` string,
>`business_code` string,
>`discount` double,
>`creation_date` string,
>`etl_time` string)stored as rcfile;
>
> CREATE TABLE `nemon.rt_dm_lpc_customer_sum_tmp4_1`(
>`customer_code` string,
>`etl_time` string) stored as rcfile;
>
> insert into nemon.rt_dm_lpc_customer_sum_tmp3_3 values
> ("logo","customer_code","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code1","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code2","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code3","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code4","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code5","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code6","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code7","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code8","brand_name","business_code",1,"creation_date","etl_time")
>
> ,("logo","customer_code9","brand_name","business_code",1,"creation_date","etl_time");
> insert into nemon.rt_dm_lpc_customer_sum_tmp4_1 values
> ("customer_code","etl_time")
>,("customer_code1","etl_time")
>,("customer_code2","etl_time")
>,("customer_code3","etl_time")
>;
> set hive.auto.convert.join.noconditionaltask.size=10;
> set hive.mapjoin.hybridgrace.hashtable=true;
> SELECT
> tt1.logo,
> tt1.customer_code,
> tt1.brand_name,
> tt1.business_code,
> tt1.discount,
> tt1.creation_date,
> date_format(from_utc_timestamp(unix_timestamp()*1000,'Asia/Shanghai'),'-MM-dd
> HH:mm:ss') etl_time
> from
> (
> SELECT
> t1.logo,
> t1.customer_code,
> t1.brand_name,
> t1.business_code,
> t1.discount,
> t1.creation_date,
> row_number() over(partition by t1.customer_code,t1.logo order by
> t1.creation_date desc) as discount_rank
> from nemon.rt_dm_lpc_customer_sum_tmp3_3 t1
> join nemon.rt_dm_lpc_customer_sum_tmp4_1 t2
> on t2.customer_code = t1.customer_code
> ) tt1
> where tt1.discount_rank = 1;
> {code}
> Error log from tez task:
> {noformat}
> 2021-11-04 10:02:47,553 [INFO] [TezChild] |exec.MapJoinOperator|: Hybrid
> Grace Hash Join: Deserializing spilled hash partition...
> 2021-11-04 10:02:47,553 [INFO] [TezChild] |exec.MapJoinOperator|: Hybrid
> Grace Hash Join: Number of rows in hashmap: 1
> 2021-11-04 10:02:47,554 [INFO] [TezChild] |exec.MapJoinOperator|: Hybrid
> Grace Hash Join: Going to process spilled big table rows in partition 5.
> Number of rows: 1
> 2021-11-04 10:02:47,561 [ERROR] [TezChild] |exec.MapJoinOperator|: Unexpected
> exception from MapJoinOperator : null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:114)
> at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:172)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldData(ColumnarStructObjectInspector.java:67)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
> at
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor.setFromRow(MapJoinBytesTableContainer.java:552)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:415)
> at
[jira] [Commented] (HIVE-21737) Upgrade Avro to version 1.10.0
[
https://issues.apache.org/jira/browse/HIVE-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211061#comment-17211061
]
Chinna Rao Lalam commented on HIVE-21737:
-
Hi [~iemejia],
Verified this patch and found these 2 test failures with below exception
{quote}avro_deserialize_map_null.q
parquet_map_null.q
{quote}
{quote}Failed with exception
java.io.IOException:org.apache.avro.AvroTypeException: Invalid default for
field avreau_col_1: null not a []
{quote}
It looks these exceptions are because of breaking backword compatability of
avro version. https://issues.apache.org/jira/browse/AVRO-2817
We tried setting *Schema.Parser.setValidateDefaults(false)* to turn of defaults
validation
Ex.
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils#getSchemaFor(java.io.File) it
did not work.
[~iemejia] any idea/workarond for this issue?
> Upgrade Avro to version 1.10.0
> --
>
> Key: HIVE-21737
> URL: https://issues.apache.org/jira/browse/HIVE-21737
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ismaël Mejía
>Assignee: Fokko Driesprong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-21737-Bump-Apache-Avro-to-1.9.2.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Avro >= 1.9.x bring a lot of fixes including a leaner version of Avro without
> Jackson in the public API and Guava as a dependency. Worth the update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23586) load data overwrite into bucket table failed
[ https://issues.apache.org/jira/browse/HIVE-23586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128132#comment-17128132 ] Chinna Rao Lalam commented on HIVE-23586: - Hi [~fsilent], Please add a testcase for this scenario and submit patch, then it will generate testreport. > load data overwrite into bucket table failed > > > Key: HIVE-23586 > URL: https://issues.apache.org/jira/browse/HIVE-23586 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 4.0.0, 3.1.2 >Reporter: zhaolong >Assignee: zhaolong >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23586.01.patch, image-2020-06-01-21-40-21-726.png, > image-2020-06-01-21-41-28-732.png > > Time Spent: 20m > Remaining Estimate: 0h > > load data overwrite into bucket table is failed if filename is not like > 00_0, but insert new data in the table. > > for example: > CREATE EXTERNAL TABLE IF NOT EXISTS test_hive2 (name string,account string) > PARTITIONED BY (logdate string) CLUSTERED BY (account) INTO 4 BUCKETS row > format delimited fields terminated by '|' STORED AS textfile; > load data inpath 'hdfs://hacluster/tmp/zltest' overwrite into table > default.test_hive2 partition (logdate='20200508'); > !image-2020-06-01-21-40-21-726.png! > load data inpath 'hdfs://hacluster/tmp/zltest' overwrite into table > default.test_hive2 partition (logdate='20200508');// should overwrite but > insert new data > !image-2020-06-01-21-41-28-732.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-23133: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-23133.6.patch committed to master. Thanks for review Gopal V. Thank you for the patch Zhenyu Zheng. > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math for compatibility in the > following UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > UDFLog2.java > UDFLog10.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107040#comment-17107040 ] Chinna Rao Lalam commented on HIVE-23133: - LGTM +1 > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math for compatibility in the > following UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > UDFLog2.java > UDFLog10.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107039#comment-17107039 ] Chinna Rao Lalam commented on HIVE-23133: - This issue fixes math functions which are failing in ARM platform like log, exp, cos, toRadians. [~Kevin_Zheng] please check other math functions if any in follow up jira. > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math for compatibility in the > following UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > UDFLog2.java > UDFLog10.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095838#comment-17095838 ] Chinna Rao Lalam commented on HIVE-23133: - Thanks for the patch. StrictMath helps portability of java programs, so using strictmath is good. I am considering these 2 things about strictmath. # Performance issue, as published jdk performance comparison differences are small. # Backward compatibility , but with strictmath it is fixing the problem completely. Any thoughts on this [~hashutosh] [~gopalv] [~jcamachorodriguez] > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Yikun Jiang >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23134) Hive & Kudu interaction not available on ARM
[ https://issues.apache.org/jira/browse/HIVE-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-23134: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-23134.2.patch committed to master. Thank you for the patch Zhenyu Zheng. > Hive & Kudu interaction not available on ARM > > > Key: HIVE-23134 > URL: https://issues.apache.org/jira/browse/HIVE-23134 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23134.1.patch, HIVE-23134.2.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ > According to the results, Hive & Kudu interaction is not available on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.kudu/ > this is because that we use Kudu version 1.10 and that version does not come > with ARM workable packages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23134) Hive & Kudu interaction not available on ARM
[ https://issues.apache.org/jira/browse/HIVE-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092294#comment-17092294 ] Chinna Rao Lalam commented on HIVE-23134: - +1 > Hive & Kudu interaction not available on ARM > > > Key: HIVE-23134 > URL: https://issues.apache.org/jira/browse/HIVE-23134 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23134.1.patch, HIVE-23134.2.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ > According to the results, Hive & Kudu interaction is not available on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.kudu/ > this is because that we use Kudu version 1.10 and that version does not come > with ARM workable packages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23134) Hive & Kudu interaction not available on ARM
[ https://issues.apache.org/jira/browse/HIVE-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092293#comment-17092293 ] Chinna Rao Lalam commented on HIVE-23134: - Currently KUDU ARM64 support is not there, so as we can not execute these tests on ARM64. Its look good to exclude these tests on arm64 platform. > Hive & Kudu interaction not available on ARM > > > Key: HIVE-23134 > URL: https://issues.apache.org/jira/browse/HIVE-23134 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23134.1.patch, HIVE-23134.2.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ > According to the results, Hive & Kudu interaction is not available on ARM > platform: > https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.kudu/ > this is because that we use Kudu version 1.10 and that version does not come > with ARM workable packages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23028) Should not use group parameter when run tests in standalone-metastore-common
[ https://issues.apache.org/jira/browse/HIVE-23028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084182#comment-17084182 ] Chinna Rao Lalam edited comment on HIVE-23028 at 4/15/20, 3:36 PM: --- HIVE-23028.1.patch committed to master. Thank you for the patch Zhenyu Zheng. was (Author: chinnalalam): +1 > Should not use group parameter when run tests in standalone-metastore-common > > > Key: HIVE-23028 > URL: https://issues.apache.org/jira/browse/HIVE-23028 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23028.1.patch > > > Should not use group parameter when run tests in standalone-metastore-common > we inherited the `group` parameter from ‘standalone-metastore’s pom.xml and > there the parameter > is set to : org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest > ([https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61]) > which only exists in metastore-server package: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/annotation/MetastoreUnitTest.java] > and for tests for metastore-common package does not have the @Catagory header: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/RetryTest.java] > > and it was actually introduced by: > [https://github.com/apache/hive/commit/7411d42579ffa0bad96e8da731a1a35afc9ff614#diff-171fcb0dda3bcba577fa13720d5b6571] > we should remove the group paramenter in > [https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml|https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23028) Should not use group parameter when run tests in standalone-metastore-common
[ https://issues.apache.org/jira/browse/HIVE-23028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084181#comment-17084181 ] Chinna Rao Lalam edited comment on HIVE-23028 at 4/15/20, 3:35 PM: --- +1 was (Author: chinnalalam): HIVE-23028.1.patch committed to master. Thank you for the patch Zhenyu Zheng. > Should not use group parameter when run tests in standalone-metastore-common > > > Key: HIVE-23028 > URL: https://issues.apache.org/jira/browse/HIVE-23028 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23028.1.patch > > > Should not use group parameter when run tests in standalone-metastore-common > we inherited the `group` parameter from ‘standalone-metastore’s pom.xml and > there the parameter > is set to : org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest > ([https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61]) > which only exists in metastore-server package: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/annotation/MetastoreUnitTest.java] > and for tests for metastore-common package does not have the @Catagory header: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/RetryTest.java] > > and it was actually introduced by: > [https://github.com/apache/hive/commit/7411d42579ffa0bad96e8da731a1a35afc9ff614#diff-171fcb0dda3bcba577fa13720d5b6571] > we should remove the group paramenter in > [https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml|https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23028) Should not use group parameter when run tests in standalone-metastore-common
[ https://issues.apache.org/jira/browse/HIVE-23028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084182#comment-17084182 ] Chinna Rao Lalam commented on HIVE-23028: - +1 > Should not use group parameter when run tests in standalone-metastore-common > > > Key: HIVE-23028 > URL: https://issues.apache.org/jira/browse/HIVE-23028 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23028.1.patch > > > Should not use group parameter when run tests in standalone-metastore-common > we inherited the `group` parameter from ‘standalone-metastore’s pom.xml and > there the parameter > is set to : org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest > ([https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61]) > which only exists in metastore-server package: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/annotation/MetastoreUnitTest.java] > and for tests for metastore-common package does not have the @Catagory header: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/RetryTest.java] > > and it was actually introduced by: > [https://github.com/apache/hive/commit/7411d42579ffa0bad96e8da731a1a35afc9ff614#diff-171fcb0dda3bcba577fa13720d5b6571] > we should remove the group paramenter in > [https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml|https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23028) Should not use group parameter when run tests in standalone-metastore-common
[ https://issues.apache.org/jira/browse/HIVE-23028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-23028: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-23028.1.patch committed to master. Thank you for the patch Zhenyu Zheng. > Should not use group parameter when run tests in standalone-metastore-common > > > Key: HIVE-23028 > URL: https://issues.apache.org/jira/browse/HIVE-23028 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23028.1.patch > > > Should not use group parameter when run tests in standalone-metastore-common > we inherited the `group` parameter from ‘standalone-metastore’s pom.xml and > there the parameter > is set to : org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest > ([https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61]) > which only exists in metastore-server package: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/annotation/MetastoreUnitTest.java] > and for tests for metastore-common package does not have the @Catagory header: > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/RetryTest.java] > > and it was actually introduced by: > [https://github.com/apache/hive/commit/7411d42579ffa0bad96e8da731a1a35afc9ff614#diff-171fcb0dda3bcba577fa13720d5b6571] > we should remove the group paramenter in > [https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml|https://github.com/apache/hive/blob/master/standalone-metastore/pom.xml#L61] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23163) Class TrustDomainAuthenticationTest should be abstract
[ https://issues.apache.org/jira/browse/HIVE-23163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-23163: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-23163.3.patch committed to master. Thank you for the patch Yikun Jiang. > Class TrustDomainAuthenticationTest should be abstract > -- > > Key: HIVE-23163 > URL: https://issues.apache.org/jira/browse/HIVE-23163 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Yikun Jiang >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23163.1.patch, HIVE-23163.2.patch, > HIVE-23163.3.patch > > > When running tests in pre-commit CI, the test parser will only identify test > classes start with 'Test' > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestPropertiesParser.java#L406] > But when running using `mvn test`, it will also parse classes such as > '*Test.java': > [http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/inclusion-exclusion.html] > so for: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TrustDomainAuthenticationTest.java#L38] > it will also be included in the test, for example: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/TrustDomainAuthenticationTest/testTrustedDomainAuthentication/] > This is because that class TrustDomainAuthenticationTest is actually a parent > class that does not have the parameters for init. The actual tests are its > children classes like the rest in > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/] > they will pass actuall parameters for init: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TestTrustDomainAuthenticationBinary.java#L26] > > we can make this class abstract so that it won't be included when running > 'mvn test', like > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractJdbcTriggersTest.java#L54] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23163) Class TrustDomainAuthenticationTest should be abstract
[ https://issues.apache.org/jira/browse/HIVE-23163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082542#comment-17082542 ] Chinna Rao Lalam commented on HIVE-23163: - +1 > Class TrustDomainAuthenticationTest should be abstract > -- > > Key: HIVE-23163 > URL: https://issues.apache.org/jira/browse/HIVE-23163 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Yikun Jiang >Priority: Major > Attachments: HIVE-23163.1.patch, HIVE-23163.2.patch, > HIVE-23163.3.patch > > > When running tests in pre-commit CI, the test parser will only identify test > classes start with 'Test' > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestPropertiesParser.java#L406] > But when running using `mvn test`, it will also parse classes such as > '*Test.java': > [http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/inclusion-exclusion.html] > so for: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TrustDomainAuthenticationTest.java#L38] > it will also be included in the test, for example: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/TrustDomainAuthenticationTest/testTrustedDomainAuthentication/] > This is because that class TrustDomainAuthenticationTest is actually a parent > class that does not have the parameters for init. The actual tests are its > children classes like the rest in > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/] > they will pass actuall parameters for init: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TestTrustDomainAuthenticationBinary.java#L26] > > we can make this class abstract so that it won't be included when running > 'mvn test', like > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractJdbcTriggersTest.java#L54] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21939) protoc:2.5.0 dependence has broken building on aarch64
[ https://issues.apache.org/jira/browse/HIVE-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-21939: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-21939.2.patch committed to master. Thank you for the patch [~Kevin_Zheng] . > protoc:2.5.0 dependence has broken building on aarch64 > --- > > Key: HIVE-21939 > URL: https://issues.apache.org/jira/browse/HIVE-21939 > Project: Hive > Issue Type: Bug >Reporter: liusheng >Assignee: Zhenyu Zheng >Priority: Blocker > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21939.1.patch, HIVE-21939.2.patch, > hive_build_with_protobuf_371.log > > Time Spent: 40m > Remaining Estimate: 0h > > When I try to build master of Hive from source code on "aarch64" server, I > met following error: > [ERROR] Failed to execute goal > com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project > hive-standalone-metastore-common: Error resolving artifact: > com.google.protobuf:protoc:2.5.0: Could not find artifact > com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0 in central > ([https://repo.maven.apache.org/maven2)] > that is because Hive using the "com.google.protobuf:protoc:2.5.0" as required > artifact, which does not have released package for "aarch64" platform. > > In order to fix this, I bumped the protobuf used in standalone-metadata to > 2.6.1 and added a new profile, this profile will identify > the hardware architecture and if it is Aarch64, it will override the protobuf > [group.id|http://group.id/] and package to com.github.os72 which > includes ARM support. For X86 platform, Hive will still download the protobuf > packages from org.google repo. With this method, > we can make Hive able to run on Aarch64 and keep the influence to existing > x86 users to the minimum. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21939) protoc:2.5.0 dependence has broken building on aarch64
[ https://issues.apache.org/jira/browse/HIVE-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059176#comment-17059176 ] Chinna Rao Lalam commented on HIVE-21939: - +1 > protoc:2.5.0 dependence has broken building on aarch64 > --- > > Key: HIVE-21939 > URL: https://issues.apache.org/jira/browse/HIVE-21939 > Project: Hive > Issue Type: Bug >Reporter: liusheng >Assignee: Zhenyu Zheng >Priority: Blocker > Labels: pull-request-available > Attachments: HIVE-21939.1.patch, HIVE-21939.2.patch, > hive_build_with_protobuf_371.log > > Time Spent: 40m > Remaining Estimate: 0h > > When I try to build master of Hive from source code on "aarch64" server, I > met following error: > [ERROR] Failed to execute goal > com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project > hive-standalone-metastore-common: Error resolving artifact: > com.google.protobuf:protoc:2.5.0: Could not find artifact > com.google.protobuf:protoc:exe:linux-aarch_64:2.5.0 in central > ([https://repo.maven.apache.org/maven2)] > that is because Hive using the "com.google.protobuf:protoc:2.5.0" as required > artifact, which does not have released package for "aarch64" platform. > > In order to fix this, I bumped the protobuf used in standalone-metadata to > 2.6.1 and added a new profile, this profile will identify > the hardware architecture and if it is Aarch64, it will override the protobuf > [group.id|http://group.id/] and package to com.github.os72 which > includes ARM support. For X86 platform, Hive will still download the protobuf > packages from org.google repo. With this method, > we can make Hive able to run on Aarch64 and keep the influence to existing > x86 users to the minimum. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22710) OOM Exception when enable CBO
[
https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-22710:
Description:
When set hive.cbo.enable=true, and execute a SQL contains union all and join to
insert the result into three tables, OOM will appear in the compilation phase.
Attached error logs.
With attached SQL file OOM will come.
Same problem observed in HIVE master branch also.
{noformat}
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_201]
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
~[?:1.8.0_201]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
~[?:1.8.0_201]
at java.lang.StringBuilder.append(StringBuilder.java:136) ~[?:1.8.0_201]
at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:167)
~[calcite-core-1.19.0.jar:1.19.0]
at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
~[calcite-core-1.19.0.jar:1.19.0]
at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_201]
at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_201]
at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109)
~[calcite-core-1.19.0.jar:1.19.0]
at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166)
~[calcite-core-1.19.0.jar:1.19.0]
at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
~[calcite-core-1.19.0.jar:1.19.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:349)
~[hive-exec-3.1.0.jar:3.1.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:319)
~[hive-exec-3.1.0.jar:3.1.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:183)
~[hive-exec-3.1.0.jar:3.1.0]
{noformat}
was:When set hive.cbo.enable=true, and execute a SQL contains union all and
join to insert the result into three tables. Outofmemoryerror will appear in
the compilation phase. For detailed SQL statements and error logs, please refer
to the attachment. The same problem exists in hive 4.0.0
> OOM Exception when enable CBO
> -
>
> Key: HIVE-22710
> URL: https://issues.apache.org/jira/browse/HIVE-22710
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Priority: Major
> Attachments: error_stack.log, sample.sql
>
>
> When set hive.cbo.enable=true, and execute a SQL contains union all and join
> to insert the result into three tables, OOM will appear in the compilation
> phase. Attached error logs.
> With attached SQL file OOM will come.
> Same problem observed in HIVE master branch also.
> {noformat}
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_201]
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> ~[?:1.8.0_201]
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> ~[?:1.8.0_201]
> at java.lang.StringBuilder.append(StringBuilder.java:136) ~[?:1.8.0_201]
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:167)
> ~[calcite-core-1.19.0.jar:1.19.0]
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
> ~[calcite-core-1.19.0.jar:1.19.0]
> at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_201]
> at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_201]
> at org.apache.calcite.rex.RexCall.appendOperands(RexCall.java:109)
> ~[calcite-core-1.19.0.jar:1.19.0]
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:166)
> ~[calcite-core-1.19.0.jar:1.19.0]
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:183)
> ~[calcite-core-1.19.0.jar:1.19.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:349)
> ~[hive-exec-3.1.0.jar:3.1.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.(HiveRelMdPredicates.java:319)
> ~[hive-exec-3.1.0.jar:3.1.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:183)
> ~[hive-exec-3.1.0.jar:3.1.0]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22710) OOM Exception when enable CBO
[ https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-22710: Summary: OOM Exception when enable CBO (was: When set CBO on, CalcitePlanner genOPTree will be Out Of Memory) > OOM Exception when enable CBO > - > > Key: HIVE-22710 > URL: https://issues.apache.org/jira/browse/HIVE-22710 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.2 >Reporter: LuGuangMing >Priority: Major > Attachments: error_stack.log, sample.sql > > > When set hive.cbo.enable=true, and execute a SQL contains union all and join > to insert the result into three tables. Outofmemoryerror will appear in the > compilation phase. For detailed SQL statements and error logs, please refer > to the attachment. The same problem exists in hive 4.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22710) When set CBO on, CalcitePlanner genOPTree will be Out Of Memory
[ https://issues.apache.org/jira/browse/HIVE-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011498#comment-17011498 ] Chinna Rao Lalam commented on HIVE-22710: - Hi [~jcamachorodriguez], We encountered this issue and when analyzing we come across these issues CALCITE-2883 And HIVE-21365. Do you think still OOM problem is there in HepPlanner of CALCITE. Please throw some insights about this issue. > When set CBO on, CalcitePlanner genOPTree will be Out Of Memory > --- > > Key: HIVE-22710 > URL: https://issues.apache.org/jira/browse/HIVE-22710 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.2 >Reporter: LuGuangMing >Priority: Major > Attachments: error_stack.log, sample.sql > > > When set hive.cbo.enable=true, and execute a SQL contains union all and join > to insert the result into three tables. Outofmemoryerror will appear in the > compilation phase. For detailed SQL statements and error logs, please refer > to the attachment. The same problem exists in hive 4.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-15431) Round(1234567891.1234567891,50) returns null, result is not consistent with Mysql.
[
https://issues.apache.org/jira/browse/HIVE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15431:
Attachment: HIVE-15431.patch
> Round(1234567891.1234567891,50) returns null, result is not consistent with
> Mysql.
> --
>
> Key: HIVE-15431
> URL: https://issues.apache.org/jira/browse/HIVE-15431
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15431.patch
>
>
> round(1234567891.1234567891,50) returns null. Result is not consistent with
> Mysql.
> Mysql output:
> {quote}
> select round(1234567891.1234567891,50);
> '1234567891.1234567891'
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15820400#comment-15820400
]
Chinna Rao Lalam commented on HIVE-15324:
-
Hi [~xuefuz],
Thanks for the review. I will check this scenario. I have one doubt please
correct me if i am wrong.
Initialize() evaluated "outputIO" for all instances based on the input type
irrespective of the scale is constant or non-constant. Will this handle result
type?
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.2.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15778618#comment-15778618
]
Chinna Rao Lalam commented on HIVE-15324:
-
Failed tests are not related to this patch.
RB request link : https://reviews.apache.org/r/55045/
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.2.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15778600#comment-15778600
]
Chinna Rao Lalam commented on HIVE-15325:
-
RB request link : https://reviews.apache.org/r/55036/
> Scale is greater than decimal values trunc(d,s) returns wrong results
> -
>
> Key: HIVE-15325
> URL: https://issues.apache.org/jira/browse/HIVE-15325
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15325.patch
>
>
> Scale is greater than decimal values trunc(d,s) returns wrong results.
> {quote}
> select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
> trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
> tablesample (1 rows);
> {quote}
> Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
> 100))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Status: Patch Available (was: Open)
updated the out file for udf_round.q.out
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.2.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Attachment: HIVE-15324.2.patch
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.2.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Status: Open (was: Patch Available)
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Release Note: Updated the function to accept non-constants also as scale
argument.
Status: Patch Available (was: Open)
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Attachment: HIVE-15324.1.patch
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.1.patch, HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15325:
Attachment: HIVE-15325.patch
> Scale is greater than decimal values trunc(d,s) returns wrong results
> -
>
> Key: HIVE-15325
> URL: https://issues.apache.org/jira/browse/HIVE-15325
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15325.patch
>
>
> Scale is greater than decimal values trunc(d,s) returns wrong results.
> {quote}
> select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
> trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
> tablesample (1 rows);
> {quote}
> Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
> 100))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15325:
Status: Patch Available (was: Open)
> Scale is greater than decimal values trunc(d,s) returns wrong results
> -
>
> Key: HIVE-15325
> URL: https://issues.apache.org/jira/browse/HIVE-15325
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15325.patch
>
>
> Scale is greater than decimal values trunc(d,s) returns wrong results.
> {quote}
> select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
> trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
> tablesample (1 rows);
> {quote}
> Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
> 100))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15325:
Description:
Scale is greater than decimal values trunc(d,s) returns wrong results.
{quote}
select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
tablesample (1 rows);
{quote}
Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
100))
was:
Scale is greater than decimal values trunc(d,s) returns wrong results.
Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
100))
> Scale is greater than decimal values trunc(d,s) returns wrong results
> -
>
> Key: HIVE-15325
> URL: https://issues.apache.org/jira/browse/HIVE-15325
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
>
> Scale is greater than decimal values trunc(d,s) returns wrong results.
> {quote}
> select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
> trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
> tablesample (1 rows);
> {quote}
> Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
> 100))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15325:
Priority: Major (was: Minor)
> Scale is greater than decimal values trunc(d,s) returns wrong results
> -
>
> Key: HIVE-15325
> URL: https://issues.apache.org/jira/browse/HIVE-15325
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>
> Scale is greater than decimal values trunc(d,s) returns wrong results.
> {quote}
> select trunc(1234567891.1234567891,15), trunc(1234567891.1234567891,25),
> trunc(1234567891.1234567891,20), trunc(1234567891.1234567891,50) FROM src
> tablesample (1 rows);
> {quote}
> Add tests with negative numbers as well as no-op (e.g. select trunc (12.34,
> 100))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[ https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-15325: Summary: Scale is greater than decimal values trunc(d,s) returns wrong results (was: Add tests with negative numbers as well as no-op tests) > Scale is greater than decimal values trunc(d,s) returns wrong results > - > > Key: HIVE-15325 > URL: https://issues.apache.org/jira/browse/HIVE-15325 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam >Priority: Minor > > Add tests with negative numbers as well as no-op (e.g. select trunc (12.34, > 100)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15325) Scale is greater than decimal values trunc(d,s) returns wrong results
[ https://issues.apache.org/jira/browse/HIVE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-15325: Description: Scale is greater than decimal values trunc(d,s) returns wrong results. Add tests with negative numbers as well as no-op (e.g. select trunc (12.34, 100)) was:Add tests with negative numbers as well as no-op (e.g. select trunc (12.34, 100)) > Scale is greater than decimal values trunc(d,s) returns wrong results > - > > Key: HIVE-15325 > URL: https://issues.apache.org/jira/browse/HIVE-15325 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam >Priority: Minor > > Scale is greater than decimal values trunc(d,s) returns wrong results. > Add tests with negative numbers as well as no-op (e.g. select trunc (12.34, > 100)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15324) Enable round() function to accept scale argument as non-constants
[
https://issues.apache.org/jira/browse/HIVE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-15324:
Attachment: HIVE-15324.patch
> Enable round() function to accept scale argument as non-constants
> -
>
> Key: HIVE-15324
> URL: https://issues.apache.org/jira/browse/HIVE-15324
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-15324.patch
>
>
> round() function should accept scale argument as non-constants, it will
> enable queries like:
> {quote}
> create table sampletable(c double, d int);
> select round(c,d) from sampletable;
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Patch Available (was: Open) Hi [~ashutoshc], Thanks for the review. I have referred round() and bround() functions, which accepts scale argument as constant. Now trunc() function is updated to accept scale argument as constant or non-constant. Please take a look at patch. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.4.patch > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Open (was: Patch Available) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686781#comment-15686781 ] Chinna Rao Lalam commented on HIVE-14582: - Created RB request : https://reviews.apache.org/r/53983/ > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.3.patch > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Patch Available (was: Open) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-14582:
Attachment: HIVE-14582.2.patch
{quote}
udf_trunc.q,
udf_trunc_number.q
{quote}
tests failed because of function description change, it was not updated in test
files. Remaining failures are not related to this patch.
Patch.2 Updated with new function description in test files.
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Open (was: Patch Available) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Patch Available (was: Open) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.1.patch Hi [~ashutoshc], Please take a look at patch. Now trunc() will accept date or number type arguments and it will behave as below trunc(date, fmt) / trunc(N,D) - Returns If input is date returns date with the time portion of the day truncated to the unit specified by the format model fmt. If you omit fmt, then date is truncated to "the nearest day. It now only supports 'MONTH'/'MON'/'MM' and 'YEAR'/''/'YY' as format. If input is a number group returns N truncated to D decimal places. If D is omitted, then N is truncated to 0 places. D can be negative to truncate (make zero) D digits left of the decimal point. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: (was: HIVE-14582.1.patch) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667030#comment-15667030 ] Chinna Rao Lalam commented on HIVE-14582: - Hi [~ashutoshc], With the current patch all positive scenarios are working and all positive tests are passed locally. Negative tests are failing udf_trunc_error1.q, udf_trunc_error2.q, as the exception message was changed and it is executing trunc number. Fixing negative tests. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.1.patch > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: (was: HIVE-14582.1.patch) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.1.patch Hi [~ashutoshc], , Updated the initial patch with overriding the function, it is not yet ready for commit, I am working on this. Sorry for the delay last 2 weeks I am in unplanned personal leave and not able to show the progress. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604345#comment-15604345 ] Chinna Rao Lalam commented on HIVE-14582: - Hi [~ashutoshc] I will update patch in 2 days. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513101#comment-15513101
]
Chinna Rao Lalam commented on HIVE-14582:
-
Hi [~niklaus.xiao]. In hive already one function is there with the name
*trunc* related to date function.
{quote}
trunc(string date, string format)
Returns date truncated to the unit specified by the format (as of Hive 1.2.0).
Supported formats: MONTH/MON/MM, YEAR//YY. Example: trunc('2015-03-17',
'MM') = 2015-03-01.
{quote}
I am checking which is better like renaming the existing one or this function,
like *truncnumber* something like this. I need to add some more tests and some
improvements. I will update along with this.
Renaming existing function name effect backward compatibility. Any suggestions.
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.patch > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14032) INSERT OVERWRITE command failed with case sensitive partition key names
[
https://issues.apache.org/jira/browse/HIVE-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468082#comment-15468082
]
Chinna Rao Lalam commented on HIVE-14032:
-
Hi [~ashutoshc],
Sorry for the late reply..
When we execute above queries this is the exception. It was failed in
validation because of case sensitive partition name(PARTITIONID).
{code}
hive> INSERT OVERWRITE TABLE sample2 PARTITION (PARTITIONID = 'one') if not
exists select a,b from sample1;
FAILED: ValidationFailureSemanticException Partition spec {PARTITIONID=one}
contains non-partition columns
{code}
This is the log..
{code}
2016-09-07T04:54:45,903 ERROR [56697602-dac6-4f1f-b49f-fd86a602883a main]:
ql.Driver (SessionState.java:printError(1053)) - FAILED:
ValidationFailureSemanticException Partition spec {PARTITIONID=one} contains
non-partition columns
org.apache.hadoop.hive.ql.metadata.Table$ValidationFailureSemanticException:
Partition spec {PARTITIONID=one} contains non-partition columns
at
org.apache.hadoop.hive.ql.metadata.Table.validatePartColumnNames(Table.java:382)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.validatePartSpec(BaseSemanticAnalyzer.java:1498)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.doPhase1(SemanticAnalyzer.java:1662)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.doPhase1(SemanticAnalyzer.java:1686)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10748)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10807)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
{code}
Same queries without *IF NOT EXISTS* clause it is successful.
{code}
INSERT OVERWRITE TABLE sample2 PARTITION (PARTITIONID = 'one') select a,b from
sample1;
{code}
> INSERT OVERWRITE command failed with case sensitive partition key names
> ---
>
> Key: HIVE-14032
> URL: https://issues.apache.org/jira/browse/HIVE-14032
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.1
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14032.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-14582: --- Assignee: Chinna Rao Lalam > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14571) Document configuration hive.msck.repair.batch.size
[
https://issues.apache.org/jira/browse/HIVE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-14571.
-
Resolution: Fixed
Updated wiki doc.
> Document configuration hive.msck.repair.batch.size
> --
>
> Key: HIVE-14571
> URL: https://issues.apache.org/jira/browse/HIVE-14571
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Labels: TODOC2.2
> Fix For: 2.2.0
>
>
> Update here
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)]
> {quote}
> When there is a large number of untracked partitions for the MSCK REPAIR
> TABLE command, there is a provision to run the msck repair table batch wise
> to avoid OOME. By giving the configured batch size for the property
> *hive.msck.repair.batch.size* it can run in the batches internally. The
> default value of the property is zero, it means it will execute all the
> partitions at one short.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10058) Log the information of cached RDD [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-10058. - Resolution: Not A Problem > Log the information of cached RDD [Spark Branch] > > > Key: HIVE-10058 > URL: https://issues.apache.org/jira/browse/HIVE-10058 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Fix For: spark-branch > > Attachments: HIVE-10058.1-spark.patch, HIVE-10058.2-spark.patch > > > Log the cached RDD Id's at info level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10058) Log the information of cached RDD [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-10058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-10058: Status: Open (was: Patch Available) > Log the information of cached RDD [Spark Branch] > > > Key: HIVE-10058 > URL: https://issues.apache.org/jira/browse/HIVE-10058 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Fix For: spark-branch > > Attachments: HIVE-10058.1-spark.patch, HIVE-10058.2-spark.patch > > > Log the cached RDD Id's at info level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-6963) Beeline logs are printing on the console
[ https://issues.apache.org/jira/browse/HIVE-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-6963. Resolution: Not A Problem Please open new jira, still it is a problem on master. > Beeline logs are printing on the console > > > Key: HIVE-6963 > URL: https://issues.apache.org/jira/browse/HIVE-6963 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Attachments: HIVE-6963.patch > > > beeline logs are not redirected to the log file. > If log is redirected to log file, only required information will print on the > console. > This way it is more easy to read the output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-6925) show query progress in Beeline
[ https://issues.apache.org/jira/browse/HIVE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-6925. Resolution: Duplicate > show query progress in Beeline > -- > > Key: HIVE-6925 > URL: https://issues.apache.org/jira/browse/HIVE-6925 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 0.12.0 >Reporter: Gwen Shapira >Assignee: Chinna Rao Lalam > > In the "old" Hive CLI, the MR output was written to screen. Making it easy to > watch the progress - map and reduce % done. > In Beeline, there is no output until the query is done (or fails). Showing > some kind of progress indicator will be nice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-6598) Importing the project into eclipse as maven project have some issues
[ https://issues.apache.org/jira/browse/HIVE-6598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-6598. Resolution: Not A Problem Please open new jira, still it is a problem on master. > Importing the project into eclipse as maven project have some issues > > > Key: HIVE-6598 > URL: https://issues.apache.org/jira/browse/HIVE-6598 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0 > Environment: Windows 8 ,Eclipse Kepler and Maven 3.1.1 >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Attachments: HIVE-6598.patch > > > Importing the project into eclipse as maven project throwing these problems. > Plugin execution not covered by lifecycle configuration: > org.apache.maven.plugins:maven-antrun-plugin:1.7:run (execution: > setup-test-dirs, phase: process-test-resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6598) Importing the project into eclipse as maven project have some issues
[ https://issues.apache.org/jira/browse/HIVE-6598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-6598: --- Status: Open (was: Patch Available) > Importing the project into eclipse as maven project have some issues > > > Key: HIVE-6598 > URL: https://issues.apache.org/jira/browse/HIVE-6598 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0 > Environment: Windows 8 ,Eclipse Kepler and Maven 3.1.1 >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Attachments: HIVE-6598.patch > > > Importing the project into eclipse as maven project throwing these problems. > Plugin execution not covered by lifecycle configuration: > org.apache.maven.plugins:maven-antrun-plugin:1.7:run (execution: > setup-test-dirs, phase: process-test-resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-3336) [Hive-Hbase Integration]:Data is not inserted into the hbase table from the hive table if the value is null even if the row key is not null
[
https://issues.apache.org/jira/browse/HIVE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-3336.
Resolution: Not A Problem
Please open new jira, still it is a problem on master.
> [Hive-Hbase Integration]:Data is not inserted into the hbase table from the
> hive table if the value is null even if the row key is not null
> ---
>
> Key: HIVE-3336
> URL: https://issues.apache.org/jira/browse/HIVE-3336
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 0.9.0
>Reporter: Kristam Subba Swathi
>Assignee: Chinna Rao Lalam
>
> Scenario
>
> 1.create a new HBase table which is to be managed by Hive, use the STORED BY
> clause on CREATE TABLE as follows
> Query:
> {noformat}
> CREATE TABLE hbase_table_join(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "xyz_join");
> {noformat}
> 2.Create two tables in hive and load data into the table as follows
> table1:
> {noformat}
> describe hive_hbase_1;
> OK
> empno int
> ename string
> deptnoint
> hive> select * from hive_hbase_1;
> OK
> 1 a 1
> 2 q 10
> 3 w 12
> 4 w 12
> 4 w 12
> 5 r 13
> {noformat}
> table2:
> {noformat}
> hive> describe hive_hbase_2;
> OK
> deptnoint
> dname string
> count int
> Time taken: 0.214 seconds
> hive> select * from hive_hbase_2;
> OK
> 11a 11
> 12q 101
> 13w 121
> 14w 121
> 2 we13NULL
> 4 w 12
> 5 r 13
> {noformat}
> 3.Insert the join result of these tables into hbase_table_join as follows
> {noformat}
> hive> insert into table hbase_table_join select /*+mapjoin(hive_hbase_1)*/
> hive_hbase_2.deptno,hive_hbase_1.ename from hive_hbase_1 right outer join
> hive_hbase_2 on hive_hbase_1.deptno=hive_hbase_2.deptno;
> {noformat}
> Actual result of the select query is
> ---
> {noformat}
> 11 NULL
> 12 w
> 12 w
> 12 w
> 13 r
> 14 NULL
> 2 NULL
> 4 NULL
> 5 NULL
> {noformat}
> But the above insert query is failing because of the following exception
> -
> {noformat}
> 2012-07-17 18:41:45,003 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"deptno":11,"dname":"a","count":11}
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"deptno":11,"dname":"a","count":11}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: No columns to insert
> at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:969)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:762)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:748)
> at
> org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat$1.write(HiveHBaseTableOutputFormat.java:98)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-3223) Alter table rename is failing since FileSystem cache is not working in hadoop 0.23.1
[ https://issues.apache.org/jira/browse/HIVE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-3223. Resolution: Not A Problem Please open new jira, still it is a problem on master. > Alter table rename is failing since FileSystem cache is not working in hadoop > 0.23.1 > > > Key: HIVE-3223 > URL: https://issues.apache.org/jira/browse/HIVE-3223 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Kristam Subba Swathi >Assignee: Chinna Rao Lalam > > The following queries are not working .. > 1)Rename Table > ALTER TABLE table_name RENAME TO new_table_name > 2)Rename partition > ALTER TABLE table_name partition (pCol1='old_part1:', pcol2='old_part2:') > rename to partition (pCol1='new_part1:', pcol2='new_part2:') -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-3222) Number of File handlers are increasing for RunJar
[ https://issues.apache.org/jira/browse/HIVE-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-3222. Resolution: Not A Problem Please open new jira, still it is a problem on master. > Number of File handlers are increasing for RunJar > - > > Key: HIVE-3222 > URL: https://issues.apache.org/jira/browse/HIVE-3222 > Project: Hive > Issue Type: Bug > Components: Server Infrastructure >Affects Versions: 0.9.0 >Reporter: Kristam Subba Swathi >Assignee: Chinna Rao Lalam > > Run an application which includes all queries like create,load and select for > about 12 hours > Check the lsof of the runjar > lsof -p $pid gives the details of the file handlers for a particular process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-2663) DynamicSerDeTypeList.serialize() method have null check for the "nullProtocol" it needs to be change.
[
https://issues.apache.org/jira/browse/HIVE-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-2663:
---
Status: Open (was: Patch Available)
> DynamicSerDeTypeList.serialize() method have null check for the
> "nullProtocol" it needs to be change.
> --
>
> Key: HIVE-2663
> URL: https://issues.apache.org/jira/browse/HIVE-2663
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hadoop 0.20.1, Hive0.9.0 and SUSE Linux Enterprise
> Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Attachments: HIVE-2663.patch
>
>
> In DynamicSerDeTypeList.serialize() method have the null check for the
> "nullProtocol" like
> if (element == null && nullProtocol != null) {
> here if element= null and nullProtocol=null in this case it will execute else
> block with element=null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-2663) DynamicSerDeTypeList.serialize() method have null check for the "nullProtocol" it needs to be change.
[
https://issues.apache.org/jira/browse/HIVE-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-2663.
Resolution: Not A Problem
Please open new jira, still it is a problem on master.
> DynamicSerDeTypeList.serialize() method have null check for the
> "nullProtocol" it needs to be change.
> --
>
> Key: HIVE-2663
> URL: https://issues.apache.org/jira/browse/HIVE-2663
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Environment: Hadoop 0.20.1, Hive0.9.0 and SUSE Linux Enterprise
> Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Attachments: HIVE-2663.patch
>
>
> In DynamicSerDeTypeList.serialize() method have the null check for the
> "nullProtocol" like
> if (element == null && nullProtocol != null) {
> here if element= null and nullProtocol=null in this case it will execute else
> block with element=null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-2512) After HIVE-2145, Hive disallow any use of function in cluster-by clause
[ https://issues.apache.org/jira/browse/HIVE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-2512: --- Status: Open (was: Patch Available) > After HIVE-2145, Hive disallow any use of function in cluster-by clause > --- > > Key: HIVE-2512 > URL: https://issues.apache.org/jira/browse/HIVE-2512 > Project: Hive > Issue Type: Bug >Reporter: Ning Zhang >Assignee: Chinna Rao Lalam > Attachments: HIVE-2512.1.patch, HIVE-2512.2.patch, HIVE-2512.patch > > > After HIVE-2145, the following query returns a semantic analysis error: > FROM src SELECT * cluster by rand(); > FAILED: Error in semantic analysis: functions are not supported in order by > Looking back at HIVE-2145, it's clear that the patch is more restrictive than > necessary. > Chinna, are you able to work on it? Please let me know if you don't have > cycles to do it now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2512) After HIVE-2145, Hive disallow any use of function in cluster-by clause
[ https://issues.apache.org/jira/browse/HIVE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2512. Resolution: Not A Problem Please open new jira, still it is a problem on master. > After HIVE-2145, Hive disallow any use of function in cluster-by clause > --- > > Key: HIVE-2512 > URL: https://issues.apache.org/jira/browse/HIVE-2512 > Project: Hive > Issue Type: Bug >Reporter: Ning Zhang >Assignee: Chinna Rao Lalam > Attachments: HIVE-2512.1.patch, HIVE-2512.2.patch, HIVE-2512.patch > > > After HIVE-2145, the following query returns a semantic analysis error: > FROM src SELECT * cluster by rand(); > FAILED: Error in semantic analysis: functions are not supported in order by > Looking back at HIVE-2145, it's clear that the patch is more restrictive than > necessary. > Chinna, are you able to work on it? Please let me know if you don't have > cycles to do it now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2469) Error message is wrong if sortby contains function
[ https://issues.apache.org/jira/browse/HIVE-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2469. Resolution: Not A Problem > Error message is wrong if sortby contains function > -- > > Key: HIVE-2469 > URL: https://issues.apache.org/jira/browse/HIVE-2469 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.9.0 > Environment: Hadoop 0.20.1, Hive0.9.0 and SUSE Linux Enterprise > Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > > select key, count(1) cnt from src group by key sort by count(1) limit 10; > Exception Message is: > FAILED: Error in semantic analysis: functions are not supported in order by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2180) Log message improvement and use StringBuilder for constructing messages instead of String
[
https://issues.apache.org/jira/browse/HIVE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-2180.
Resolution: Not A Problem
> Log message improvement and use StringBuilder for constructing messages
> instead of String
> -
>
> Key: HIVE-2180
> URL: https://issues.apache.org/jira/browse/HIVE-2180
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.5.0, 0.8.0
> Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
> Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Attachments: HIVE-2180.patch
>
>
> Need to properly format/present the error messages for better traceability.
> For example in classes like ScriptOperator, FetchOperator, PartitionPruner
> etc,
> {noformat}
> LG.info("Executing " + Arrays.asList(wrappedCmdArgs));
> LOG.info("tablename="+ hconf.get(HiveConf.ConfVars.HIVETABLENAME.varname));
> {noformat}
> The two lines which are being logged separately can be clubbed using string
> builder and logged at once like
> {noformat}
> StringBuilder sb = new StringBuilder("Executing ");
> sb.append(Arrays.asList(wrappedCmdArgs));
> sb.append("\ntablename=");
> sb.append(hconf.get(HiveConf.ConfVars.HIVETABLENAME.varname));
> LOG.info(sb.toString());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-2205) Miscellaneous code improvements in all the packages
[ https://issues.apache.org/jira/browse/HIVE-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2205. Resolution: Not A Problem > Miscellaneous code improvements in all the packages > --- > > Key: HIVE-2205 > URL: https://issues.apache.org/jira/browse/HIVE-2205 > Project: Hive > Issue Type: Bug > Components: CLI, Query Processor, Serializers/Deserializers, Server > Infrastructure > Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise > Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Attachments: HIVE-2205.patch > > > Miscellaneous code improvements from all the packages and some code cleanup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2179) Miscellaneous code improvements in the ql,serde,metastore and contrib packages
[ https://issues.apache.org/jira/browse/HIVE-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2179. Resolution: Not A Problem > Miscellaneous code improvements in the ql,serde,metastore and contrib packages > -- > > Key: HIVE-2179 > URL: https://issues.apache.org/jira/browse/HIVE-2179 > Project: Hive > Issue Type: Bug > Components: Contrib, Metastore, Query Processor, > Serializers/Deserializers >Affects Versions: 0.5.0, 0.8.0 > Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise > Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam >Priority: Minor > > Miscellaneous code improvements in the ql,serde,metastore and contrib > packages like > Avoided creation of variables inside loop and log message improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2177) JLint Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2177. Resolution: Not A Problem > JLint Comments fixes > > > Key: HIVE-2177 > URL: https://issues.apache.org/jira/browse/HIVE-2177 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.5.0, 0.8.0 > Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise > Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam > Attachments: HIVE-2177.patch > > > Fix JLint Comments. > About JLint Tool : > Jlint will check your Java code and find bugs, inconsistencies and > synchronization problems by doing data flow analysis and building the lock > graph. > http://jlint.sourceforge.net/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2111) NullPointerException on select * with table using RegexSerDe and partitions
[ https://issues.apache.org/jira/browse/HIVE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2111. Resolution: Not A Problem Please open new jira, still it is a problem on master. > NullPointerException on select * with table using RegexSerDe and partitions > --- > > Key: HIVE-2111 > URL: https://issues.apache.org/jira/browse/HIVE-2111 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.7.0 > Environment: Amazon Elastic Mapreduce >Reporter: Marc Harris >Assignee: Chinna Rao Lalam > Attachments: HIVE-2111.1.patch, HIVE-2111.2.patch, HIVE-2111.patch > > > When querying against a table that is partitioned, and uses RegexSerde, > select with explicit columns works, but "select *" results in a > NullPointerException > To reproduce: > 1) create a table containing the following text (notice the blank line): > start > fillerdatafillerdatafiller > fillerdata2fillerdata2filler > =end= > 2) copy the file to hdfs: > hadoop dfs -put foo.txt test/part1=x/foo.txt > 3) run the following hive commands to create a table: > add jar s3://elasticmapreduce/samples/hive/jars/hive_contrib.jar; > drop table test; > create external table test(col1 STRING, col2 STRING) > partitioned by (part1 STRING) > row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' > with serdeproperties ( "input.regex" = "^\(.*data\)\(.*data\).*$") > stored as textfile > location 'hdfs:///user/hadoop/test'; > alter table test add partition (part1='x'); > (Note that the text processor seems to have mangled the regex a bit. Inside > each pair of parentheses should be dot star data. After the second pair of > parentheses should be dot start dollar). > 4) select from it with explicit columns: > select part1, col1, col2 from test; > outputs: > OK > x fillerdata fillerdata > x NULLNULL > x fillerdata 2fillerdata > 5) select from it with * columns > select * from test; > outputs: > Failed with exception java.io.IOException:java.lang.NullPointerException > 11/04/12 14:28:27 ERROR CliDriver: Failed with exception > java.io.IOException:java.lang.NullPointerException > java.io.IOException: java.lang.NullPointerException > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:149) > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1039) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) > at > org.apache.hadoop.hive.cli.CliDriver.processLineInternal(CliDriver.java:228) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:209) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:398) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.NullPointerException > at java.util.ArrayList.addAll(ArrayList.java:472) > at > org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:144) > at > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:357) > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:141) > ... 10 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2099) GROUP BY rules not applied correctly for select *
[ https://issues.apache.org/jira/browse/HIVE-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2099. Resolution: Cannot Reproduce Please open new jira, still it is a problem on master. > GROUP BY rules not applied correctly for select * > - > > Key: HIVE-2099 > URL: https://issues.apache.org/jira/browse/HIVE-2099 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.6.0, 0.7.0, 0.8.0, 0.9.0 >Reporter: John Sichi >Assignee: Chinna Rao Lalam > Attachments: HIVE-2099.1.patch, HIVE-2099.patch > > > This fails as expected: > select foo, bar from pokes group by foo; > This succeeds, which is incorrect: > select * from pokes group by foo; > I verified this as far back as 0.6, so maybe it has always been this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2041) Log message improvement for the ScriptOperator.java
[ https://issues.apache.org/jira/browse/HIVE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-2041. Resolution: Invalid > Log message improvement for the ScriptOperator.java > --- > > Key: HIVE-2041 > URL: https://issues.apache.org/jira/browse/HIVE-2041 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.7.0 > Environment: Hadoop 0.20.1, Hive0.7.0 and SUSE Linux Enterprise > Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5). >Reporter: Chinna Rao Lalam >Assignee: Chinna Rao Lalam >Priority: Minor > Attachments: HIVE-2041.Patch > > > Added some new log messages and updated exception message in the existing log > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2017) Driver.execute() should maintaining SessionState in case of runtime errors
[
https://issues.apache.org/jira/browse/HIVE-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-2017.
Resolution: Not A Problem
This is an old issue. This is not an issue on the master.
> Driver.execute() should maintaining SessionState in case of runtime errors
> --
>
> Key: HIVE-2017
> URL: https://issues.apache.org/jira/browse/HIVE-2017
> Project: Hive
> Issue Type: Bug
>Reporter: Ning Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-2017.1.patch, HIVE-2017.2.patch, HIVE-2017.3.patch,
> HIVE-2017.4.patch
>
>
> Here's a snippet from Driver.execute():
> {code}
> // TODO: This error messaging is not very informative. Fix that.
> errorMessage = "FAILED: Execution Error, return code " + exitVal
> + " from "
> + tsk.getClass().getName();
> SQLState = "08S01";
> console.printError(errorMessage);
> if (running.size() != 0) {
> taskCleanup();
> }
> return 9;
> {code}
> I simply returned in case of runtime errors without maintaining SessionState.
> It could cause resource leak mentioned in HIVE-1959.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13604) Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
[
https://issues.apache.org/jira/browse/HIVE-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-13604:
Priority: Major (was: Trivial)
> Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
> ---
>
> Key: HIVE-13604
> URL: https://issues.apache.org/jira/browse/HIVE-13604
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Yuriy Plysyuk
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-13604.1.patch, HIVE-13604.2.patch, HIVE-13604.patch
>
>
> When trying to create view that exists with statement:
> CREATE VIEW IF NOT EXISTS dummy_table ...
> Next error is logged:
> ERROR RetryingHMSHandler:190 - AlreadyExistsException(message:Table
> dummy_view already exists)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1296)
> ...
> The same when creating schema using:
> CREATE SCHEMA IF NOT EXISTS ...
> Error should not be logged as it confuses.
> For
> CREATE TABLE IF NOT EXISTS ...
> it works fine. I checked that there is code to handle this in:
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> could you please add similar check for creating views and schema?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14571) Document configuration hive.msck.repair.batch.size
[
https://issues.apache.org/jira/browse/HIVE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431311#comment-15431311
]
Chinna Rao Lalam commented on HIVE-14571:
-
Thanks for the review [~leftylev]. Please give me edit permissions for the
wiki. My username "chinnaraol" .
> Document configuration hive.msck.repair.batch.size
> --
>
> Key: HIVE-14571
> URL: https://issues.apache.org/jira/browse/HIVE-14571
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Labels: TODOC2.2
> Fix For: 2.2.0
>
>
> Update here
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)]
> {quote}
> When there is a large number of untracked partitions for the MSCK REPAIR
> TABLE command, there is a provision to run the msck repair table batch wise
> to avoid OOME. By giving the configured batch size for the property
> *hive.msck.repair.batch.size* it can run in the batches internally. The
> default value of the property is zero, it means it will execute all the
> partitions at one short.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-12077: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Fix For: 2.2.0 > > Attachments: HIVE-12077.1.patch, HIVE-12077.2.patch, > HIVE-12077.3.patch, HIVE-12077.4.patch, HIVE-12077.5.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
