[jira] [Commented] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690086#comment-17690086
]
iBenny commented on HIVE-27021:
---
[~zabetak] Thanks for the suggestion.
{quote}Those interested to enforce permissions before dropping the table can
use [metastore
pre-listeners|https://github.com/apache/hive/blob/29dc08172ba9ae3a3f51320f656994c20198c5f0/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L360].
There are also already implemented handlers that perform checks based on the
filesystem directory permissions.
{quote}
Do you mean there are already implemented handlers that perform checks based on
the filesystem directory permissions? Could you tell me which handler I should
refer to? I can only find a few PreEventListener but they are not related.
[https://github.com/apache/hive/search?q=MetaStorePreEventListener]
Or do I have to build a new PreEventListener to implement it?
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689719#comment-17689719
]
iBenny commented on HIVE-27021:
---
[~zabetak]
I am thinking before we really delete the filesystem directory, could we do
some permission checks for the filesystem directory, if the user permission is
not allowed to do the deletion, HMS can return an error to the client. I think
it can avoid inconsistency in most cases.
Current flow:
delete table metadata (return error if fails) -> delete filesystem directory
(not return error if fails)
New flow:
check permission for deleting filesystem directory (return error if fails) ->
delete table metadata (return error if fails) -> delete filesystem directory
(not return error if fails)
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688519#comment-17688519
]
iBenny edited comment on HIVE-27021 at 2/14/23 2:18 PM:
[~zabetak]
{quote}Assuming that we change the behavior for drop operations what should
happen if we delete successfully the filesystem directory of the table but then
fail to update the metadata?
{quote}
The filesystem directory of the table will be deleted and the table metadata
remains the same, but an ERROR will be returned to the HMS client. Do you
think it is acceptable?
was (Author: ibenny):
[~zabetak]
{quote}Assuming that we change the behavior for drop operations what should
happen if we delete successfully the filesystem directory of the table but then
fail to update the metadata?
{quote}
The filesystem directory of the table will be deleted and the table metadata
remains the same, but the ERROR will be returned to the HMS client. Do you
think it is acceptable?
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688519#comment-17688519
]
iBenny edited comment on HIVE-27021 at 2/14/23 2:17 PM:
[~zabetak]
{quote}Assuming that we change the behavior for drop operations what should
happen if we delete successfully the filesystem directory of the table but then
fail to update the metadata?
{quote}
The filesystem directory of the table will be deleted and the table metadata
remains the same, but the ERROR will be returned to the HMS client. Do you
think it is acceptable?
was (Author: ibenny):
[~zabetak]
{quote}Assuming that we change the behavior for drop operations what should
happen if we delete successfully the filesystem directory of the table but then
fail to update the metadata?
{quote}
The filesystem directory of the table will be deleted (permanently) and the
table metadata remains, but the ERROR will be returned to the HMS client. Do
you think it is acceptable?
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688519#comment-17688519
]
iBenny commented on HIVE-27021:
---
[~zabetak]
{quote}Assuming that we change the behavior for drop operations what should
happen if we delete successfully the filesystem directory of the table but then
fail to update the metadata?
{quote}
The filesystem directory of the table will be deleted (permanently) and the
table metadata remains, but the ERROR will be returned to the HMS client. Do
you think it is acceptable?
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685869#comment-17685869
]
iBenny edited comment on HIVE-27021 at 2/8/23 12:03 PM:
[~zabetak]
For requests create_table and add_partition, HMS call mkdir
{color:#ff}before{color} changing the metadata, and the requests return
{color:#ff}fails{color} when the exception occurs during mkdir.
For requests drop_table and drop_partition, HMS call deletedir
{color:#ff}after{color} changing the metadata, and the requests return
{color:#ff}success{color} even if the exception occurs during deletedir.
Do you think we can change the behavior of drop_table and drop_partition to the
same as create_table and add_partition's in other to ensure operation
consistency？
was (Author: ibenny):
[~zabetak]
For requests create_table and add_partition, HMS call mkdir
{color:#FF}before{color} changing the metadata, and the requests return
{color:#FF}fails{color} when the exception occurs during mkdir.
For requests drop_table and drop_partition, HMS call deletedir
{color:#FF}after{color} changing the metadata, and the requests return
{color:#FF}success{color} even if the exception occurs during deletedir.
Do you think we can change the drop_table and drop_partition to the same as
create_table and add_partition, to ensure the operation consistency？
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27021) Drop table should not be success on purge enabled tables if underlying HDFS data delete fails
[
https://issues.apache.org/jira/browse/HIVE-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685869#comment-17685869
]
iBenny commented on HIVE-27021:
---
[~zabetak]
For requests create_table and add_partition, HMS call mkdir
{color:#FF}before{color} changing the metadata, and the requests return
{color:#FF}fails{color} when the exception occurs during mkdir.
For requests drop_table and drop_partition, HMS call deletedir
{color:#FF}after{color} changing the metadata, and the requests return
{color:#FF}success{color} even if the exception occurs during deletedir.
Do you think we can change the drop_table and drop_partition to the same as
create_table and add_partition, to ensure the operation consistency？
> Drop table should not be success on purge enabled tables if underlying HDFS
> data delete fails
> -
>
> Key: HIVE-27021
> URL: https://issues.apache.org/jira/browse/HIVE-27021
> Project: Hive
> Issue Type: Bug
>Reporter: iBenny
>Priority: Minor
> Labels: backward-incompatible
>
> User A has drop privs for a table, but does not have delete privs on table
> HDFS path & table purge is enabled. When User A fires drop table command, HMS
> metadata is removed but HDFS files are left as-is & drop is reported
> successful leaving stale files.
> For example, the drop table query is completed successfully.
> {code:java}
> 0: jdbc:hive2://> drop table testtb_ex1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.032 seconds
> INFO : Executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125):
> drop table testtb_ex1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20221213121153_67cc8de3-9bd1-47c5-b2ec-13f6a0cd5125);
> Time taken: 0.413 seconds
> INFO : OK
> No rows affected (0.516 seconds) {code}
> However, the table HDFS data deletion fails
> {code:java}
> 2022-12-13 12:11:53,700 ERROR
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils: [pool-6-thread-73]:
> Got exception: org.apache.hadoop.security.AccessControlException Permission
> denied by sticky bit: user
> =oozie,
> path="/warehouse/tablespace/external/hive/testtb_ex1":hive:hive:drwxr-xr-x,
> parent="/warehouse/tablespace/external/hive":hive:hive:drwxrwxrwt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[ https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491960#comment-17491960 ] iBenny commented on HIVE-23556: --- [~kgyrtkirk] Open a new PR: https://github.com/apache/hive/pull/3021 > Support hive.metastore.limit.partition.request for get_partitions_ps > > > Key: HIVE-23556 > URL: https://issues.apache.org/jira/browse/HIVE-23556 > Project: Hive > Issue Type: Improvement >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23556.2.patch, HIVE-23556.3.patch, > HIVE-23556.4.patch, HIVE-23556.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-13884 added the configuration hive.metastore.limit.partition.request to > limit the number of partitions that can be requested. > Currently, it takes in effect for the following MetaStore APIs > * get_partitions, > * get_partitions_with_auth, > * get_partitions_by_filter, > * get_partitions_spec_by_filter, > * get_partitions_by_expr, > but not for > * get_partitions_ps, > * get_partitions_ps_with_auth. > This issue proposes to apply the configuration also to get_partitions_ps and > get_partitions_ps_with_auth. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[ https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491790#comment-17491790 ] iBenny commented on HIVE-23556: --- Since some parts of HiveMetastore.java have been moved into a file named HMSHandler.java, I am going to reopen a PR, > Support hive.metastore.limit.partition.request for get_partitions_ps > > > Key: HIVE-23556 > URL: https://issues.apache.org/jira/browse/HIVE-23556 > Project: Hive > Issue Type: Improvement >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-23556.2.patch, HIVE-23556.3.patch, > HIVE-23556.4.patch, HIVE-23556.patch > > > HIVE-13884 added the configuration hive.metastore.limit.partition.request to > limit the number of partitions that can be requested. > Currently, it takes in effect for the following MetaStore APIs > * get_partitions, > * get_partitions_with_auth, > * get_partitions_by_filter, > * get_partitions_spec_by_filter, > * get_partitions_by_expr, > but not for > * get_partitions_ps, > * get_partitions_ps_with_auth. > This issue proposes to apply the configuration also to get_partitions_ps and > get_partitions_ps_with_auth. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25794) CombineHiveRecordReader: log statements in a loop leads to memory pressure
[
https://issues.apache.org/jira/browse/HIVE-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457799#comment-17457799
]
iBenny edited comment on HIVE-25794 at 12/12/21, 3:01 AM:
--
Can we have this commit request by [~abstractdog] get approval?
was (Author: ibenny):
Can we have this Jira get approval?
> CombineHiveRecordReader: log statements in a loop leads to memory pressure
> --
>
> Key: HIVE-25794
> URL: https://issues.apache.org/jira/browse/HIVE-25794
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: iBenny
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Similar to HIVE-16150, a huge string will be built in a loop, even the log
> level is INFO. That leads to memory pressure when processing a big number of
> split files.
> From
> [CombineHiveRecordReader.java|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveRecordReader.java#L116],
> the following needs to be fixed.
> LOG.debug("Found spec for " + path + " " + otherPart + " from " +
> pathToPartInfo);
> {code}
> "TezChild" #26 daemon prio=5 os_prio=0 tid=0x7f5fd1716000 nid=0x2118a
> runnable [0x7f5f8c411000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at java.util.AbstractMap.toString(AbstractMap.java:557)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.extractSinglePartSpec(CombineHiveRecordReader.java:119)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:88)
> at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:257)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:144)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.run(MergeFileRecordProcessor.java:153)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25794) CombineHiveRecordReader: log statements in a loop leads to memory pressure
[
https://issues.apache.org/jira/browse/HIVE-25794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457799#comment-17457799
]
iBenny commented on HIVE-25794:
---
Can we have this Jira get approval?
> CombineHiveRecordReader: log statements in a loop leads to memory pressure
> --
>
> Key: HIVE-25794
> URL: https://issues.apache.org/jira/browse/HIVE-25794
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: iBenny
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Similar to HIVE-16150, a huge string will be built in a loop, even the log
> level is INFO. That leads to memory pressure when processing a big number of
> split files.
> From
> [CombineHiveRecordReader.java|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveRecordReader.java#L116],
> the following needs to be fixed.
> LOG.debug("Found spec for " + path + " " + otherPart + " from " +
> pathToPartInfo);
> {code}
> "TezChild" #26 daemon prio=5 os_prio=0 tid=0x7f5fd1716000 nid=0x2118a
> runnable [0x7f5f8c411000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at java.util.AbstractMap.toString(AbstractMap.java:557)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.extractSinglePartSpec(CombineHiveRecordReader.java:119)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:88)
> at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:257)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:144)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.run(MergeFileRecordProcessor.java:153)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-24528) Wrong implicit type conversion when comparing decimals and strings
[
https://issues.apache.org/jira/browse/HIVE-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437388#comment-17437388
]
iBenny commented on HIVE-24528:
---
[~zabetak] The issue makes different query behavior compared with hive2.1. Can
we fix this issue in a later version?
> Wrong implicit type conversion when comparing decimals and strings
> --
>
> Key: HIVE-24528
> URL: https://issues.apache.org/jira/browse/HIVE-24528
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In many cases when comparing decimals and strings (literals/columns) the
> comparison is done using doubles which can create some quite unexpected
> results in the answers of queries.
> {code:sql}
> create table t_str (str_col string);
> insert into t_str values ('1208925742523269458163819');
> select * from t_str where str_col=1208925742523269479013976;
> {code}
> The SELECT query brings up one row while the filtering value is not the same
> with the one present in the string column of the table. The problem is that
> both types are converted to doubles and due to loss of precision the values
> are deemed equal.
> The same happens during the join of a decimal with a string type.
> {code:sql}
> create table t_dec (dec_col decimal(25,0));
> insert into t_dec values (1208925742523269479013976);
> select * from t_dec inner join t_str on dec_col=str_col;
> {code}
> The join result contains one row although the values are not equal.
> Implicit type conversions are working differently in every DBMS and for some
> of them (e.g., mysql) the above behavior is normal or not allowed at all
> (e.g. Postgres).
> In the past, Hive used to compare decimal with string columns by converting
> to decimals but this behavior changed in 2.3.0 (with HIVE-13380). It seems
> that this behavior change was not intentional since following jiras (e.g.,
> HIVE-18434) imply that comparison of decimals with strings should be done
> using decimals. Since decimal is an exact numeric it appears a more
> appropriate type for comparing decimals and strings.
> The goal of this issue is to change the implicit conversion of decimals with
> strings to doubles and use decimals instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-17648) Delegation token can't be passed through from Hive to HBase when required by UDF
[ https://issues.apache.org/jira/browse/HIVE-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] iBenny updated HIVE-17648: -- Issue Type: Bug (was: Improvement) > Delegation token can't be passed through from Hive to HBase when required by > UDF > > > Key: HIVE-17648 > URL: https://issues.apache.org/jira/browse/HIVE-17648 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.1.1 >Reporter: iBenny > > When using Hive CLI to query a Hive table with UDF which needs to access > HBase with Kerberos, the delegation token can't be passed through to HBase > from Hive CLI. Usually, if we access a Hbase directly from Hive CLI, Hive CLI > will get the delegation token before the job starting. But in this case hive > CLI don't know this job need a HBase delegation token until the UDF was > running in the map task and then error happened in the map job: > GSS initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos ...)] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
