[jira] [Comment Edited] (HIVE-27712) GenericUDAFNumericStatsEvaluator throws NPE
[
https://issues.apache.org/jira/browse/HIVE-27712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793539#comment-17793539
]
liang yu edited comment on HIVE-27712 at 12/6/23 7:56 AM:
--
[~zhangbutao]
Thanks for your comments.
I checked the patch, it deprecate the function compute_stats, and use other
functions to get the same result, this made too many changes.
as I have mentioned in my solution and description, this is just a bug for
functions compute_stats, which can be easily solved, if we made so many new
changes, there might be more problems and bugs.
was (Author: JIRAUSER299608):
[~zhangbutao]
I checked the patch, it deprecate the function compute_stats, and use other
functions to get the same result, this made too many changes.
as I have mentioned in my solution and description, this is just a bug for
functions compute_stats, which can be easily solved, if we made so many new
changes, there might be more problems and bugs.
> GenericUDAFNumericStatsEvaluator throws NPE
> ---
>
> Key: HIVE-27712
> URL: https://issues.apache.org/jira/browse/HIVE-27712
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-09-19-16-33-49-881.png
>
>
> using Hadoop 3.3.4
> Hive 3.1.3
> when I set the config:
> {code:java}
> set hive.groupby.skewindata=true;
> set hive.map.aggr=true; {code}
> and execute a sql with groupby execution and join execution, I got a
> NullPointerException below:
>
> !image-2023-09-19-16-33-49-881.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27712) GenericUDAFNumericStatsEvaluator throws NPE
[
https://issues.apache.org/jira/browse/HIVE-27712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793539#comment-17793539
]
liang yu commented on HIVE-27712:
-
[~zhangbutao]
I checked the patch, it deprecate the function compute_stats, and use other
functions to get the same result, this made too many changes.
as I have mentioned in my solution and description, this is just a bug for
functions compute_stats, which can be easily solved, if we made so many new
changes, there might be more problems and bugs.
> GenericUDAFNumericStatsEvaluator throws NPE
> ---
>
> Key: HIVE-27712
> URL: https://issues.apache.org/jira/browse/HIVE-27712
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-09-19-16-33-49-881.png
>
>
> using Hadoop 3.3.4
> Hive 3.1.3
> when I set the config:
> {code:java}
> set hive.groupby.skewindata=true;
> set hive.map.aggr=true; {code}
> and execute a sql with groupby execution and join execution, I got a
> NullPointerException below:
>
> !image-2023-09-19-16-33-49-881.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27712) GenericUDAFNumericStatsEvaluator throws NPE
[
https://issues.apache.org/jira/browse/HIVE-27712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17791523#comment-17791523
]
liang yu commented on HIVE-27712:
-
Solution:
add an check to verify if variable _'minFieldOI'_ and _'maxFieldOI'_ is null,
if null，call _getValueObjectInspector with no arguments._
> GenericUDAFNumericStatsEvaluator throws NPE
> ---
>
> Key: HIVE-27712
> URL: https://issues.apache.org/jira/browse/HIVE-27712
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-09-19-16-33-49-881.png
>
>
> using Hadoop 3.3.4
> Hive 3.1.3
> when I set the config:
> {code:java}
> set hive.groupby.skewindata=true;
> set hive.map.aggr=true; {code}
> and execute a sql with groupby execution and join execution, I got a
> NullPointerException below:
>
> !image-2023-09-19-16-33-49-881.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27712) GenericUDAFNumericStatsEvaluator throws NPE
[
https://issues.apache.org/jira/browse/HIVE-27712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang yu reassigned HIVE-27712:
---
Assignee: liang yu
> GenericUDAFNumericStatsEvaluator throws NPE
> ---
>
> Key: HIVE-27712
> URL: https://issues.apache.org/jira/browse/HIVE-27712
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-19-16-33-49-881.png
>
>
> using Hadoop 3.3.4
> Hive 3.1.3
> when I set the config:
> {code:java}
> set hive.groupby.skewindata=true;
> set hive.map.aggr=true; {code}
> and execute a sql with groupby execution and join execution, I got a
> NullPointerException below:
>
> !image-2023-09-19-16-33-49-881.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27917) hive metastore: MetaStoreDirectSql waste time on Testing connection to MYSQL Database
liang yu created HIVE-27917:
---
Summary: hive metastore: MetaStoreDirectSql waste time on Testing
connection to MYSQL Database
Key: HIVE-27917
URL: https://issues.apache.org/jira/browse/HIVE-27917
Project: Hive
Issue Type: Improvement
Reporter: liang yu
Assignee: liang yu
Using Hive-3.1.3. HADOOP-3.3.4.
Description:
Using hive-cli to execute sql, and then exit. hive client will take more than 1
minute to finish the connection to hive metastore. This caused my serial
execution stuck beacause each execution will create a new client and then exit.
Analysis:
When a client try to connect hive metastore server, using direct sql, it will
init a MetaStoreDirectSql Object, following is a trace of how it is created:
{code:java}
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getProductName(MetaStoreDirectSql.java:184)
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.
java:184)
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:499)
org.apache.hadoop.hive.metastore.ObjectStore.initialize(0bjectStore.java:421)
org.apache.hadoop.hive.metastore.ObjectStore.setConff(ObjectStore.java:376)
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:139)
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
org.apache.hadoop.hive.metastore.HiveMetaStoreSHMSHandler.newRawStoreForConf
(HiveMetaStore.java:720)
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf
(HiveMetaStore.java:698)
org.apache.hadoop.hive.metastore.HiveMetaStoreSHMSHandler.getMS
(HiveMetaStore.java:692) {code}
{code:java}
org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.ensureDbInit(MetaStoreDirectSql.
java:240)
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.
java:184)
org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:499)
org.apache.hadoop.hive.metastore.ObjectStore.initialize(0bjectStore.java:421)
org.apache.hadoop.hive.metastore.ObjectStore.setConff(ObjectStore.java:376)
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79)
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:139)
org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
org.apache.hadoop.hive.metastore.HiveMetaStoreSHMSHandler.newRawStoreForConf
(HiveMetaStore.java:720)
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf
(HiveMetaStore.java:698)
org.apache.hadoop.hive.metastore.HiveMetaStoreSHMSHandler.getMS
(HiveMetaStore.java:692){code}
In these two places, the hive metastore server will try to connect to mysql and
execute a simple sql, this will take tens of milliseconds which is a waste of
time cause there is a Lock hold during the init. When there gets more and more
clients try to connect the server, it will cause block and client will get
stuck.
Solution:
We use the MYSQL as our database, so we don't need to use getProductName to
determine the database type, and add a new Configuration to pass the database
type.
We have our DBA, so we don't need to use ensureDbInit to make sure that the
mysql database is ready to connect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27712) GenericUDAFNumericStatsEvaluator throws NPE
liang yu created HIVE-27712:
---
Summary: GenericUDAFNumericStatsEvaluator throws NPE
Key: HIVE-27712
URL: https://issues.apache.org/jira/browse/HIVE-27712
Project: Hive
Issue Type: Bug
Reporter: liang yu
Attachments: image-2023-09-19-16-33-49-881.png
using Hadoop 3.3.4
Hive 3.1.3
when I set the config:
{code:java}
set hive.groupby.skewindata=true;
set hive.map.aggr=true; {code}
and execute a sql with groupby execution and join execution, I got a
NullPointerException below:
!image-2023-09-19-16-33-49-881.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764575#comment-17764575
]
liang yu edited comment on HIVE-27688 at 9/13/23 10:36 AM:
---
In branch-3.1
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
Solution:
I changed the code in org/apache/hadoop/hive/ql/metadata/Hive.java: line 3771
{code:java}
throw getHiveException(e, msg); {code}
to
{code:java}
throw getHiveException(e, e.getMessage(), msg){code}
and it returns the correct error message
In master branch, it's in org/apache/hadoop/hive/ql/metadata/Hive.java: line
5110
was (Author: JIRAUSER299608):
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
Solution:
I changed the code in org/apache/hadoop/hive/ql/metadata/Hive.java: line 3771
{code:java}
throw getHiveException(e, msg); {code}
to
{code:java}
throw getHiveException(e, e.getMessage(), msg){code}
and it returns the correct error message
> hive MoveTask cannot show the correct exception message
[jira] [Comment Edited] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764585#comment-17764585
]
liang yu edited comment on HIVE-27688 at 9/13/23 10:31 AM:
---
cc [~aturoczy] [~dkuzmenko]
was (Author: JIRAUSER299608):
cc [~aturoczy]
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> I am using hive.version=3.1.3; hadoop.version=3.3.4.
> Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql.
> When I try to insert *overwrite* to an {*}existing table partition{*}, I get
> the exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
>
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
>
> But when I try to insert *into* an {*}existing table partition{*}, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
>
> {code:java}
> FAILED: Execution Error, return code l from org.
> apache.hadoop.hive.ql.metadata.HiveException: org.
> apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
> master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
> hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
> the file is not owned by hive and load data is also not ran as hive{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764585#comment-17764585
]
liang yu commented on HIVE-27688:
-
cc [~aturoczy]
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> I am using hive.version=3.1.3; hadoop.version=3.3.4.
> Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql.
> When I try to insert *overwrite* to an {*}existing table partition{*}, I get
> the exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
>
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
>
> But when I try to insert *into* an {*}existing table partition{*}, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
>
> {code:java}
> FAILED: Execution Error, return code l from org.
> apache.hadoop.hive.ql.metadata.HiveException: org.
> apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
> master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
> hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
> the file is not owned by hive and load data is also not ran as hive{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764575#comment-17764575
]
liang yu edited comment on HIVE-27688 at 9/13/23 10:19 AM:
---
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
Solution:
I changed the code in org/apache/hadoop/hive/ql/metadata/Hive.java: line 3771
{code:java}
throw getHiveException(e, msg); {code}
to
{code:java}
throw getHiveException(e, e.getMessage(), msg){code}
and it returns the correct error message
was (Author: JIRAUSER299608):
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments:
[jira] [Updated] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang yu updated HIVE-27688:
Description:
I am using hive.version=3.1.3; hadoop.version=3.3.4.
Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql.
When I try to insert *overwrite* to an {*}existing table partition{*}, I get
the exception that: Unable to move source /path/to/source to destination
/path/to/dest, which is a very regular error message, gives me no helpful
information.
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
But when I try to insert *into* an {*}existing table partition{*}, I get the
exception that: Unable to move source /path/to/source to destination
/path/to/dest as the file is not owned by hive and load data is also not ran as
hive. which gives me a very helpful error message that I should change the
hive.load.data.owner to hive.
{code:java}
FAILED: Execution Error, return code l from org.
apache.hadoop.hive.ql.metadata.HiveException: org.
apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
the file is not owned by hive and load data is also not ran as hive{code}
was:
Setting hive.load.data.owner to hive.
When I try to insert *overwrite* to an {*}existing table partition{*}, I get
the exception that: Unable to move source /path/to/source to destination
/path/to/dest, which is a very regular error message, gives me no helpful
information.
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
But when I try to insert *into* an {*}existing table partition{*}, I get the
exception that: Unable to move source /path/to/source to destination
/path/to/dest as the file is not owned by hive and load data is also not ran as
hive. which gives me a very helpful error message that I should change the
hive.load.data.owner to hive.
{code:java}
FAILED: Execution Error, return code l from org.
apache.hadoop.hive.ql.metadata.HiveException: org.
apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
the file is not owned by hive and load data is also not ran as hive{code}
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> I am using hive.version=3.1.3; hadoop.version=3.3.4.
> Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql.
> When I try to insert *overwrite* to an {*}existing table partition{*}, I get
> the exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
>
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
>
> But when I try to insert *into* an {*}existing table partition{*}, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
>
> {code:java}
> FAILED: Execution Error, return code l from org.
> apache.hadoop.hive.ql.metadata.HiveException: org.
> apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
> master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
> hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
> the file is not owned by hive and load data is also not ran as hive{code}
>
>
>
--
This message was
[jira] [Updated] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang yu updated HIVE-27688:
Description:
Setting hive.load.data.owner to hive.
When I try to insert *overwrite* to an {*}existing table partition{*}, I get
the exception that: Unable to move source /path/to/source to destination
/path/to/dest, which is a very regular error message, gives me no helpful
information.
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
hdfs://xl/user/ubd master/ubd_b_dwa.
db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
But when I try to insert *into* an {*}existing table partition{*}, I get the
exception that: Unable to move source /path/to/source to destination
/path/to/dest as the file is not owned by hive and load data is also not ran as
hive. which gives me a very helpful error message that I should change the
hive.load.data.owner to hive.
{code:java}
FAILED: Execution Error, return code l from org.
apache.hadoop.hive.ql.metadata.HiveException: org.
apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
the file is not owned by hive and load data is also not ran as hive{code}
was:
Setting hive.load.data.owner to hive.
When I try to insert overwrite to an existing table partition, I get the
exception that: Unable to move source /path/to/source to destination
/path/to/dest, which is a very regular error message, gives me no helpful
information.
!image-2023-09-13-17-40-02-981.png!
But when I try to insert into an existing table partition, I get the exception
that: Unable to move source /path/to/source to destination /path/to/dest as the
file is not owned by hive and load data is also not ran as hive. which gives me
a very helpful error message that I should change the hive.load.data.owner to
hive.
!image-2023-09-13-17-39-27-864.png!
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> Setting hive.load.data.owner to hive.
> When I try to insert *overwrite* to an {*}existing table partition{*}, I get
> the exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
>
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
>
>
>
> But when I try to insert *into* an {*}existing table partition{*}, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
>
> {code:java}
> FAILED: Execution Error, return code l from org.
> apache.hadoop.hive.ql.metadata.HiveException: org.
> apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
> master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
> hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as
> the file is not owned by hive and load data is also not ran as hive{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764575#comment-17764575
]
liang yu edited comment on HIVE-27688 at 9/13/23 10:06 AM:
---
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
was (Author: JIRAUSER299608):
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)-> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
392)->
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)-> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
392)->
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> Setting hive.load.data.owner to hive.
> When I try to insert overwrite to an existing table partition, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message,
[jira] [Commented] (HIVE-27688) hive MoveTask cannot show the correct exception message
[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764575#comment-17764575
]
liang yu commented on HIVE-27688:
-
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)-> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
392)->
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)-> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
392)->
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
> hive MoveTask cannot show the correct exception message
> ---
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
>Reporter: liang yu
>Assignee: liang yu
>Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> Setting hive.load.data.owner to hive.
> When I try to insert overwrite to an existing table partition, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
> !image-2023-09-13-17-40-02-981.png!
>
> But when I try to insert into an existing table partition, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
> !image-2023-09-13-17-39-27-864.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27688) hive MoveTask cannot show the correct exception message
[ https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang yu updated HIVE-27688: Attachment: image-2023-09-13-17-40-02-981.png image-2023-09-13-17-39-27-864.png image-2023-09-13-17-38-58-508.png Description: Setting hive.load.data.owner to hive. When I try to insert overwrite to an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest, which is a very regular error message, gives me no helpful information. !image-2023-09-13-17-40-02-981.png! But when I try to insert into an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest as the file is not owned by hive and load data is also not ran as hive. which gives me a very helpful error message that I should change the hive.load.data.owner to hive. !image-2023-09-13-17-39-27-864.png! was: Setting hive.load.data.owner to hive. When I try to insert overwrite to an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest, which is a very regular error message, gives me no helpful information. But when I try to insert into an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest as the file is not owned by hive and load data is also not ran as hive. which gives me a very helpful error message that I should change the hive.load.data.owner to hive. > hive MoveTask cannot show the correct exception message > --- > > Key: HIVE-27688 > URL: https://issues.apache.org/jira/browse/HIVE-27688 > Project: Hive > Issue Type: Bug >Reporter: liang yu >Assignee: liang yu >Priority: Major > Attachments: image-2023-09-13-17-39-27-864.png, > image-2023-09-13-17-40-02-981.png > > > Setting hive.load.data.owner to hive. > When I try to insert overwrite to an existing table partition, I get the > exception that: Unable to move source /path/to/source to destination > /path/to/dest, which is a very regular error message, gives me no helpful > information. > !image-2023-09-13-17-40-02-981.png! > > But when I try to insert into an existing table partition, I get the > exception that: Unable to move source /path/to/source to destination > /path/to/dest as the file is not owned by hive and load data is also not ran > as hive. which gives me a very helpful error message that I should change the > hive.load.data.owner to hive. > !image-2023-09-13-17-39-27-864.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27688) hive MoveTask cannot show the correct exception message
[ https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang yu updated HIVE-27688: Attachment: (was: image-2023-09-13-17-38-58-508.png) > hive MoveTask cannot show the correct exception message > --- > > Key: HIVE-27688 > URL: https://issues.apache.org/jira/browse/HIVE-27688 > Project: Hive > Issue Type: Bug >Reporter: liang yu >Assignee: liang yu >Priority: Major > Attachments: image-2023-09-13-17-39-27-864.png, > image-2023-09-13-17-40-02-981.png > > > Setting hive.load.data.owner to hive. > When I try to insert overwrite to an existing table partition, I get the > exception that: Unable to move source /path/to/source to destination > /path/to/dest, which is a very regular error message, gives me no helpful > information. > !image-2023-09-13-17-40-02-981.png! > > But when I try to insert into an existing table partition, I get the > exception that: Unable to move source /path/to/source to destination > /path/to/dest as the file is not owned by hive and load data is also not ran > as hive. which gives me a very helpful error message that I should change the > hive.load.data.owner to hive. > !image-2023-09-13-17-39-27-864.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27688) hive MoveTask cannot show the correct exception message
liang yu created HIVE-27688: --- Summary: hive MoveTask cannot show the correct exception message Key: HIVE-27688 URL: https://issues.apache.org/jira/browse/HIVE-27688 Project: Hive Issue Type: Bug Reporter: liang yu Assignee: liang yu Setting hive.load.data.owner to hive. When I try to insert overwrite to an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest, which is a very regular error message, gives me no helpful information. But when I try to insert into an existing table partition, I get the exception that: Unable to move source /path/to/source to destination /path/to/dest as the file is not owned by hive and load data is also not ran as hive. which gives me a very helpful error message that I should change the hive.load.data.owner to hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
