[
https://issues.apache.org/jira/browse/HIVE-25025?focusedWorklogId=584939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584939
]
ASF GitHub Bot logged work on HIVE-25025:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Apr/21 07:45
Start Date: 19/Apr/21 07:45
Worklog Time Spent: 10m
Work Description: wanghualei opened a new pull request #2187:
URL: https://github.com/apache/hive/pull/2187
after set Run as end user instead of Hive user , when execute insert
overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE and
source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp
method, it may cause tmp stats file lost.
example:
set hive.exec.copyfile.maxsize=0;
set hive.exec.copyfile.maxnumfiles=0;
insert overwrite table abc_new select * from abc;
select count(1) from abc_new ;
select * from abc_new ;
then the count(1) result will be 0, but select * will display real data,
because stats info lost.
fix:
distcp method different from FileUtil.copy, because distcp method ignore
parent directory, then may overwrite stats files,cause stats info lost.?
modify dst value, add parent directory to dst path, make distcp method run
result as same as?FileUtil.copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 584939)
Remaining Estimate: 71h 50m (was: 72h)
Time Spent: 10m
> Distcp In MoveTask may cause stats info lost
> --------------------------------------------
>
> Key: HIVE-25025
> URL: https://issues.apache.org/jira/browse/HIVE-25025
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: WangHualei
> Assignee: WangHualei
> Priority: Major
> Original Estimate: 72h
> Time Spent: 10m
> Remaining Estimate: 71h 50m
>
> after set _Run_ _as_ _end_ _user_ _instead_ _of_ _Hive_ _user_ , when
> execute insert overwrite , In MoveTask ,if source byte >
> HIVE_EXEC_COPYFILE_MAXSIZE and source file count>
> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause
> tmp stats file lost.
> example:
> set hive.exec.copyfile.maxsize=0;
> set hive.exec.copyfile.maxnumfiles=0;
> insert overwrite table abc_new select * from abc;
> select count(1) from abc_new ;
> select * from abc_new ;
> then the count(1) result will be 0, but select * will display real data,
> because stats info lost.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
