[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=446448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446448 ] ASF GitHub Bot logged work on HIVE-21446: - Author: ASF GitHub Bot Created on: 16/Jun/20 11:44 Start Date: 16/Jun/20 11:44 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #569: URL: https://github.com/apache/hive/pull/569 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446448) Time Spent: 3h (was: 2h 50m) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, > HIVE-21446.03.patch > > Time Spent: 3h > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=442794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-442794 ] ASF GitHub Bot logged work on HIVE-21446: - Author: ASF GitHub Bot Created on: 09/Jun/20 15:59 Start Date: 09/Jun/20 15:59 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #569: URL: https://github.com/apache/hive/pull/569#issuecomment-641143834 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 442794) Time Spent: 2h 50m (was: 2h 40m) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, > HIVE-21446.03.patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215343
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 09:26
Start Date: 19/Mar/19 09:26
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266793956
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215343)
Time Spent: 2h 40m (was: 2.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215305
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 08:17
Start Date: 19/Mar/19 08:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266770131
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
I think, handling this case is redundant if it doesn't exist today and it is
dead code. Also, assert is the best way to represent such assumptions. I think,
this is just a copy utility class which is expected to copy the list of source
files to target directory. If at all any filtering to be done, it should be
done before we make this map.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215305)
Time Spent: 2.5h (was: 2h 20m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215284
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 06:57
Start Date: 19/Mar/19 06:57
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266751216
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
i think instead of assert putting a log is better option as the scenario is
handled
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215284)
Time Spent: 2h 20m (was: 2h 10m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215247
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 05:44
Start Date: 19/Mar/19 05:44
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266738404
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
In that case, can we just change it to assert? and add a comment for this
assumption.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215247)
Time Spent: 2h 10m (was: 2h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215240
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 05:26
Start Date: 19/Mar/19 05:26
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266735716
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
currently no ..but if some filters are added and the files to be copied are
removed then this could lead to empty list
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215240)
Time Spent: 2h (was: 1h 50m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=215223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215223
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 19/Mar/19 04:24
Start Date: 19/Mar/19 04:24
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266726914
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -57,30 +57,37 @@
private final long maxNumberOfFiles;
private final boolean hiveInTest;
private final String copyAsUser;
+ private FileSystem destinationFs;
- public CopyUtils(String distCpDoAsUser, HiveConf hiveConf) {
+ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf, FileSystem
destinationFs) {
this.hiveConf = hiveConf;
maxNumberOfFiles =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXNUMFILES);
maxCopyFileSize =
hiveConf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE);
hiveInTest = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_IN_TEST);
this.copyAsUser = distCpDoAsUser;
+this.destinationFs = destinationFs;
}
// Used by replication, copy files from source to destination. It is
possible source file is
// changed/removed during copy, so double check the checksum after copy,
// if not match, copy again from cm
- public void copyAndVerify(FileSystem destinationFs, Path destRoot,
+ public void copyAndVerify(Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
UserGroupInformation proxyUser = getProxyUser();
try {
for (Map.Entry>>
entry : map.entrySet()) {
-FileSystem sourceFs = entry.getKey();
Map> destMap = entry.getValue();
for (Map.Entry> destMapEntry :
destMap.entrySet()) {
Path destination = destMapEntry.getKey();
List fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
+ if (fileInfoList.isEmpty()) {
Review comment:
In what scenario, fileInfoList can be empty?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215223)
Time Spent: 1h 50m (was: 1h 40m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214605=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214605
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316109
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -238,23 +249,54 @@ private boolean isSourceFileMismatch(FileSystem
sourceFs, ReplChangeManager.File
return false;
}
+ private UserGroupInformation getProxyUser() throws LoginException,
IOException {
+if (copyAsUser == null) {
+ return null;
+}
+UserGroupInformation proxyUser = null;
+int currentRetry = 0;
+while (currentRetry <= MAX_IO_RETRY) {
+ try {
+UserGroupInformation ugi = Utils.getUGI();
+String currentUser = ugi.getShortUserName();
+if (!currentUser.equals(copyAsUser)) {
+ proxyUser = UserGroupInformation.createProxyUser(
+ copyAsUser, UserGroupInformation.getLoginUser());
+}
+return proxyUser;
+ } catch (IOException e) {
+currentRetry++;
+if (currentRetry <= MAX_IO_RETRY) {
+ LOG.warn("Unable to get UGI info", e);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214605)
Time Spent: 1h 20m (was: 1h 10m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214607
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:19
Start Date: 18/Mar/19 07:19
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316131
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
+if (!checkIfPathExist(sourcePath, proxyUser)) {
+ LOG.warn("Source path is missing. Ignoring exception.");
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214607)
Time Spent: 1.5h (was: 1h 20m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214608
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:19
Start Date: 18/Mar/19 07:19
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316160
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
Review comment:
change to info
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214608)
Time Spent: 1h 40m (was: 1.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214603
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266315969
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
-for (Map.Entry>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map> destMap = entry.getValue();
- for (Map.Entry> destMapEntry :
destMap.entrySet()) {
-Path destination = destMapEntry.getKey();
-List fileInfoList =
destMapEntry.getValue();
-boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
-if (!destinationFs.exists(destination)
-&& !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
-}
+UserGroupInformation proxyUser = getProxyUser();
+try {
+ for (Map.Entry>>
entry : map.entrySet()) {
+FileSystem sourceFs = entry.getKey();
Review comment:
The hive retry logic is for very limited local functionality ..for all other
places ..in case of file system error ..dlm will retry
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214603)
Time Spent: 1h (was: 50m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214604
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316079
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -136,7 +143,11 @@ private void doCopyRetry(FileSystem sourceFs,
List s
}
// looks like some network outrage, reset the file system object and
retry.
- FileSystem.closeAllForUGI(Utils.getUGI());
+ if (proxyUser == null) {
+FileSystem.closeAllForUGI(Utils.getUGI());
+ } else {
+FileSystem.closeAllForUGI(proxyUser);
+ }
sourceFs = pathList.get(0).getFileSystem(hiveConf);
destinationFs = destination.getFileSystem(hiveConf);
Review comment:
no need ..dlm or any other system calling repl load should retry in case of
file system error
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214604)
Time Spent: 1h 10m (was: 1h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214575
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266298085
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
-for (Map.Entry>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map> destMap = entry.getValue();
- for (Map.Entry> destMapEntry :
destMap.entrySet()) {
-Path destination = destMapEntry.getKey();
-List fileInfoList =
destMapEntry.getValue();
-boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
-if (!destinationFs.exists(destination)
-&& !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
-}
+UserGroupInformation proxyUser = getProxyUser();
+try {
+ for (Map.Entry>>
entry : map.entrySet()) {
+FileSystem sourceFs = entry.getKey();
Review comment:
The sourceFs should be re-get from HDFS if previous copy closed all file
systems object for proxy user.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214575)
Time Spent: 40m (was: 0.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214576
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297006
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -238,23 +249,54 @@ private boolean isSourceFileMismatch(FileSystem
sourceFs, ReplChangeManager.File
return false;
}
+ private UserGroupInformation getProxyUser() throws LoginException,
IOException {
+if (copyAsUser == null) {
+ return null;
+}
+UserGroupInformation proxyUser = null;
+int currentRetry = 0;
+while (currentRetry <= MAX_IO_RETRY) {
+ try {
+UserGroupInformation ugi = Utils.getUGI();
+String currentUser = ugi.getShortUserName();
+if (!currentUser.equals(copyAsUser)) {
+ proxyUser = UserGroupInformation.createProxyUser(
+ copyAsUser, UserGroupInformation.getLoginUser());
+}
+return proxyUser;
+ } catch (IOException e) {
+currentRetry++;
+if (currentRetry <= MAX_IO_RETRY) {
+ LOG.warn("Unable to get UGI info", e);
Review comment:
Can log the retry number in the log msg.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214576)
Time Spent: 40m (was: 0.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214574
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296535
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
+if (!checkIfPathExist(sourcePath, proxyUser)) {
+ LOG.warn("Source path is missing. Ignoring exception.");
Review comment:
Can be just info log as it is possible valid scenario in case of external
tables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214574)
Time Spent: 0.5h (was: 20m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214577
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297902
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -136,7 +143,11 @@ private void doCopyRetry(FileSystem sourceFs,
List s
}
// looks like some network outrage, reset the file system object and
retry.
- FileSystem.closeAllForUGI(Utils.getUGI());
+ if (proxyUser == null) {
+FileSystem.closeAllForUGI(Utils.getUGI());
+ } else {
+FileSystem.closeAllForUGI(proxyUser);
+ }
sourceFs = pathList.get(0).getFileSystem(hiveConf);
destinationFs = destination.getFileSystem(hiveConf);
Review comment:
We need to propagate the destinationFs to caller as well as caller still
points to old file system object which is invalid. As caller uses this object
in a loop, it is necessary to propagate it back to caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214577)
Time Spent: 50m (was: 40m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214573
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 05:27
Start Date: 18/Mar/19 05:27
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296453
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
Review comment:
It can be just info or debug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214573)
Time Spent: 20m (was: 10m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=212885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-212885 ] ASF GitHub Bot logged work on HIVE-21446: - Author: ASF GitHub Bot Created on: 14/Mar/19 03:51 Start Date: 14/Mar/19 03:51 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications URL: https://github.com/apache/hive/pull/569 … This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 212885) Time Spent: 10m Remaining Estimate: 0h > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
