[
https://issues.apache.org/jira/browse/HIVE-24718?focusedWorklogId=555670&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555670
]
ASF GitHub Bot logged work on HIVE-24718:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Feb/21 08:12
Start Date: 22/Feb/21 08:12
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #1936:
URL: https://github.com/apache/hive/pull/1936#discussion_r580051218
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpWork.java
##########
@@ -184,42 +190,86 @@ public void setResultValues(List<String> resultValues) {
this.resultValues = resultValues;
}
- public List<Task<?>> externalTableCopyTasks(TaskTracker tracker, HiveConf
conf) {
+ public List<Task<?>> externalTableCopyTasks(TaskTracker tracker, HiveConf
conf) throws IOException {
if (conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_SKIP_IMMUTABLE_DATA_COPY))
{
return Collections.emptyList();
}
List<Task<?>> tasks = new ArrayList<>();
- while (externalTblCopyPathIterator.hasNext() && tracker.canAddMoreTasks())
{
- DirCopyWork dirCopyWork = new DirCopyWork(metricCollector,
currentDumpPath.toString());
- dirCopyWork.loadFromString(externalTblCopyPathIterator.next());
- Task<DirCopyWork> task = TaskFactory.get(dirCopyWork, conf);
- tasks.add(task);
- tracker.addTask(task);
- LOG.debug("added task for {}", dirCopyWork);
+ Retryable retryable = Retryable.builder()
+ .withHiveConf(conf)
+ .withRetryOnException(UncheckedIOException.class).build();
+ try {
+ retryable.executeCallable((Callable<Void>) ()-> {
+ try{
+ int numEntriesToSkip = tasks == null ? 0 : tasks.size();
+ while (externalTblCopyPathIterator.hasNext() &&
tracker.canAddMoreTasks()) {
+ if(numEntriesToSkip > 0) {
+ //skip tasks added in previous attempts of this retryable block
+ externalTblCopyPathIterator.next();
+ numEntriesToSkip--;
+ continue;
+ }
+ DirCopyWork dirCopyWork = new DirCopyWork(metricCollector,
currentDumpPath.toString());
+ dirCopyWork.loadFromString(externalTblCopyPathIterator.next());
+ Task<DirCopyWork> task = TaskFactory.get(dirCopyWork, conf);
+ tasks.add(task);
+ tracker.addTask(task);
Review comment:
Also they are not added again during retires. This is done depending on
the size of the 'tasks' list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 555670)
Time Spent: 3h (was: 2h 50m)
> Moving to file based iteration for copying data
> -----------------------------------------------
>
> Key: HIVE-24718
> URL: https://issues.apache.org/jira/browse/HIVE-24718
> Project: Hive
> Issue Type: Bug
> Reporter: Arko Sharma
> Assignee: Arko Sharma
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-24718.01.patch, HIVE-24718.02.patch,
> HIVE-24718.04.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
