[
https://issues.apache.org/jira/browse/TAJO-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110176#comment-14110176
]
Jihoon Son commented on TAJO-986:
---------------------------------
[~mhthanh], thanks for nice finding.
I added the task size to SchedulerContext to pass it from SubQuery to
TaskScheduler. As you said, the task size is not used in DefaultTaskScheduler,
but used in only LazyTaskScheduler. However, I also modified
DefaultTaskScheduler to use the same interface.
In my opinion, the task size is not important in DefaultTaskScheduler, because
the input of each task is mapped to an HDFS block. Also, I don't have any plans
to utilize task size information in DefaultTaskScheduler.
Actually, LazyTaskScheduler is not used currently, but I expect that
LazyTaskScheduler will be useful for some works such as ETL. So, I'm preparing
the performance evaluation of LazyTaskScheduler. After the evaluation, I have a
plan to clean up the code.
Anyway, it would be better to remove meaningless code such as
{{subQuery.schedulerContext.setTaskSize(fragments.size());}}, because it might
make users confusing.
Thanks,
Jihoon
> Task scheduler gets incorrect task size
> ---------------------------------------
>
> Key: TAJO-986
> URL: https://issues.apache.org/jira/browse/TAJO-986
> Project: Tajo
> Issue Type: Bug
> Reporter: Mai Hai Thanh
> Assignee: Mai Hai Thanh
> Attachments: TAJO-986.140812.patch.txt
>
>
> In function {{scheduleFragmentsForLeafQuery}} in file SubQuery.java, the
> following 2 lines exist
> {code}
> subQuery.schedulerContext.setTaskSize(fragments.size());
> ...
> subQuery.schedulerContext.setTaskSize(conf.getIntVar(ConfVars.TASK_DEFAULT_SIZE)
> * 1024 * 1024);
> {code}
> It is very likely that one of them is not correct.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
