[
https://issues.apache.org/jira/browse/TAJO-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105450#comment-14105450
]
Jihoon Son commented on TAJO-986:
---------------------------------
Hi [~mhthanh], thanks for nice finding and your efforts.
The first one is absolutely a bug. Actually, the task size of the scheduler
context means the input size for each task. Thus, it should be changed to have
the length of a fragment instead of the total fragment length.
For the second one, I should first introduce lazy task scheduler. Tajo has two
kinds of task schedulers, default task scheduler and lazy task scheduler. The
latter is used only for HDFS scan, and proposed to reduce the number of tasks
by allowing a task to process several HDFS blocks. Users can set the data size
processed per task by configuring ConfVars.TASK_DEFAULT_SIZE. The second one is
related with lazy task scheduler. In my opinion, we need to improve the coding
convention, but it will work well.
If you have any questions, please feel free to ask me.
Thanks!
> Task scheduler gets incorrect task size
> ---------------------------------------
>
> Key: TAJO-986
> URL: https://issues.apache.org/jira/browse/TAJO-986
> Project: Tajo
> Issue Type: Bug
> Reporter: Mai Hai Thanh
> Assignee: Mai Hai Thanh
> Attachments: TAJO-986.140812.patch.txt
>
>
> In function {{scheduleFragmentsForLeafQuery}} in file SubQuery.java, the
> following 2 lines exist
> {code}
> subQuery.schedulerContext.setTaskSize(fragments.size());
> ...
> subQuery.schedulerContext.setTaskSize(conf.getIntVar(ConfVars.TASK_DEFAULT_SIZE)
> * 1024 * 1024);
> {code}
> It is very likely that one of them is not correct.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
