[
https://issues.apache.org/jira/browse/PIG-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1290:
--------------------------------
Attachment: PIG-1290.patch
Attached patch removes the check in WeightedRangePartitioner to check that the
input is empty when quantile file is empty. There is already a test
-testEmptyStore in TestEvalPipeline2 to test that pig handles order by on empty
files fine - so this patch does not include any new tests.
> WeightedRangePartitioner should not check if input is empty if quantile file
> is empty
> -------------------------------------------------------------------------------------
>
> Key: PIG-1290
> URL: https://issues.apache.org/jira/browse/PIG-1290
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1290.patch
>
>
> Currently WeightedRangePartitioner checks if the input is also empty if the
> quantile file is empty. For this it tries to read the input (which under the
> covers will result in creating splits for the input etc). If the input is a
> directory with many files, this could result in many calls to the namenode
> from each task - this can be avoided.
> If the input is non empty and quantile file is empty, then we would error out
> anyway (this should be confirmed). Also while fixing this jira we should
> ensure that pig can still do order by on empty input.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
