[
https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978997#action_12978997
]
Ning Zhang commented on HIVE-1900:
----------------------------------
Namit, do you mean bucketized sort-merge join? In that case don't you need to
use a specialized InputFormat and RecordReader? If we allow mappers get inputs
from multiple partitions, we need to ensure HiveInputFormat and
CombineHiveInputFormat and the RecordReaders be partition aware.
2) is important because we don't want to merge different partitions in one
file. Otherwise you need a dynamic partition insert for the merge which may
generate multiple small files for a partition again.
3) If TableScanOperator can take multiple partitions, the stats has to be
gathered according to the input partition column values. Currently the
partition column value is checked for the 1st row and assumes all the rows have
the same partitioning column value. If we allow multiple partitions in a
mapper, we have to check the partition column values for each row.
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, a mapper only spans a single partition which creates a problem in
> the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across
> multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
