[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796261#comment-17796261
]
Sourabh Badhya commented on TEZ-3291:
-------------------------------------
Hi everyone,
I stumbled upon this Jira while investigating a small files issue in S3. If the
files are small and if we are trying to generate grouped splits then grouping
is not appropriate since the locality returned by S3 is the same and the patch
does not allow grouping the splits into a single split based on
tez.grouping.min-size.
Can we introduce a feature flag to disallow using locality and to generate
appropriate splits based on tez.grouping.min-size as expected since this can
improve in cases related to small files?
cc [~rajesh.balamohan] [~abstractdog] [~dkuzmenko]
> Optimize splits grouping when locality information is not available
> -------------------------------------------------------------------
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Fix For: 0.9.0, 0.8.4
>
> Attachments: TEZ-3291.004.patch, TEZ-3291.2.patch, TEZ-3291.3.patch,
> TEZ-3291.4.patch, TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==============
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==================
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
