[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796261#comment-17796261
]
Sourabh Badhya commented on TEZ-3291:
-
Hi everyone,
I stumbled upon this Jira while investigating a small files issue in S3. If the
files are small and if we are trying to generate grouped splits then grouping
is not appropriate since the locality returned by S3 is the same and the patch
does not allow grouping the splits into a single split based on
tez.grouping.min-size.
Can we introduce a feature flag to disallow using locality and to generate
appropriate splits based on tez.grouping.min-size as expected since this can
improve in cases related to small files?
cc [~rajesh.balamohan] [~abstractdog] [~dkuzmenko]
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 0.9.0, 0.8.4
>
> Attachments: TEZ-3291.004.patch, TEZ-3291.2.patch, TEZ-3291.3.patch,
> TEZ-3291.4.patch, TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352171#comment-15352171
]
Siddharth Seth commented on TEZ-3291:
-
[~rajesh.balamohan] - Can this be closed. I think it's been committed.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.004.patch, TEZ-3291.2.patch, TEZ-3291.3.patch,
> TEZ-3291.4.patch, TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342883#comment-15342883
]
TezQA commented on TEZ-3291:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12812298/TEZ-3291.004.patch
against master revision 2d72a04.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1809//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1809//console
This message is automatically generated.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.004.patch, TEZ-3291.2.patch, TEZ-3291.3.patch,
> TEZ-3291.4.patch, TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342655#comment-15342655
]
Rajesh Balamohan commented on TEZ-3291:
---
Thanks [~bikassaha]. Created TEZ-3310. Will reattach 04 patch for jenkins.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.4.patch,
> TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342583#comment-15342583
]
Bikas Saha commented on TEZ-3291:
-
Sure. lets create a follow up jira.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.4.patch,
> TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342512#comment-15342512
]
Rajesh Balamohan commented on TEZ-3291:
---
[~bikassaha] - 0.4 patch has been working fine. Should we have a follow up jira
to fix it for long term and until then have this in master?
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.4.patch,
> TEZ-3291.5.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328326#comment-15328326
]
Bikas Saha commented on TEZ-3291:
-
I am with Gopal on the fragility of this workaround. Single machine is
affected. We assume localhost will not be used but it could. [~gopalv]
[~rajesh.balamohan] can we please evaluate an extension of fileSizeEstimator or
something similar to handled this. My gut feeling is that this is not the first
s3 related issue we will hit and having an abstraction in place might make
handling future issues better.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.4.patch,
> TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326714#comment-15326714
]
Gopal V commented on TEZ-3291:
--
The numDistinctLocations worries me since this impl leaks into HDFS runs as
well.
S3 and WASB return "localhost" for the hostnames (causing much damage with YARN
container allocation), while all other impls which provide actual locality
information instead of providing a dummy entry - in particular, using the
actual "127.0.0.1" IP address instead of using hostnames.
The text entry of "localhost" could be special-cased, so that this change
cannot impact HDFS installs.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.3.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326689#comment-15326689
]
Bikas Saha commented on TEZ-3291:
-
The comment could be more explicit like "this is a workaround for systems like
S3 that pass the same fake hostname for all splits"
The log could log the newDesiredSplits and also the final value of desired
splits such that we get all the info in one log.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326684#comment-15326684
]
Bikas Saha commented on TEZ-3291:
-
Would the split not have the URLs with S3 in them? Wondering how ORC split
estimator works? If it cases the spit into ORCSplit and inspects internal
members then perhaps the S3 split could also be cast into the correct object to
look at the URLs?
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.2.patch, TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326649#comment-15326649
]
Bikas Saha commented on TEZ-3291:
-
Why the numLoc=1 check only in the size < min case?
A comment before the code, explaining the above workaround would be useful.
Also a log statement.
This may affect single node cases because numLoc=1 in that case too. Is there
any way we can find out if the splits are coming from an S3 like source and use
that information instead. E.g. something similar to splitSizeEstimator that can
look at the split and return if its locations are potentially fake.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326302#comment-15326302
]
Rajesh Balamohan commented on TEZ-3291:
---
It is ready for review [~bikassaha]. haven't renamed the patch.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325748#comment-15325748
]
Bikas Saha commented on TEZ-3291:
-
[~rajesh.balamohan] Is the patch still WIP or ready for final review?
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323264#comment-15323264
]
Bikas Saha commented on TEZ-3291:
-
I will take a quick look at the patch by EOD. Looks like the main issue was
that there was some split size heuristic that needed an update to account for
cases where locations are invalid. The patch is using distinctlocations=1 as a
proxy for invalid locations. Unless this negatively affects a real single node
cluster scenario, this should be fine.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323031#comment-15323031
]
Steve Loughran commented on TEZ-3291:
-
I don't see any problem with adding fake locations; we know that it hasn't
caused problems on Azure, and it isn't going to interfere with things that
don't care about locality. If that's all that's needed for a fix, it's easily
done.
That doesn't mean that the Tez-side patch won't be good: it will work for other
filesystems. A quick check of {{RawLocalFileSystem}} shows it hard codes to
local too... if ever someone tried to run Tez against a large NFS cluster or
other distributed FS accessed via the native OS, it'd replicate the problem
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322839#comment-15322839
]
Hitesh Shah commented on TEZ-3291:
--
bq. Generating random location can be addressed
[~rajesh.balamohan] My premise is that null != null. i.e. if a split1 and
split2 say that the block location is null or unknown, we should treat the
locations of split1 and split2 as diff locations ( today they seem to be
treated as the same dummy localhost location creating the problem ) and
following the same grouping logic applied when valid locations are supplied. If
all splits with invalid locations are assigned to dummy random values, the
grouping logic would then no need to be special cases for distinct location
counts.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322781#comment-15322781
]
Rajesh Balamohan commented on TEZ-3291:
---
[~bikassaha] - The fix is the single line check in "distinctLocations.size() >
1" in the patch. Rest of them are rearragement in order to make use of the
variables.
[~hitesh] - Generating random location can be addressed via HADOOP-12878 (as
mentioned earlier). In that case, tez can go ahead in normal codepath (it might
generate slightly lower splits than the the current patch, where it sets the
desiredSplits).
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319263#comment-15319263
]
Bikas Saha commented on TEZ-3291:
-
Then that would be a bug to fix. Hopefully thats what the patch is doing.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319219#comment-15319219
]
Hitesh Shah commented on TEZ-3291:
--
bq. it will group as if all splits are on the same machine.
Seems like a better solution would be to generate a random location string for
each split ( "InvalidHost_N" where N auto-increments ) and then let the rest of
the grouping code do what it does today? Would need some better handling to
convert the split location into a host-ask against the RM I guess.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319172#comment-15319172
]
Gopal V commented on TEZ-3291:
--
bq. it will group as if all splits are on the same machine.
I think that's the exact issue here - they are not.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318941#comment-15318941
]
Bikas Saha commented on TEZ-3291:
-
IIRC they should because localhost will be treated as a valid machine name and
it will group as if all splits are on the same machine. The code itself does
the same thing by adding a same bogus machine location name for all splits that
have no location. Thereafter the code works identically for splits that have
real locations and other that have fake locations.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316870#comment-15316870
]
Bikas Saha commented on TEZ-3291:
-
Since the data fits within the max size for a grouped split its creating 1
split. Whats the issue here?
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3291) Optimize splits grouping when locality information is not available
[
https://issues.apache.org/jira/browse/TEZ-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316098#comment-15316098
]
Gopal V commented on TEZ-3291:
--
The specific scenario to detect is probably not the numDistinctLocations, but
the presence of a bogus location like "localhost".
HDFS/Isilon etc hand out explicit IP addresses, while S3 & WASB hand out the
bogus "localhost".
A simple check for the bogus location might be a better way than the total # of
locations.
> Optimize splits grouping when locality information is not available
> ---
>
> Key: TEZ-3291
> URL: https://issues.apache.org/jira/browse/TEZ-3291
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: TEZ-3291.WIP.patch
>
>
> There are scenarios where splits might not contain the location details. S3
> is an example, where all splits would have "localhost" for the location
> details. In such cases, curent split computation does not go through the
> rack local and allow-small groups optimizations and ends up creating small
> number of splits. Depending on clusters this can end creating long running
> map jobs.
> Example with hive:
> ==
> 1. Inventory table in tpc-ds dataset is partitioned and is relatively a small
> table.
> 2. With query-22, hive requests with the original splits count as 52 and
> overall length of splits themselves is around 12061817 bytes.
> {{tez.grouping.min-size}} was set to 16 MB.
> 3. In tez splits grouping, this ends up creating a single split with 52+
> files be processed in the split. In clusters with split locations, this
> would have landed up with multiple splits since {{allowSmallGroups}} would
> have kicked in.
> But in S3, since everything would have "localhost" all splits get added to
> single group. This makes things a lot worse.
> 4. Depending on the dataset and the format, this can be problematic. For
> instance, file open calls and random seeks can be expensive in S3.
> 5. In this case, 52 files have to be opened and processed by single task in
> sequential fashion. Had it been processed by multiple tasks, response time
> would have drastically reduced.
> E.g log details
> {noformat}
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Grouping splits in Tez
> 2016-06-01 13:48:08,353 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired splits: 110 too large. Desired
> splitLength: 109652 Min splitLength: 16777216 New desired splits: 1 Total
> length: 12061817 Original splits: 52
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Desired numSplits: 1 lengthPerGroup: 12061817
> numLocations: 1 numSplitsPerLocation: 52 numSplitsInGroup: 52 totalLength:
> 12061817 numOriginalSplits: 52 . Grouping by length: true count: false
> 2016-06-01 13:48:08,354 [INFO] [InputInitializer {Map 2} #0]
> |split.TezMapredSplitsGrouper|: Number of splits desired: 1 created: 1
> splitsProcessed: 52
> {noformat}
> Alternate options:
> ==
> 1. Force Hadoop to provide bogus locations for S3. But not sure, if that
> would be accepted anytime soon. Ref: HADOOP-12878
> 2. Set {{tez.grouping.min-size}} to very very low value. But should the end
> user always be doing this on query to query basis?
> 3. When {{(lengthPerGroup < "tez.grouping.min-size")}}, recompute
> desiredNumSplits only when number of distinct locations in the splits is > 1.
> This would force more number of splits to be generated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
