[
https://issues.apache.org/jira/browse/TEZ-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CruiseCheung updated TEZ-4301:
------------------------------
Description:
When previous span perItem is much large than others, reserved.remaining() is
large。This may create too many min sort span，cause now we use perItem length
to deduce next sort span's metaSize. In production, we may have 1 million
sort span !
Previous sort span:
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496,
reserved.metasize=16777216
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
1400122
[TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677,
time=0
[TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
(span=Span[16777216,947882594]),
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem
= 442, counter:533341
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
416
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem
= 416, counter:533342
was:
When previous span perItem is much large than others, reserved.remaining() is
large。This may create too many min sort span，cause now we use perItem length
to deduce next sort span's metaSize.
Previous sort span:
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496,
reserved.metasize=16777216
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
1400122
[TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677,
time=0
[TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
(span=Span[16777216,947882594]),
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem
= 442, counter:533341
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
416
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem
= 416, counter:533342
> TezChlid OOM cause PipelineSorter may create too many sort span
> ---------------------------------------------------------------
>
> Key: TEZ-4301
> URL: https://issues.apache.org/jira/browse/TEZ-4301
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Environment:
> Reporter: CruiseCheung
> Priority: Minor
>
> When previous span perItem is much large than others, reserved.remaining()
> is large。This may create too many min sort span，cause now we use perItem
> length to deduce next sort span's metaSize. In production, we may have 1
> million sort span !
>
> Previous sort span:
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=965738496, reserved.metasize=16777216
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
> 1400122
> [TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0,
> length=677, time=0
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
> (span=Span[16777216,947882594]),
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem =
> 442
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276247288, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1,
> perItem = 442, counter:533341
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
> 416
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276246856, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1,
> perItem = 416, counter:533342
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
