[
https://issues.apache.org/jira/browse/TEZ-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736732#comment-17736732
]
László Bodor commented on TEZ-4301:
-----------------------------------
thanks for this patch [~CruiseCheung], sorry for the late response, we forgot
this, can you please open a PR?
> TezChlid OOM cause PipelineSorter may create too many small sort span
> ---------------------------------------------------------------------
>
> Key: TEZ-4301
> URL: https://issues.apache.org/jira/browse/TEZ-4301
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Environment:
> Reporter: CruiseCheung
> Priority: Minor
> Attachments: TEZ-4301.1.patch, TEZ-4301.2.patch, TEZ-4301.3.patch
>
>
> When previous span perItem is much large than others, reserved.remaining()
> is large。This may create too many small sort span，cause now we use perItem
> length to deduce next sort span's metaSize. In production, we may have 1
> million sort span !
>
> Previous sort span:
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=965738496, reserved.metasize=16777216
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
> 1400122
> [TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0,
> length=677, time=0
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
> (span=Span[16777216,947882594]),
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem =
> 442
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276247288, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1,
> perItem = 442, counter:533341
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
> 416
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276246856, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1,
> perItem = 416, counter:533342
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
