[
https://issues.apache.org/jira/browse/ASTERIXDB-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410643#comment-16410643
]
Ali Alsuliman commented on ASTERIXDB-2336:
------------------------------------------
The issue is that the record size can be big enough that it occupies a huge
part of the memory budget leaving tiny space for other run files. It ends up
being the only run file in the sort buffer. It does not currently fail because
in all our test cases this does not happen. However, there is one test case
with big tuples. The query for that test case is ordering the result on the
primary key. The compiler will remove the local sort in each partition as it's
already sorted and it will only do the global merge through sort-merge
connector. The sort-merge connector does not have memory restriction. So, that
test case passes. When you just change that query to sort on non-primary keys,
it fails right away.
I'm facing this issue right now in my parallel sort work. As you know, the new
plan generated by the parallel sort eliminates the sort-merge connector and
keeps the local sort (of course after partitioning the tuples across the
cluster). This means there is no way to sort locally if you happen to have a
big tuple that would almost take up the memory budget. It does not have to be
that big, actually. It's the aggregate frame sizes of each run. For example, if
you have 5 runs to merge and Run1 has a frame size that is 65% of the budget,
and all the rest have a frame size that is 40% of the budget, the system would
not be able to merge them.
> Sorting big tuples fails due to memory budget
> ---------------------------------------------
>
> Key: ASTERIXDB-2336
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2336
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: HYR - Hyracks
> Reporter: Ali Alsuliman
> Assignee: Ali Alsuliman
> Priority: Major
>
> Currently, when sorting data in a partition, and during merging the sorted
> run files, the system throws an exception if one of the run files has a huge
> tuple. The reason is that such a huge tuple will take almost the entire
> budget allocated for sorting. For such cases, allow exceeding the budget in
> order to sort the data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
