[ https://issues.apache.org/jira/browse/CALCITE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184954#comment-17184954 ]
Rui Wang edited comment on CALCITE-4193 at 8/26/20, 6:23 AM: ------------------------------------------------------------- >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? Calcite does not care about how serialize and deserialize values, etc. is done? was (Author: amaliujia): >In particular, we have no mechanism to serialize and deserialize values, and >no on-disk format, and no scheduler that can handle asynchronous i/o. Will it work if hide such details under EnumerableExternalSort.current(), EnumerableExternalSort.next(), etc.? > Implement new sort operator: EnumerableExternalSort > --------------------------------------------------- > > Key: CALCITE-4193 > URL: https://issues.apache.org/jira/browse/CALCITE-4193 > Project: Calcite > Issue Type: Improvement > Components: core > Reporter: Ruben Q L > Priority: Major > > Sometimes we need to sort a big volume of data which does not fit into > memory. In this situation EnumerableSort will cause an OutOfMemoryError. > The solution for such a scenario will be using a different sorting algorithm: > [External Sort|https://en.wikipedia.org/wiki/External_sorting]. > The goal of the current ticket is to implement a new operator > (EnumerableExternalSort) to provide this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)