[ https://issues.apache.org/jira/browse/IGNITE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416350#comment-16416350 ]
Denis Magda commented on IGNITE-7526: ------------------------------------- [~vozerov], according to the summary the optimizations will be applied to the client applications (who are reducers in 95% of the time). However, don't we need to do the same on the mappers side? I do remember the reportings when servers went down just because someone executed "SELECT * FROM table;" query. > SQL: Introduce memory region for reducer merge results with disk offload > ------------------------------------------------------------------------ > > Key: IGNITE-7526 > URL: https://issues.apache.org/jira/browse/IGNITE-7526 > Project: Ignite > Issue Type: Task > Components: sql > Reporter: Vladimir Ozerov > Priority: Major > > Currently all results received from map nodes are stored inside reducer's > heap memory. What is worse, in case of complex queries, such as having sorts > or groupings, need to collect all results from mappers first before final > processing could be applied. In case of big results set (or intermediate > results) this could easily lead to OOME on reducer. > To mitigate this we should introduce special memory area where intermediate > results could be stored. All final processing should be stored in the same > area as well. This area should be of limited size and should be able to > offload results to disk in case of overflow. > We could start with our B+Tree and free list and store results in some K-V > form. -- This message was sent by Atlassian JIRA (v7.6.3#76005)