[ https://issues.apache.org/jira/browse/PHOENIX-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267965#comment-15267965 ]
James Taylor commented on PHOENIX-2405: --------------------------------------- For an end-to-end test, yes spinning up a mini cluster is necessary. We amortize this cost by spinning up a single one and running lots of test suites on the same one. You can write lower level tests too that test at the iterator level. See MergeSortResultIteratorTest , SpoolingResultIteratorTest, and others in org.apache.phoenix.iterate package as well. Also, please make sure to @ mention your primary mentor, [~maryannxue]. > Improve performance and stability of server side sort for ORDER BY > ------------------------------------------------------------------ > > Key: PHOENIX-2405 > URL: https://issues.apache.org/jira/browse/PHOENIX-2405 > Project: Phoenix > Issue Type: Bug > Reporter: James Taylor > Assignee: Haoran Zhang > Labels: gsoc2016 > Fix For: 4.8.0 > > > We currently use memory mapped files to buffer data as it's being sorted in > an ORDER BY (see MappedByteBufferQueue). The following types of exceptions > have been seen to occur: > {code} > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904) > {code} > [~apurtell] has read that memory mapped files are not cleaned up after very > well in Java: > {quote} > "Map failed" means the JVM ran out of virtual address space. If you search > around stack overflow for suggestions on what to do when your app (in this > case Phoenix) encounters this issue when using mapped buffers, the answers > tend toward manually cleaning up the mapped buffers or explicitly triggering > a full GC. See > http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer > for example. There are apparently long standing JVM/JRE problems with > reclamation of mapped buffers. I think we may want to explore in Phoenix a > different way to achieve what the current code is doing. > {quote} > Instead of using memory mapped files, we could use heap memory, or perhaps > there are other mechanisms too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)