Paul Rogers created DRILL-5163:
----------------------------------
Summary: External sort on Mac creates a separate child process per
spill via HDFS FS
Key: DRILL-5163
URL: https://issues.apache.org/jira/browse/DRILL-5163
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor
The external sort operator spills to disk. Spill files are created and written
using the HDFS file system. For performance, HDFS uses native libraries to
access the file system. These native libraries are not available on the Mac. As
a result, some operations are implemented using a shower, Java-only path. One
of these operations (need details) is implemented by forking a child process.
When run in a debugger on the Mac, the behavior shows up as the furious
creation and deletion of threads to manage the child processes: one per spill.
Because of this behavior, performance of external sort is slow. Of course, no
production code uses Drill on a Mac, so this is more of a nuisance than a real
bug, which is why it is marked as an improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
