Hi drillers, I made a small architectural change in my first iteration of the SQL parser/validator/optimizer. I changed the LogicalPlan specification to allow query output to go to a java.util.Queue data structure rather than a file.
It was convenient for my purposes because I didn't need to allocate a temporary file, wait til the process had finished, and then start returning results. I know that in the big data world, many queries will produce large results and the results will not pass through just one processing node. But still, many queries will produce small results. A logical plan is a graph, and there is an argument that it should be symmetric. A logical plan has sources, so why shouldn't it declare sinks? I hacked the change in [ see my changes to ROPConverter in https://github.com/apache/incubator-drill/commit/7f294adb649064e26dd2f28864260b17b07315ef ] , creating a "write" operator with a special file name "socket:0", and I modified ROPConverter to create a QueueSinkROP rather than the usual JSONWriter. Let's discuss what is the correct architecture for this. Julian
