paul-rogers commented on PR #12745: URL: https://github.com/apache/druid/pull/12745#issuecomment-1176615784
@FrankChen021, @gianm, comment on the row vs. columnar discussion. Over on Apache Drill, our internal "value vector" formats are roughly similar to Gian's columnar frame. What we found was that, for local operators such as project, select and so on, columnar is fast. (Adding or removing a column requires adding or removing just one vector, the rest are unchanged.) Filtering is a wash. Yes, we can apply a predicate only on the target columns, but we still have to copy the entire row when removing the "misses". That is actually more costly with columns than with rows because of all the per-column twiddling needed. The killer, however, is on exchanges, such as a hash exchange. You have a batch of, say 1000 rows. You want to send it to 100 different receivers. You've got to do row-based slices based on hash, then build up an "outgoing" batch. If you buffer, say, 100 rows for each of your 100 servers (so the batch is efficient), you end up buffering 10K rows, and causing a pipeline stall. Our theory is that row-based exchanges (as we had in Apache Impala) would be more efficient: buffer when you can, but if x ms. goes by with no additional data, send what you have to keep things flowing. A row based format makes this more feasible. (Obviously lots of details that I'm skipping over.) There was a very lively debate on this over at the Drill project, if anyone is interested. In short, I like the approach of having both options in the frame abstraction: it provides tuning options. Even better would be if the surrounding code could be agnostic: a column writer is a column writer, independent of format. (There may be two different implementations, but the API is the same.) That way, a planner could decide when it is faster to be row-based vs. column based for any given operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
