Hi Weijie, As it turns out, vectorized processing in Drill is more aspirational than operational at this point in time.
The code used in Drill is not actually vector-based even though the data itself is columnar. Drill generally does row-wise operations because row-wise operations fit the SQL semantics better than column-wise operations. There is generally a "loop over all rows" block of code that calls into a "do something for column a" block, followed by a "do something for column b" block, etc. For vectorized processing, the loops have to be inverted: "do all column a" followed by "do all column b". That is not always possible, however. Further, many of Drill's readers produce Nullable types. In this case, every value carries a null/not-null flag which must be checked for each data value. It is unlikely that CPU instructions exist for this case. So, a first step is to research how various operators could be vectorized. For example, how would we handle a "WHERE x = 10" case in a way that would benefit from vectorization? How about a "SUM(x)" case? Once that is sorted out (there are likely research papers that explain how others have done it), we can move onto changing the generated code (the loop-over-all-rows code) to use the newer design. Thanks, - Paul On Friday, June 29, 2018, 10:30:04 AM PDT, Aman Sinha <amansi...@gmail.com> wrote: Hi Weijie, the Panama project is an OpenJDK initialitve, right [1] ? not Intel specific. It would be quite a bit of work to test and certify with Intel's JVM which may be still in the experimental stage. Also, you may have seen the Gandiva project for Apache Arrow which aims to improve vectorization for operations on Arrow buffers (this requires integration with Arrow). I assume the test program or workload you were running was already written to exploit vectorization. Have you also looked into Drill's code-gen to see which ones are amenable to vectorization ? We could start with some small use case and expand. [1] http://www.oracle.com/technetwork/java/jvmls2016-ajila-vidstedt-3125545.pdf On Fri, Jun 29, 2018 at 3:23 AM weijie tong <tongweijie...@gmail.com> wrote: > HI all: > > I have investigate some vector friendly java codes's jit assembly code by > the JITWatch tool . Then I found that JVM did not generate the expected AVX > code.According to some conclusion from the JVM expert , JVM only supply > some restrict usage case to generate AVX code. > > I found Intel have fired a project called panama, which supply the > intrinsic vector API to actual execute AVX code. Here is the reference ( > > https://software.intel.com/en-us/articles/vector-api-developer-program-for-java > ) > . It also supports offheap calculation. From our JVM team's message, the > vector api will be released at JDK11. > > So I wonder whether we can distribute Intel's current JVM as a supplied > default JVM to users (like spark distribution with a default scala) and as > a option to rewrite parts of our operator codes according to this new > vector api. >