[Java] impact of java 10 on spark dev

2018-05-16 Thread xmehaut
Hello, i would like to know what coudl be the impacts of java 10+ on spark. I know that spark is written in scala, but the last versions of java include many improvements, especially in the jvm or in the delivery process (modules, jit, memory mngt, ...) which could benefit to spark. regards

Re: [Arrow][Dremio]

2018-05-14 Thread xmehaut
Hi Michaƫl, I'm not an expert of Dremio, i just try to evaluate the potential of this techno and what impacts it could have on spark, and how they can work together, or how spark could use even further arrow internally along the existing algorithms. Dremio has already a quite rich api set

[Arrow][Dremio]

2018-05-13 Thread xmehaut
Hello, I've some question about Spark and Apache Arrow. Up to now, Arrow is only used for sharing data between Python and Spark executors instead of transmitting them through sockets. I'm studying currently Dremio as an interesting way to access multiple sources of data, and as a potential