Thanks for bringing this up, Saisai. I tried to do this a couple of months ago, but ran into a problem with dependency locks. I couldn't get two different versions of Spark packages in the build with baseline, but maybe I was missing something. If you can get it working, I think it's a great idea to get this into master.
Otherwise, I was thinking about proposing an 0.8.0 release in the next month or so based on Spark 2.4. Then we could merge the branch into master and do another release for Spark 3.0 when it's ready. rb On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao <sai.sai.s...@gmail.com> wrote: > Hi team, > > I was thinking of merging spark-3 branch into master, also per the > discussion before we could make spark-2 and spark-3 coexisted into 2 > different sub-modules. With this, one build could generate both spark-2 and > spark-3 runtime jars, user could pick either at preference. > > One concern is that they share lots of common code in read/write path, > this will increase the maintenance overhead to keep consistency of two > copies. > > So I'd like to hear your thoughts, any suggestions on it? > > Thanks > Saisai > -- Ryan Blue Software Engineer Netflix
