Re: [SS] full outer stream-stream join

2020-11-22 Thread Jungtaek Lim
Adding rationalization here, my request for raising the thead to dev mailing list is, to figure out possible reasons not having full outer join at the moment when adding left/right outer join. This is rather historical knowledge, so I have no idea about this. Most likely a limited number of folks

Seeking committers' help to review on SS PR

2020-11-22 Thread Jungtaek Lim
Hi devs, I have been struggling to find reviewers who are committers, to get my PR [1] for SPARK-27237 [2] reviewed. The PR was submitted on Mar. 2019 (1.5 years ago), and somehow it got two approvals from contributors working on the SS area, but still doesn't get any committers' traction to

Custom file datasource implementations

2020-11-22 Thread Navin Viswanath
Hi, I’m looking into upgrading our Spark version from 2.4 to 3.0 and noticed that there was an API change that seems unavoidable. I wanted to check whether our implementation is the ideal way to do this and if this change is necessary. We have files on HDFS in thrift format and implement custom