Re: Spark 2.3 V2 Datasource API questions

2018-04-06 Thread Thakrar, Jayesh
Thank you Jose for the quick reply! I have made myself a watcher on them. From: Joseph Torres Date: Friday, April 6, 2018 at 10:41 AM To: "Thakrar, Jayesh" Cc: "dev@spark.apache.org" Subject: Re: Spark 2.3 V2 Datasource API questions Thanks for trying it out! We haven

Re: Spark 2.3 V2 Datasource API questions

2018-04-06 Thread Joseph Torres
Thanks for trying it out! We haven't hooked continuous streaming up to query.status or query.recentProgress yet - commit() should be called under the hood, we just don't yet report that it is. I've filed SPARK-23886 and SPARK-23887 to track the work to add those things. The issue with printing wa

Spark 2.3 V2 Datasource API questions

2018-04-06 Thread Thakrar, Jayesh
First of all thank you to the Spark dev team for coming up with the standardized and intuitive API interfaces. I am sure it will encourage integrating a lot more new datasource integration. I have been creating playing with the API and have some questions on the continuous streaming API (see htt