+1 hangout ________________________________ From: Xiao Li <gatorsm...@gmail.com> Sent: Wednesday, January 31, 2018 10:46:26 PM To: Ryan Blue Cc: Reynold Xin; dev; Wenchen Fen; Russell Spitzer Subject: Re: data source v2 online meetup
Hi, Ryan, wow, your Iceberg already used data source V2 API! That is pretty cool! I am just afraid these new APIs are not stable. We might deprecate or change some data source v2 APIs in the next version (2.4). Sorry for the inconvenience it might introduce. Thanks for your feedback always, Xiao 2018-01-31 15:54 GMT-08:00 Ryan Blue <rb...@netflix.com.invalid<mailto:rb...@netflix.com.invalid>>: Thanks for suggesting this, I think it's a great idea. I'll definitely attend and can talk about the changes that we've made DataSourceV2 to enable our new table format, Iceberg<https://github.com/Netflix/iceberg#about-iceberg>. On Wed, Jan 31, 2018 at 2:35 PM, Reynold Xin <r...@databricks.com<mailto:r...@databricks.com>> wrote: Data source v2 API is one of the larger main changes in Spark 2.3, and whatever that has already been committed is only the first version and we'd need more work post-2.3 to improve and stablize it. I think at this point we should stop making changes to it in branch-2.3, and instead focus on using the existing API and getting feedback for 2.4. Would people be interested in doing an online hangout to discuss this, perhaps in the month of Feb? It'd be more productive if people attending the hangout have tried the API by implementing some new sources or porting an existing source over. -- Ryan Blue Software Engineer Netflix
