Re: how to get partition column info in Data Source V2 writer

2019-12-18 Thread aakash aakash
Thanks Wenchen! On Wed, Dec 18, 2019 at 7:25 PM Wenchen Fan wrote: > Hi Aakash, > > You can try the latest DS v2 with the 3.0 preview, and the API is in a > quite stable shape now. With the latest API, a Writer is created from a > Table, and the Table has the partitioning information. > >

Re: how to get partition column info in Data Source V2 writer

2019-12-18 Thread Wenchen Fan
Hi Aakash, You can try the latest DS v2 with the 3.0 preview, and the API is in a quite stable shape now. With the latest API, a Writer is created from a Table, and the Table has the partitioning information. Thanks, Wenchen On Wed, Dec 18, 2019 at 3:22 AM aakash aakash wrote: > Thanks

Re: how to get partition column info in Data Source V2 writer

2019-12-17 Thread aakash aakash
Thanks Andrew! It seems there is a drastic change in 3.0, going through it. -Aakash On Tue, Dec 17, 2019 at 11:01 AM Andrew Melo wrote: > Hi Aakash > > On Tue, Dec 17, 2019 at 12:42 PM aakash aakash > wrote: > >> Hi Spark dev folks, >> >> First of all kudos on this new Data Source v2, API

Re: how to get partition column info in Data Source V2 writer

2019-12-17 Thread Andrew Melo
Hi Aakash On Tue, Dec 17, 2019 at 12:42 PM aakash aakash wrote: > Hi Spark dev folks, > > First of all kudos on this new Data Source v2, API looks simple and it > makes easy to develop a new data source and use it. > > With my current work, I am trying to implement a new data source V2 writer >

how to get partition column info in Data Source V2 writer

2019-12-17 Thread aakash aakash
Hi Spark dev folks, First of all kudos on this new Data Source v2, API looks simple and it makes easy to develop a new data source and use it. With my current work, I am trying to implement a new data source V2 writer with Spark 2.3 and I was wondering how I will get the info about partition by