Reader.MyDataSourceReader:
>> Instantiatedcom.shubham.reader.MyDataSourceReader@3095c449
>> MyDataSourceReader.planBatchInputPartitions:
>> com.shubham.reader.MyDataSourceReader@3095c449 schema: null
>> +++
>> |col1|col2|
>> +++
>> +++
&
correct way?
>
> Thanks,
> Shubham
>
>> On Tue, Oct 9, 2018 at 4:43 PM Mendelson, Assaf
>> wrote:
>> I am using v2.4.0-RC2
>>
>>
>>
>> The code as is wouldn’t run (e.g. planBatchInputPartitions returns null).
>> How are you calling it?
>&
> Shubham
>
> On Tue, Oct 9, 2018 at 4:43 PM Mendelson, Assaf
> wrote:
>
>> I am using v2.4.0-RC2
>>
>>
>>
>> The code as is wouldn’t run (e.g. planBatchInputPartitions returns null).
>> How are you calling it?
>>
>>
>>
>>
?
>
>
>
> When I do:
>
> Val df = spark.read.format(mypackage).load().show()
>
> I am getting a single creation, how are you creating the reader?
>
>
>
> Thanks,
>
> Assaf
>
>
>
> *From:* Shubham Chaurasia [mailto:shubh.chaura...@gmail.c
[mailto:shubh.chaura...@gmail.com]
Sent: Tuesday, October 9, 2018 2:02 PM
To: Mendelson, Assaf; user@spark.apache.org
Subject: Re: DataSourceV2 APIs creating multiple instances of DataSourceReader
and hence not preserving the state
[EXTERNAL EMAIL]
Please report any suspicious attachments, links
Thanks Assaf, you tried with *tags/v2.4.0-rc2?*
Full Code:
MyDataSource is the entry point which simply creates Reader and Writer
public class MyDataSource implements DataSourceV2, WriteSupport,
ReadSupport, SessionConfigSupport {
@Override public DataSourceReader createReader(DataSourceOptio
Could you add a fuller code example? I tried to reproduce it in my environment
and I am getting just one instance of the reader…
Thanks,
Assaf
From: Shubham Chaurasia [mailto:shubh.chaura...@gmail.com]
Sent: Tuesday, October 9, 2018 9:31 AM
To: user@spark.apache.org
Subject: DataSourceV2
