Yes, in fact those are the settings that cause this behaviour. If set to false,
everything goes fine since the implementation in spark sources in this case is
pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
Best regards,
Sergey Ivanychev
> 11 нояб. 2021 г., в 13:58, Mich Talebzadeh <mich.talebza...@gmail.com>
> написал(а):
>
>
> Have you tried the following settings:
>
> spark.conf.set("spark.sql.execution.arrow.pysppark.enabled", "true")
> spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled","true")
>
> HTH
>
> view my Linkedin profile
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
>
>> On Thu, 4 Nov 2021 at 18:06, Mich Talebzadeh <mich.talebza...@gmail.com>
>> wrote:
>> Ok so it boils down on how spark does create toPandas() DF under the bonnet.
>> How many executors are involved in k8s cluster. In this model spark will
>> create executors = no of nodes - 1
>>
>>> On Thu, 4 Nov 2021 at 17:42, Sergey Ivanychev <sergeyivanyc...@gmail.com>
>>> wrote:
>>> > Just to confirm with Collect() alone, this is all on the driver?
>>>
>>> I shared the screenshot with the plan in the first email. In the collect()
>>> case the data gets fetched to the driver without problems.
>>>
>>> Best regards,
>>>
>>>
>>> Sergey Ivanychev
>>>
>>>> 4 нояб. 2021 г., в 20:37, Mich Talebzadeh <mich.talebza...@gmail.com>
>>>> написал(а):
>>>>
>>>
>>>> Just to confirm with Collect() alone, this is all on the driver?
>> --
>>
>>
>> view my Linkedin profile
>>
>>
>> Disclaimer: Use it at your own risk. Any and all responsibility for any
>> loss, damage or destruction of data or any other property which may arise
>> from relying on this email's technical content is explicitly disclaimed. The
>> author will in no case be liable for any monetary damages arising from such
>> loss, damage or destruction.
>>
