I think I found the Coalesce you were talking about, but this is a catalyst
class that I think is not available from pyspark
Regards,
Olivier.
Le mer. 22 avr. 2015 à 11:56, Olivier Girardot <
o.girar...@lateral-thoughts.com> a écrit :
> Where should this *coalesce* come from ? Is it related to the partition
> manipulation coalesce method ?
> Thanks !
>
> Le lun. 20 avr. 2015 à 22:48, Reynold Xin <r...@databricks.com> a écrit :
>
>> Ah ic. You can do something like
>>
>>
>> df.select(coalesce(df("a"), lit(0.0)))
>>
>> On Mon, Apr 20, 2015 at 1:44 PM, Olivier Girardot <
>> o.girar...@lateral-thoughts.com> wrote:
>>
>>> From PySpark it seems to me that the fillna is relying on Java/Scala
>>> code, that's why I was wondering.
>>> Thank you for answering :)
>>>
>>> Le lun. 20 avr. 2015 à 22:22, Reynold Xin <r...@databricks.com> a
>>> écrit :
>>>
>>>> You can just create fillna function based on the 1.3.1 implementation
>>>> of fillna, no?
>>>>
>>>>
>>>> On Mon, Apr 20, 2015 at 2:48 AM, Olivier Girardot <
>>>> o.girar...@lateral-thoughts.com> wrote:
>>>>
>>>>> a UDF might be a good idea no ?
>>>>>
>>>>> Le lun. 20 avr. 2015 à 11:17, Olivier Girardot <
>>>>> o.girar...@lateral-thoughts.com> a écrit :
>>>>>
>>>>> > Hi everyone,
>>>>> > let's assume I'm stuck in 1.3.0, how can I benefit from the *fillna*
>>>>> API
>>>>> > in PySpark, is there any efficient alternative to mapping the records
>>>>> > myself ?
>>>>> >
>>>>> > Regards,
>>>>> >
>>>>> > Olivier.
>>>>> >
>>>>>
>>>>
>>>>
>>
