748)
> ---
> Any idea about this error?
>
> Thanks
> Jason
>
> On Mon, 30 Nov 2020 at 19:34, Jia, Ke A wrote:
>
>> The fromRow method is removed in spark3.0. And the new API is :
>>
>> val encoder = RowEncoder(schema)
>>
>> v
e new API is :
>
> val encoder = RowEncoder(schema)
>
> val row = encoder.createDeserializer().apply(internalRow)
>
>
>
> Thanks,
>
> Jia Ke
>
>
>
> *From:* Wenchen Fan
> *Sent:* Friday, November 27, 2020 9:32 PM
> *To:* Jason Jun
> *Cc:* Spark
The fromRow method is removed in spark3.0. And the new API is :
val encoder = RowEncoder(schema)
val row = encoder.createDeserializer().apply(internalRow)
Thanks,
Jia Ke
From: Wenchen Fan
Sent: Friday, November 27, 2020 9:32 PM
To: Jason Jun
Cc: Spark dev list
Subject: Re: How to convert
InternalRow is an internal/developer API that might change overtime.
Right now, the way to convert it to Row is to use `RowEncoder`, but you
need to know the data schema:
val encoder = RowEncoder(schema)
val row = encoder.fromRow(internalRow)
On Fri, Nov 27, 2020 at 6:16 AM Jason Jun wrote:
>
Hi dev,
i'm working on generating custom pipeline on the fly, which means I
generate SparkPlan along with each node in my pipeline.
So, my pipeline end up with PipeLineRelation extending BaseRelation like:
case class PipeLineRelation(schema: StructType, pipeLinePlan:
LogicalPlan)(@transient
