Re: Question on List to DF
Hi Jayesh,
thanks found your email quite interesting :)
Regards,
Gourav
On Wed, Mar 16, 2022 at 8:02 AM Bitfox wrote:
> Thank you. that makes sense.
>
> On Wed, Mar 16, 2022 at 2:03 PM Lalwani, Jayesh
> wrote:
>
>> The toDF function in scala uses a bit of Scala magic that allows you to
>> add methods to existing classes. Here’s a link to explanation
>> https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s11.html
>>
>>
>>
>> In short, you can implement a class that extends the List class and add
>> methods to your list class, and you can implement an implicit converter
>> that converts from List to your class. When the Scala compiler sees that
>> you are calling a function on a List object that doesn’t exist in the List
>> class, it will look for implicit converters that convert List object to
>> another object that has the function, and will automatically call it.
>>
>> So, if you have a class
>>
>> Class MyList extends List {
>> def toDF(colName: String): DataFrame{
>> …..
>> }
>> }
>>
>> and a implicit converter
>> implicit def convertListToMyList(list: List): MyList {
>>
>> ….
>> }
>>
>> when you do
>> List("apple","orange","cherry").toDF("fruit")
>>
>>
>>
>> Internally, Scala will generate the code as
>> convertListToMyList(List("apple","orange","cherry")).toDF("fruit")
>>
>>
>>
>>
>>
>> *From: *Bitfox
>> *Date: *Wednesday, March 16, 2022 at 12:06 AM
>> *To: *"user @spark"
>> *Subject: *[EXTERNAL] Question on List to DF
>>
>>
>>
>> *CAUTION*: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>>
>>
>> I am wondering why the list in scala spark can be converted into a
>> dataframe directly?
>>
>>
>>
>> scala> val df = List("apple","orange","cherry").toDF("fruit")
>>
>> *df*: *org.apache.spark.sql.DataFrame* = [fruit: string]
>>
>>
>>
>> scala> df.show
>>
>> +--+
>>
>> | fruit|
>>
>> +--+
>>
>> | apple|
>>
>> |orange|
>>
>> |cherry|
>>
>> +--+
>>
>>
>>
>> I don't think pyspark can convert that as well.
>>
>>
>>
>> Thank you.
>>
>
Re: Question on List to DF
Thank you. that makes sense.
On Wed, Mar 16, 2022 at 2:03 PM Lalwani, Jayesh wrote:
> The toDF function in scala uses a bit of Scala magic that allows you to
> add methods to existing classes. Here’s a link to explanation
> https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s11.html
>
>
>
> In short, you can implement a class that extends the List class and add
> methods to your list class, and you can implement an implicit converter
> that converts from List to your class. When the Scala compiler sees that
> you are calling a function on a List object that doesn’t exist in the List
> class, it will look for implicit converters that convert List object to
> another object that has the function, and will automatically call it.
>
> So, if you have a class
>
> Class MyList extends List {
> def toDF(colName: String): DataFrame{
> …..
> }
> }
>
> and a implicit converter
> implicit def convertListToMyList(list: List): MyList {
>
> ….
> }
>
> when you do
> List("apple","orange","cherry").toDF("fruit")
>
>
>
> Internally, Scala will generate the code as
> convertListToMyList(List("apple","orange","cherry")).toDF("fruit")
>
>
>
>
>
> *From: *Bitfox
> *Date: *Wednesday, March 16, 2022 at 12:06 AM
> *To: *"user @spark"
> *Subject: *[EXTERNAL] Question on List to DF
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> I am wondering why the list in scala spark can be converted into a
> dataframe directly?
>
>
>
> scala> val df = List("apple","orange","cherry").toDF("fruit")
>
> *df*: *org.apache.spark.sql.DataFrame* = [fruit: string]
>
>
>
> scala> df.show
>
> +--+
>
> | fruit|
>
> +--+
>
> | apple|
>
> |orange|
>
> |cherry|
>
> +--+
>
>
>
> I don't think pyspark can convert that as well.
>
>
>
> Thank you.
>
Re: Question on List to DF
The toDF function in scala uses a bit of Scala magic that allows you to add
methods to existing classes. Here’s a link to explanation
https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s11.html
In short, you can implement a class that extends the List class and add methods
to your list class, and you can implement an implicit converter that converts
from List to your class. When the Scala compiler sees that you are calling a
function on a List object that doesn’t exist in the List class, it will look
for implicit converters that convert List object to another object that has the
function, and will automatically call it.
So, if you have a class
Class MyList extends List {
def toDF(colName: String): DataFrame{
…..
}
}
and a implicit converter
implicit def convertListToMyList(list: List): MyList {
….
}
when you do
List("apple","orange","cherry").toDF("fruit")
Internally, Scala will generate the code as
convertListToMyList(List("apple","orange","cherry")).toDF("fruit")
From: Bitfox
Date: Wednesday, March 16, 2022 at 12:06 AM
To: "user @spark"
Subject: [EXTERNAL] Question on List to DF
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
I am wondering why the list in scala spark can be converted into a dataframe
directly?
scala> val df = List("apple","orange","cherry").toDF("fruit")
df: org.apache.spark.sql.DataFrame = [fruit: string]
scala> df.show
+--+
| fruit|
+--+
| apple|
|orange|
|cherry|
+--+
I don't think pyspark can convert that as well.
Thank you.
Question on List to DF
I am wondering why the list in scala spark can be converted into a
dataframe directly?
scala> val df = List("apple","orange","cherry").toDF("fruit")
*df*: *org.apache.spark.sql.DataFrame* = [fruit: string]
scala> df.show
+--+
| fruit|
+--+
| apple|
|orange|
|cherry|
+--+
I don't think pyspark can convert that as well.
Thank you.
