From: Magnus Nilsson
Sent: Sunday, February 24, 2019 4:47 PM
To: Yeikel
Cc: user@spark.apache.org
Subject: Re: How can I parse an "unnamed" json array present in a column?
Well, I'm guessing the file is small enough so you don't have any memory issues.
If you're u
(if possible) to extract the function from
> the original source code and run it on an older version of Spark?
>
>
>
>
>
> *From:* Magnus Nilsson
> *Sent:* Sunday, February 24, 2019 5:34 AM
> *To:* Yeikel
> *Cc:* user@spark.apache.org
> *Subject:* Re: How can I p
@spark.apache.org
Subject: Re: How can I parse an "unnamed" json array present in a column?
That's a bummer, if you're unable to upgrade to Spark 2.3+ your best bet is
probably to prepend/append the jsonarray-string and locate the json array as
the value of a root attribute in a
ossible to export a function from 2.3 to 2.1? What other
> options do I have?
>
>
>
> Thank you.
>
>
>
>
>
> *From:* Magnus Nilsson
> *Sent:* Saturday, February 23, 2019 3:43 PM
> *Cc:* user@spark.apache.org
> *Subject:* Re: How can I parse an "unnamed" js
ed").show(false)
Is it viable/possible to export a function from 2.3 to 2.1? What other options
do I have?
Thank you.
From: Magnus Nilsson
Sent: Saturday, February 23, 2019 3:43 PM
Cc: user@spark.apache.org
Subject: Re: How can I parse an "unnamed" json arra
Use spark.sql.types.ArrayType instead of a Scala Array as the root type
when you define the schema and it will work.
Regards,
Magnus
On Fri, Feb 22, 2019 at 11:15 PM Yeikel wrote:
> I have an "unnamed" json array stored in a *column*.
>
> The format is the following :
>
> column name : news
>
I have an "unnamed" json array stored in a *column*.
The format is the following :
column name : news
Data :
[
{
"source": "source1",
"name": "News site1"
},
{
"source": "source2",
"name": "News site2"
}
]
Ideally , I'd like to parse it as :
news ARRAY>
I've tr
