Thank you for answering, this is all very helpful!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/advantages-of-SparkSQL-tp19661p19753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
ache-spark-user-list.1001560.n3.nabble.com/advantages-of-SparkSQL-tp19661.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
-
To unsubscribe, e-mail: user-unsub
;
>> Something like:
>> data = sqc.parquetFile(path)
>> results = data.map(lambda x: applyfunc(x.field))
>>
>> Is this faster/more optimised than having the data stored as a text file
>> and
>> using Spark (non-SQL) to process it?
>>
>>
>>
>
park (non-SQL) to process it?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/advantages-of-SparkSQL-tp19661.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> --
than having the data stored as a text file and
using Spark (non-SQL) to process it?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/advantages-of-SparkSQL-tp19661.html
Sent from the Apache Spark User List mailing list archive at Nabble.com