nt to stick to Dataframe you can always repartition the data after
> you apply the limit.
>
> ..Manas
>
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark
.
..Manas
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark-sql-tp25789p25797.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
is less or equal then 1.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark-sql-tp25789.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
Hi,
I am using spark sql in a way like this:
sqlContext.sql(“select * from table limit 1”).map(...).collect()
The problem is that the limit clause will collect all the 10,000 records into a
single partition, resulting the map afterwards running only in one partition
and being really slow.I
o call persist() instead?
From: tiandiwo...@icloud.com<mailto:tiandiwo...@icloud.com>
Subject: Problem using limit clause in spark sql
Date: Wed, 23 Dec 2015 21:26:51 +0800
To: user@spark.apache.org<mailto:user@spark.apache.org>
Hi,
I am using spark sql in a way like this:
sponse.
>>
>>> 在 2015年12月24日，上午8:22，Hudong Wang <justupl...@hotmail.com
>>> <mailto:justupl...@hotmail.com>> 写道：
>>>
>>> When you call collect() it will bring all the data to the driver. Do you
>>> mean to call persist() instead?
&g
he driver. Do you
> mean to call persist() instead?
>
> --
> From: tiandiwo...@icloud.com
> Subject: Problem using limit clause in spark sql
> Date: Wed, 23 Dec 2015 21:26:51 +0800
> To: user@spark.apache.org
>
> Hi,
> I am using spark sql in a
gt;
> From: tiandiwo...@icloud.com
> Subject: Problem using limit clause in spark sql
> Date: Wed, 23 Dec 2015 21:26:51 +0800
> To: user@spark.apache.org
>
> Hi,
> I am using spark sql in a way like this:
>
> sqlContext.sql(“select * from table limit 1”).map(...).col
