(for example, you might be
able to avoid a shuffle when doing joins on tables that are already
bucketed by exposing more metastore information to the planner).
Can you provide more input on how to implement this functionality so that i
can speed up join between 2 hive tables, both with few bill
I reviewed and merged that PR, in case you want to try out the fix.
- Josh
On December 22, 2014 at 10:40:35 AM, Marcelo Vanzin (van...@cloudera.com) wrote:
https://github.com/apache/spark/pull/3705
On Mon, Dec 22, 2014 at 10:19 AM, Cody Koeninger wrote:
> Is there a reason not to go ahead an
I’m investigating this issue and left some comments on the proposed fix:
https://github.com/apache/spark/pull/3345#issuecomment-68014353
To summarize, I agree with your description of the problem but think that the
right fix may be a bit more involved than what’s proposed in that PR (that PR’s
Hi All,
I am starting to use Spark. I am having trouble getting the latest code
from git.
I am using Intellij as suggested in the below link,
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-StarterTasks
The below link isn't working as well,
http://sp
What error are you getting?
Tim
Sent from my iPhone
> On Dec 24, 2014, at 8:59 PM, Naveen Madhire wrote:
>
> Hi All,
>
> I am starting to use Spark. I am having trouble getting the latest code
> from git.
> I am using Intellij as suggested in the below link,
>
> https://cwiki.apache.org/conf
Hi all,
In my occasion, I have a huge HashMap[(Int, Long), (Double, Double,
Double)], say several GB to tens of GB, after each iteration, I need to
collect() this HashMap and perform some calculation, and then broadcast()
it to every node. Now I have 20GB for each executor and after it
performances
Hi Will,
When you call collect() the item you are collecting needs to fit in
memory on the driver. Is it possible your driver program does not have
enough memory?
- Patrick
On Wed, Dec 24, 2014 at 9:34 PM, Will Yang wrote:
> Hi all,
> In my occasion, I have a huge HashMap[(Int, Long), (Double,
The correct docs link is:
https://spark.apache.org/docs/1.2.0/building-spark.html
Where did you get that bad link from?
Nick
On Thu Dec 25 2014 at 12:00:53 AM Naveen Madhire
wrote:
> Hi All,
>
> I am starting to use Spark. I am having trouble getting the latest code
> from git.
> I am using I
Hi,
We have such requirements to save RDD output to HDFS with saveAsTextFile like
API, but need to overwrite the data if existed. I'm not sure if current Spark
support such kind of operations, or I need to check this manually?
There's a thread in mailing list discussed about this
(http://apach
Is it sufficient to set "spark.hadoop.validateOutputSpecs" to false?
http://spark.apache.org/docs/latest/configuration.html
- Patrick
On Wed, Dec 24, 2014 at 10:52 PM, Shao, Saisai wrote:
> Hi,
>
>
>
> We have such requirements to save RDD output to HDFS with saveAsTextFile
> like API, but need
I am wondering if we can provide more friendly API, other than configuration
for this purpose. What do you think Patrick?
Cheng Hao
-Original Message-
From: Patrick Wendell [mailto:pwend...@gmail.com]
Sent: Thursday, December 25, 2014 3:22 PM
To: Shao, Saisai
Cc: u...@spark.apache.org;
So the behavior of overwriting existing directories IMO is something
we don't want to encourage. The reason why the Hadoop client has these
checks is that it's very easy for users to do unsafe things without
them. For instance, a user could overwrite an RDD that had 100
partitions with an RDD that
After a long talk with Patrick and TD (thanks guys), I opened the following
jira
https://issues.apache.org/jira/browse/SPARK-4964
Sample PR has an impementation for the batch and the dstream case, and a
link to a project with example usage.
On Fri, Dec 19, 2014 at 4:36 PM, Koert Kuipers wrote:
In general such discussions happen or is posted on the dev lists. Could you
please post a summary? Thanks.
Thanks, Hari
On Wed, Dec 24, 2014 at 11:46 PM, Cody Koeninger
wrote:
> After a long talk with Patrick and TD (thanks guys), I opened the following
> jira
> https://issues.apache.org/jir
Thanks Patrick for your detailed explanation.
BR
Jerry
-Original Message-
From: Patrick Wendell [mailto:pwend...@gmail.com]
Sent: Thursday, December 25, 2014 3:43 PM
To: Cheng, Hao
Cc: Shao, Saisai; u...@spark.apache.org; dev@spark.apache.org
Subject: Re: Question on saveAsTextFile with
15 matches
Mail list logo
