Re: A Spark Compilation Question

2014-09-26 Thread Yanbo Liang
Hi Hansu, I have encountered the same problem. Maven compiled avro file and generated corresponding Java file in new directory which is not source file directory of the project. I have modified pom.xml file and it can be work. The line marked as red is added, you can add them to your

executorAdded event to DAGScheduler

2014-09-26 Thread praveen seluka
Can someone explain the motivation behind passing executorAdded event to DAGScheduler ? *DAGScheduler *does *submitWaitingStages *when *executorAdded *method is called by *TaskSchedulerImpl*. I see some issue in the below code, *TaskSchedulerImpl.scala code* if (!executorsByHost.contains(o.host))

Re: executorAdded event to DAGScheduler

2014-09-26 Thread praveen seluka
Some corrections. On Fri, Sep 26, 2014 at 5:32 PM, praveen seluka praveen.sel...@gmail.com wrote: Can someone explain the motivation behind passing executorAdded event to DAGScheduler ? *DAGScheduler *does *submitWaitingStages *when *executorAdded *method is called by *TaskSchedulerImpl*. I

Re: executorAdded event to DAGScheduler

2014-09-26 Thread Nan Zhu
just a quick reply, we cannot start two executors in the same host for a single application in the standard deployment (one worker per machine) I’m not sure if it will create an issue when you have multiple workers in the same host, as submitWaitingStages is called everywhere and I never try

Re: executorAdded event to DAGScheduler

2014-09-26 Thread praveen seluka
In Yarn, we can easily have multiple containers allocated in the same node. On Fri, Sep 26, 2014 at 6:05 PM, Nan Zhu zhunanmcg...@gmail.com wrote: just a quick reply, we cannot start two executors in the same host for a single application in the standard deployment (one worker per machine)

Re: PARSING_ERROR from kryo

2014-09-26 Thread Arun Ahuja
I am seeing the same error as well since upgrading to Spark1.1: 14/09/26 15:35:05 ERROR executor.Executor: Exception in task 1032.0 in stage 5.1 (TID 22449) com.esotericsoftware.kryo.KryoException: java.io.IOException: failed to uncompress the chunk: PARSING_ERROR(2) at

FYI: jenkins systems patched to fix bash exploit

2014-09-26 Thread shane knapp
all of our systems were affected by the shellshock bug, and i've just patched everything w/the latest fix from redhat: https://access.redhat.com/articles/1200223 we're not running bash.x86_64 0:4.1.2-15.el6_5.2 on all of our systems. shane

Re: FYI: jenkins systems patched to fix bash exploit

2014-09-26 Thread shane knapp
we're not running bash.x86_64 0:4.1.2-15.el6_5.2 on all of our systems. s/not/now :)

thank you for reviewing our patches

2014-09-26 Thread Nicholas Chammas
I recently came across this mailing list post by Linus Torvalds https://lkml.org/lkml/2004/12/20/255 about the value of reviewing even “trivial” patches. The following passages stood out to me: I think that much more important than the patch is the fact that people get used to the notion that

Re: thank you for reviewing our patches

2014-09-26 Thread Reynold Xin
Keep the patches coming :) On Fri, Sep 26, 2014 at 1:50 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: I recently came across this mailing list post by Linus Torvalds https://lkml.org/lkml/2004/12/20/255 about the value of reviewing even “trivial” patches. The following passages

Re: SparkSQL: map type MatchError when inserting into Hive table

2014-09-26 Thread Cheng Lian
Would you mind to provide the DDL of this partitioned table together with the query you tried? The stacktrace suggests that the query was trying to cast a map into something else, which is not supported in Spark SQL. And I doubt whether Hive support casting a complex type to some other type.

Re: SparkSQL: map type MatchError when inserting into Hive table

2014-09-26 Thread Cheng Lian
Would you mind to provide the DDL of this partitioned table together with the query you tried? The stacktrace suggests that the query was trying to cast a map into something else, which is not supported in Spark SQL. And I doubt whether Hive support casting a complex type to some other type.