Re: Represent your project at ApacheCon

Thanks for the feedback, Chris. I will submit the proposal tomorrow. Time is the priceless commodity for all of us, but I think Spark/Shark have great potentials to be the next generation platform for _all_ parallel and highly scalable analytics. hence, I am totally ok with spending energy to help

Re: Represent your project at ApacheCon

Hi Matei, There have been several tracks on Big Data technologies at the past few ApacheCons and they're definitely worth attending. Cheers, Chris -Original Message- From: Matei Zaharia Reply-To: "dev@spark.incubator.apache.org" Date: Monday, January 27, 2014 9:40 PM To: "dev@spark.in

Re: Represent your project at ApacheCon

That certainly sounds good. I’m not sure if people at Berkeley or Databricks will go there, but it may make sense. (Basically just not sure how much the audience there is interested in big data.. but perhaps just the ability to hack parallel stuff in Python and Scala will be interesting.) Matei

Re: Represent your project at ApacheCon

Cos go for it -- if you have the time and energy I personally at least would welcome your submission to ApacheCon! Cheers, Chris -Original Message- From: Konstantin Boudnik Reply-To: "dev@spark.incubator.apache.org" Date: Monday, January 27, 2014 9:29 PM To: "dev@spark.incubator.apac

Re: Fwd: Represent your project at ApacheCon

I have a solid story of our commercial support for Spark & Shark stack for real-world customer. Did that talk at Spark summit a couple of months. Would be happy to make it more OSS oriented and present. Would it be of any interest? Cos On Mon, Jan 27, 2014 at 04:51PM, Nick Pentreath wrote: > Is S

Question regarding prepareCommand in WorkerRunnableUtil

Hi there, When reading Spark on YARN code, I have one question regarding prepareCommand in WorkerRunnableUtil: This function is used to build command to run in executor, why doesn't it follow the same way as ExectorRunner(Standalone mode) where the "run" script is used to run the command? Is t

Re: Moving to Typesafe Config?

Hi Patrick, That sounds great, I just fired a JIRA ticket for this (https://spark-project.atlassian.net/browse/SPARK-1048). We are setting up spark in one of our data center (~100 machines) and would like to have this feature ASAP. Please let me know if there are things we can help. Thanks -Sheng

Re: real world streaming code

thanks, Mayur. I will read this workload's code first. On Mon, Jan 27, 2014 at 12:37 PM, Mayur Rustagi wrote: > Now add these jars to the lib folder of the streaming project as well as in > the sparkstreamingcontext object jar list: > https://www.dropbox.com/sh/00sy9mv8qsefwc1/vsEXF0aHsJ > These

Re: Problems while moving from 0.8.0 to 0.8.1

The code would not call null.equals(b); only a.equals(null) is allowable (we check for k.eq(null) at the top). a.equals(null) returning false is part of the Object.equals() contract, so it is not invalid to us

Re: Problems while moving from 0.8.0 to 0.8.1

On Tue, Jan 28, 2014 at 1:25 AM, Aaron Davidson wrote: > Looks like your MyCustomKeyType.equals() method doesn't correctly handle a > null argument. In general, the contract of equals is to return false if > called with a null argument, which this code currently relies on. > > Ok. But what if it

Re: Problems while moving from 0.8.0 to 0.8.1

Looks like your MyCustomKeyType.equals() method doesn't correctly handle a null argument. In general, the contract of equals is to return false if called with a null argument, which this code currently relies on. This could still be patched for the sake of backwards compatibility with similarly in

Re: Problems while moving from 0.8.0 to 0.8.1

ERROR executor.Executor: Exception in task ID 20 java.lang.NullPointerException at com.xyz.spark.common.collection.MyCustomKeyType.equals(MyCustomKeyType.java:200) at org.apache.spark.util.AppendOnlyMap.changeValue(AppendOnlyMap.scala:122) at org.apache.spark.Aggregator.comb

Re: [VOTE] Graduation of Apache Spark

+1 On 2014-01-26 4:50 PM, "Matei Zaharia" wrote: > Hi guys, > > Discussion has proceeded positively, so I'm calling for a community VOTE > for the graduation of Apache Spark (incubating) into a top level project. > If this VOTE is successful, then I'll call an Incubator PMC VOTE in 72 > hours, an

Re: Moving to Typesafe Config?

Hey Heiko, Spark 0.9 introduced a common config class for Spark applications. It also (initially) supported loading config files in the nested typesafe format, but this was removed last minute due to a bug. In 1.0 we'll probably add support for config files, though it may not support typesafe's tr

Re: real world streaming code

I can help you setup streaming with algebird for Uniques.. I suppose you can extend that to top K using algebird functions. First why dont you setup spark streaming on your machine using this guide: http://docs.sigmoidanalytics.com/index.php/Running_A_Simple_Streaming_Job_in_Local_Machine Then lemm

Re: real world streaming code

Now add these jars to the lib folder of the streaming project as well as in the sparkstreamingcontext object jar list: https://www.dropbox.com/sh/00sy9mv8qsefwc1/vsEXF0aHsJ These are algebird jars. This also contains the algebird scala for streaming uniques: https://www.dropbox.com/s/ydyn7kd75hhnn

Re: Problems while moving from 0.8.0 to 0.8.1

Do you mind pasting the whole stack trace for the NPE? On Mon, Jan 27, 2014 at 6:44 AM, Archit Thakur wrote: > Hi, > > Implementation of aggregation logic has been changed with 0.8.1 > (Aggregator.scala) > > It is now using AppendOnlyMap as compared to java.util.HashMap in 0.8.0 > release. > >

Fwd: real world streaming code

This email, which includes my questions about spark streaming, is forwarded from user@mailing-list. Sorry about this, because I haven't got any reply yet. thanks, dachuan. -- Forwarded message -- From: dachuan Date: Fri, Jan 24, 2014 at 10:28 PM Subject: real world streaming code

Re: Moving to Typesafe Config?

Thanks. I found the discussion myself ;) /heiko > Am 27.01.2014 um 17:34 schrieb Mark Hamstra : > > And it would be more helpful if I gave you a usable link > http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html > > Sent from my iPhone > >> O

Re: Moving to Typesafe Config?

And it would be more helpful if I gave you a usable link http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html Sent from my iPhone > On Jan 27, 2014, at 8:13 AM, Heiko Braun wrote: > > Thanks Mark. > >> On 27 Jan 2014, at 17:05, Mark Hamstra

Re: Moving to Typesafe Config?

Thanks Mark. On 27 Jan 2014, at 17:05, Mark Hamstra wrote: > Been done and undone, and will probably be redone for 1.0. See > https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 > > > On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: > >> >> Is there any interest in moving

Re: Moving to Typesafe Config?

Been done and undone, and will probably be redone for 1.0. See https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: > > Is there any interest in moving to a more structured approach for > configuring spark components? I.e. movin

Moving to Typesafe Config?

Is there any interest in moving to a more structured approach for configuring spark components? I.e. moving to the typesafe config [1]. Since spark already leverages akka, this seems to be a reasonable choice IMO. [1] https://github.com/typesafehub/config Regards, Heiko

Re: Represent your project at ApacheCon

We have a couple of customer stories around Spark would that be something that would be interesting here? Also what is the process of submitting talks? Regards Mayur Mayur Rustagi Ph: +919632149971 h ttp://www.sigmoidanalytics.com https://twitter.com/mayur_rustag

Fwd: Represent your project at ApacheCon

Is Spark active in submitting anything for this? -- Forwarded message -- From: Rich Bowen Date: Mon, Jan 27, 2014 at 4:20 PM Subject: Represent your project at ApacheCon To: committ...@apache.org Folks, 5 days from the end of the CFP, we have only 50 talks submitted. We need t

Problems while moving from 0.8.0 to 0.8.1

Hi, Implementation of aggregation logic has been changed with 0.8.1 (Aggregator.scala) It is now using AppendOnlyMap as compared to java.util.HashMap in 0.8.0 release. Aggregator.scala def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]]) : Iterator[(K, C)] = { val combiners = new Appe