Re: [External] Re: no stdout output from worker

Hi Ranjan, Whatever code is being passed as closure to spark operations like map, flatmap, filter etc are part of task All others are in driver. Thanks, Sourav On Mon, Mar 10, 2014 at 12:03 PM, Sen, Ranjan [USA] sen_ran...@bah.comwrote: Hi Patrick How do I know which part of the code is

Re: Streaming JSON string from REST Api in Spring

Thanks Mayur for your clarification. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Streaming-JSON-string-from-REST-Api-in-Spring-tp2358p2451.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

subscribe

hi

subscribe

hi

Re: subscribe

send this to 'user-request', not 'user' 2014-03-10 17:32 GMT+08:00 hequn cheng chenghe...@gmail.com: hi

Using flume to create stream for spark streaming.

Hey, I am using the following flume flow, Flume agent 1 consisting of Rabbitmq- source, files- channet, avro- sink sending data to a slave node of spark cluster. Flume agent 2, slave node of spark cluster, consisting of avro- source, files- channel, now for the sink i tried avro, hdfs,

Re: [External] Re: no stdout output from worker

Hi Sourav That makes so much sense. Thanks much. Ranjan From: Sourav Chandra sourav.chan...@livestream.commailto:sourav.chan...@livestream.com Reply-To: user@spark.apache.orgmailto:user@spark.apache.org user@spark.apache.orgmailto:user@spark.apache.org Date: Sunday, March 9, 2014 at 10:37 PM

Log Analyze

Hi Guys, Could anyone help me to understand this piece of log in red? Why is this happened? Thanks 14/03/10 16:55:20 INFO SparkContext: Starting job: first at NetworkWordCount.scala:87 14/03/10 16:55:20 INFO JobScheduler: Finished job streaming job 1394466892000 ms.0 from job set of time

Unsubscribe

Room for rent in Aptos

Hello , My name is Arjun and i am 30 years old and I was inquiring about the room ad that you have put up on craigslist in Aptos. I am very much interested in the room and can move in pretty early . My annual income is around 105K and I am a software engineer working in the silicon valley for

Re: Sbt Permgen

hey sandy, i think that pulreq is not relevant to the 0.9 branch i am using switching to java 7 for sbt/sbt test made it work. not sure why... On Sun, Mar 9, 2014 at 11:44 PM, Sandy Ryza sandy.r...@cloudera.com wrote: There was an issue related to this fixed recently:

RE: Pig on Spark

Hi Mayur,We are planning to upgrade our distribution MR1 MR2 (YARN) and the goal is to get SPROK set up next month. I will keep you posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark

Re: [BLOG] Spark on Cassandra w/ Calliope

We are happy that you found Calliope useful and glad we could help. *Founder CEO, **Tuplejump, Inc.* www.tuplejump.com *The Data Engineering Platform* On Sat, Mar 8, 2014 at 2:18 AM, Brian O'Neill b...@alumni.brown.edu wrote: FWIW - I posted some notes to help

Java example of using broadcast

Hi Patrick Yes I get it. I have a different question now - (changed the sub) Can anyone point me to a Java example of using broadcast variables? - Ranjan From: Patrick Wendell pwend...@gmail.commailto:pwend...@gmail.com Reply-To: user@spark.apache.orgmailto:user@spark.apache.org

Re: [External] Re: no stdout output from worker

Hey Sen, Suarav is right, and I think all of your print statements are inside of the driver program rather than inside of a closure. How are you running your program (i.e. what do you run that starts this job)? Where you run the driver you should expect to see the output. - Patrick On Mon, Mar

computation slows down 10x because of cached RDDs

hello all, i am observing a strange result. i have a computation that i run on a cached RDD in spark-standalone. it typically takes about 4 seconds. but when other RDDs that are not relevant to the computation at hand are cached in memory (in same spark context), the computation takes 40 seconds

Re: computation slows down 10x because of cached RDDs

hey matei, it happens repeatedly. we are currently runnning on java 6 with spark 0.9. i will add -XX:+PrintGCDetails and collect details, and also look into java 7 G1. thanks On Mon, Mar 10, 2014 at 6:27 PM, Matei Zaharia matei.zaha...@gmail.comwrote: Does this happen repeatedly if you

Re: Too many open files exception on reduceByKey

Hey Matt, The best way is definitely just to increase the ulimit if possible, this is sort of an assumption we make in Spark that clusters will be able to move it around. You might be able to hack around this by decreasing the number of reducers but this could have some performance implications

How to create RDD from Java in-memory data?

I would like to construct an RDD from data I already have in memory as POJO objects. Is this possible? For example, is it possible to create an RDD from IterableString? I'm running Spark from Java as a stand-alone application. The JavaWordCount example runs fine. In the example, the initial

Unsubscribe

Unsubscribe

if there is shark 0.9 build can be download?

Does anyone know if there is shark 0.9 build can be download? if not, when there will be shark 0.9 build?

Re: How to create RDD from Java in-memory data?

I was right ... I was missing something obvious. The answer to my question is to use JavaSparkContext.parallelize which works with ListT or ListTuple2lt;K,V. -- View this message in context:

Re: [BLOG] Spark on Cassandra w/ Calliope

+1 that we have been using calliope for few months and its working out really great for us. Any plans on integrating into spark? On Mar 10, 2014 1:58 PM, Rohit Rai ro...@tuplejump.com wrote: We are happy that you found Calliope useful and glad we could help. *Founder CEO, **Tuplejump, Inc.*

Re: Sharing SparkContext

Which version of Spark are you using? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi https://twitter.com/mayur_rustagi On Mon, Mar 10, 2014 at 6:49 PM, abhinav chowdary abhinav.chowd...@gmail.com wrote: for any one who is interested to know about job

Re: Sharing SparkContext

0.8.1 we used branch 0.8 and pull request into our local repo. I remember we have to deal with few issues but once we are thought that its working great. On Mar 10, 2014 6:51 PM, Mayur Rustagi mayur.rust...@gmail.com wrote: Which version of Spark are you using? Mayur Rustagi Ph: +1 (760)

Re: Sharing SparkContext

Are you using it with HDFS? What version of Hadoop? 1.0.4? Ognen On 3/10/14, 8:49 PM, abhinav chowdary wrote: for any one who is interested to know about job server from Ooyala.. we started using it recently and been working great so far.. On Feb 25, 2014 9:23 PM, Ognen Duzlevski

is spark 0.9.0 HA?

is spark 0.9.0 HA? we only have one master server , i think is is not . so, Does anyone know how to support HA for spark?

Re: Sharing SparkContext

hdfs 1.0.4 but we primarily use Cassandra + Spark (calliope). I tested it with both Are you using it with HDFS? What version of Hadoop? 1.0.4? Ognen On 3/10/14, 8:49 PM, abhinav chowdary wrote: for any one who is interested to know about job server from Ooyala.. we started using it recently and

Re: SPARK_JAVA_OPTS not picked up by the application

have your send spark-env.sh to the slave nodes ? 2014-03-11 6:47 GMT+08:00 Linlin linlin200...@gmail.com: Hi, I have a java option (-Xss) setting specified in SPARK_JAVA_OPTS in spark-env.sh, noticed after stop/restart the spark cluster, the master/worker daemon has the setting being

Re: is spark 0.9.0 HA?

Spark 0.9.0 does include standalone scheduler HA, but it requires running multiple masters. The docs are located here: https://spark.apache.org/docs/0.9.0/spark-standalone.html#high-availability 0.9.0 also includes driver HA (for long-running normal or streaming jobs), allowing you to submit a

Re: SPARK_JAVA_OPTS not picked up by the application

The properties in spark-env.sh are machine-specific. so need to specify in you worker as well. I guess you ask is the System.setproperty(). you can call it before you initialize your sparkcontext. Best Regards, Chen Jingci On Tue, Mar 11, 2014 at 6:47 AM, Linlin linlin200...@gmail.com wrote:

Re: SPARK_JAVA_OPTS not picked up by the application

my cluster only has 1 node (master/worker). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-JAVA-OPTS-not-picked-up-by-the-application-tp2483p2506.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: how to use the log4j for the standalone app

Thanks, but I do not to log myself program info, I just do not want spark output all the info to my console, I want the spark output the log into to some file which I specified. On Tue, Mar 11, 2014 at 11:49 AM, Robin Cjc cjcro...@gmail.com wrote: Hi lihu, you can extends the

Re: SPARK_JAVA_OPTS not picked up by the application

Thanks! since my worker is on the same node, -Xss JVM option is for setting thread maximum stack size, my worker does show this option now. now I realized I accidently run the the app run in local mode as I didn't give the master URL when initializing the spark context, for local mode, how to

Re: SPARK_JAVA_OPTS not picked up by the application

Thanks! so SPARK_DAEMON_JAVA_OPTS is for worker? and SPARK_JAVA_OPTS is for master? I only set SPARK_JAVA_OPTS in spark-env.sh, and the JVM opt is applied to both master/worker daemon. -- View this message in context: