How to prevent webUI from coming up

2013-10-04 Thread Jesvin Jose
I want it as a workaround to Failed to bind web UI to port x errors in 0.7.2. What are the corresponding options for standalone (./run) master, worker and a sbt-compiled driver? I am mostly bothered by the driver throwing up: 13/10/04 06:09:29 ERROR worker.Worker: Failed to create web UI

Naive Bayes Classifier with mllib

2013-10-04 Thread Aslan Bekirov
Hi All, While I was examining MLBase documentation, I could not see whether it supports naive bayes classifier or not? Does it support naive bayes classifier, if yes , any example will be very helpful? BR, Aslan

Re: Naive Bayes Classifier with mllib

2013-10-04 Thread Reynold Xin
It does not yet support naive bayes. On Friday, October 4, 2013, Aslan Bekirov wrote: Hi All, While I was examining MLBase documentation, I could not see whether it supports naive bayes classifier or not? Does it support naive bayes classifier, if yes , any example will be very helpful?

Re: Roadblock with Spark 0.8.0 ActorStream

2013-10-04 Thread Prashant Sharma
On Thu, Oct 3, 2013 at 10:15 PM, Paul Snively psniv...@icloud.com wrote: Hi Prahant, First, thanks so much for taking the time to investigate this! On Oct 3, 2013, at 3:59 AM, Prashant Sharma wrote: Trying to replicate it following change to ActorWordCount can reproduce it.

Re: Troubleshooting and how to interpret the logs

2013-10-04 Thread Ashish Rangole
Hi Matei, Thanks! Will look out for long GC pauses. On Oct 3, 2013 3:00 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi Ashish, Those removing messages mean that the node in question didn't communicate with your application for 45 seconds. Most likely the executor process on the node

Re: How to prevent webUI from coming up

2013-10-04 Thread Shangyu Luo
Are you trying to start multiple workers on one machine? 2013/10/4 Jesvin Jose frank.einst...@gmail.com I want it as a workaround to Failed to bind web UI to port x errors in 0.7.2. What are the corresponding options for standalone (./run) master, worker and a sbt-compiled driver? I am

Re: Roadblock with Spark 0.8.0 ActorStream

2013-10-04 Thread Paul Snively
Hi Prashant!That's very interesting. So this appears to be a general issue with ActorStream, then?With all DStream's I suppose.OK, so that seems like a bug independent of the issue around having no ActorSystem in scope for runJob(), then.I have not spent enough time on spray-io yet, but if it is

Re: Roadblock with Spark 0.8.0 ActorStream

2013-10-04 Thread Matei Zaharia
Hi Paul, Just FYI, I'm not sure Akka was designed to pass ActorSystems across closures the way you're doing. Also, there's a bit of a misunderstanding about closures on RDDs. Consider this change you made to ActorWordCount: lines.flatMap(_.split(\\s+)).map(x = (x, 1)).reduceByKey(_ +

Re: Roadblock with Spark 0.8.0 ActorStream

2013-10-04 Thread Matei Zaharia
Ah, I see, I thought calling ! on an ActorRef required an implicit ActorSystem to be in scope, but maybe that's not so. It seems like the actual issue is that ActorRef can't be sent around with just Java Serialization, but rather need to be sent as an Akka message. Specifically, when they are

Loss was due to com.esotericsoftware.kryo.KryoException: Buffer overflow.

2013-10-04 Thread Ryan Compton
When I turn on Kryo serialization in 0.8 my jobs fail with these errors and don't understand what's going wrong. Any ideas? I've got these properties: //my usual spark props System.setProperty(spark.serializer, org.apache.spark.serializer.KryoSerializer)

spark-ec2 launch script ... some issues and comments

2013-10-04 Thread Shay Seng
Hi, I've been trying to use the spark-ec2 launch scripts have have some comments on it, not sure if this is the best place to post ... (1) On the AMI image, most of the modeule's init.sh file has the following idiom: if [ -d spark ]; then echo Spark seems to be installed. Exiting. exit 0

Re: Loss was due to com.esotericsoftware.kryo.KryoException: Buffer overflow.

2013-10-04 Thread Ryan Compton
Some hints: I'm doing collect() on a large (~10g??) dataset. If I shrink that down, I have no problems. Ive tried System.setProperty(spark.akka.frameSize, 15420) But then I get: 13/10/04 18:49:33 ERROR client.Client$ClientActor: Failed to connect to master

Re: spark-ec2 launch script ... some issues and comments

2013-10-04 Thread Shivaram Venkataraman
Hi Thanks for reporting this. Some relevant changes that address these issues: - (1) is being fixed by Josh Rosen in https://github.com/mesos/spark-ec2/pull/22 - For (2), Patrick had a change that we discussed before at https://github.com/mesos/spark-ec2/pull/17 . I think he is re-working that