Linear Regression with SGD

Hi User group, We are using spark Linear Regression with SGD as the optimization technique and we are achieving very sub-optimal results. Can anyone shed some light on why this implementation seems to produce such poor results vs our own implementation? We are using a very small dataset, but

Where does partitioning and data loading happen?

A colleague and I were having a discussion and we were disagreeing about something in Spark/Mesos that perhaps someone can shed some light into. We have a mesos cluster that runs spark via a sparkHome, rather than downloading an executable and such. My colleague says that say we have parquet

RE: swap tuple

Yea, I wouldn't try and modify the current since RDDs are suppose to be immutable, just create a new one... val newRdd = oldRdd.map(r = (r._2(), r._1())) or something of that nature... Steve From: Evo Eftimov [evo.efti...@isecc.com] Sent: Thursday, May 14, 2015

Re: Spark on Mesos

at 11:11 AM, Stephen Carman scar...@coldlight.com wrote: So I installed spark on each of the slaves 1.3.1 built with hadoop2.6 I just basically got the pre-built from the spark website… I placed those compiled spark installs on each slave at /opt/spark My spark properties seem to be getting

Re: Spark on Mesos

, 2015 at 10:50 AM, Stephen Carman scar...@coldlight.commailto:scar...@coldlight.com wrote: Sander, I eventually solved this problem via the --[no-]switch_user flag, which is set to true by default. I set this to false, which would have the user that owns the process run the job, otherwise it was my

Re: Spark on Mesos

So I installed spark on each of the slaves 1.3.1 built with hadoop2.6 I just basically got the pre-built from the spark website… I placed those compiled spark installs on each slave at /opt/spark My spark properties seem to be getting picked up on my side fine…

Spark on Mesos

So I can’t for the life of me to get something even simple working for Spark on Mesos. I installed a 3 master, 3 slave mesos cluster, which is all configured, but I can’t for the life of me even get the spark shell to work properly. I get errors like this org.apache.spark.SparkException: Job

Spark Memory Utilities

they can be made public inside the library or have some interface to them such that children classes can make use of them? Thanks, Stephen Carman, M.S. AI Engineer, Coldlight Solutions, LLC Cell - 267 240 0363 This e-mail is intended solely for the above-mentioned recipient and it may contain