Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Andrew Or
+1 (binding) Ran the same tests I did for RC3: Tested the standalone cluster mode REST submission gateway - submit / status / kill Tested simple applications on YARN client / cluster modes with and without --jars Tested python applications on YARN client / cluster modes with and without

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Calvin Jia
+1 Tested with input from Tachyon and persist off heap. On Thu, Jun 4, 2015 at 6:26 PM, Timothy Chen tnac...@gmail.com wrote: +1 Been testing cluster mode and client mode with mesos with 6 nodes cluster. Everything works so far. Tim On Jun 4, 2015, at 5:47 PM, Andrew Or

Re: Anyone facing problem in incremental building of individual project

2015-06-04 Thread Meethu Mathew
Hi, I added ​ createDependencyReducedPom in my pom.xml and the problem is solved. !-- Work around MSHADE-148 -- + ​​ createDependencyReducedPomfalse/createDependencyReducedPom ​Thank you @Steve​ and @Ted Regards, Meethu Mathew Senior Engineer Flytxt On Thu, Jun 4, 2015 at 9:51

PySpark on PyPi

2015-06-04 Thread Olivier Girardot
Hi everyone, Considering the python API as just a front needing the SPARK_HOME defined anyway, I think it would be interesting to deploy the Python part of Spark on PyPi in order to handle the dependencies in a Python project needing PySpark via pip. For now I just symlink the python/pyspark in

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Eron Wright
I saw something like this last night, with a similar message. Is this what you’re referring to? [error] org.deeplearning4j#dl4j-spark-ml;0.0.3.3.4.alpha1-SNAPSHOT!dl4j-spark-ml.jar origin location must be absolute:

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread shane knapp
interesting... i definitely haven't seen it happen that often in our build system, and when it has happened, i wasn't able to determine the cause. On Thu, Jun 4, 2015 at 10:16 AM, Marcelo Vanzin van...@cloudera.com wrote: On Thu, Jun 4, 2015 at 10:04 AM, shane knapp skn...@berkeley.edu wrote:

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Marcelo Vanzin
On Thu, Jun 4, 2015 at 10:04 AM, shane knapp skn...@berkeley.edu wrote: this has occasionally happened on our jenkins as well (twice since last august), and deleting the cache fixes it right up. Yes deleting the cache fixes things, but it's kinda annoying to have to do that. And yesterday

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Marcelo Vanzin
Here's one of the types of exceptions I get (this one when running VersionsSuite from sql/hive): [info] - 13: create client *** FAILED *** (1 second, 946 milliseconds) [info] java.lang.RuntimeException: [download failed: org.apache.httpcomponents#httpclient;4.2.5!httpclient.jar, download

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Matei Zaharia
+1 Tested on Mac OS X On Jun 4, 2015, at 1:09 PM, Patrick Wendell pwend...@gmail.com wrote: I will give +1 as well. On Wed, Jun 3, 2015 at 11:59 PM, Reynold Xin r...@databricks.com wrote: Let me give you the 1st +1 On Tue, Jun 2, 2015 at 10:47 PM, Patrick Wendell

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread shane knapp
this has occasionally happened on our jenkins as well (twice since last august), and deleting the cache fixes it right up. On Thu, Jun 4, 2015 at 4:29 AM, Sean Owen so...@cloudera.com wrote: I've definitely seen the dependency path must be relative problem, and fixed it by deleting the ivy

Re: Anyone facing problem in incremental building of individual project

2015-06-04 Thread Ted Yu
Andrew Or put in this workaround : diff --git a/pom.xml b/pom.xml index 0b1aaad..d03d33b 100644 --- a/pom.xml +++ b/pom.xml @@ -1438,6 +1438,8 @@ version2.3/version configuration shadedArtifactAttachedfalse/shadedArtifactAttached + !-- Work around MSHADE-148

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Marcelo Vanzin
They're my local builds, so I wouldn't be able to send you any links... and the error is generally from sbt, not the unit tests. But if there's any info I can collect when I see the error, let me know. I'll try spark.jars.ivy. I wonder if we should just set that to the system properties in

Fwd: How to pass system properties in spark ?

2015-06-04 Thread Ashwin Shankar
Trying spark-dev mailing list to see if anyone knows. -- Forwarded message -- From: Ashwin Shankar ashwinshanka...@gmail.com Date: Wed, Jun 3, 2015 at 5:38 PM Subject: How to pass system properties in spark ? To: u...@spark.apache.org u...@spark.apache.org Hi, I'm trying to use

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Burak Yavuz
Hi Marcelo, This is interesting. Can you please send me links to any failing builds if you see that problem please. For now you can set a conf: `spark.jars.ivy` to use a path except `~/.ivy2` for Spark. Thanks, Burak On Thu, Jun 4, 2015 at 4:29 AM, Sean Owen so...@cloudera.com wrote: I've

Re: Where is the JIRA filter for new contributers?

2015-06-04 Thread Sean Owen
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark ... which contains ... https://issues.apache.org/jira/browse/SPARK-7993?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened) On Thu, Jun 4, 2015 at

Where is the JIRA filter for new contributers?

2015-06-04 Thread Ravi Desai
I am new to Spark and would like to contribute to it. I recall seeing somewhere on the website a link to a JIRA filter for new contributers, but can't find that anymore. Could someone point me to it? Thanks, -ravi - To

Spark Packages: using sbt-spark-package tool with R

2015-06-04 Thread Chris Freeman
Hey everyone, I’m looking to develop a package for use with SparkR. This package would include custom R and Scala code and I was wondering if anyone had any insight into how I might be able to use the sbt-spark-package tool to publish something that needs to include an R package as well as a

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Reynold Xin
Let me give you the 1st +1 On Tue, Jun 2, 2015 at 10:47 PM, Patrick Wendell pwend...@gmail.com wrote: He all - a tiny nit from the last e-mail. The tag is v1.4.0-rc4. The exact commit and all other information is correct. (thanks Shivaram who pointed this out). On Tue, Jun 2, 2015 at

RE: MLlib: Anybody working on hierarchical topic models like HLDA?

2015-06-04 Thread Yang, Yuhao
Hi DB Tsai, Not for now. My primary reference is http://jmlr.csail.mit.edu/proceedings/papers/v15/wang11a/wang11a.pdf . And I'm seeking a way to maximum code reuse. Any suggestion will be welcome. Thanks. Regards, yuhao -Original Message- From: DB Tsai [mailto:dbt...@dbtsai.com]

Anyone facing problem in incremental building of individual project

2015-06-04 Thread Meethu Mathew
Hi all, ​I added some new code to MLlib. When I am trying to build only the mllib project using *mvn --projects mllib/ -DskipTests clean install* *​ *after setting export S PARK_PREPEND_CLASSES=true ​, the build is getting stuck with the following message. Excluding

Re: Ivy support in Spark vs. sbt

2015-06-04 Thread Sean Owen
I've definitely seen the dependency path must be relative problem, and fixed it by deleting the ivy cache, but I don't know more than this. On Thu, Jun 4, 2015 at 1:33 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, I've been bit by something really weird lately and I'm starting to

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Patrick Wendell
I will give +1 as well. On Wed, Jun 3, 2015 at 11:59 PM, Reynold Xin r...@databricks.com wrote: Let me give you the 1st +1 On Tue, Jun 2, 2015 at 10:47 PM, Patrick Wendell pwend...@gmail.com wrote: He all - a tiny nit from the last e-mail. The tag is v1.4.0-rc4. The exact commit and all

Re: Anyone facing problem in incremental building of individual project

2015-06-04 Thread Steve Loughran
On 4 Jun 2015, at 11:16, Meethu Mathew meethu.mat...@flytxt.commailto:meethu.mat...@flytxt.com wrote: Hi all, ​I added some new code to MLlib. When I am trying to build only the mllib project using mvn --projects mllib/ -DskipTests clean install ​ after setting export S