Easy win: SBT plugin config expert to help on SPARK-3359?

2014-10-21 Thread Sean Owen
This one can be resolved, I think, with a bit of help from someone who understands SBT + plugin config: https://issues.apache.org/jira/browse/SPARK-3359 Just a matter of figuring out how to set a property on the plugin. This would make Java 8 javadoc work much more nicely. Minor but useful!

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Cheng Lian
Hm, seems that 7u71 comes back again. Observed similar Kinesis compilation error just now: https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/410/consoleFull Checked Jenkins slave nodes, saw /usr/java/latest points to jdk1.7.0_71. However, /usr/bin/javac -version says:

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Nan Zhu
just curious…what is this “NewSparkPullRequestBuilder”? Best, -- Nan Zhu On Tuesday, October 21, 2014 at 8:30 AM, Cheng Lian wrote: Hm, seems that 7u71 comes back again. Observed similar Kinesis compilation error just now:

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Cheng Lian
It's a new pull request builder written by Josh, integrated into our state-of-the-art PR dashboard :) On 10/21/14 9:33 PM, Nan Zhu wrote: just curious…what is this “NewSparkPullRequestBuilder”? Best, -- Nan Zhu On Tuesday, October 21, 2014 at 8:30 AM, Cheng Lian wrote: Hm, seems that 7u71

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Nan Zhu
weird…..two buildings (one triggered by New, one triggered by Old) were executed in the same node, amp-jenkins-slave-01, one compiles, one not… Best, -- Nan Zhu On Tuesday, October 21, 2014 at 9:39 AM, Nan Zhu wrote: seems that all PRs built by NewSparkPRBuilder suffers from 7u71, while

[MLlib] Contributing Algorithm for Outlier Detection

2014-10-21 Thread Ashutosh
Hi, I am new to Apache Spark (any open source project). I want to contribute to it. I found that MLlib has no algorithm for outlier detection yet. By literature review I found the algorithm Attribute Value Frequency (AVF) is promising. Here is the link DOI: 10.1109/ICTAI.2007.125 By following

Re: [MLlib] Contributing Algorithm for Outlier Detection

2014-10-21 Thread Xiangrui Meng
Hi Ashutosh, The process you described is correct, with details documented in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark . There is no outlier detection algorithm in MLlib. Before you start coding, please open an JIRA and let's discuss which algorithms are appropriate

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Josh Rosen
I find it concerning that there’s a JDK version that breaks out build, since we’re supposed to support Java 7.  Is 7u71 an upgrade or downgrade from the JDK that we used before?  Is there an easy way to fix our build so that it compiles with 7u71’s stricter settings? I’m not sure why the “New”

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Sean Owen
Given the nature of the error, I would be really, really shocked if Java 7u71 were actually being used in the failing build, so no I do not thing the problem has to do with 7u71 per se. As I'd expect I see no changes to javac in this update from 7u65, and no chatter about crazy javac regressions.

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Josh Rosen
Ah, that makes sense.  I had forgotten that there was a JIRA for this: https://issues.apache.org/jira/browse/SPARK-4021   On October 21, 2014 at 10:08:58 AM, Patrick Wendell (pwend...@gmail.com) wrote: Josh - the errors that broke our build indicated that JDK5 was being used. Somehow the

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Patrick Wendell
Josh - the errors that broke our build indicated that JDK5 was being used. Somehow the upgrade caused our build to use a much older Java version. See the JIRA for more details. On Tue, Oct 21, 2014 at 10:05 AM, Josh Rosen rosenvi...@gmail.com wrote: I find it concerning that there's a JDK

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread Nan Zhu
I agree with Sean I just compiled spark core successfully with 7u71 in Mac OS X On Tue, Oct 21, 2014 at 1:11 PM, Josh Rosen rosenvi...@gmail.com wrote: Ah, that makes sense. I had forgotten that there was a JIRA for this: https://issues.apache.org/jira/browse/SPARK-4021 On October 21,

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread shane knapp
i'm currently in a meeting and will be starting to do some tests in ~1 hour or so. On Tue, Oct 21, 2014 at 11:07 AM, Nan Zhu zhunanmcg...@gmail.com wrote: I agree with Sean I just compiled spark core successfully with 7u71 in Mac OS X On Tue, Oct 21, 2014 at 1:11 PM, Josh Rosen

Re: [MLlib] Contributing Algorithm for Outlier Detection

2014-10-21 Thread Ashutosh
Hi Xiangrui, Thanks for the reply. AVF is not so difficult to implement in parallel. It just calculate the frequency of each attribute and calculate the overall 'score' of the datapoint. Low score points are considered outlier. One advantage of it is that it does not calculate distance, so in

Re: something wrong with Jenkins or something untested merged?

2014-10-21 Thread shane knapp
ok, i did some testing and found out what's happening. https://issues.apache.org/jira/browse/SPARK-4021 here's the TL;DR: jenkins ignores what JDKs are installed via the web interface when there's more than one defined, and falls back to whatever is default on the slave the test is run on. in

Re: short jenkins downtime -- trying to get to the bottom of the git fetch timeouts

2014-10-21 Thread shane knapp
i've seen a few more builds fail w/timeouts and it appears that we're definitely NOT hitting any rate limiting. https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22005/console [jenkins@amp-jenkins-slave-01 ~]$ curl -i -H Authorization: token REDACTED https://api.github.com |