Re: proposal: replace lift-json with spray-json

Evan, Excuse me but that's WRONG that play-json pulls all play deps! PLAY/JSON has NO HEAVY DEP ON PLAY! I personally worked to make it an independent module in play! So play/json has just one big dep which is Jackson! I agree that jackson is the right way to go as a beginning. But for scala de

[GitHub] incubator-spark pull request: [java8API] SPARK-964 Investigate the...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34732078 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12676/

[GitHub] incubator-spark pull request: [java8API] SPARK-964 Investigate the...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34732076 Merged build finished.

[GitHub] incubator-spark pull request: Graph primitives2

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/580#issuecomment-34731728 Can one of the admins verify this patch?

[GitHub] incubator-spark pull request: Graph primitives2

GitHub user semihsalihoglu opened a pull request: https://github.com/apache/incubator-spark/pull/580 Graph primitives2 Hi guys, I'm following Joey and Ankur's suggestions to add collectEdges and pickRandomVertex. I'm also adding the tests for collectEdges and refactoring o

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 -- Matt Massie UC, Berkeley AMPLab Twitter: @matt_massie , @amplab https://amplab.cs.berkeley.edu/ On Mon, Feb 10, 2014 at 11:12 PM, Zongheng Yang wrote: > +1 > > On Mon, Feb 10, 2014 at 10:21 PM, Reynold Xin wrote: > > Actually I

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 10:21 PM, Reynold Xin wrote: > Actually I made a mistake by saying binding. > > Just +1 here. > > > On Mon, Feb 10, 2014 at 10:20 PM, Mattmann, Chris A (3980) < > chris.a.mattm...@jpl.nasa.gov> wrote: > >> Hi Nathan, anybody is welcome to to VOTE. Thank you. >> Only VOT

[GitHub] incubator-spark pull request: [javaAPI] SPARK-964 Investigate the ...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34730943 Merged build triggered.

[GitHub] incubator-spark pull request: [javaAPI] SPARK-964 Investigate the ...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34730944 Merged build started.

[GitHub] incubator-spark pull request: "in the source DStream" rather than ...

Github user CrazyJvm closed the pull request at: https://github.com/apache/incubator-spark/pull/579

[GitHub] incubator-spark pull request: [javaAPI] SPARK-964 Investigate the ...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34729545 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12675/

[GitHub] incubator-spark pull request: [javaAPI] SPARK-964 Investigate the ...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34729544 Merged build finished.

[GitHub] incubator-spark pull request: "in the source DStream" rather than ...

Github user rxin commented on the pull request: https://github.com/apache/incubator-spark/pull/579#issuecomment-34729446 Thanks. Merged.

[GitHub] incubator-spark pull request: "in the source DStream" rather than ...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/579#issuecomment-34729420 Can one of the admins verify this patch?

[GitHub] incubator-spark pull request: "in the source DStream" rather than ...

GitHub user CrazyJvm opened a pull request: https://github.com/apache/incubator-spark/pull/579 "in the source DStream" rather than "int the source DStream" "flatMap is a one-to-many DStream operation that creates a new DStream by generating multiple new records from each record int

Re: [VOTE] Graduation of Apache Spark from the Incubator

Actually I made a mistake by saying binding. Just +1 here. On Mon, Feb 10, 2014 at 10:20 PM, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > Hi Nathan, anybody is welcome to to VOTE. Thank you. > Only VOTEs from the Incubator PMC are what is considered "binding", but > I welc

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 To clarify to others, this is an IPCM vote so only the IPCM votes are binding :) On Mon, Feb 10, 2014 at 10:02 PM, Sandy Ryza wrote: > +1 > > > On Mon, Feb 10, 2014 at 9:57 PM, Mark Hamstra wrote: > >> +1 >> >> >> On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann >> wrote: >> >> > Hi Everyone,

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Feb 10, 2014 8:28 PM, "Chris Mattmann" wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks! > > [ ] +1 Graduate Apach

Re: [VOTE] Graduation of Apache Spark from the Incubator

Hi Nathan, anybody is welcome to to VOTE. Thank you. Only VOTEs from the Incubator PMC are what is considered "binding", but I welcome and will tally all VOTEs provided. Cheers, Chris -Original Message- From: Nathan Kronenfeld Reply-To: "dev@spark.incubator.apache.org" Date: Monday,

[GitHub] incubator-spark pull request: [WIP] [javaAPI] SPARK-964 Investigat...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34728599 Merged build started.

[GitHub] incubator-spark pull request: [WIP] [javaAPI] SPARK-964 Investigat...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/539#issuecomment-34728598 Merged build triggered.

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 9:57 PM, Mark Hamstra wrote: > +1 > > > On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann > wrote: > > > Hi Everyone, > > > > This is a new VOTE to decide if Apache Spark should graduate > > from the Incubator. Please VOTE on the resolution pasted below > > the ballot.

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks! > > [ ] +1 Gradua

Re: [VOTE] Graduation of Apache Spark from the Incubator

Who is allowed to vote on stuff like this? On Mon, Feb 10, 2014 at 11:27 PM, Chris Mattmann wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 9:35 PM, Nan Zhu wrote: > +1 > > -- > Nan Zhu > > > On Tuesday, February 11, 2014 at 12:30 AM, Ameet Talwalkar wrote: > > > +1 > > > > > > On Mon, Feb 10, 2014 at 9:28 PM, Evan Sparks evan.spa...@gmail.com)> wrote: > > > > > +1 > > > > > > > On Feb 10, 2014, at 9:20

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 9:34 PM, Azuryy Yu wrote: > +1 > > > On Tue, Feb 11, 2014 at 12:27 PM, Chris Mattmann >wrote: > > > Hi Everyone, > > > > This is a new VOTE to decide if Apache Spark should graduate > > from the Incubator. Please VOTE on the resolution pasted below > > the ballot. I'

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Tue, Feb 11, 2014 at 12:27 PM, Chris Mattmann wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks! > > [ ] +1 Gradua

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 -- Nan Zhu On Tuesday, February 11, 2014 at 12:30 AM, Ameet Talwalkar wrote: > +1 > > > On Mon, Feb 10, 2014 at 9:28 PM, Evan Sparks (mailto:evan.spa...@gmail.com)> wrote: > > > +1 > > > > > On Feb 10, 2014, at 9:20 PM, Shivaram Venkataraman < > > shiva...@eecs.berkeley.edu (mailto:sh

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 9:28 PM, Evan Sparks wrote: > +1 > > > On Feb 10, 2014, at 9:20 PM, Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > > > > +1 > > > >> On Mon, Feb 10, 2014 at 9:05 PM, Prashant Sharma > wrote: > >> +1 > >> > >> > >>> On Tue, Feb 11, 2014 at 10:30 AM, Aa

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 > On Feb 10, 2014, at 9:20 PM, Shivaram Venkataraman > wrote: > > +1 > >> On Mon, Feb 10, 2014 at 9:05 PM, Prashant Sharma >> wrote: >> +1 >> >> >>> On Tue, Feb 11, 2014 at 10:30 AM, Aaron Davidson wrote: >>> >>> +1 >>> >>> On Mon, Feb 10, 2014 at 8:58 PM, Reynold Xin wrote: >

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 9:05 PM, Prashant Sharma wrote: > +1 > > > On Tue, Feb 11, 2014 at 10:30 AM, Aaron Davidson wrote: > >> +1 >> >> >> On Mon, Feb 10, 2014 at 8:58 PM, Reynold Xin wrote: >> >> > +1 (binding) >> > >> > >> > On Mon, Feb 10, 2014 at 8:56 PM, Henry Saputra > > >wrote: >> >

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Tue, Feb 11, 2014 at 10:30 AM, Aaron Davidson wrote: > +1 > > > On Mon, Feb 10, 2014 at 8:58 PM, Reynold Xin wrote: > > > +1 (binding) > > > > > > On Mon, Feb 10, 2014 at 8:56 PM, Henry Saputra > >wrote: > > > > > +1 (binding) > > > > > > > > > - Henry > > > > > > On Mon, Feb 10, 2014 a

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Mon, Feb 10, 2014 at 8:58 PM, Reynold Xin wrote: > +1 (binding) > > > On Mon, Feb 10, 2014 at 8:56 PM, Henry Saputra >wrote: > > > +1 (binding) > > > > > > - Henry > > > > On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann > > wrote: > > > Hi Everyone, > > > > > > This is a new VOTE to dec

Re: Proposal: Clarifying minor points of Scala style

Alright, makes sense -- consistency is more important than special casing for possible readability benefits. That is one of the main points behind a style guide after all. I switch my vote for (1) to Shivaram's proposal as well. On Mon, Feb 10, 2014 at 4:40 PM, Evan Chan wrote: > +1 to the prop

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 (binding) On Mon, Feb 10, 2014 at 8:56 PM, Henry Saputra wrote: > +1 (binding) > > > - Henry > > On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann > wrote: > > Hi Everyone, > > > > This is a new VOTE to decide if Apache Spark should graduate > > from the Incubator. Please VOTE on the resolutio

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 (binding) - Henry On Mon, Feb 10, 2014 at 8:27 PM, Chris Mattmann wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks!

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Tue, Feb 11, 2014 at 10:20 AM, Mosharaf Chowdhury < mosharafka...@gmail.com> wrote: > +1 > > -- > Mosharaf Chowdhury > http://www.mosharaf.com/ > > > On Mon, Feb 10, 2014 at 8:45 PM, Matei Zaharia >wrote: > > > +1 > > > > On Feb 10, 2014, at 8:27 PM, Chris Mattmann wrote: > > > > > Hi Ev

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 -- Mosharaf Chowdhury http://www.mosharaf.com/ On Mon, Feb 10, 2014 at 8:45 PM, Matei Zaharia wrote: > +1 > > On Feb 10, 2014, at 8:27 PM, Chris Mattmann wrote: > > > Hi Everyone, > > > > This is a new VOTE to decide if Apache Spark should graduate > > from the Incubator. Please VOTE on the

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1 On Feb 10, 2014, at 8:27 PM, Chris Mattmann wrote: > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks! > > [ ] +1 Graduate

Re: [VOTE] Graduation of Apache Spark from the Incubator

+1! 2014-02-10 20:27 GMT-08:00 Chris Mattmann : > Hi Everyone, > > This is a new VOTE to decide if Apache Spark should graduate > from the Incubator. Please VOTE on the resolution pasted below > the ballot. I'll leave this VOTE open for at least 72 hours. > > Thanks! > > [ ] +1 Graduate Apache S

[VOTE] Graduation of Apache Spark from the Incubator

Hi Everyone, This is a new VOTE to decide if Apache Spark should graduate from the Incubator. Please VOTE on the resolution pasted below the ballot. I'll leave this VOTE open for at least 72 hours. Thanks! [ ] +1 Graduate Apache Spark from the Incubator. [ ] +0 Don't care. [ ] -1 Don't graduate

[GitHub] incubator-spark pull request: Adding an option to persist Spark RD...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/468#issuecomment-34722556 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12674/

[GitHub] incubator-spark pull request: [SPARK-979] a LRU scheduler for load...

Github user CodingCat commented on the pull request: https://github.com/apache/incubator-spark/pull/548#issuecomment-34722573 added a test case

[GitHub] incubator-spark pull request: Adding an option to persist Spark RD...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/468#issuecomment-34722555 Build finished.

[GitHub] incubator-spark pull request: Adding an option to persist Spark RD...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/468#issuecomment-34722512 Build triggered.

[GitHub] incubator-spark pull request: Adding an option to persist Spark RD...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/468#issuecomment-34722513 Build started.

[GitHub] incubator-spark pull request: Adding an option to persist Spark RD...

Github user haoyuan commented on the pull request: https://github.com/apache/incubator-spark/pull/468#issuecomment-34722448 Jenkins, test this please.

[GitHub] incubator-spark pull request: ROC AUC and Average precision metric...

Github user schmit commented on the pull request: https://github.com/apache/incubator-spark/pull/550#issuecomment-34720671 https://spark-project.atlassian.net/browse/MLLIB-23

Re: proposal: replace lift-json with spray-json

Evan, yes! Luis linked your blog post earlier and it was really helpful. The other advantage of json4s-jackson is that the interface is mostly compatible with lift-json. I made the (few and trivial) changes necessary to get everything in Spark switched over earlier today and will write it up

[GitHub] incubator-spark pull request: Fix typos in Spark Streaming program...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/536#issuecomment-34719031 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12673/

[GitHub] incubator-spark pull request: Fix typos in Spark Streaming program...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/536#issuecomment-34719030 Merged build finished.

[GitHub] incubator-spark pull request: Added parquetFileAsJSON to read Parq...

Github user laserson commented on the pull request: https://github.com/apache/incubator-spark/pull/576#issuecomment-34718389 No, this actually constructs Avro `GenericRecord` objects in memory. The problem is that if you want access to the Parquet data through PySpark, there is no ob

[GitHub] incubator-spark pull request: Fix typos in Spark Streaming program...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/536#issuecomment-34717536 Merged build started.

[GitHub] incubator-spark pull request: Fix typos in Spark Streaming program...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/536#issuecomment-34717534 Merged build triggered.

[GitHub] incubator-spark pull request: Hadoop jar name

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/522#issuecomment-34716681 Can one of the admins verify this patch?

[GitHub] incubator-spark pull request: Hadoop jar name

GitHub user bijaybisht reopened a pull request: https://github.com/apache/incubator-spark/pull/522 Hadoop jar name This pull request is a copy of #121 - Fix for hadoop client jar name, which got changed from 1.*. The other one was from master, which is wrong way of generating t

[GitHub] incubator-spark pull request: Default log4j initialization causes ...

Github user prb commented on the pull request: https://github.com/apache/incubator-spark/pull/573#issuecomment-34715973 The infinite loop was caused by the fact that the list of appenders is *always* empty when the slf4j mock implementation of log4j is in place.

[GitHub] incubator-spark pull request: Hadoop jar name

Github user bijaybisht closed the pull request at: https://github.com/apache/incubator-spark/pull/522

[GitHub] incubator-spark pull request: Added parquetFileAsJSON to read Parq...

Github user velvia commented on the pull request: https://github.com/apache/incubator-spark/pull/576#issuecomment-34715532 My concern with this is that Parquet is typically used for high performance OLAP queries, and changing it to JSON makes it much slower. Out of curiosity, I have

Re: Proposal: Clarifying minor points of Scala style

+1 to the proposal. On Mon, Feb 10, 2014 at 2:56 PM, Michael Armbrust wrote: > +1 to Shivaram's proposal. I think we should try to avoid functions with > many args as much as possible so having a high vertical cost here isn't the > worst thing. I also like the visual consistency. > > FWIW, (bas

[GitHub] incubator-spark pull request: Hadoop jar name

Github user berngp commented on the pull request: https://github.com/apache/incubator-spark/pull/522#issuecomment-34715269 Is there anything we can do to facilitate the merge into master. I am also looking forward for this change.

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34715135 Wow nice writeup. (Is Breeze benchmarked too somewhere? don't see it there). Totally agree. That's why I would use JBlas at least for the complex operations. Alth

Re: proposal: replace lift-json with spray-json

By the way, I did a benchmark on JSON parsing performance recently. Based on that, spray-json was about 10x slower than the Jackson-based parsers. I recommend json4s-jackson, because jackson is almost certainly already a dependency of Sparks (many other Java libraries use it), so the dependencies

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user mengxr commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34714242 @srowen Thanks for the information! I believe native BLAS/LAPACK libraries performs much better than Java implementation for level 2 and level 3 operations, but f

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user mengxr commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34712992 @debasish83 Are you speaking of the benchmark I posted to the JIRA? BLAS/LAPACK cannot be used for dense vector + sparse vector. Those are designed for dense-only

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34711528 I see the other discussion -- https://github.com/mesos/spark/pull/736 ? I didn't see the benchmark but maybe missed it. I think there was an impression th

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user debasish83 commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34710100 @mengxr as long as the interface is clean and we can bring in netlib-java, start with mahout-math does not seem like a bad idea...netlib-java uses jni while i

[GitHub] incubator-spark pull request: Adding assignRanks and assignUniqueI...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/578#issuecomment-34710311 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12672/

[GitHub] incubator-spark pull request: Adding assignRanks and assignUniqueI...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/578#issuecomment-34710309 Merged build finished.

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user mengxr commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34707127 @sscdotopen @debasish83 , I'm okay with copying VectorWritable and remove mahout-core from dependencies. @srowen Just as you mentioned, the sparse vector

[GitHub] incubator-spark pull request: Adding assignRanks and assignUniqueI...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/578#issuecomment-34705055 Merged build triggered.

[GitHub] incubator-spark pull request: Adding assignRanks and assignUniqueI...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/578#issuecomment-34705056 Merged build started.

[GitHub] incubator-spark pull request: Adding assignRanks and assignUniqueI...

GitHub user mengxr opened a pull request: https://github.com/apache/incubator-spark/pull/578 Adding assignRanks and assignUniqueIds to RDD Assign ranks to an ordered or unordered data set is a common operation. This could be done by first counting records in each partition and then

[GitHub] incubator-spark pull request: Default log4j initialization causes ...

Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/573#issuecomment-34701207 I think I misunderstood the nature of the infinite loop and thought it had to do with querying for the appenders. If not, yeah, removing the guard does not affect

[GitHub] incubator-spark pull request: Default log4j initialization causes ...

Github user pwendell commented on the pull request: https://github.com/apache/incubator-spark/pull/573#issuecomment-34698102 The issue is a user reported problems when they were writing from slf4j to Logback, they had log4j-over-slf4j on the classpath, and this initialization code kic

Re: Proposal: Clarifying minor points of Scala style

+1 to Shivaram's proposal. I think we should try to avoid functions with many args as much as possible so having a high vertical cost here isn't the worst thing. I also like the visual consistency. FWIW, (based on a cursory inspection) in the scala compiler they don't seem to ever orphan the ret

Re: Proposal: Clarifying minor points of Scala style

Yeah that was my proposal - Essentially we can just have two styles: The entire function + parameterList + return type fits in one line or when it doesn't we wrap parameters into lines. I agree that it makes the code a more verbose, but it'll make code style more consistent. Shivaram On Mon, Feb

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user srowen commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34692729 The mahout-math implementation of vectors is encumbered with a few bad design choices, Hadoop stuff that's not needed here, dependence on that old fork of colt co

Re: Proposal: Clarifying minor points of Scala style

Shivaram, is your recommendation to wrap the parameter list even if it fits, but just the return type doesn't? Personally, I think the cost of moving from a single-line parameter list to an n-ine list is pretty high, as it takes up a lot more space. I am even in favor of allowing a parameter list t

[GitHub] incubator-spark pull request: SPARK-1051. Executors should doAs su...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/538#issuecomment-34686449 Merged build finished.

[GitHub] incubator-spark pull request: SPARK-1051. Executors should doAs su...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/538#issuecomment-34686450 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12671/

Re: Proposal: Clarifying minor points of Scala style

For the 1st case wouldn't it be better to just wrap the parameters to the next line as we do in other cases ? For example def longMethodName( param1, param2, ...) : Long = { } Are there a lot functions which use the old format ? Can we just stick to the above for new functions ? Thanks S

[GitHub] incubator-spark pull request: SPARK-1072 Use binary search when ne...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/571#issuecomment-34684204 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12670/

[GitHub] incubator-spark pull request: SPARK-1072 Use binary search when ne...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/571#issuecomment-34684202 Merged build finished.

[GitHub] incubator-spark pull request: SPARK-1051. Executors should doAs su...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/538#issuecomment-34683487 Merged build triggered.

[GitHub] incubator-spark pull request: SPARK-1051. Executors should doAs su...

Github user sryza commented on the pull request: https://github.com/apache/incubator-spark/pull/538#issuecomment-34683445 Updated the patch to work with yarn-standalone mode as well. Does a doAs in the application master when running the user class.

[GitHub] incubator-spark pull request: SPARK-1051. Executors should doAs su...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/538#issuecomment-34683488 Merged build started.

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user debasish83 commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34682539 I agree...depending on mahout-math is much better than bringing in the mahout-core...mahout-math code I think will compile fine with Apache Hadoop, CDH and HD

[GitHub] incubator-spark pull request: SPARK-1072 Use binary search when ne...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/571#issuecomment-34681467 Merged build triggered.

[GitHub] incubator-spark pull request: SPARK-1072 Use binary search when ne...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/571#issuecomment-34681468 Merged build started.

[GitHub] incubator-spark pull request: SPARK-1075 Fix doc in the Spark Stre...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/577#issuecomment-34678627 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12669/

[GitHub] incubator-spark pull request: SPARK-1075 Fix doc in the Spark Stre...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/577#issuecomment-34678626 Merged build finished.

[GitHub] incubator-spark pull request: SPARK-1075 Fix doc in the Spark Stre...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/577#issuecomment-34675689 Merged build started.

[GitHub] incubator-spark pull request: SPARK-1075 Fix doc in the Spark Stre...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/577#issuecomment-34675686 Merged build triggered.

[GitHub] incubator-spark pull request: SPARK-1075 Fix doc in the Spark Stre...

GitHub user hsaputra opened a pull request: https://github.com/apache/incubator-spark/pull/577 SPARK-1075 Fix doc in the Spark Streaming custom receiver closing bracket in the class constructor The closing parentheses in the constructor in the first code block example is reversed:

[GitHub] incubator-spark pull request: [Proposal] Adding sparse data suppor...

Github user sscdotopen commented on the pull request: https://github.com/apache/incubator-spark/pull/575#issuecomment-34674441 I think making making the heavyweight mahout-core a dependency just for access to the sparse vectors is no good idea. A better way would be to just depend on

Re: Proposal: Clarifying minor points of Scala style

+1 on both On Mon, Feb 10, 2014 at 1:34 AM, Aaron Davidson wrote: > There are a few bits of the Scala style that are underspecified by > both the Scala > style guide and our own supplemental > notes< > https://cwiki.apache.org/confluence/display/SPARK/Spark+C

[GitHub] incubator-spark pull request: MLI-2: Add k-fold cross validation t...

Github user holdenk commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-34671873 Sure, I'll take a look at that tonight. From the earlier pull request that was abandoned someone had asked that its PartionedRDD (which only did it for k=2

[GitHub] incubator-spark pull request: MLI-2: Add k-fold cross validation t...

Github user mengxr commented on the pull request: https://github.com/apache/incubator-spark/pull/572#issuecomment-34668194 @holdenk , the PartitionwiseSampledRDD was designed with this use case in mind. Both the folded RDD and its complement can be represented by PartitionwiseSampledR

[GitHub] incubator-spark pull request: Added parquetFileAsJSON to read Parq...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/incubator-spark/pull/576#issuecomment-34665329 Can one of the admins verify this patch?

  1   2   >