[jira] [Commented] (SPARK-7726) Maven Install Breaks When Upgrading Scala 2.11.2--[2.11.3 or higher]

2015-08-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680885#comment-14680885 ] Patrick Wendell commented on SPARK-7726: [~srowen] [~dragos] This is cropping up

[jira] [Commented] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-08-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660796#comment-14660796 ] Patrick Wendell commented on SPARK-1517: Hey Ryan, IIRC - the Apache snapshot

[jira] [Commented] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-08-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660420#comment-14660420 ] Patrick Wendell commented on SPARK-1517: Hey Ryan, For the maven snapshot

Avoiding unnecessary build changes until tests are in better shape

2015-08-05 Thread Patrick Wendell
Hey All, Was wondering if people would be willing to avoid merging build changes until we have put the tests in better shape. The reason is that build changes are the most likely to cause downstream issues with the test matrix and it's very difficult to reverse engineer which patches caused which

Re: How to help for 1.5 release?

2015-08-04 Thread Patrick Wendell
Hey Meihua, If you are a user of Spark, one thing that is really helpful is to run Spark 1.5 on your workload and report any issues, performance regressions, etc. - Patrick On Mon, Aug 3, 2015 at 11:49 PM, Akhil Das ak...@sigmoidanalytics.com wrote: I think you can start from here

Re: PSA: Maven 3.3.3 now required to build

2015-08-03 Thread Patrick Wendell
Yeah the best bet is to use ./build/mvn --force (otherwise we'll still use your system maven). - Patrick On Mon, Aug 3, 2015 at 1:26 PM, Sean Owen so...@cloudera.com wrote: That statement is true for Spark 1.4.x. But you've reminded me that I failed to update this doc for 1.5, to say Maven

[jira] [Created] (SPARK-9547) Allow testing pull requests with different Hadoop versions

2015-08-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-9547: -- Summary: Allow testing pull requests with different Hadoop versions Key: SPARK-9547 URL: https://issues.apache.org/jira/browse/SPARK-9547 Project: Spark

[jira] [Updated] (SPARK-9545) Run Maven tests in pull request builder if title has [maven-test] in it

2015-08-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-9545: --- Issue Type: Improvement (was: Bug) Run Maven tests in pull request builder if title has

[jira] [Created] (SPARK-9545) Run Maven tests in pull request builder if title has [maven-test] in it

2015-08-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-9545: -- Summary: Run Maven tests in pull request builder if title has [maven-test] in it Key: SPARK-9545 URL: https://issues.apache.org/jira/browse/SPARK-9545 Project

Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-08-01 Thread Patrick Wendell
Hey All, I got it up and running - it was a newly surfaced bug in the build scripts. - Patrick On Wed, Jul 29, 2015 at 6:05 AM, Bharath Ravi Kumar reachb...@gmail.com wrote: Hey Patrick, Any update on this front please? Thanks, Bharath On Fri, Jul 24, 2015 at 8:38 PM, Patrick Wendell

Re: Should spark-ec2 get its own repo?

2015-07-31 Thread Patrick Wendell
Hey All, I've mostly kept quiet since I am not very active in maintaining this code anymore. However, it is a bit odd that the project is split-brained with a lot of the code being on github and some in the Spark repo. If the consensus is to migrate everything to github, that seems okay with me.

Re: Data source aliasing

2015-07-30 Thread Patrick Wendell
Yeah this could make sense - allowing data sources to register a short name. What mechanism did you have in mind? To use the jar service loader? The only issue is that there could be conflicts since many of these are third party packages. If the same name were registered twice I'm not sure what

[jira] [Resolved] (SPARK-9423) Why do every other spark comiter keep suggesting to use spark-submit script

2015-07-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-9423. Resolution: Invalid Why do every other spark comiter keep suggesting to use spark-submit

[jira] [Commented] (SPARK-9423) Why do every other spark comiter keep suggesting to use spark-submit script

2015-07-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645495#comment-14645495 ] Patrick Wendell commented on SPARK-9423: This is not a valid issue for JIRA (we

Re: ReceiverTrackerSuite failing in master build

2015-07-28 Thread Patrick Wendell
Thanks ted for pointing this out. CC to Ryan and TD On Tue, Jul 28, 2015 at 8:25 AM, Ted Yu yuzhih...@gmail.com wrote: Hi, I noticed that ReceiverTrackerSuite is failing in master Jenkins build for both hadoop profiles. The failure seems to start with:

Protocol for build breaks

2015-07-25 Thread Patrick Wendell
Hi All, If there is a build break (i.e. a compile issue or consistently failing test) that somehow makes it into master, the best protocol is: 1. Revert the offending patch. 2. File a JIRA and assign it to the committer of the offending patch. The JIRA should contain links to broken builds.

[jira] [Created] (SPARK-9304) Improve backwards compatibility of SPARK-8401

2015-07-24 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-9304: -- Summary: Improve backwards compatibility of SPARK-8401 Key: SPARK-9304 URL: https://issues.apache.org/jira/browse/SPARK-9304 Project: Spark Issue Type

Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-07-24 Thread Patrick Wendell
noticed the last (1.5) build has a timestamp of 16th July. Have nightly builds been discontinued since then? Thanks, Bharath On Sun, May 24, 2015 at 1:11 PM, Patrick Wendell pwend...@gmail.com wrote: Hi All, This week I got around to setting up nightly builds for Spark on Jenkins. I'd like

Policy around backporting bug fixes

2015-07-24 Thread Patrick Wendell
Hi All, A few times I've been asked about backporting and when to backport and not backport fix patches. Since I have managed this for many of the past releases, I wanted to point out the way I have been thinking about it. If we have some consensus I can put it on the wiki. The trade off when

[jira] [Updated] (SPARK-8703) Add CountVectorizer as a ml transformer to convert document to words count vector

2015-07-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8703: --- Issue Type: Sub-task (was: New Feature) Parent: SPARK-8521 Add CountVectorizer

[jira] [Updated] (SPARK-8564) Add the Python API for Kinesis

2015-07-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8564: --- Target Version/s: 1.5.0 Add the Python API for Kinesis

Re: KinesisStreamSuite failing in master branch

2015-07-19 Thread Patrick Wendell
I think we should just revert this patch on all affected branches. No reason to leave the builds broken until a fix is in place. - Patrick On Sun, Jul 19, 2015 at 6:03 PM, Josh Rosen rosenvi...@gmail.com wrote: Yep, I emailed TD about it; I think that we may need to make a change to the pull

Re: Foundation policy on releases and Spark nightly builds

2015-07-19 Thread Patrick Wendell
: Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose

Re: Foundation policy on releases and Spark nightly builds

2015-07-19 Thread Patrick Wendell
Hey Sean, One other thing I'd be okay doing is moving the main text about nightly builds to the wiki and just have header called Nightly builds at the end of the downloads page that says For developers, Spark maintains nightly builds. More information is available on the [Spark developer

Re: [discuss] Removing individual commit messages from the squash commit message

2015-07-18 Thread Patrick Wendell
+1 from me too On Sat, Jul 18, 2015 at 3:32 AM, Ted Yu yuzhih...@gmail.com wrote: +1 to removing commit messages. On Jul 18, 2015, at 1:35 AM, Sean Owen so...@cloudera.com wrote: +1 to removing them. Sometimes there are 50+ commits because people have been merging from master into their

Re: Slight API incompatibility caused by SPARK-4072

2015-07-15 Thread Patrick Wendell
One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also):

Re: Slight API incompatibility caused by SPARK-4072

2015-07-15 Thread Patrick Wendell
Actually the java one is a concrete class. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell pwend...@gmail.com wrote: One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility

Announcing Spark 1.4.1!

2015-07-15 Thread Patrick Wendell
Hi All, I'm happy to announce the Spark 1.4.1 maintenance release. We recommend all users on the 1.4 branch upgrade to this release, which contain several important bug fixes. Download Spark 1.4.1 - http://spark.apache.org/downloads.html Release notes -

Announcing Spark 1.4.1!

2015-07-15 Thread Patrick Wendell
Hi All, I'm happy to announce the Spark 1.4.1 maintenance release. We recommend all users on the 1.4 branch upgrade to this release, which contain several important bug fixes. Download Spark 1.4.1 - http://spark.apache.org/downloads.html Release notes -

[jira] [Updated] (SPARK-7920) Make MLlib ChiSqSelector Serializable ( Fix Related Documentation Example).

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7920: --- Labels: (was: spark.tc) Make MLlib ChiSqSelector Serializable ( Fix Related Documentation

[jira] [Updated] (SPARK-8927) Doc format wrong for some config descriptions

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8927: --- Labels: (was: spark.tc) Doc format wrong for some config descriptions

[jira] [Updated] (SPARK-7985) Remove fittingParamMap references. Update ML Doc Estimator, Transformer, and Param examples.

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7985: --- Labels: (was: spark.tc) Remove fittingParamMap references. Update ML Doc Estimator

[jira] [Updated] (SPARK-7969) Drop method on Dataframes should handle Column

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7969: --- Labels: (was: spark.tc) Drop method on Dataframes should handle Column

[jira] [Updated] (SPARK-7830) ML doc cleanup: logreg, classification link

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7830: --- Labels: (was: spark.tc) ML doc cleanup: logreg, classification link

[jira] [Updated] (SPARK-8343) Improve the Spark Streaming Guides

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8343: --- Labels: (was: spark.tc) Improve the Spark Streaming Guides

[jira] [Updated] (SPARK-7977) Disallow println

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7977: --- Labels: starter (was: spark.tc starter) Disallow println

[jira] [Updated] (SPARK-8570) Improve MLlib Local Matrix Documentation.

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8570: --- Labels: (was: spark.tc) Improve MLlib Local Matrix Documentation

[jira] [Updated] (SPARK-7883) Fixing broken trainImplicit example in MLlib Collaborative Filtering documentation.

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7883: --- Labels: (was: spark.tc) Fixing broken trainImplicit example in MLlib Collaborative

[jira] [Updated] (SPARK-7426) spark.ml AttributeFactory.fromStructField should allow other NumericTypes

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7426: --- Labels: (was: spark.tc) spark.ml AttributeFactory.fromStructField should allow other

[jira] [Updated] (SPARK-8639) Instructions for executing jekyll in docs/README.md could be slightly more clear, typo in docs/api.md

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8639: --- Labels: (was: spark.tc) Instructions for executing jekyll in docs/README.md could

[jira] [Updated] (SPARK-7357) Improving HBaseTest example

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7357: --- Labels: (was: spark.tc) Improving HBaseTest example

[jira] [Updated] (SPARK-8746) Need to update download link for Hive 0.13.1 jars (HiveComparisonTest)

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8746: --- Labels: documentation test (was: documentation spark.tc test) Need to update download link

[jira] [Updated] (SPARK-6485) Add CoordinateMatrix/RowMatrix/IndexedRowMatrix in PySpark

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6485: --- Labels: (was: spark.tc) Add CoordinateMatrix/RowMatrix/IndexedRowMatrix in PySpark

[jira] [Updated] (SPARK-7744) Distributed matrix section in MLlib Data Types documentation should be reordered.

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7744: --- Labels: (was: spark.tc) Distributed matrix section in MLlib Data Types documentation

[jira] [Updated] (SPARK-6785) DateUtils can not handle date before 1970/01/01 correctly

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6785: --- Labels: (was: spark.tc) DateUtils can not handle date before 1970/01/01 correctly

[jira] [Updated] (SPARK-5562) LDA should handle empty documents

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5562: --- Labels: starter (was: spark.tc starter) LDA should handle empty documents

[jira] [Updated] (SPARK-7265) Improving documentation for Spark SQL Hive support

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7265: --- Labels: (was: spark.tc) Improving documentation for Spark SQL Hive support

[jira] [Updated] (SPARK-2859) Update url of Kryo project in related docs

2015-07-14 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2859: --- Labels: (was: spark.tc) Update url of Kryo project in related docs

[jira] [Resolved] (SPARK-1403) Spark on Mesos does not set Thread's context class loader

2015-07-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1403. Resolution: Fixed Target Version/s: (was: 1.5.0) Hey All, This issue should

[jira] [Comment Edited] (SPARK-1403) Spark on Mesos does not set Thread's context class loader

2015-07-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625739#comment-14625739 ] Patrick Wendell edited comment on SPARK-1403 at 7/14/15 2:59 AM

[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-13 Thread Patrick Wendell
This vote passes with 14 +1 (7 binding) votes and no 0 or -1 votes. +1 (14): Patrick Wendell Reynold Xin Sean Owen Burak Yavuz Mark Hamstra Michael Armbrust Andrew Or York, Brennon Krishna Sankar Luciano Resende Holden Karau Tom Graves Denny Lee Sean McNamara - Patrick On Wed, Jul 8, 2015 at 10

Re: Foundation policy on releases and Spark nightly builds

2015-07-12 Thread Patrick Wendell
pretty good to me. Mark it developers-only, not formally tested by the community, etc.) On Sun, Jul 12, 2015 at 7:50 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-12 Thread Patrick Wendell
I think we can close this vote soon. Any addition votes/testing would be much appreciated! On Fri, Jul 10, 2015 at 11:30 AM, Sean McNamara sean.mcnam...@webtrends.com wrote: +1 Sean On Jul 8, 2015, at 11:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2015-07-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624086#comment-14624086 ] Patrick Wendell commented on SPARK-2089: Yeah - we can open it again later

Re: Foundation policy on releases and Spark nightly builds

2015-07-12 Thread Patrick Wendell
Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease visibility in a way that is not beneficial to the project - this feature was specifically requested by developers from other projects that integrate with Spark. If the

SparkHub: a new community site for Apache Spark

2015-07-10 Thread Patrick Wendell
Hi All, Today, I'm happy to announce SparkHub (http://sparkhub.databricks.com), a service for the Apache Spark community to easily find the most relevant Spark resources on the web. SparkHub is a curated list of Spark news, videos and talks, package releases, upcoming events around the world,

[jira] [Created] (SPARK-8957) Backport Hive 1.X support to Branch 1.4

2015-07-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8957: -- Summary: Backport Hive 1.X support to Branch 1.4 Key: SPARK-8957 URL: https://issues.apache.org/jira/browse/SPARK-8957 Project: Spark Issue Type

[jira] [Updated] (SPARK-8957) Backport Hive 1.X support to Branch 1.4

2015-07-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8957: --- Priority: Critical (was: Major) Backport Hive 1.X support to Branch 1.4

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Patrick Wendell
+1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2015-07-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620051#comment-14620051 ] Patrick Wendell commented on SPARK-2089: Yeah - I think let's get SPARK-4352

[jira] [Created] (SPARK-8949) Remove references to preferredNodeLocalityData in javadoc and print warning when used

2015-07-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8949: -- Summary: Remove references to preferredNodeLocalityData in javadoc and print warning when used Key: SPARK-8949 URL: https://issues.apache.org/jira/browse/SPARK-8949

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues

[jira] [Comment Edited] (SPARK-8768) SparkSubmitSuite fails on Hadoop 1.x builds due to java.lang.VerifyError in Akka Protobuf

2015-07-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619681#comment-14619681 ] Patrick Wendell edited comment on SPARK-8768 at 7/9/15 1:04 AM

[jira] [Commented] (SPARK-8768) SparkSubmitSuite fails on Hadoop 1.x builds due to java.lang.VerifyError in Akka Protobuf

2015-07-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619687#comment-14619687 ] Patrick Wendell commented on SPARK-8768: I created SPARK-8933 to track

[jira] [Created] (SPARK-8933) Provide a --force flag to build/mvn that always uses downloaded maven

2015-07-08 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8933: -- Summary: Provide a --force flag to build/mvn that always uses downloaded maven Key: SPARK-8933 URL: https://issues.apache.org/jira/browse/SPARK-8933 Project

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
Hey All, The issue that Josh pointed out is not just a test failure, it's an issue with an important bug fix that was not correctly back-ported into the 1.4 branch. Unfortunately the overall state of the 1.4 branch tests on Jenkins was not in great shape so this was missed earlier on. Given that

[jira] [Commented] (SPARK-8768) SparkSubmitSuite fails on Hadoop 1.x builds due to java.lang.VerifyError in Akka Protobuf

2015-07-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619681#comment-14619681 ] Patrick Wendell commented on SPARK-8768: So it turns out that build/mvn still uses

[VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-08 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2):

[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
This vote is cancelled in favor of RC4. - Patrick On Tue, Jul 7, 2015 at 12:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http

[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-07 Thread Patrick Wendell
Hey All, This vote is cancelled in favor of RC3. - Patrick On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here

[jira] [Updated] (SPARK-6805) ML Pipeline API in SparkR

2015-07-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6805: --- Priority: Critical (was: Major) ML Pipeline API in SparkR

Re: Can not build master

2015-07-04 Thread Patrick Wendell
-03 17:35 GMT-07:00 Krishna Sankar ksanka...@gmail.com: Patrick, I assume an RC3 will be out for folks like me to test the distribution. As usual, I will run the tests when you have a new distribution. Cheers k/ On Fri, Jul 3, 2015 at 4:38 PM, Patrick Wendell pwend...@gmail.com wrote

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
-Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package this also gave the ‘Dependency-reduced POM’ loop Robin On 3 Jul 2015, at 23:41, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Let's continue the disucssion on the other thread relating to the master build. On Fri, Jul 3, 2015 at 4:13 PM, Patrick Wendell pwend...@gmail.com wrote: Thanks - it appears this is just a legitimate issue with the build, affecting all versions of Maven. On Fri, Jul 3, 2015 at 4:02 PM

Re: Can not build master

2015-07-03 Thread Patrick Wendell
: Doesn't change anything for me. On Fri, Jul 3, 2015 at 3:45 PM Patrick Wendell pwend...@gmail.com wrote: Can you try using the built in maven build/mvn...? All of our builds are passing on Jenkins so I wonder if it's a maven version issue: https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA

Re: Can not build master

2015-07-03 Thread Patrick Wendell
://github.com/apache/spark/commit/bc51bcaea734fe64a90d007559e76f5ceebfea9e On Fri, Jul 3, 2015 at 4:36 PM, Patrick Wendell pwend...@gmail.com wrote: Okay I did some forensics with Sean Owen. Some things about this bug: 1. The underlying cause is that we added some code to make the tests of sub

Re: Can not build master

2015-07-03 Thread Patrick Wendell
Can you try using the built in maven build/mvn...? All of our builds are passing on Jenkins so I wonder if it's a maven version issue: https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/ - Patrick On Fri, Jul 3, 2015 at 3:14 PM, Ted Yu yuzhih...@gmail.com wrote: Please take a look at

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
, family: mac Let me nuke it and reinstall maven. Cheers k/ On Fri, Jul 3, 2015 at 3:41 PM, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven

[RESULT] [VOTE] Release Apache Spark 1.4.1

2015-07-03 Thread Patrick Wendell
the time of the RC voting is an interesting topic, Sean I like your most recent proposal. Maybe we can put that on the wiki or start a DISCUSS thread to cover that topic. On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate

[VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7):

[jira] [Resolved] (SPARK-8649) Mapr repository is not defined properly

2015-06-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-8649. Resolution: Fixed Fix Version/s: 1.5.0 Mapr repository is not defined properly

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-28 Thread Patrick Wendell
/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted

[jira] [Commented] (SPARK-8667) Improve Spark UI behavior at scale

2015-06-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604000#comment-14604000 ] Patrick Wendell commented on SPARK-8667: Thanks Sean. I looked for a while

[jira] [Resolved] (SPARK-8667) Improve Spark UI behavior at scale

2015-06-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-8667. Resolution: Duplicate Improve Spark UI behavior at scale

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-26 Thread Patrick Wendell
something that is definitely being worked on for 1.4.1? On Wed, Jun 24, 2015 at 6:56 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted

[jira] [Created] (SPARK-8667) Improve Spark UI behavior at scale

2015-06-26 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8667: -- Summary: Improve Spark UI behavior at scale Key: SPARK-8667 URL: https://issues.apache.org/jira/browse/SPARK-8667 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-8667) Improve Spark UI behavior at scale

2015-06-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8667: --- Component/s: Web UI Improve Spark UI behavior at scale

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-24 Thread Patrick Wendell
of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here

[VOTE] Release Apache Spark 1.4.1

2015-06-23 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5):

[jira] [Updated] (SPARK-8494) ClassNotFoundException when running with sbt, scala 2.10.4, spray 1.3.3

2015-06-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8494: --- Assignee: (was: Patrick Wendell) ClassNotFoundException when running with sbt, scala

[jira] [Updated] (SPARK-7292) Provide operator to truncate lineage without persisting RDD's

2015-06-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7292: --- Assignee: Andrew Or Provide operator to truncate lineage without persisting RDD's

[jira] [Commented] (SPARK-8416) Thread dump page should highlight Spark executor threads

2015-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592411#comment-14592411 ] Patrick Wendell commented on SPARK-8416: It would also be nice to put those

[jira] [Updated] (SPARK-8434) Add a pretty parameter to show

2015-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8434: --- Component/s: SQL Add a pretty parameter to show

[jira] [Updated] (SPARK-8450) PySpark write.parquet raises Unsupported datatype DecimalType()

2015-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8450: --- Component/s: SQL PySpark PySpark write.parquet raises Unsupported datatype

[jira] [Updated] (SPARK-8427) Incorrect ACL checking for partitioned table in Spark SQL-1.4

2015-06-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8427: --- Priority: Critical (was: Blocker) Incorrect ACL checking for partitioned table in Spark SQL

[jira] [Updated] (SPARK-5787) Protect JVM from some not-important exceptions

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5787: --- Target Version/s: 1.5.0 (was: 1.4.0) Protect JVM from some not-important exceptions

[jira] [Updated] (SPARK-7448) Implement custom bye array serializer for use in PySpark shuffle

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7448: --- Target Version/s: 1.5.0 (was: 1.4.0) Implement custom bye array serializer for use

[jira] [Updated] (SPARK-7078) Cache-aware binary processing in-memory sort

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7078: --- Target Version/s: 1.5.0 (was: 1.4.0) Cache-aware binary processing in-memory sort

[jira] [Updated] (SPARK-7041) Avoid writing empty files in BypassMergeSortShuffleWriter

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7041: --- Target Version/s: 1.5.0 (was: 1.4.0) Avoid writing empty files

[jira] [Commented] (SPARK-6393) Extra RPC to the AM during killExecutor invocation

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590426#comment-14590426 ] Patrick Wendell commented on SPARK-6393: [~sandyryza] I'm un-targeting

<    1   2   3   4   5   6   7   8   9   10   >