[jira] [Updated] (SPARK-5735) Replace uses of EasyMock with Mockito

2015-02-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5735: --- Assignee: Josh Rosen Replace uses of EasyMock with Mockito

Re: Mail to u...@spark.apache.org failing

2015-02-09 Thread Patrick Wendell
Ah - we should update it to suggest mailing the dev@ list (and if there is enough traffic maybe do something else). I'm happy to add you if you can give an organization name, URL, a list of which Spark components you are using, and a short description of your use case.. On Mon, Feb 9, 2015 at

Re: New Metrics Sink class not packaged in spark-assembly jar

2015-02-09 Thread Patrick Wendell
Hi Judy, If you have added source files in the sink/ source folder, they should appear in the assembly jar when you build. One thing I noticed is that you are looking inside the /dist folder. That only gets populated if you run make-distribution. The normal development process is just to do mvn

[jira] [Resolved] (SPARK-1142) Allow adding jars on app submission, outside of code

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1142. Resolution: Not a Problem Allow adding jars on app submission, outside of code

[jira] [Updated] (SPARK-5195) when hive table is query with alias the cache data lose effectiveness.

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5195: --- Assignee: yixiaohua when hive table is query with alias the cache data lose effectiveness

[jira] [Created] (SPARK-5690) Flaky test:

2015-02-09 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5690: -- Summary: Flaky test: Key: SPARK-5690 URL: https://issues.apache.org/jira/browse/SPARK-5690 Project: Spark Issue Type: Bug Components: Tests

[jira] [Updated] (SPARK-5690) Flaky test: org.apache.spark.deploy.rest.StandaloneRestSubmitSuite.simple submit until completion

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5690: --- Summary: Flaky test: org.apache.spark.deploy.rest.StandaloneRestSubmitSuite.simple submit

[jira] [Updated] (SPARK-5689) Document what can be run in different YARN modes

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5689: --- Issue Type: Documentation (was: Improvement) Document what can be run in different YARN

[jira] [Commented] (SPARK-1142) Allow adding jars on app submission, outside of code

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312608#comment-14312608 ] Patrick Wendell commented on SPARK-1142: This already exists - you can use

Re: Keep or remove Debian packaging in Spark?

2015-02-09 Thread Patrick Wendell
I have wondered whether we should sort of deprecated it more officially, since otherwise I think people have the reasonable expectation based on the current code that Spark intends to support complete Debian packaging as part of the upstream build. Having something that's sort-of maintained but no

[jira] [Resolved] (SPARK-2892) Socket Receiver does not stop when streaming context is stopped

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2892. Resolution: Fixed Fix Version/s: 1.2.1 I believe this is fixed by SPARK-5035, so I'm

Re: multi-line comment style

2015-02-09 Thread Patrick Wendell
can find Thanks Shivaram [1] https://google-styleguide.googlecode.com/svn/trunk/javaguide.html#s4.8.6.1-block-comment-style On Wed, Feb 4, 2015 at 2:05 PM, Patrick Wendell pwend...@gmail.com wrote: Personally I have no opinion, but agree it would be nice to standardize. - Patrick

[ANNOUNCE] Apache Spark 1.2.1 Released

2015-02-09 Thread Patrick Wendell
Hi All, I've just posted the 1.2.1 maintenance release of Apache Spark. We recommend all 1.2.0 users upgrade to this release, as this release includes stability fixes across all components of Spark. - Download this release: http://spark.apache.org/downloads.html - View the release notes:

[jira] [Commented] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313030#comment-14313030 ] Patrick Wendell commented on SPARK-4423: [~joshrosen] Is this specific to foreach

[jira] [Commented] (SPARK-5696) HiveThriftServer2Suite fails because of extra log4j.properties in the driver classpath

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313005#comment-14313005 ] Patrick Wendell commented on SPARK-5696: Wow - this must have been a substantial

[jira] [Commented] (SPARK-5647) Output metrics do not show up for older hadoop versions ( 2.5)

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313020#comment-14313020 ] Patrick Wendell commented on SPARK-5647: Isn't it just possible to get the file

[jira] [Updated] (SPARK-5647) Output metrics do not show up for older hadoop versions ( 2.5)

2015-02-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5647: --- Target Version/s: 1.4.0 Output metrics do not show up for older hadoop versions ( 2.5

Re: Improving metadata in Spark JIRA

2015-02-08 Thread Patrick Wendell
:52 PM Patrick Wendell pwend...@gmail.com wrote: Per Nick's suggestion I added two components: 1. Spark Submit 2. Spark Scheduler I figured I would just add these since if we decide later we don't want them, we can simply merge them into Spark Core. On Fri, Feb 6, 2015 at 11:53 AM

[RESULT] [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-08 Thread Patrick Wendell
This vote passes with 5 +1 votes (3 binding) and no 0 or -1 votes. +1 Votes: Krishna Sankar Sean Owen* Chip Senkbeil Matei Zaharia* Patrick Wendell* 0 Votes: (none) -1 Votes: (none) On Fri, Feb 6, 2015 at 5:12 PM, Patrick Wendell pwend...@gmail.com wrote: I'll add a +1 as well. On Fri, Feb

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-08 Thread Patrick Wendell
I'll add a +1 as well. On Fri, Feb 6, 2015 at 2:38 PM, Matei Zaharia matei.zaha...@gmail.com wrote: +1 Tested on Mac OS X. Matei On Feb 2, 2015, at 8:57 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.2.1

Unit tests

2015-02-08 Thread Patrick Wendell
Hey All, The tests are in a not-amazing state right now due to a few compounding factors: 1. We've merged a large volume of patches recently. 2. The load on jenkins has been relatively high, exposing races and other behavior not seen at lower load. For those not familiar, the main issue is

[jira] [Commented] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311475#comment-14311475 ] Patrick Wendell commented on SPARK-761: --- I think the main thing to catch would

[jira] [Updated] (SPARK-4687) SparkContext#addFile doesn't keep file folder information

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4687: --- Component/s: Spark Core SparkContext#addFile doesn't keep file folder information

[jira] [Updated] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5299: --- Component/s: (was: Deploy) Build Is http://www.apache.org/dist/spark

[jira] [Updated] (SPARK-3033) [Hive] java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3033: --- Component/s: (was: Spark Core) [Hive] java.math.BigDecimal cannot be cast

[jira] [Updated] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-761: -- Labels: starter (was: ) Print a nicer error message when incompatible Spark binaries try

[jira] [Updated] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-761: -- Description: As a starter task, it would be good to audit the current behavior for different

[jira] [Commented] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311490#comment-14311490 ] Patrick Wendell commented on SPARK-761: --- [~aash] right now we don't explicitly encode

[jira] [Updated] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5659: --- Component/s: Tests Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

[jira] [Created] (SPARK-5679) Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads and .input metrics with mixed read method

2015-02-08 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5679: -- Summary: Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads and .input metrics with mixed read method Key: SPARK-5679 URL: https

[jira] [Updated] (SPARK-5679) Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads and input metrics with mixed read method

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5679: --- Description: Please audit these and see if there are any assumptions with respect to File IO

[jira] [Updated] (SPARK-5679) Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads and input metrics with mixed read method

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5679: --- Summary: Flaky tests in InputOutputMetricsSuite: input metrics with interleaved reads

[jira] [Updated] (SPARK-4896) Don't redundantly copy executor dependencies in Utils.fetchFile

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4896: --- Component/s: Spark Core Don't redundantly copy executor dependencies in Utils.fetchFile

[jira] [Updated] (SPARK-5355) SparkConf is not thread-safe

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5355: --- Component/s: Spark Core SparkConf is not thread-safe

[jira] [Updated] (SPARK-5289) Backport publishing of repl, yarn into branch-1.2

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5289: --- Component/s: Build Backport publishing of repl, yarn into branch-1.2

[jira] [Updated] (SPARK-5254) Update the user guide to make clear that spark.mllib is not being deprecated

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5254: --- Component/s: MLlib Update the user guide to make clear that spark.mllib is not being

[jira] [Updated] (SPARK-5308) MD5 / SHA1 hash format doesn't match standard Maven output

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5308: --- Fix Version/s: (was: 1.2.1) MD5 / SHA1 hash format doesn't match standard Maven output

[jira] [Updated] (SPARK-5308) MD5 / SHA1 hash format doesn't match standard Maven output

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5308: --- Fix Version/s: 1.2.1 MD5 / SHA1 hash format doesn't match standard Maven output

[jira] [Updated] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5524: --- Component/s: Spark Core Remove messy dependencies to log4j

[jira] [Commented] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1430#comment-1430 ] Patrick Wendell commented on SPARK-5524: [~nchammas] I don't think this is related

[jira] [Updated] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5524: --- Component/s: (was: Build) Remove messy dependencies to log4j

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309784#comment-14309784 ] Patrick Wendell commented on SPARK-5388: On DELETE, I'll defer to you guys, have

Re: Improving metadata in Spark JIRA

2015-02-06 Thread Patrick Wendell
Per Nick's suggestion I added two components: 1. Spark Submit 2. Spark Scheduler I figured I would just add these since if we decide later we don't want them, we can simply merge them into Spark Core. On Fri, Feb 6, 2015 at 11:53 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Do we

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309825#comment-14309825 ] Patrick Wendell commented on SPARK-5388: One the boolean and numeric values. I

[jira] [Updated] (SPARK-4874) Report number of records read/written in a task

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4874: --- Component/s: Web UI Spark Core Report number of records read/written

[jira] [Resolved] (SPARK-4874) Report number of records read/written in a task

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4874. Resolution: Fixed Fix Version/s: 1.3.0 Target Version/s: 1.3.0 Report

[jira] [Updated] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5659: --- Labels: flaky-test (was: ) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

[jira] [Resolved] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5388. Resolution: Fixed Fix Version/s: 1.3.0 Provide a stable application submission

[jira] [Updated] (SPARK-5662) Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5662: --- Priority: Critical (was: Major) Flaky test

[jira] [Created] (SPARK-5662) Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream

2015-02-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5662: -- Summary: Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream Key: SPARK-5662 URL: https://issues.apache.org/jira/browse/SPARK-5662

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308635#comment-14308635 ] Patrick Wendell commented on SPARK-5388: I think it's reasonable to use DELETE per

[jira] [Commented] (SPARK-5557) spark-shell failed to start

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308346#comment-14308346 ] Patrick Wendell commented on SPARK-5557: I can send a fix for this shortly

Re: PSA: Maven supports parallel builds

2015-02-05 Thread Patrick Wendell
I've done this in the past, but back when I wasn't using Zinc it didn't make a big difference. It's worth doing this in our jenkins environment though. - Patrick On Thu, Feb 5, 2015 at 4:52 PM, Dirceu Semighini Filho dirceu.semigh...@gmail.com wrote: Thanks Nicholas, I didn't knew this.

[jira] [Commented] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308423#comment-14308423 ] Patrick Wendell commented on SPARK-5607: This may have actually caused more

[jira] [Updated] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5557: --- Summary: Servlet API classes now missing after jetty shading (was: spark-shell failed

[jira] [Comment Edited] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308391#comment-14308391 ] Patrick Wendell edited comment on SPARK-5557 at 2/6/15 1:07 AM

[jira] [Commented] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308391#comment-14308391 ] Patrick Wendell commented on SPARK-5557: I'm sorry that this affected so many

[jira] [Resolved] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5557. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Patrick Wendell Servlet

[jira] [Commented] (SPARK-5626) Spurious test failures due to NullPointerException in EasyMock test code

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308482#comment-14308482 ] Patrick Wendell commented on SPARK-5626: [~joshrosen] I may have caused

[jira] [Reopened] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-5607: NullPointerException in objenesis - Key

[jira] [Commented] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308493#comment-14308493 ] Patrick Wendell commented on SPARK-5607: I've reverted my patch since it may have

[jira] [Resolved] (SPARK-5474) curl should support URL redirection in build/mvn

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5474. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Guoqiang Li curl should

[jira] [Updated] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5594: --- Priority: Critical (was: Major) SparkException: Failed to get broadcast (TorrentBroadcast

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305657#comment-14305657 ] Patrick Wendell commented on SPARK-5594: I've seen this occasionally in unit tests

Re: 1.2.1-rc3 - Avro input format for Hadoop 2 broken/fix?

2015-02-04 Thread Patrick Wendell
Hi Markus, That won't be included in 1.2.1 most likely because the release votes have already started, and at that point we don't hold the release except for major regression issues from 1.2.0. However, if this goes through we can backport it into the 1.2 branch and it will end up in a future

Re: multi-line comment style

2015-02-04 Thread Patrick Wendell
Personally I have no opinion, but agree it would be nice to standardize. - Patrick On Wed, Feb 4, 2015 at 1:58 PM, Sean Owen so...@cloudera.com wrote: One thing Marcelo pointed out to me is that the // style does not interfere with commenting out blocks of code with /* */, which is a small

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Priority: Blocker (was: Critical) Automatically provide sqlContext in Spark shell

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Assignee: (was: Patrick Wendell) Automatically provide sqlContext in Spark shell

[jira] [Resolved] (SPARK-5411) Allow SparkListeners to be specified in SparkConf and loaded when creating SparkContext

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5411. Resolution: Fixed Fix Version/s: 1.3.0 Allow SparkListeners to be specified

[jira] [Resolved] (SPARK-5607) NullPointerException in objenesis

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5607. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Patrick Wendell

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Labels: flaky-test (was: ) Flaky test: Python regression

[jira] [Created] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5585: -- Summary: Flaky test: Python regression Key: SPARK-5585 URL: https://issues.apache.org/jira/browse/SPARK-5585 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Affects Version/s: 1.3.0 Flaky test: Python regression

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Priority: Critical (was: Major) Flaky test: Python regression

[jira] [Updated] (SPARK-5341) Support maven coordinates in spark-shell and spark-submit

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5341: --- Assignee: Burak Yavuz Support maven coordinates in spark-shell and spark-submit

[jira] [Resolved] (SPARK-5341) Support maven coordinates in spark-shell and spark-submit

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5341. Resolution: Fixed Fix Version/s: 1.3.0 Support maven coordinates in spark-shell

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Priority: Critical (was: Major) Automatically provide sqlContext in Spark shell

[jira] [Created] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5586: -- Summary: Automatically provide sqlContext in Spark shell Key: SPARK-5586 URL: https://issues.apache.org/jira/browse/SPARK-5586 Project: Spark Issue Type

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Fix Version/s: (was: 1.3.0) Automatically provide sqlContext in Spark shell

[jira] [Updated] (SPARK-5140) Two RDDs which are scheduled concurrently should be able to wait on parent in all cases

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5140: --- Fix Version/s: (was: 1.2.1) (was: 1.3.0) Two RDDs which

[jira] [Created] (SPARK-5569) Checkpoints cannot reference classes defined outside of Spark's assembly

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5569: -- Summary: Checkpoints cannot reference classes defined outside of Spark's assembly Key: SPARK-5569 URL: https://issues.apache.org/jira/browse/SPARK-5569 Project

[ANNOUNCE] branch-1.3 has been cut

2015-02-03 Thread Patrick Wendell
Hey All, Just wanted to announce that we've cut the 1.3 branch which will become the 1.3 release after community testing. There are still some features that will go in (in higher level libraries, and some stragglers in spark core), but overall this indicates the end of major feature development

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302999#comment-14302999 ] Patrick Wendell commented on SPARK-4550: The doc alludes to having to (at some

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304010#comment-14304010 ] Patrick Wendell commented on SPARK-5388: The intention for this is really just

Re: Spark Master Maven with YARN build is broken

2015-02-02 Thread Patrick Wendell
It's my fault, I'm sending a hot fix now. On Mon, Feb 2, 2015 at 1:44 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ Is this is a known issue? It seems to have

[jira] [Commented] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302290#comment-14302290 ] Patrick Wendell commented on SPARK-3778: /cc [~hshreedharan] newAPIHadoopRDD

[jira] [Updated] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3778: --- Priority: Blocker (was: Critical) newAPIHadoopRDD doesn't properly pass credentials

[jira] [Updated] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4550: --- Target Version/s: 1.4.0 In sort-based shuffle, store map outputs in serialized form

[jira] [Updated] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4550: --- Priority: Critical (was: Major) In sort-based shuffle, store map outputs in serialized form

[jira] [Updated] (SPARK-5195) when hive table is query with alias the cache data lose effectiveness.

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5195: --- Fix Version/s: (was: 1.2.1) when hive table is query with alias the cache data lose

[jira] [Updated] (SPARK-4508) Native Date type for SQL92 Date

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4508: --- Fix Version/s: (was: 1.3.0) Native Date type for SQL92 Date

[jira] [Created] (SPARK-5541) Allow running Maven or SBT in the Spark build

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5541: -- Summary: Allow running Maven or SBT in the Spark build Key: SPARK-5541 URL: https://issues.apache.org/jira/browse/SPARK-5541 Project: Spark Issue Type

[jira] [Updated] (SPARK-5541) Allow running Maven or SBT in run-tests

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5541: --- Summary: Allow running Maven or SBT in run-tests (was: Allow running Maven or SBT

[jira] [Reopened] (SPARK-4508) Native Date type for SQL92 Date

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4508: This has caused several date-related test failures in the master and pull request builds, so

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302326#comment-14302326 ] Patrick Wendell commented on SPARK-4550: Yeah, this is a good idea. I don't see

[jira] [Created] (SPARK-5542) Decouple publishing, packaging, and tagging in release script

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5542: -- Summary: Decouple publishing, packaging, and tagging in release script Key: SPARK-5542 URL: https://issues.apache.org/jira/browse/SPARK-5542 Project: Spark

Temporary jenkins issue

2015-02-02 Thread Patrick Wendell
Hey All, I made a change to the Jenkins configuration that caused most builds to fail (attempting to enable a new plugin), I've reverted the change effective about 10 minutes ago. If you've seen recent build failures like below, this was caused by that change. Sorry about that. ERROR:

[jira] [Resolved] (SPARK-5542) Decouple publishing, packaging, and tagging in release script

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5542. Resolution: Fixed Fix Version/s: 1.3.0 Decouple publishing, packaging, and tagging

[jira] [Created] (SPARK-5548) Flaky test: org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5548: -- Summary: Flaky test: org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server Key: SPARK-5548 URL: https://issues.apache.org/jira/browse/SPARK-5548

[RESULT] [VOTE] Release Apache Spark 1.2.1 (RC2)

2015-02-02 Thread Patrick Wendell
This is cancelled in favor of RC2. On Mon, Feb 2, 2015 at 8:50 PM, Patrick Wendell pwend...@gmail.com wrote: The windows issue reported only affects actually running Spark on Windows (not job submission). However, I agree it's worth cutting a new RC. I'm going to cancel this vote and propose

<    5   6   7   8   9   10   11   12   13   14   >