[jira] [Updated] (SPARK-6393) Extra RPC to the AM during killExecutor invocation

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6393: --- Target Version/s: (was: 1.4.0) Extra RPC to the AM during killExecutor invocation

[jira] [Updated] (SPARK-8390) Update DirectKafkaWordCount examples to show how offset ranges can be used

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8390: --- Issue Type: Improvement (was: Bug) Update DirectKafkaWordCount examples to show how offset

[jira] [Updated] (SPARK-8389) Expose KafkaRDDs offsetRange in Java and Python

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8389: --- Issue Type: New Feature (was: Bug) Expose KafkaRDDs offsetRange in Java and Python

[jira] [Updated] (SPARK-7689) Deprecate spark.cleaner.ttl

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7689: --- Target Version/s: 1.4.1 (was: 1.4.0) Deprecate spark.cleaner.ttl

[jira] [Commented] (SPARK-7521) Allow all required release credentials to be specified with env vars

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590451#comment-14590451 ] Patrick Wendell commented on SPARK-7521: Sort of - I still need to actually

[jira] [Updated] (SPARK-8397) Allow custom configuration for TestHive

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8397: --- Component/s: SQL Allow custom configuration for TestHive

[jira] [Updated] (SPARK-8388) The script docs/_plugins/copy_api_dirs.rb should be run anywhere

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8388: --- Target Version/s: (was: 1.4.0) The script docs/_plugins/copy_api_dirs.rb should be run

[jira] [Updated] (SPARK-8325) Ability to provide role based row level authorization through Spark SQL

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8325: --- Target Version/s: (was: 1.4.0) Ability to provide role based row level authorization

[jira] [Updated] (SPARK-8388) The script docs/_plugins/copy_api_dirs.rb should be run anywhere

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8388: --- Fix Version/s: (was: 1.4.1) The script docs/_plugins/copy_api_dirs.rb should be run

[jira] [Commented] (SPARK-8388) The script docs/_plugins/copy_api_dirs.rb should be run anywhere

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590408#comment-14590408 ] Patrick Wendell commented on SPARK-8388: Hi [~kaixin9ok] - please don't set fix

[jira] [Updated] (SPARK-8325) Ability to provide role based row level authorization through Spark SQL

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8325: --- Fix Version/s: (was: 1.4.1) Ability to provide role based row level authorization

[jira] [Updated] (SPARK-8324) Register Query as view through JDBC interface

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8324: --- Target Version/s: (was: 1.4.0) Register Query as view through JDBC interface

[jira] [Updated] (SPARK-8324) Register Query as view through JDBC interface

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8324: --- Fix Version/s: (was: 1.4.1) Register Query as view through JDBC interface

[jira] [Updated] (SPARK-7025) Create a Java-friendly input source API

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7025: --- Target Version/s: 1.5.0 (was: 1.4.0) Create a Java-friendly input source API

[jira] [Updated] (SPARK-6208) executor-memory does not work when using local cluster

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6208: --- Target Version/s: (was: 1.4.0) executor-memory does not work when using local cluster

[jira] [Updated] (SPARK-7019) Build docs on doc changes

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7019: --- Target Version/s: 1.5.0 (was: 1.4.0) Build docs on doc changes

[jira] [Updated] (SPARK-7018) Refactor dev/run-tests-jenkins into Python

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7018: --- Target Version/s: 1.5.0 (was: 1.4.0) Refactor dev/run-tests-jenkins into Python

[jira] [Updated] (SPARK-5647) Output metrics do not show up for older hadoop versions ( 2.5)

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5647: --- Target Version/s: (was: 1.4.0) Output metrics do not show up for older hadoop versions

[jira] [Resolved] (SPARK-4227) Document external shuffle service

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4227. Resolution: Not A Problem Resolving since I think the conclusion is that this works

[jira] [Updated] (SPARK-8369) Support dependency jar and files on HDFS in standalone cluster mode

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8369: --- Target Version/s: (was: 1.3.1, 1.4.0) Support dependency jar and files on HDFS

[jira] [Updated] (SPARK-7355) FlakyTest - o.a.s.DriverSuite

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7355: --- Target Version/s: 1.4.1 (was: 1.4.0) FlakyTest - o.a.s.DriverSuite

[jira] [Updated] (SPARK-6026) Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6026: --- Target Version/s: (was: 1.4.0) Eliminate the bypassMergeThreshold parameter and associated

[jira] [Updated] (SPARK-7021) JUnit output for Python tests

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7021: --- Target Version/s: 1.5.0 (was: 1.4.0) JUnit output for Python tests

[jira] [Updated] (SPARK-7016) Refactor dev/run-tests(-jenkins) from Bash to Python

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7016: --- Target Version/s: 1.5.0 (was: 1.4.0) Refactor dev/run-tests(-jenkins) from Bash to Python

[jira] [Updated] (SPARK-5915) Spillable should check every N bytes rather than every 32 elements

2015-06-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5915: --- Target Version/s: (was: 1.4.0) Spillable should check every N bytes rather than every 32

Re: Remove Hadoop 1 support (Hadoop 2.2) for Spark 1.5?

2015-06-13 Thread Patrick Wendell
is that it's much more efficient for us as the Spark maintainers to pay this cost rather than to force a lot of our users to deal with painful upgrades. On Sat, Jun 13, 2015 at 1:39 AM, Steve Loughran ste...@hortonworks.com wrote: On 12 Jun 2015, at 17:12, Patrick Wendell pwend...@gmail.com

[jira] [Updated] (SPARK-8318) Spark Streaming Starter JIRAs

2015-06-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8318: --- Issue Type: Improvement (was: Bug) Spark Streaming Starter JIRAs

Re: Remove Hadoop 1 support (Hadoop 2.2) for Spark 1.5?

2015-06-12 Thread Patrick Wendell
I feel this is quite different from the Java 6 decision and personally I don't see sufficient cause to do it. I would like to understand though Sean - what is the proposal exactly? Hadoop 2 itself supports all of the Hadoop 1 API's, so things like removing the Hadoop 1 variant of sc.hadoopFile,

Re: Fully in-memory shuffles

2015-06-11 Thread Patrick Wendell
...@gmail.com wrote: Ok so it is the case that small shuffles can be done without hitting any disk. Is this the same case for the aux shuffle service in yarn? Can that be done without hitting disk? On Wed, Jun 10, 2015 at 9:17 PM, Patrick Wendell pwend...@gmail.com wrote: In many cases the shuffle

[jira] [Commented] (SPARK-8311) saveAsTextFile with Hadoop1 could lead to errors

2015-06-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582741#comment-14582741 ] Patrick Wendell commented on SPARK-8311: Is this related to or the same as SPARK

[ANNOUNCE] Announcing Spark 1.4

2015-06-11 Thread Patrick Wendell
Hi All, I'm happy to announce the availability of Spark 1.4.0! Spark 1.4.0 is the fifth release on the API-compatible 1.X line. It is Spark's largest release ever, with contributions from 210 developers and more than 1,000 commits! A huge thanks go to all of the individuals and organizations

[ANNOUNCE] Announcing Spark 1.4

2015-06-11 Thread Patrick Wendell
Hi All, I'm happy to announce the availability of Spark 1.4.0! Spark 1.4.0 is the fifth release on the API-compatible 1.X line. It is Spark's largest release ever, with contributions from 210 developers and more than 1,000 commits! A huge thanks go to all of the individuals and organizations

[jira] [Commented] (SPARK-6511) Publish hadoop provided build with instructions for different distros

2015-06-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582158#comment-14582158 ] Patrick Wendell commented on SPARK-6511: Thanks [~RPCMoritz] - The --config option

[RESULT] [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-10 Thread Patrick Wendell
This vote passes! Thanks to everyone who voted. I will get the release artifacts and notes up within a day or two. +1 (23 votes): Reynold Xin* Patrick Wendell* Matei Zaharia* Andrew Or* Timothy Chen Calvin Jia Burak Yavuz Krishna Sankar Hari Shreedharan Ram Sriharsha* Kousuke Saruta Sandy Ryza

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-06-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581297#comment-14581297 ] Patrick Wendell commented on SPARK-5594: /cc [~zsxwing] SparkException: Failed

Re: Fully in-memory shuffles

2015-06-10 Thread Patrick Wendell
In many cases the shuffle will actually hit the OS buffer cache and not ever touch spinning disk if it is a size that is less than memory on the machine. - Patrick On Wed, Jun 10, 2015 at 5:06 PM, Corey Nolet cjno...@gmail.com wrote: So with this... to help my understanding of Spark under the

Re: Jcenter / bintray support for spark packages?

2015-06-10 Thread Patrick Wendell
Hey Hector, It's not a bad idea. I think we'd want to do this by virtue of allowing custom repositories, so users can add bintray or others. - Patrick On Wed, Jun 10, 2015 at 6:23 PM, Hector Yee hector@gmail.com wrote: Hi Spark devs, Is it possible to add jcenter or bintray support for

[jira] [Updated] (SPARK-6945) Provide SQL tab in the Spark UI

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6945: --- Assignee: Andrew Or Provide SQL tab in the Spark UI

[jira] [Updated] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7261: --- Labels: starter (was: ) Change default log level to WARN in the REPL

[jira] [Updated] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7261: --- Target Version/s: 1.5.0 (was: 1.4.0) Change default log level to WARN in the REPL

[jira] [Commented] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579359#comment-14579359 ] Patrick Wendell commented on SPARK-7261: Darn - this slipped (pretty simple change

[jira] [Assigned] (SPARK-6511) Publish hadoop provided build with instructions for different distros

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reassigned SPARK-6511: -- Assignee: Patrick Wendell Publish hadoop provided build with instructions

[jira] [Resolved] (SPARK-8061) Document how to use hadoop provided builds

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-8061. Resolution: Duplicate Document how to use hadoop provided builds

[jira] [Resolved] (SPARK-4356) Test Scala 2.11 on Jenkins

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4356. Resolution: Fixed There is now a 2.11 compile build: https://amplab.cs.berkeley.edu

[jira] [Updated] (SPARK-6946) Add visualization of logical and physical plans for SQL/DataFrames

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6946: --- Assignee: Andrew Or Add visualization of logical and physical plans for SQL/DataFrames

[jira] [Updated] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7261: --- Assignee: Shixiong Zhu (was: Patrick Wendell) Change default log level to WARN in the REPL

[jira] [Updated] (SPARK-8164) transformExpressions should support nested expression sequence

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8164: --- Component/s: SQL transformExpressions should support nested expression sequence

[jira] [Updated] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7261: --- Description: We should add a log4j properties file for the repl (log4j-defaults

[jira] [Updated] (SPARK-7261) Change default log level to WARN in the REPL

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7261: --- Priority: Blocker (was: Minor) Change default log level to WARN in the REPL

[jira] [Resolved] (SPARK-3079) Hive build should depend on parquet serdes

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3079. Resolution: Invalid I think we either fixed this or it's no longer an issue. Closing it now

[jira] [Resolved] (SPARK-5693) Install Pandas on Jenkins machines and enable to_pandas doctest for DataFrames

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5693. Resolution: Fixed This is super old. Pandas is no on all the Jenkins boxes. Install

[jira] [Resolved] (SPARK-6511) Publish hadoop provided build with instructions for different distros

2015-06-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6511. Resolution: Fixed Fix Version/s: 1.4.0 Publish hadoop provided build

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-08 Thread Patrick Wendell
Hi All, Thanks for the continued voting! I'm going to leave this thread open for another few days to continue to collect feedback. - Patrick On Tue, Jun 2, 2015 at 8:53 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version

Re: Scheduler question: stages with non-arithmetic numbering

2015-06-07 Thread Patrick Wendell
Hey Mike, Stage ID's are not guaranteed to be sequential because of the way the DAG scheduler works (only increasing). In some cases stage ID numbers are skipped when stages are generated. Any stage/ID that appears in the Spark UI is an actual stage, so if you see ID's in there, but they are not

[jira] [Updated] (SPARK-2883) Spark Support for ORCFile format

2015-06-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2883: --- Issue Type: New Feature (was: Bug) Spark Support for ORCFile format

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

2015-06-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5342: --- Issue Type: New Feature (was: Bug) Allow long running Spark apps to run on secure YARN/HDFS

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2015-06-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3266: --- Issue Type: Bug (was: Improvement) JavaDoubleRDD doesn't contain max

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2015-06-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3266: --- Issue Type: Improvement (was: Bug) JavaDoubleRDD doesn't contain max

[DISCUSS] Minimize use of MINOR, BUILD, and HOTFIX w/ no JIRA

2015-06-06 Thread Patrick Wendell
Hey All, Just a request here - it would be great if people could create JIRA's for any and all merged pull requests. The reason is that when patches get reverted due to build breaks or other issues, it is very difficult to keep track of what is going on if there is no JIRA. Here is a list of 5

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-04 Thread Patrick Wendell
I will give +1 as well. On Wed, Jun 3, 2015 at 11:59 PM, Reynold Xin r...@databricks.com wrote: Let me give you the 1st +1 On Tue, Jun 2, 2015 at 10:47 PM, Patrick Wendell pwend...@gmail.com wrote: He all - a tiny nit from the last e-mail. The tag is v1.4.0-rc4. The exact commit and all

[jira] [Deleted] (SPARK-8073) Directory traversal vulnerability

2015-06-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell deleted SPARK-8073: --- Directory traversal vulnerability - Key

[RESULT] [VOTE] Release Apache Spark 1.4.0 (RC3)

2015-06-02 Thread Patrick Wendell
randomSplit 9a88be1 [SPARK-6013] [ML] Add more Python ML examples for spark.ml 2bd4460 [SPARK-7954] [SPARKR] Create SparkContext in sparkRSQL init On Fri, May 29, 2015 at 4:40 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.0

[VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-02 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc3 (commit 22596c5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 22596c534a38cfdda91aef18aa9037ab101e4251 The release files, including signatures, digests, etc.

[jira] [Updated] (SPARK-8061) Document how to use hadoop provided builds

2015-06-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8061: --- Target Version/s: 1.4.1, 1.5.0 (was: 1.4.1) Document how to use hadoop provided builds

[jira] [Created] (SPARK-8061) Document how to use hadoop provided builds

2015-06-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8061: -- Summary: Document how to use hadoop provided builds Key: SPARK-8061 URL: https://issues.apache.org/jira/browse/SPARK-8061 Project: Spark Issue Type

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

2015-06-02 Thread Patrick Wendell
He all - a tiny nit from the last e-mail. The tag is v1.4.0-rc4. The exact commit and all other information is correct. (thanks Shivaram who pointed this out). On Tue, Jun 2, 2015 at 8:53 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache

[jira] [Resolved] (SPARK-8021) DataFrameReader/Writer in Python does not match Scala

2015-06-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-8021. Resolution: Fixed Fix Version/s: 1.4.0 Target Version/s: 1.4.0 (was: 1.4.1

[jira] [Commented] (SPARK-7988) Mechanism to control receiver scheduling

2015-06-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568613#comment-14568613 ] Patrick Wendell commented on SPARK-7988: Sure - that would provide more control

[jira] [Updated] (SPARK-8013) Get JDBC server working with Scala 2.11

2015-06-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8013: --- Target Version/s: 1.5.0 Get JDBC server working with Scala 2.11

[jira] [Updated] (SPARK-8013) Get JDBC server working with Scala 2.11

2015-06-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8013: --- Priority: Critical (was: Major) Get JDBC server working with Scala 2.11

[jira] [Created] (SPARK-8013) Get JDBC server working with Scala 2.11

2015-06-01 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-8013: -- Summary: Get JDBC server working with Scala 2.11 Key: SPARK-8013 URL: https://issues.apache.org/jira/browse/SPARK-8013 Project: Spark Issue Type: Sub

Re: [VOTE] Release Apache Spark 1.4.0 (RC3)

2015-06-01 Thread Patrick Wendell
:978) ... but maybe I missed the memo about how to build for Hive? do I still need another Hive profile? Other tests, signatures, etc look good. On Sat, May 30, 2015 at 12:40 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark

[jira] [Updated] (SPARK-8023) Random Number Generation inconsistent in projections in DataFrame

2015-06-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-8023: --- Target Version/s: 1.4.0 Random Number Generation inconsistent in projections in DataFrame

[jira] [Updated] (SPARK-7944) Spark-Shell 2.11 1.4.0-RC-03 does not add jars to class path

2015-05-31 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7944: --- Priority: Critical (was: Major) Spark-Shell 2.11 1.4.0-RC-03 does not add jars to class

[jira] [Commented] (SPARK-7987) TransportContext.createServer(int port) is missing in Spark 1.4

2015-05-31 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566860#comment-14566860 ] Patrick Wendell commented on SPARK-7987: Ah I see - since this is all bytecode

[jira] [Resolved] (SPARK-7987) TransportContext.createServer(int port) is missing in Spark 1.4

2015-05-31 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7987. Resolution: Not A Problem Closing as Not an Issue because this was a private module

[jira] [Created] (SPARK-7987) TransportContext.createServer(int port) is missing in Spark 1.4

2015-05-31 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-7987: -- Summary: TransportContext.createServer(int port) is missing in Spark 1.4 Key: SPARK-7987 URL: https://issues.apache.org/jira/browse/SPARK-7987 Project: Spark

[jira] [Closed] (SPARK-7959) Uneven distribution of receivers in the cluster

2015-05-31 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-7959. -- Resolution: Not A Problem The scheduling of receivers is not deterministic and based on Spark's

[RESULT] [VOTE] Release Apache Spark 1.4.0 (RC2)

2015-05-29 Thread Patrick Wendell
Thanks for all the discussion on the vote thread. I am canceling this vote in favor of RC3. On Sun, May 24, 2015 at 12:22 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc2 (commit

[VOTE] Release Apache Spark 1.4.0 (RC3)

2015-05-29 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc3 (commit dd109a8): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=dd109a8746ec07c7c83995890fc2c0cd7a693730 The release files, including signatures, digests, etc.

[jira] [Commented] (SPARK-7933) Patrick's username / password shouldn't be the defaults in the merge script

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564074#comment-14564074 ] Patrick Wendell commented on SPARK-7933: Thanks - this was a dummy password I

[jira] [Updated] (SPARK-7890) Document that Spark 2.11 now supports Kafka

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7890: --- Assignee: Sean Owen (was: Iulian Dragos) Document that Spark 2.11 now supports Kafka

[jira] [Resolved] (SPARK-7930) Shutdown hook deletes rool local dir before SparkContext is stopped, throwing errors

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7930. Resolution: Fixed Fix Version/s: 1.4.0 Shutdown hook deletes rool local dir before

[jira] [Commented] (SPARK-7890) Document that Spark 2.11 now supports Kafka

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564231#comment-14564231 ] Patrick Wendell commented on SPARK-7890: No - the JDBC component is not supported

[jira] [Resolved] (SPARK-7895) Move Kafka examples from scala-2.10/src to src

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7895. Resolution: Fixed Fix Version/s: 1.4.0 Move Kafka examples from scala-2.10/src

[jira] [Updated] (SPARK-7895) Move Kafka examples from scala-2.10/src to src

2015-05-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7895: --- Assignee: Shixiong Zhu Move Kafka examples from scala-2.10/src to src

[jira] [Resolved] (SPARK-7873) Serializer re-use + Kryo autoReset disabled leads to AraryIndexOutOfBounds exception in sort-shuffle bypassMergeSort path

2015-05-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7873. Resolution: Fixed Fix Version/s: 1.4.0 Serializer re-use + Kryo autoReset disabled

[jira] [Updated] (SPARK-6294) PySpark task may hang while call take() on in Java/Scala

2015-05-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6294: --- Fix Version/s: 1.3.1 1.4.0 PySpark task may hang while call take

Re: [VOTE] Release Apache Spark 1.4.0 (RC2)

2015-05-27 Thread Patrick Wendell
Hi James, As I said before that is not a blocker issue for this release, thanks. Separately, there are some comments in this code review that indicate you may be facing a bug in your own code rather than with Spark: https://github.com/apache/spark/pull/5688#issuecomment-104491410 Please follow

[jira] [Resolved] (SPARK-7896) IndexOutOfBoundsException in ChainedBuffer

2015-05-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7896. Resolution: Fixed Fix Version/s: 1.4.0 IndexOutOfBoundsException in ChainedBuffer

[jira] [Created] (SPARK-7890) Document that Spark 2.11 now supports Kafka

2015-05-27 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-7890: -- Summary: Document that Spark 2.11 now supports Kafka Key: SPARK-7890 URL: https://issues.apache.org/jira/browse/SPARK-7890 Project: Spark Issue Type

[jira] [Reopened] (SPARK-7042) Spark version of akka-actor_2.11 is not compatible with the official akka-actor_2.11 2.3.x

2015-05-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-7042: Hey [~srowen] and [~Abk1024], I had to revert this because it broke the scala 2.11 build. I

[jira] [Updated] (SPARK-7845) Bump Hadoop 1 tests to version 1.2.0

2015-05-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7845: --- Description: A small number of API's in Hadoop were added between 1.0.4 and 1.2.0

[jira] [Updated] (SPARK-7845) Bump Hadoop 1 tests to version 1.2.0

2015-05-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7845: --- Assignee: Yin Huai Bump Hadoop 1 tests to version 1.2.0

Re: [VOTE] Release Apache Spark 1.4.0 (RC1)

2015-05-24 Thread Patrick Wendell
Hey jameszhouyi, Since SPARK-7119 is not a regression from earlier versions, we won't hold the release for it. However, please comment on the JIRA if it is affecting you... it will help us prioritize the bug. - Patrick On Fri, May 22, 2015 at 8:41 PM, jameszhouyi yiaz...@gmail.com wrote: We

[RESULT] [VOTE] Release Apache Spark 1.4.0 (RC1)

2015-05-24 Thread Patrick Wendell
This vote is cancelled in favor of RC2. On Tue, May 19, 2015 at 9:10 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc1 (commit 777a081): https://git-wip-us.apache.org/repos/asf?p

[VOTE] Release Apache Spark 1.4.0 (RC2)

2015-05-24 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.0! The tag to be voted on is v1.4.0-rc2 (commit 03fb26a3): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=03fb26a3e50e00739cc815ba4e2e82d71d003168 The release files, including signatures, digests, etc.

[ANNOUNCE] Nightly maven and package builds for Spark

2015-05-24 Thread Patrick Wendell
Hi All, This week I got around to setting up nightly builds for Spark on Jenkins. I'd like feedback on these and if it's going well I can merge the relevant automation scripts into Spark mainline and document it on the website. Right now I'm doing: 1. SNAPSHOT's of Spark master and release

[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.

2015-05-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557675#comment-14557675 ] Patrick Wendell commented on SPARK-6907: Hey [~ste...@apache.org] - my guess

Re: [IMPORTANT] Committers please update merge script

2015-05-23 Thread Patrick Wendell
, Patrick Wendell pwend...@gmail.com wrote: Hi All - unfortunately the fix introduced another bug, which is that fixVersion was not updated properly. I've updated the script and had one other person test it. So committers please pull from master again thanks! - Patrick On Tue, May 12, 2015

<    1   2   3   4   5   6   7   8   9   10   >