[jira] [Updated] (SPARK-5297) File Streams do not work with custom key/values

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5297: --- Assignee: Saisai Shao File Streams do not work with custom key/values

[jira] [Updated] (SPARK-4996) Memory leak?

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4996: --- Priority: Major (was: Blocker) Memory leak? Key: SPARK-4996

[jira] [Commented] (SPARK-4996) Memory leak?

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285255#comment-14285255 ] Patrick Wendell commented on SPARK-4996: I'm de-escalating this right now because

[jira] [Updated] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4959: --- Target Version/s: 1.3.0, 1.2.1 (was: 1.3.0) Attributes are case sensitive when using

[jira] [Updated] (SPARK-4296) Throw Expression not in GROUP BY when using same expression in group by clause and select clause

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4296: --- Fix Version/s: 1.2.1 1.3.0 Throw Expression not in GROUP BY when using

[jira] [Resolved] (SPARK-5275) pyspark.streaming is not included in assembly jar

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5275. Resolution: Fixed Assignee: Davies Liu pyspark.streaming is not included in assembly

[jira] [Issue Comment Deleted] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4959: --- Comment: was deleted (was: Note that in the 1.2 branch this was fixed by https://github.com

[jira] [Commented] (SPARK-4296) Throw Expression not in GROUP BY when using same expression in group by clause and select clause

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285263#comment-14285263 ] Patrick Wendell commented on SPARK-4296: Note this was fixed in https://github.com

[jira] [Updated] (SPARK-4660) JavaSerializer uses wrong classloader

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4660: --- Assignee: Piotr Kołaczkowski JavaSerializer uses wrong classloader

[jira] [Resolved] (SPARK-4660) JavaSerializer uses wrong classloader

2015-01-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4660. Resolution: Fixed Fix Version/s: 1.2.1 1.1.2

[jira] [Updated] (SPARK-5297) File Streams do not work with custom key/values

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5297: --- Description: The following code: {code} stream_context.K,V,SequenceFileInputFormatK

[jira] [Updated] (SPARK-5270) Provide isEmpty utility function in RDD API

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5270: --- Summary: Provide isEmpty utility function in RDD API (was: Elegantly check if RDD is empty

[jira] [Updated] (SPARK-5270) Provide isEmpty() function in RDD API

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5270: --- Summary: Provide isEmpty() function in RDD API (was: Provide isEmpty utility function in RDD

[jira] [Updated] (SPARK-5270) Provide isEmpty utility function in RDD API

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5270: --- Assignee: Sean Owen Provide isEmpty utility function in RDD API

[jira] [Resolved] (SPARK-5270) Provide isEmpty() function in RDD API

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5270. Resolution: Fixed Fix Version/s: 1.3.0 Provide isEmpty() function in RDD API

[jira] [Resolved] (SPARK-2595) The driver run garbage collection, when the executor throws OutOfMemoryError exception

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2595. Resolution: Won't Fix Per PR comment, closing this for now. The driver run garbage

[jira] [Resolved] (SPARK-5088) Use spark-class for running executors directly on mesos

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5088. Resolution: Fixed Assignee: Jongyoul Lee Use spark-class for running executors

[jira] [Resolved] (SPARK-5217) Spark UI should report pending stages during job execution on AllStagesPage.

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5217. Resolution: Fixed Fix Version/s: 1.3.0 Spark UI should report pending stages during

[jira] [Resolved] (SPARK-4417) New API: sample RDD to fixed number of items

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4417. Resolution: Won't Fix Assignee: Ilya Ganelin [~ilganeli] ended up taking a crack

[jira] [Resolved] (SPARK-3758) Script style checking

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3758. Resolution: Won't Fix This patch ended up being so large, I think we're gonna pass

[jira] [Resolved] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3288. Resolution: Fixed Fix Version/s: 1.3.0 Target Version/s: 1.3.0 (was: 1.2.0

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2015-01-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3288: --- Assignee: Ilya Ganelin (was: Dale Richardson) All fields in TaskMetrics should be private

Re: Semantics of LGTM

2015-01-19 Thread Patrick Wendell
The wiki does not seem to be operational ATM, but I will do this when it is back up. On Mon, Jan 19, 2015 at 12:00 PM, Patrick Wendell pwend...@gmail.com wrote: Okay - so given all this I was going to put the following on the wiki tentatively: ## Reviewing Code Community code review

Re: Semantics of LGTM

2015-01-19 Thread Patrick Wendell
the latter unless qualified in some other way. I don't have any opinion on the specific characters, but I agree with Aaron that it would be nice to have some sort of abbreviation for both the strong and weak forms of approval. -Sandy On Jan 17, 2015, at 7:25 PM, Patrick Wendell

[jira] [Updated] (SPARK-5249) In SparkConf accept value with Any type and perform string conversion

2015-01-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5249: --- Summary: In SparkConf accept value with Any type and perform string conversion (was: Added

[jira] [Commented] (SPARK-5249) In SparkConf accept value with Any type and perform string conversion

2015-01-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282062#comment-14282062 ] Patrick Wendell commented on SPARK-5249: I also updated the title to reflect what

[jira] [Resolved] (SPARK-5249) Added setX functions to set a Boolean, Int, Float and Double parameters with a specialized function.

2015-01-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5249. Resolution: Won't Fix Per discussion on the issue we've decided to just ask users

Re: Bouncing Mails

2015-01-17 Thread Patrick Wendell
Akhil, Those are handled by ASF infrastructure, not anyone in the Spark project. So this list is not the appropriate place to ask for help. - Patrick On Sat, Jan 17, 2015 at 12:56 AM, Akhil Das ak...@sigmoidanalytics.com wrote: My mails to the mailing list are getting rejected, have opened a

Re: Bouncing Mails

2015-01-17 Thread Patrick Wendell
Akhil, Those are handled by ASF infrastructure, not anyone in the Spark project. So this list is not the appropriate place to ask for help. - Patrick On Sat, Jan 17, 2015 at 12:56 AM, Akhil Das ak...@sigmoidanalytics.com wrote: My mails to the mailing list are getting rejected, have opened a

[jira] [Resolved] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2015-01-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3694. Resolution: Duplicate Allow printing object graph of tasks/RDD's with a debug flag

[jira] [Resolved] (SPARK-5096) SparkBuild.scala assumes you are at the spark root dir

2015-01-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5096. Resolution: Fixed Fix Version/s: 1.3.0 SparkBuild.scala assumes you

[jira] [Updated] (SPARK-5096) SparkBuild.scala assumes you are at the spark root dir

2015-01-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5096: --- Target Version/s: (was: 1.0.3) SparkBuild.scala assumes you are at the spark root dir

[jira] [Resolved] (SPARK-5289) Backport publishing of repl, yarn into branch-1.2

2015-01-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5289. Resolution: Fixed Backport publishing of repl, yarn into branch-1.2

Semantics of LGTM

2015-01-17 Thread Patrick Wendell
Hey All, Just wanted to ping about a minor issue - but one that ends up having consequence given Spark's volume of reviews and commits. As much as possible, I think that we should try and gear towards Google Style LGTM on reviews. What I mean by this is that LGTM has the following semantics: I

[jira] [Resolved] (SPARK-4357) Modify release publishing to work with Scala 2.11

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4357. Resolution: Fixed Sorry this is actually working now. We now publish artifacts for Scala

[jira] [Updated] (SPARK-5270) Elegantly check if RDD is empty

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5270: --- Target Version/s: 1.3.0 Elegantly check if RDD is empty

[jira] [Updated] (SPARK-5260) Expose JsonRDD.allKeysWithValueTypes() in a utility class

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5260: --- Fix Version/s: (was: 1.3.0) Expose JsonRDD.allKeysWithValueTypes() in a utility class

[jira] [Created] (SPARK-5289) Backport publishing of repl, yarn, and hive-thriftserver into branch-1.2

2015-01-16 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5289: -- Summary: Backport publishing of repl, yarn, and hive-thriftserver into branch-1.2 Key: SPARK-5289 URL: https://issues.apache.org/jira/browse/SPARK-5289 Project

[jira] [Updated] (SPARK-5289) Backport publishing of repl, yarn into branch-1.2

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5289: --- Summary: Backport publishing of repl, yarn into branch-1.2 (was: Backport publishing of repl

[jira] [Updated] (SPARK-5289) Backport publishing of repl, yarn into branch-1.2

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5289: --- Description: In SPARK-3452 we did some clean-up of published artifacts that turned out

[jira] [Updated] (SPARK-5289) Backport publishing of repl, yarn into branch-1.2

2015-01-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5289: --- Description: In SPARK-3452 we did some clean-up of published artifacts that turned out

[jira] [Resolved] (SPARK-4857) Add Executor Events to SparkListener

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4857. Resolution: Fixed Fix Version/s: 1.3.0 Add Executor Events to SparkListener

[jira] [Resolved] (SPARK-2630) Input data size of CoalescedRDD is incorrect

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2630. Resolution: Duplicate I think this is a dup of SPARK-4092. Input data size

[jira] [Updated] (SPARK-4955) Dynamic allocation doesn't work in YARN cluster mode

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4955: --- Priority: Blocker (was: Critical) Dynamic allocation doesn't work in YARN cluster mode

[jira] [Updated] (SPARK-4955) Dynamic allocation doesn't work in YARN cluster mode

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4955: --- Target Version/s: 1.3.0 Dynamic allocation doesn't work in YARN cluster mode

[jira] [Commented] (SPARK-5216) Spark Ui should report estimated time remaining for each stage.

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279863#comment-14279863 ] Patrick Wendell commented on SPARK-5216: This has been proposed before

[jira] [Updated] (SPARK-5176) Thrift server fails with confusing error message when deploy-mode is cluster

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5176: --- Labels: starter (was: ) Thrift server fails with confusing error message when deploy-mode

[jira] [Commented] (SPARK-5176) Thrift server fails with confusing error message when deploy-mode is cluster

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279869#comment-14279869 ] Patrick Wendell commented on SPARK-5176: Yes, we should add a check here similar

[jira] [Comment Edited] (SPARK-5176) Thrift server fails with confusing error message when deploy-mode is cluster

2015-01-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279869#comment-14279869 ] Patrick Wendell edited comment on SPARK-5176 at 1/16/15 6:28 AM

Re: Accumulator value in Spark UI

2015-01-14 Thread Patrick Wendell
It should appear in the page for any stage in which accumulators are updated. On Wed, Jan 14, 2015 at 6:46 PM, Justin Yip yipjus...@prediction.io wrote: Hello, From accumulator documentation, it says that if the accumulator is named, it will be displayed in the WebUI. However, I cannot find

[jira] [Resolved] (SPARK-5078) Allow setting Akka host name from env vars

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5078. Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Allow setting Akka

[jira] [Resolved] (SPARK-5102) CompressedMapStatus needs to be registered with Kryo

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5102. Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Fixed by: https

[jira] [Resolved] (SPARK-5172) spark-examples-***.jar shades a wrong Hadoop distribution

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5172. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Sean Owen spark-examples

[jira] [Commented] (SPARK-4923) Maven build should keep publishing spark-repl

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274239#comment-14274239 ] Patrick Wendell commented on SPARK-4923: Hey All, Sorry this has caused

[jira] [Comment Edited] (SPARK-4923) Maven build should keep publishing spark-repl

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274239#comment-14274239 ] Patrick Wendell edited comment on SPARK-4923 at 1/12/15 9:58 PM

[jira] [Commented] (SPARK-4923) Maven build should keep publishing spark-repl

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274263#comment-14274263 ] Patrick Wendell commented on SPARK-4923: [~senkwich] definitely prefer github

[jira] [Updated] (SPARK-5102) CompressedMapStatus needs to be registered with Kryo

2015-01-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5102: --- Target Version/s: 1.2.1 Assignee: Lianhui Wang CompressedMapStatus needs

[jira] [Updated] (SPARK-3340) Deprecate ADD_JARS and ADD_FILES

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3340: --- Labels: starter (was: ) Deprecate ADD_JARS and ADD_FILES

[jira] [Resolved] (SPARK-3450) Enable specifiying the --jars CLI option multiple times

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3450. Resolution: Won't Fix I'd prefer not to do this one, it complicates our parsing

Re: Job priority

2015-01-11 Thread Patrick Wendell
Priority scheduling isn't something we've supported in Spark and we've opted to support FIFO and Fair scheduling and asked users to try and fit these to the needs of their applications. In practice from what I've seen of priority schedulers, such as the linux CPU scheduler, is that strict

[jira] [Commented] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273178#comment-14273178 ] Patrick Wendell commented on SPARK-1422: Good call NIck - yeah let's close

[jira] [Resolved] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1422. Resolution: Won't Fix Add scripts for launching Spark on Google Compute Engine

[jira] [Resolved] (SPARK-4399) Support multiple cloud providers

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4399. Resolution: Won't Fix We'll let the community take this one on. Support multiple cloud

[jira] [Updated] (SPARK-5166) Stabilize Spark SQL APIs

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5166: --- Priority: Blocker (was: Critical) Stabilize Spark SQL APIs

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2015-01-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273225#comment-14273225 ] Patrick Wendell commented on SPARK-3561: So if the question is: Is Spark only API

[jira] [Resolved] (SPARK-5032) MimaExcludes should not exclude GraphX

2015-01-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5032. Resolution: Fixed Fix Version/s: 1.3.0 MimaExcludes should not exclude GraphX

[jira] [Commented] (SPARK-4737) Prevent serialization errors from ever crashing the DAG scheduler

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272214#comment-14272214 ] Patrick Wendell commented on SPARK-4737: It's great to see this go in. Thanks

[jira] [Resolved] (SPARK-1143) ClusterSchedulerSuite (soon to be TaskSchedulerImplSuite) does not actually test the ClusterScheduler/TaskSchedulerImpl

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1143. Resolution: Fixed Assignee: Kay Ousterhout (was: Nan Zhu) ClusterSchedulerSuite

[jira] [Resolved] (SPARK-5163) Load properties from configuration file for example spark-defaults.conf when creating SparkConf object

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5163. Resolution: Won't Fix I'd prefer not to accept this patch for now - the spark-defaults.conf

[jira] [Updated] (SPARK-5073) spark.storage.memoryMapThreshold has two default values

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5073: --- Summary: spark.storage.memoryMapThreshold has two default values

[jira] [Resolved] (SPARK-5136) Improve documentation around setting up Spark IntelliJ project

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5136. Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Improve

[jira] [Updated] (SPARK-5073) spark.storage.memoryMapThreshold has two default value

2015-01-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5073: --- Summary: spark.storage.memoryMapThreshold has two default value

[jira] [Resolved] (SPARK-4048) Enhance and extend hadoop-provided profile

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4048. Resolution: Fixed Fix Version/s: 1.3.0 Enhance and extend hadoop-provided profile

[jira] [Comment Edited] (SPARK-5152) Let metrics.properties file take an hdfs:// path

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270616#comment-14270616 ] Patrick Wendell edited comment on SPARK-5152 at 1/9/15 6:19 AM

Re: Spark development with IntelliJ

2015-01-08 Thread Patrick Wendell
Nick - yes. Do you mind moving it? I should have put it in the Contributing to Spark page. On Thu, Jan 8, 2015 at 3:22 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Side question: Should this section

Re: Spark development with IntelliJ

2015-01-08 Thread Patrick Wendell
Actually I went ahead and did it. On Thu, Jan 8, 2015 at 10:25 PM, Patrick Wendell pwend...@gmail.com wrote: Nick - yes. Do you mind moving it? I should have put it in the Contributing to Spark page. On Thu, Jan 8, 2015 at 3:22 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Side

[jira] [Commented] (SPARK-5152) Let metrics.properties file take an hdfs:// path

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270616#comment-14270616 ] Patrick Wendell commented on SPARK-5152: Should we be loading the metrics

[jira] [Updated] (SPARK-2620) case class cannot be used as key for reduce

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2620: --- Assignee: Tobias Schlatter case class cannot be used as key for reduce

[jira] [Commented] (SPARK-5136) Improve documentation around setting up Spark IntelliJ project

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270626#comment-14270626 ] Patrick Wendell commented on SPARK-5136: I've updated it to be in the new location

[jira] [Commented] (SPARK-5136) Improve documentation around setting up Spark IntelliJ project

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270624#comment-14270624 ] Patrick Wendell commented on SPARK-5136: Hey Guys, I wrote that on the wiki quite

[jira] [Created] (SPARK-5158) Allow for keytab-based HDFS security in Standalone mode

2015-01-08 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5158: -- Summary: Allow for keytab-based HDFS security in Standalone mode Key: SPARK-5158 URL: https://issues.apache.org/jira/browse/SPARK-5158 Project: Spark

[jira] [Updated] (SPARK-5158) Allow for keytab-based HDFS security in Standalone mode

2015-01-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5158: --- Description: There have been a handful of patches for allowing access to Kerberized HDFS

[jira] [Updated] (SPARK-5097) Adding data frame APIs to SchemaRDD

2015-01-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5097: --- Priority: Critical (was: Major) Adding data frame APIs to SchemaRDD

[jira] [Commented] (SPARK-1529) Support setting spark.local.dirs to a hadoop FileSystem

2015-01-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267419#comment-14267419 ] Patrick Wendell commented on SPARK-1529: Hey Sean, From what I remember

[jira] [Commented] (SPARK-1529) Support setting spark.local.dirs to a hadoop FileSystem

2015-01-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267424#comment-14267424 ] Patrick Wendell commented on SPARK-1529: BTW - I think if MapR wants to have

[jira] [Created] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

2015-01-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5113: -- Summary: Audit and document use of hostnames and IP addresses in Spark Key: SPARK-5113 URL: https://issues.apache.org/jira/browse/SPARK-5113 Project: Spark

[jira] [Updated] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

2015-01-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5113: --- Description: Spark has multiple network components that start servers and advertise

[jira] [Updated] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

2015-01-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5113: --- Description: Spark has multiple network components that start servers and advertise

[jira] [Updated] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

2015-01-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5113: --- Description: Spark has multiple network components that start servers and advertise

[jira] [Updated] (SPARK-5113) Audit and document use of hostnames and IP addresses in Spark

2015-01-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5113: --- Description: Spark has multiple network components that start servers and advertise

[jira] [Updated] (SPARK-4737) Prevent serialization errors from ever crashing the DAG scheduler

2015-01-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4737: --- Affects Version/s: 1.0.2 1.1.1 Prevent serialization errors from ever

[jira] [Commented] (SPARK-4687) SparkContext#addFile doesn't keep file folder information

2015-01-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265416#comment-14265416 ] Patrick Wendell commented on SPARK-4687: I spent some more time looking

Re: Spark UI history job duration is wrong

2015-01-05 Thread Patrick Wendell
Thanks for reporting this - it definitely sounds like a bug. Please open a JIRA for it. My guess is that we define the start or end time of the job based on the current time instead of looking at data encoded in the underlying event stream. That would cause it to not work properly when loading

Re: Spark driver main thread hanging after SQL insert

2015-01-02 Thread Patrick Wendell
Hi Alessandro, Can you create a JIRA for this rather than reporting it on the dev list? That's where we track issues like this. Thanks!. - Patrick On Wed, Dec 31, 2014 at 8:48 PM, Alessandro Baretta alexbare...@gmail.com wrote: Here's what the console shows: 15/01/01 01:12:29 INFO

[jira] [Updated] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2014-12-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5008: --- Component/s: EC2 Persistent HDFS does not recognize EBS Volumes

[jira] [Updated] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2014-12-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5008: --- Labels: (was: amazon aws ec2 hdfs persistent) Persistent HDFS does not recognize EBS

[jira] [Created] (SPARK-5025) Write a guide for creating well-formed packages for Spark

2014-12-30 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5025: -- Summary: Write a guide for creating well-formed packages for Spark Key: SPARK-5025 URL: https://issues.apache.org/jira/browse/SPARK-5025 Project: Spark

[jira] [Updated] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4908: --- Target Version/s: 1.2.1 Spark SQL built for Hive 13 fails under concurrent metadata queries

Re: Long-running job cleanup

2014-12-28 Thread Patrick Wendell
What do you mean when you say the overhead of spark shuffles start to accumulate? Could you elaborate more? In newer versions of Spark shuffle data is cleaned up automatically when an RDD goes out of scope. It is safe to remove shuffle data at this point because the RDD can no longer be

Re: action progress in ipython notebook?

2014-12-28 Thread Patrick Wendell
Hey Eric, I'm just curious - which specific features in 1.2 do you find most help with usability? This is a theme we're focusing on for 1.3 as well, so it's helpful to hear what makes a difference. - Patrick On Sun, Dec 28, 2014 at 1:36 AM, Eric Friedman eric.d.fried...@gmail.com wrote: Hi

<    7   8   9   10   11   12   13   14   15   16   >