[jira] [Reopened] (SPARK-5213) Pluggable SQL Parser Support

2015-04-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-5213: I believe this introduced a build break so I reopened it https://amplab.cs.berkeley.edu

Re: Mima test failure in the master branch?

2015-04-30 Thread Patrick Wendell
I reverted the patch that I think was causing this: SPARK-5213 Thanks On Thu, Apr 30, 2015 at 7:59 PM, zhazhan zzh...@hortonworks.com wrote: Any PR open for this? -- View this message in context:

[jira] [Reopened] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

2015-04-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-5342: I had to revert this patch because it broke old Hadoop versions https://amplab.cs.berkeley.edu

[jira] [Resolved] (SPARK-7205) Support local ivy cache in --packages

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7205. Resolution: Fixed Assignee: Burak Yavuz Support local ivy cache in --packages

[jira] [Updated] (SPARK-7213) Exception while copying Hadoop config files due to permission issues

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7213: --- Component/s: YARN Exception while copying Hadoop config files due to permission issues

[jira] [Updated] (SPARK-7224) Mock repositories for testing with --packages

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7224: --- Assignee: Burak Yavuz Mock repositories for testing with --packages

[jira] [Commented] (SPARK-7230) Make RDD API private in SparkR for Spark 1.4

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520058#comment-14520058 ] Patrick Wendell commented on SPARK-7230: I think this is a good idea. We should

[jira] [Updated] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7233: --- Priority: Critical (was: Major) ClosureCleaner#clean blocks concurrent job submitter

[jira] [Commented] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520202#comment-14520202 ] Patrick Wendell commented on SPARK-7233: Thanks this is a great find. As a simple

[jira] [Created] (SPARK-7260) Support changing Spark's log level programatically

2015-04-29 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-7260: -- Summary: Support changing Spark's log level programatically Key: SPARK-7260 URL: https://issues.apache.org/jira/browse/SPARK-7260 Project: Spark Issue

[jira] [Created] (SPARK-7261) Change default log level to WARN in the REPL

2015-04-29 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-7261: -- Summary: Change default log level to WARN in the REPL Key: SPARK-7261 URL: https://issues.apache.org/jira/browse/SPARK-7261 Project: Spark Issue Type

[jira] [Updated] (SPARK-7237) Many user provided closures are not actually cleaned

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7237: --- Component/s: Spark Core Many user provided closures are not actually cleaned

[jira] [Commented] (SPARK-7233) ClosureCleaner#clean blocks concurrent job submitter threads

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520368#comment-14520368 ] Patrick Wendell commented on SPARK-7233: I think we should just create a lazy val

[jira] [Commented] (SPARK-7261) Change default log level to WARN in the REPL

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520467#comment-14520467 ] Patrick Wendell commented on SPARK-7261: Yeah, but SPARK-7260 is super simple, I

[jira] [Commented] (SPARK-7230) Make RDD API private in SparkR for Spark 1.4

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520478#comment-14520478 ] Patrick Wendell commented on SPARK-7230: Yeah the goal is absolutely to support

[jira] [Updated] (SPARK-7228) SparkR public API for 1.4 release

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7228: --- Priority: Blocker (was: Critical) SparkR public API for 1.4 release

[jira] [Comment Edited] (SPARK-7230) Make RDD API private in SparkR for Spark 1.4

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520635#comment-14520635 ] Patrick Wendell edited comment on SPARK-7230 at 4/30/15 1:13 AM

[jira] [Commented] (SPARK-7230) Make RDD API private in SparkR for Spark 1.4

2015-04-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520635#comment-14520635 ] Patrick Wendell commented on SPARK-7230: Yes - removing API's is really difficult

[jira] [Created] (SPARK-7204) Call sites in UI are not accurate for DataFrame operations

2015-04-28 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-7204: -- Summary: Call sites in UI are not accurate for DataFrame operations Key: SPARK-7204 URL: https://issues.apache.org/jira/browse/SPARK-7204 Project: Spark

[jira] [Commented] (SPARK-7180) SerializationDebugger fails with ArrayOutOfBoundsException

2015-04-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516536#comment-14516536 ] Patrick Wendell commented on SPARK-7180: /cc [~rxin] SerializationDebugger fails

[jira] [Commented] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514862#comment-14514862 ] Patrick Wendell commented on SPARK-7108: I see - when I wrote that, I meant

[jira] [Commented] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514822#comment-14514822 ] Patrick Wendell commented on SPARK-7108: Yes - the case I'm referring

[jira] [Comment Edited] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514822#comment-14514822 ] Patrick Wendell edited comment on SPARK-7108 at 4/27/15 7:50 PM

[jira] [Commented] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514865#comment-14514865 ] Patrick Wendell commented on SPARK-7108: Actually to correct what I said

[jira] [Commented] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514403#comment-14514403 ] Patrick Wendell commented on SPARK-7108: Hey Marcelo, Your interpretation

[jira] [Comment Edited] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14514403#comment-14514403 ] Patrick Wendell edited comment on SPARK-7108 at 4/27/15 4:33 PM

[jira] [Resolved] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-04-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4925. Resolution: Fixed Fix Version/s: 1.3.1 1.4.0 Publish Spark SQL

Re: WebUI shows poor locality when task scheduling

2015-04-26 Thread Patrick Wendell
Hi Eric - please direct this to the user@ list. This list is for development of Spark itself. On Sun, Apr 26, 2015 at 1:12 AM, eric wong win19...@gmail.com wrote: Hi developers, I have sent to user mail list but no response... When running a exprimental KMeans job for expriment, the

Re: Design docs: consolidation and discoverability

2015-04-26 Thread Patrick Wendell
are on Google Docs. Perhaps Apache should consider opening up parts of the wiki to a larger group, to better serve this use case. Punya On Fri, Apr 24, 2015 at 5:01 PM Patrick Wendell pwend...@gmail.com wrote: Using our ASF git repository as a working area for design docs, it seems potentially

Re: Spark timeout issue

2015-04-26 Thread Patrick Wendell
Hi Deepak - please direct this to the user@ list. This list is for development of Spark itself. On Sun, Apr 26, 2015 at 12:42 PM, Deepak Gopalakrishnan dgk...@gmail.com wrote: Hello All, I'm trying to process a 3.5GB file on standalone mode using spark. I could run my spark job succesfully on

[jira] [Updated] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2015-04-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7108: --- Summary: spark.local.dir is no longer honored in Standalone mode (was: Setting

Reminder about Spark 1.4.0 deadline of May 1st

2015-04-25 Thread Patrick Wendell
Hey All, Just a friendly reminder that May 1st is the feature freeze for Spark 1.4, meaning major outstanding changes will need to land in the next week. After May 1st we'll package a release for testing and then go into the normal triage process where bugs are prioritized and some smaller

[jira] [Reopened] (SPARK-6752) Allow StreamingContext to be recreated from checkpoint and existing SparkContext

2015-04-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-6752: I had to revert this because it caused test failures with the Hadoop 1.0 build. To reproduce

Re: Contributing Documentation Changes

2015-04-25 Thread Patrick Wendell
It is true that in the past we've posted community tutorials on the site. Spark has grown a lot since then and it might be a better fit at this point to curate community tutorials on the wiki (something like the powered by page) and link to them from the documentation website. The documentation

[jira] [Comment Edited] (SPARK-7108) Setting spark.local.dir in driver no longer overrides the standalone worker's local directory setting

2015-04-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512280#comment-14512280 ] Patrick Wendell edited comment on SPARK-7108 at 4/25/15 6:01 AM

[jira] [Comment Edited] (SPARK-7108) Setting spark.local.dir in driver no longer overrides the standalone worker's local directory setting

2015-04-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512280#comment-14512280 ] Patrick Wendell edited comment on SPARK-7108 at 4/25/15 6:02 AM

Re: Should we let everyone set Assignee?

2015-04-24 Thread Patrick Wendell
It's a bit of a digression - but Steve's suggestion that we have a mailing list for new issues is a great idea and we can do it easily. We could nave new-issues@s.a.o or something (we already have issues@s.a.o). - Patrick On Fri, Apr 24, 2015 at 9:50 AM, Ted Yu yuzhih...@gmail.com wrote: bq.

[jira] [Updated] (SPARK-7123) support table.star in sqlcontext

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7123: --- Issue Type: New Feature (was: Bug) support table.star in sqlcontext

[jira] [Commented] (SPARK-6961) Cannot save data to parquet files when executing from Windows from a Maven Project

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512272#comment-14512272 ] Patrick Wendell commented on SPARK-6961: Hey [~bogdannb] - so Spark actually uses

[jira] [Commented] (SPARK-7108) Setting spark.local.dir in driver no longer overrides the standalone worker's local directory setting

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512280#comment-14512280 ] Patrick Wendell commented on SPARK-7108: Hey I think [~joshrosen] actually

[jira] [Commented] (SPARK-7103) SparkContext.union crashed when some RDDs have no partitioner

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511689#comment-14511689 ] Patrick Wendell commented on SPARK-7103: Escalated the priority since IMO

[jira] [Commented] (SPARK-7108) Setting spark.local.dir in driver no longer overrides the standalone worker's local directory setting

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511687#comment-14511687 ] Patrick Wendell commented on SPARK-7108: Ping [~vanzin] who authored SPARK-4834

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Patrick Wendell
Using our ASF git repository as a working area for design docs, it seems potentially concerning to me. It's difficult process wise because all commits need to go through committers and also, we'd pollute our git history a lot with random incremental design updates. The git history is used a lot

[jira] [Updated] (SPARK-7120) ClosureCleaner lacks documentation

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7120: --- Issue Type: Improvement (was: Bug) ClosureCleaner lacks documentation

[jira] [Updated] (SPARK-7103) SparkContext.union crashed when some RDDs have no partitioner

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7103: --- Target Version/s: 1.3.2, 1.4.0 SparkContext.union crashed when some RDDs have no partitioner

[jira] [Updated] (SPARK-7103) SparkContext.union crashed when some RDDs have no partitioner

2015-04-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7103: --- Priority: Critical (was: Minor) SparkContext.union crashed when some RDDs have

[jira] [Updated] (SPARK-6921) Spark SQL API saveAsParquetFile will output tachyon file with different block size

2015-04-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6921: --- Priority: Critical (was: Blocker) Spark SQL API saveAsParquetFile will output tachyon file

Re: Should we let everyone set Assignee?

2015-04-22 Thread Patrick Wendell
One over arching issue is that it's pretty unclear what Assigned to X in JIAR means from a process perspective. Personally I actually feel it's better for this to be more historical - i.e. who ended up submitting a patch for this feature that was merged - rather than creating an exclusive

[jira] [Updated] (SPARK-7064) Adding binary sparse vector support

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7064: --- Component/s: ML Adding binary sparse vector support

[jira] [Updated] (SPARK-7065) Clear the cached locations mapping after every stage to avoid inconsistent status

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7065: --- Component/s: Spark Core Clear the cached locations mapping after every stage to avoid

[jira] [Commented] (SPARK-7065) Clear the cached locations mapping after every stage to avoid inconsistent status

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508433#comment-14508433 ] Patrick Wendell commented on SPARK-7065: It would be helpful to have a bit more

[jira] [Resolved] (SPARK-7054) Spark jobs hang for ~15 mins when a node goes down

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7054. Resolution: Invalid Hey There, Please send this to the Spark users list to get feedback

[jira] [Updated] (SPARK-7063) when lz4 compression is used, it causes core dump

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7063: --- Environment: IBM JDK when lz4 compression is used, it causes core dump

[jira] [Resolved] (SPARK-7062) Parquet compression does not work for Spark SQL loading

2015-04-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7062. Resolution: Duplicate Parquet compression does not work for Spark SQL loading

Re: Should we let everyone set Assignee?

2015-04-22 Thread Patrick Wendell
: Agreed. The Spark project and community that Vinod describes do not resemble the ones with which I am familiar. On Wed, Apr 22, 2015 at 1:20 PM, Patrick Wendell pwend...@gmail.com wrote: Hi Vinod, Thanks for you thoughts - However, I do not agree with your sentiment

Re: Should we let everyone set Assignee?

2015-04-22 Thread Patrick Wendell
at Apache. +Vinod On Apr 22, 2015, at 12:32 PM, Patrick Wendell pwend...@gmail.com wrote: One over arching issue is that it's pretty unclear what Assigned to X in JIAR means from a process perspective. Personally I actually feel it's better for this to be more historical - i.e. who ended up

[jira] [Resolved] (SPARK-5081) Shuffle write increases

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5081. Resolution: Duplicate I'm pretty sure this is fixed via SPARK-6905. Closing this and we can

[jira] [Comment Edited] (SPARK-5081) Shuffle write increases

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504442#comment-14504442 ] Patrick Wendell edited comment on SPARK-5081 at 4/21/15 6:36 AM

[jira] [Resolved] (SPARK-2044) Pluggable interface for shuffles

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2044. Resolution: Fixed Fix Version/s: 1.1.2 Pluggable interface for shuffles

[jira] [Updated] (SPARK-2044) Pluggable interface for shuffles

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2044: --- Fix Version/s: (was: 1.1.2) 1.1.0 Pluggable interface for shuffles

[jira] [Updated] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6980: --- Assignee: Harsh Gupta Akka timeout exceptions indicate which conf controls them

[jira] [Commented] (SPARK-6980) Akka timeout exceptions indicate which conf controls them

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505404#comment-14505404 ] Patrick Wendell commented on SPARK-6980: [~harshg] I just assigned you Akka

Re: Spark 1.2.2 prebuilt release for Hadoop 2.4 didn't get deployed

2015-04-21 Thread Patrick Wendell
Good catch Olivier - I'll take care of it. Tracking this on SPARK-7027. On Tue, Apr 21, 2015 at 6:06 AM, Olivier Girardot o.girar...@lateral-thoughts.com wrote: Hi everyone, It seems the some of the Spark 1.2.2 prebuilt versions (I tested mainly for Hadoop 2.4 and later) didn't get deploy on

[jira] [Updated] (SPARK-7027) Spark 1.2.2 Hadoop 2.4 download is missing

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7027: --- Summary: Spark 1.2.2 Hadoop 2.4 download is missing (was: Spark 1.2.2 Hadoop 2.4 download

[jira] [Updated] (SPARK-7027) Spark 1.2.2 Hadoop 2.4 download is missing

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7027: --- Priority: Critical (was: Major) Spark 1.2.2 Hadoop 2.4 download is missing

[jira] [Resolved] (SPARK-7030) spark-submit in cluster mode with supervisor flag, not able to transfer application jar to driver node

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7030. Resolution: Invalid This is filed as a bug report, but this is not currently a feature

[jira] [Assigned] (SPARK-7027) Spark 1.2.2 Hadoop 2.4 download is missing

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reassigned SPARK-7027: -- Assignee: Patrick Wendell Spark 1.2.2 Hadoop 2.4 download is missing

[jira] [Updated] (SPARK-6958) Add Pandas style sort operator

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6958: --- Summary: Add Pandas style sort operator (was: sort) Add Pandas style sort operator

[jira] [Updated] (SPARK-6957) Add Pandas style group by operator

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6957: --- Summary: Add Pandas style group by operator (was: groupby) Add Pandas style group

[jira] [Updated] (SPARK-7016) Refactor dev/run-tests(-jenkins) from Bash to Python

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7016: --- Priority: Critical (was: Major) Refactor dev/run-tests(-jenkins) from Bash to Python

Re: Is spark-ec2 for production use?

2015-04-21 Thread Patrick Wendell
It could be a good idea to document this a bit. The original goals were to give people an easy way to get started with Spark and also to provide a consistent environment for our own experiments and benchmarking of Spark at the AMPLab. Over time I've noticed a huge amount of scope increase in terms

[jira] [Resolved] (SPARK-6995) Spark driver program throws exception cant get Master Kerberose principal for use as renewer

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6995. Resolution: Invalid Can you reach out on the user list with this question? Thanks! Spark

[jira] [Updated] (SPARK-7020) Restrict module testing based on commit contents

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7020: --- Priority: Critical (was: Major) Restrict module testing based on commit contents

[jira] [Resolved] (SPARK-7027) Spark 1.2.2 Hadoop 2.4 download is missing

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-7027. Resolution: Fixed Fix Version/s: 1.2.2 It's fixed now, but will take several hours

[jira] [Updated] (SPARK-6990) Add Java linting script

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6990: --- Labels: starter (was: ) Add Java linting script

[jira] [Updated] (SPARK-1684) Merge script should standardize SPARK-XXX prefix

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1684: --- Assignee: Michelle Casbon (was: Patrick Wendell) Merge script should standardize SPARK-XXX

[jira] [Closed] (SPARK-1684) Merge script should standardize SPARK-XXX prefix

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-1684. -- Resolution: Fixed Fix Version/s: 1.4.0 Merge script should standardize SPARK-XXX prefix

[jira] [Commented] (SPARK-6889) Streamline contribution process with update to Contribution wiki, JIRA rules

2015-04-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506195#comment-14506195 ] Patrick Wendell commented on SPARK-6889: Hey Sean - I just read over your changes

[jira] [Updated] (SPARK-6999) infinite recursion with createDataFrame(JavaRDD[Row], java.util.List[String])

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6999: --- Component/s: SQL infinite recursion with createDataFrame(JavaRDD[Row], java.util.List[String

[jira] [Updated] (SPARK-7003) Improve reliability of connection failure detection between Netty block transfer service endpoints

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-7003: --- Component/s: Spark Core Improve reliability of connection failure detection between Netty

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502323#comment-14502323 ] Patrick Wendell commented on SPARK-3630: [~arahuja] If you are able to upgrade

[jira] [Updated] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6968: --- Priority: Critical (was: Minor) Make maniuplating an underlying RDD of a DataFrame easier

[jira] [Updated] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6968: --- Labels: (was: DataFrame) Make maniuplating an underlying RDD of a DataFrame easier

[jira] [Updated] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6968: --- Component/s: SQL Make maniuplating an underlying RDD of a DataFrame easier

[jira] [Updated] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6968: --- Component/s: (was: Spark Core) Make maniuplating an underlying RDD of a DataFrame easier

[jira] [Updated] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6968: --- Priority: Minor (was: Critical) Make maniuplating an underlying RDD of a DataFrame easier

[jira] [Commented] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502386#comment-14502386 ] Patrick Wendell commented on SPARK-6968: So coalesce() is added to the DataFrame

[jira] [Commented] (SPARK-6961) Cannot save data to parquet files when executing from Windows from a Maven Project

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502316#comment-14502316 ] Patrick Wendell commented on SPARK-6961: Based on looking at the stack trace, I

[jira] [Updated] (SPARK-6959) Support for datetime comparisions in filter for dataframes in pyspark

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6959: --- Issue Type: Improvement (was: Bug) Support for datetime comparisions in filter

[jira] [Comment Edited] (SPARK-6968) Make maniuplating an underlying RDD of a DataFrame easier

2015-04-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502386#comment-14502386 ] Patrick Wendell edited comment on SPARK-6968 at 4/20/15 5:56 AM

[jira] [Updated] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2015-04-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3266: --- Target Version/s: (was: 2+) JavaDoubleRDD doesn't contain max

[jira] [Updated] (SPARK-5825) Failure stopping Services while command line argument is too long

2015-04-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5825: --- Priority: Critical (was: Blocker) Failure stopping Services while command line argument

Announcing Spark 1.3.1 and 1.2.2

2015-04-17 Thread Patrick Wendell
Hi All, I'm happy to announce the Spark 1.3.1 and 1.2.2 maintenance releases. We recommend all users on the 1.3 and 1.2 Spark branches upgrade to these releases, which contain several important bug fixes. Download Spark 1.3.1 or 1.2.2: http://spark.apache.org/downloads.html Release notes:

Announcing Spark 1.3.1 and 1.2.2

2015-04-17 Thread Patrick Wendell
Hi All, I'm happy to announce the Spark 1.3.1 and 1.2.2 maintenance releases. We recommend all users on the 1.3 and 1.2 Spark branches upgrade to these releases, which contain several important bug fixes. Download Spark 1.3.1 or 1.2.2: http://spark.apache.org/downloads.html Release notes:

[jira] [Created] (SPARK-6971) Each Jenkins build should use a distinct Zinc port

2015-04-16 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6971: -- Summary: Each Jenkins build should use a distinct Zinc port Key: SPARK-6971 URL: https://issues.apache.org/jira/browse/SPARK-6971 Project: Spark Issue

[RESULT] [VOTE] Release Apache Spark 1.2.2

2015-04-16 Thread Patrick Wendell
on OS X +1 Sean On Apr 14, 2015, at 10:59 PM, Patrick Wendell pwend...@gmail.com wrote: I'd like to close this vote to coincide with the 1.3.1 release, however, it would be great to have more people test this release first. I'll leave it open for a bit longer and see if others can give

[jira] [Updated] (SPARK-3468) Provide timeline view in Job and Stage UI pages

2015-04-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3468: --- Assignee: Kousuke Saruta Provide timeline view in Job and Stage UI pages

[jira] [Updated] (SPARK-6418) Add simple per-stage visualization to the UI

2015-04-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6418: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-6942 Add simple per-stage

[jira] [Created] (SPARK-6946) Add visualization of logical and physical plans for SQL/DataFrames

2015-04-15 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6946: -- Summary: Add visualization of logical and physical plans for SQL/DataFrames Key: SPARK-6946 URL: https://issues.apache.org/jira/browse/SPARK-6946 Project: Spark

[jira] [Created] (SPARK-6945) Provide SQL tab in the Spark UI

2015-04-15 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6945: -- Summary: Provide SQL tab in the Spark UI Key: SPARK-6945 URL: https://issues.apache.org/jira/browse/SPARK-6945 Project: Spark Issue Type: Sub-task

<    1   2   3   4   5   6   7   8   9   10   >