[jira] [Commented] (SPARK-8297) Scheduler backend is not notified in case node fails in YARN

2015-06-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581038#comment-14581038 ] Mridul Muralidharan commented on SPARK-8297: Spark on mesos handles

[jira] [Updated] (SPARK-8297) Scheduler backend is not notified in case node fails in YARN

2015-06-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-8297: --- Affects Version/s: 1.5.0 Scheduler backend is not notified in case node fails

OT: Key types which have potential issues

2015-05-19 Thread Mridul Muralidharan
Hi, I vaguely remember issues with using float/double as keys in MR (and spark ?). But cant seem to find documentation/analysis about the same. Does anyone have some resource/link I can refer to ? Thanks, Mridul - To

Re: Change for submitting to yarn in 1.3.1

2015-05-11 Thread Mridul Muralidharan
That works when it is launched from same process - which is unfortunately not our case :-) - Mridul On Sun, May 10, 2015 at 9:05 PM, Manku Timma manku.tim...@gmail.com wrote: sc.applicationId gives the yarn appid. On 11 May 2015 at 08:13, Mridul Muralidharan mri...@gmail.com wrote: We had

Re: YARN mode startup takes too long (10+ secs)

2015-05-11 Thread Mridul Muralidharan
For tiny/small clusters (particularly single tenet), you can set it to lower value. But for anything reasonably large or multi-tenet, the request storm can be bad if large enough number of applications start aggressively polling RM. That is why the interval is set to configurable. - Mridul On

Re: Change for submitting to yarn in 1.3.1

2015-05-10 Thread Mridul Muralidharan
We had a similar requirement, and as a stopgap, I currently use a suboptimal impl specific workaround - parsing it out of the stdout/stderr (based on log config). A better means to get to this is indeed required ! Regards, Mridul On Sun, May 10, 2015 at 7:33 PM, Ron's Yahoo!

[jira] [Assigned] (SPARK-7323) Use insertAll instead of individual insert while merging combiners

2015-05-02 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-7323: -- Assignee: Mridul Muralidharan Use insertAll instead of individual insert

[jira] [Created] (SPARK-7323) Use insertAll instead of individual insert while merging combiners

2015-05-02 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-7323: -- Summary: Use insertAll instead of individual insert while merging combiners Key: SPARK-7323 URL: https://issues.apache.org/jira/browse/SPARK-7323 Project

Re: [discuss] ending support for Java 6?

2015-05-02 Thread Mridul Muralidharan
We could build on minimum jdk we support for testing pr's - which will automatically cause build failures in case code uses newer api ? Regards, Mridul On Fri, May 1, 2015 at 2:46 PM, Reynold Xin r...@databricks.com wrote: It's really hard to inspect API calls since none of us have the Java

Re: [discuss] ending support for Java 6?

2015-05-02 Thread Mridul Muralidharan
... ;) On Sat, May 2, 2015 at 1:09 PM, Mridul Muralidharan mri...@gmail.com wrote: We could build on minimum jdk we support for testing pr's - which will automatically cause build failures in case code uses newer api ? Regards, Mridul On Fri, May 1, 2015 at 2:46 PM, Reynold Xin r

Re: Why does SortShuffleWriter write to disk always?

2015-05-02 Thread Mridul Muralidharan
I agree, this is better handled by the filesystem cache - not to mention, being able to do zero copy writes. Regards, Mridul On Sat, May 2, 2015 at 10:26 PM, Reynold Xin r...@databricks.com wrote: I've personally prototyped completely in-memory shuffle for Spark 3 times. However, it is unclear

[jira] [Created] (SPARK-7317) ShuffleHandle needs to be exposed

2015-05-01 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-7317: -- Summary: ShuffleHandle needs to be exposed Key: SPARK-7317 URL: https://issues.apache.org/jira/browse/SPARK-7317 Project: Spark Issue Type

[jira] [Created] (SPARK-7313) Allow for configuring max_samples in range partitioner.

2015-05-01 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-7313: -- Summary: Allow for configuring max_samples in range partitioner. Key: SPARK-7313 URL: https://issues.apache.org/jira/browse/SPARK-7313 Project: Spark

Re: Should we let everyone set Assignee?

2015-04-24 Thread Mridul Muralidharan
This is a great suggestion - definitely makes sense to have it. Regards, Mridul On Fri, Apr 24, 2015 at 11:08 AM, Patrick Wendell pwend...@gmail.com wrote: It's a bit of a digression - but Steve's suggestion that we have a mailing list for new issues is a great idea and we can do it easily.

[jira] [Closed] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-27 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-6168. -- Resolution: Won't Fix Expose some of the collection classes as DeveloperApi

[jira] [Updated] (SPARK-1476) 2GB limit in spark for blocks

2015-03-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-1476: --- Assignee: (was: Mridul Muralidharan) 2GB limit in spark for blocks

[jira] [Updated] (SPARK-2532) Fix issues with consolidated shuffle

2015-03-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-2532: --- Assignee: (was: Mridul Muralidharan) Fix issues with consolidated shuffle

[jira] [Reopened] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-25 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reopened SPARK-6168: Assignee: Mridul Muralidharan Expose some of the collection classes

[jira] [Commented] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-25 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380759#comment-14380759 ] Mridul Muralidharan commented on SPARK-6168: Looks like I did not get

Re: broadcast hang out

2015-03-15 Thread Mridul Muralidharan
Cross region as in different data centers ? - Mridul On Sun, Mar 15, 2015 at 8:08 PM, lonely Feb lonely8...@gmail.com wrote: Hi all, i meet up with a problem that torrent broadcast hang out in my spark cluster (1.2, standalone) , particularly serious when driver and executors are

Re: Spark config option 'expression language' feedback request

2015-03-13 Thread Mridul Muralidharan
Let me try to rephrase my query. How can a user specify, for example, what the executor memory should be or number of cores should be. I dont want a situation where some variables can be specified using one set of idioms (from this PR for example) and another set cannot be. Regards, Mridul

Re: May we merge into branch-1.3 at this point?

2015-03-13 Thread Mridul Muralidharan
Who is managing 1.3 release ? You might want to coordinate with them before porting changes to branch. Regards Mridul On Friday, March 13, 2015, Sean Owen so...@cloudera.com wrote: Yeah, I'm guessing that is all happening quite literally as we speak. The Apache git tag is the one of

Re: Release Scala version vs Hadoop version (was: [VOTE] Release Apache Spark 1.3.0 (RC3))

2015-03-09 Thread Mridul Muralidharan
In ideal situation, +1 on removing all vendor specific builds and making just hadoop version specific - that is what we should depend on anyway. Though I hope Sean is correct in assuming that vendor specific builds for hadoop 2.4 are just that; and not 2.4- or 2.4+ which cause incompatibilities

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2015-03-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352377#comment-14352377 ] Mridul Muralidharan commented on SPARK-1239: Hitting akka framesize for map

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2015-03-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352284#comment-14352284 ] Mridul Muralidharan commented on SPARK-1239: [~pwendell] Is there any update

Re: enum-like types in Spark

2015-03-05 Thread Mridul Muralidharan
While I dont have any strong opinions about how we handle enum's either way in spark, I assume the discussion is targetted at (new) api being designed in spark. Rewiring what we already have exposed will lead to incompatible api change (StorageLevel for example, is in 1.0). Regards, Mridul On

[jira] [Commented] (SPARK-6169) Shuffle based join

2015-03-05 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348995#comment-14348995 ] Mridul Muralidharan commented on SPARK-6169: I have not done too much research

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-03-05 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348987#comment-14348987 ] Mridul Muralidharan commented on SPARK-4879: [~joshrosen] The former

[jira] [Updated] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-03-05 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-4879: --- Affects Version/s: 1.3.0 Missing output partitions after job completes

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-03-05 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348522#comment-14348522 ] Mridul Muralidharan commented on SPARK-4879: With 1.3 RC, we are still seeing

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-03-05 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348527#comment-14348527 ] Mridul Muralidharan commented on SPARK-4879: Just to add - we are seeing

Re: enum-like types in Spark

2015-03-05 Thread Mridul Muralidharan
I have a strong dislike for java enum's due to the fact that they are not stable across JVM's - if it undergoes serde, you end up with unpredictable results at times [1]. One of the reasons why we prevent enum's from being key : though it is highly possible users might depend on it internally

[jira] [Updated] (SPARK-6166) Add config to limit number of concurrent outbound connections for shuffle fetch

2015-03-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-6166: --- Priority: Minor (was: Major) Add config to limit number of concurrent outbound

[jira] [Created] (SPARK-6166) Add config to limit number of concurrent outbound connections for shuffle fetch

2015-03-04 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6166: -- Summary: Add config to limit number of concurrent outbound connections for shuffle fetch Key: SPARK-6166 URL: https://issues.apache.org/jira/browse/SPARK-6166

[jira] [Created] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-04 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6168: -- Summary: Expose some of the collection classes as DeveloperApi Key: SPARK-6168 URL: https://issues.apache.org/jira/browse/SPARK-6168 Project: Spark

[jira] [Updated] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-6168: --- Description: It would be very useful to mark some of the collection and util classes

[jira] [Created] (SPARK-6169) Shuffle based join

2015-03-04 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6169: -- Summary: Shuffle based join Key: SPARK-6169 URL: https://issues.apache.org/jira/browse/SPARK-6169 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-6170) Add support for skew join

2015-03-04 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6170: -- Summary: Add support for skew join Key: SPARK-6170 URL: https://issues.apache.org/jira/browse/SPARK-6170 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-6165) Aggregate and reduce should spool to disk and complete

2015-03-04 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6165: -- Summary: Aggregate and reduce should spool to disk and complete Key: SPARK-6165 URL: https://issues.apache.org/jira/browse/SPARK-6165 Project: Spark

[jira] [Updated] (SPARK-6165) Aggregate and reduce should be able to work with very large number of tasks.

2015-03-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-6165: --- Summary: Aggregate and reduce should be able to work with very large number of tasks

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2015-02-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341997#comment-14341997 ] Mridul Muralidharan commented on SPARK-3889: [~adav] I have seen this a lot

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-27 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340379#comment-14340379 ] Mridul Muralidharan commented on SPARK-6050: [~tgraves] You are right, cpu

[jira] [Comment Edited] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339686#comment-14339686 ] Mridul Muralidharan edited comment on SPARK-6050 at 2/27/15 5:24 AM

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339798#comment-14339798 ] Mridul Muralidharan commented on SPARK-6050: With more verbose debug added

[jira] [Created] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-26 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-6050: -- Summary: Spark on YARN does not work --executor-cores is specified Key: SPARK-6050 URL: https://issues.apache.org/jira/browse/SPARK-6050 Project: Spark

[jira] [Comment Edited] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339686#comment-14339686 ] Mridul Muralidharan edited comment on SPARK-6050 at 2/27/15 3:50 AM

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-02-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339686#comment-14339686 ] Mridul Muralidharan commented on SPARK-6050: Thanks to [~tgraves] for helping

Re: 2GB limit for partitions?

2015-02-04 Thread Mridul Muralidharan
, seems promising. thanks, Imran On Tue, Feb 3, 2015 at 7:32 PM, Mridul Muralidharan mri...@gmail.com javascript:_e(%7B%7D,'cvml','mri...@gmail.com'); wrote: That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul

Re: 2GB limit for partitions?

2015-02-04 Thread Mridul Muralidharan
, seems promising. thanks, Imran On Tue, Feb 3, 2015 at 7:32 PM, Mridul Muralidharan mri...@gmail.com javascript:_e(%7B%7D,'cvml','mri...@gmail.com'); wrote: That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul

Re: Welcoming three new committers

2015-02-03 Thread Mridul Muralidharan
Congratulations ! Keep up the good work :-) Regards Mridul On Tuesday, February 3, 2015, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in

Re: 2GB limit for partitions?

2015-02-03 Thread Mridul Muralidharan
That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul On Tuesday, February 3, 2015, Imran Rashid iras...@cloudera.com wrote: Thanks for the explanations, makes sense. For the record looks like this was worked on a while

Re: 2GB limit for partitions?

2015-02-03 Thread Mridul Muralidharan
That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul On Tuesday, February 3, 2015, Imran Rashid iras...@cloudera.com wrote: Thanks for the explanations, makes sense. For the record looks like this was worked on a while

Re: keeping PR titles / descriptions up to date

2014-12-02 Thread Mridul Muralidharan
I second that ! Would also be great if the JIRA was updated accordingly too. Regards, Mridul On Wed, Dec 3, 2014 at 1:53 AM, Kay Ousterhout kayousterh...@gmail.com wrote: Hi all, I've noticed a bunch of times lately where a pull request changes to be pretty different from the original pull

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178162#comment-14178162 ] Mridul Muralidharan commented on SPARK-4030: We have also needed to use

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173413#comment-14173413 ] Mridul Muralidharan commented on SPARK-3948: Not exactly, what I

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173489#comment-14173489 ] Mridul Muralidharan commented on SPARK-3948: Damn, this sucks

[jira] [Comment Edited] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173489#comment-14173489 ] Mridul Muralidharan edited comment on SPARK-3948 at 10/16/14 7:39 AM

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172116#comment-14172116 ] Mridul Muralidharan commented on SPARK-3948: [~joshrosen] Assuming

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172134#comment-14172134 ] Mridul Muralidharan commented on SPARK-3948: [~jerryshao] Just to clarify

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172225#comment-14172225 ] Mridul Muralidharan commented on SPARK-3948: That is weird, I tried a stripped

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172226#comment-14172226 ] Mridul Muralidharan commented on SPARK-3948: Note, t is just some file

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172585#comment-14172585 ] Mridul Muralidharan commented on SPARK-3948: [~jerryshao] great work ! I agree

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167303#comment-14167303 ] Mridul Muralidharan commented on SPARK-3889: The status says fixed - what

Re: Breaking the previous large-scale sort record with Spark

2014-10-10 Thread Mridul Muralidharan
Brilliant stuff ! Congrats all :-) This is indeed really heartening news ! Regards, Mridul On Fri, Oct 10, 2014 at 8:24 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi folks, I interrupt your regularly scheduled user / dev list to bring you some pretty cool news for the project, which

Re: Breaking the previous large-scale sort record with Spark

2014-10-10 Thread Mridul Muralidharan
Brilliant stuff ! Congrats all :-) This is indeed really heartening news ! Regards, Mridul On Fri, Oct 10, 2014 at 8:24 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi folks, I interrupt your regularly scheduled user / dev list to bring you some pretty cool news for the project, which

[jira] [Commented] (SPARK-3785) Support off-loading computations to a GPU

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163326#comment-14163326 ] Mridul Muralidharan commented on SPARK-3785: [~sowen] We had prototyped

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163351#comment-14163351 ] Mridul Muralidharan commented on SPARK-3561: [~pwendell] If I understood

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163376#comment-14163376 ] Mridul Muralidharan commented on SPARK-3561: [~ozhurakousky] I think

[jira] [Commented] (SPARK-3847) Enum.hashCode is only consistent within the same JVM

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163407#comment-14163407 ] Mridul Muralidharan commented on SPARK-3847: Wow, nice bug

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164095#comment-14164095 ] Mridul Muralidharan commented on SPARK-3561: I agree with [~pwendell

[jira] [Commented] (SPARK-3847) Enum.hashCode is only consistent within the same JVM

2014-10-08 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164775#comment-14164775 ] Mridul Muralidharan commented on SPARK-3847: [~joshrosen] array hashcode might

[jira] [Commented] (SPARK-3714) Spark workflow scheduler

2014-09-29 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151486#comment-14151486 ] Mridul Muralidharan commented on SPARK-3714: Most of the drawbacks mentioned

[jira] [Commented] (SPARK-3714) Spark workflow scheduler

2014-09-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151211#comment-14151211 ] Mridul Muralidharan commented on SPARK-3714: Have you tried using oozie

[jira] [Commented] (SPARK-1956) Enable shuffle consolidation by default

2014-09-07 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125027#comment-14125027 ] Mridul Muralidharan commented on SPARK-1956: The recent changes

[jira] [Commented] (SPARK-3019) Pluggable block transfer (data plane communication) interface

2014-09-02 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119362#comment-14119362 ] Mridul Muralidharan commented on SPARK-3019: Just went over the proposal

[jira] [Commented] (SPARK-3019) Pluggable block transfer (data plane communication) interface

2014-09-02 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119365#comment-14119365 ] Mridul Muralidharan commented on SPARK-3019: I will try to push the version we

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

2014-09-02 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119385#comment-14119385 ] Mridul Muralidharan commented on SPARK-1476: WIP version pushed to https

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113741#comment-14113741 ] Mridul Muralidharan commented on SPARK-3277: This looks like unrelated changes

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Priority: Blocker (was: Major) LZ4 compression cause the the ExternalSort

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Affects Version/s: 1.2.0 1.1.0 LZ4 compression cause

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114014#comment-14114014 ] Mridul Muralidharan commented on SPARK-3277: [~matei] Attaching a patch which

[jira] [Comment Edited] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114022#comment-14114022 ] Mridul Muralidharan edited comment on SPARK-3277 at 8/28/14 5:37 PM

[jira] [Updated] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-3277: --- Attachment: test_lz4_bug.patch Against master, though I noticed similar changes

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114026#comment-14114026 ] Mridul Muralidharan commented on SPARK-3277: [~hzw] did you notice

[jira] [Commented] (SPARK-3277) LZ4 compression cause the the ExternalSort exception

2014-08-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114484#comment-14114484 ] Mridul Muralidharan commented on SPARK-3277: Sounds great, thx ! I suspect

Re: [VOTE] Release Apache Spark 1.1.0 (RC1)

2014-08-28 Thread Mridul Muralidharan
Is SPARK-3277 applicable to 1.1 ? If yes, until it is fixed, I am -1 on the release (I am on break, so can't verify or help fix, sorry). Regards Mridul On 28-Aug-2014 9:33 pm, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version

Re: [VOTE] Release Apache Spark 1.1.0 (RC1)

2014-08-28 Thread Mridul Muralidharan
and we'll patch it and spin a new RC. We can also update the test coverage to cover LZ4. - Patrick On Thu, Aug 28, 2014 at 9:27 AM, Mridul Muralidharan mri...@gmail.com wrote: Is SPARK-3277 applicable to 1.1 ? If yes, until it is fixed, I am -1 on the release (I am on break, so can't

[jira] [Commented] (SPARK-3175) Branch-1.1 SBT build failed for Yarn-Alpha

2014-08-23 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107923#comment-14107923 ] Mridul Muralidharan commented on SPARK-3175: Please add more information

Re: is Branch-1.1 SBT build broken for yarn-alpha ?

2014-08-21 Thread Mridul Muralidharan
Weird that Patrick did not face this while creating the RC. Essentially the yarn alpha pom.xml has not been updated properly in the 1.1 branch. Just change version to '1.1.1-SNAPSHOT' for yarn/alpha/pom.xml (to make it same as any other pom). Regards, Mridul On Thu, Aug 21, 2014 at 5:09 AM,

[jira] [Commented] (SPARK-3115) Improve task broadcast latency for small tasks

2014-08-19 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102142#comment-14102142 ] Mridul Muralidharan commented on SPARK-3115: I had a tab open with pretty much

[jira] [Commented] (SPARK-3019) Pluggable block transfer (data plane communication) interface

2014-08-17 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099909#comment-14099909 ] Mridul Muralidharan commented on SPARK-3019: I am yet to go through

[jira] [Commented] (SPARK-3019) Pluggable block transfer (data plane communication) interface

2014-08-17 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099910#comment-14099910 ] Mridul Muralidharan commented on SPARK-3019: Btw, can we do something about

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

2014-08-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099072#comment-14099072 ] Mridul Muralidharan commented on SPARK-1476: Based on discussions we had

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2014-08-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099115#comment-14099115 ] Mridul Muralidharan commented on SPARK-2089: For a general case, wont

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2014-08-13 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095247#comment-14095247 ] Mridul Muralidharan commented on SPARK-2089: Since I am not maintaining

[jira] [Commented] (SPARK-2962) Suboptimal scheduling in spark

2014-08-11 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092807#comment-14092807 ] Mridul Muralidharan commented on SPARK-2962: On further investigation

[jira] [Created] (SPARK-2962) Suboptimal scheduling in spark

2014-08-10 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-2962: -- Summary: Suboptimal scheduling in spark Key: SPARK-2962 URL: https://issues.apache.org/jira/browse/SPARK-2962 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2962) Suboptimal scheduling in spark

2014-08-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092427#comment-14092427 ] Mridul Muralidharan commented on SPARK-2962: To give more context; a) Our

[jira] [Commented] (SPARK-2962) Suboptimal scheduling in spark

2014-08-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092430#comment-14092430 ] Mridul Muralidharan commented on SPARK-2962: Hi [~matei], I am referencing

[jira] [Commented] (SPARK-2962) Suboptimal scheduling in spark

2014-08-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092431#comment-14092431 ] Mridul Muralidharan commented on SPARK-2962: Note, I dont think

<    3   4   5   6   7   8   9   10   11   12   >