[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

2018-03-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Improvement (was: Documentation) > Document using docker in h

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

2018-03-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Task (was: Improvement) > Document using docker in host network

[jira] [Created] (SPARK-23718) Document using docker in host networking mode in hadoop 3

2018-03-16 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-23718: --- Summary: Document using docker in host networking mode in hadoop 3 Key: SPARK-23718 URL: https://issues.apache.org/jira/browse/SPARK-23718 Project

[jira] [Created] (SPARK-23717) Leverage docker support in Hadoop 3

2018-03-16 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-23717: --- Summary: Leverage docker support in Hadoop 3 Key: SPARK-23717 URL: https://issues.apache.org/jira/browse/SPARK-23717 Project: Spark Issue Type

Re: Welcoming some new committers

2018-03-03 Thread Mridul Muralidharan
Congratulations ! Regards, Mridul On Fri, Mar 2, 2018 at 2:41 PM, Matei Zaharia wrote: > Hi everyone, > > The Spark PMC has recently voted to add several new committers to the > project, based on their contributions to Spark 2.3 and other past work: > > - Anirudh Ramanathan (contributor to Ku

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

2018-02-22 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373202#comment-16373202 ] Mridul Muralidharan commented on YARN-7935: --- Hi [~tgraves],    Ini

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

2018-02-21 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372320#comment-16372320 ] Mridul Muralidharan commented on YARN-7935: --- The hostname executor's bi

Re: [Core][Suggestion] sortWithinPartitions and aggregateWithinPartitions for RDD

2018-02-01 Thread Mridul Muralidharan
On Wed, Jan 31, 2018 at 1:15 AM, Ruifeng Zheng wrote: > HI all: > > > >1, Dataset API supports operation “sortWithinPartitions”, but in RDD > API there is no counterpart (I know there is > “repartitionAndSortWithinPartitions”, but I don’t want to repartition the > RDD), I have to convert R

Re: Kubernetes backend and docker images

2018-01-05 Thread Mridul Muralidharan
We should definitely clean this up and make it the default, nicely done Marcelo ! Thanks, Mridul On Fri, Jan 5, 2018 at 5:06 PM Marcelo Vanzin wrote: > Hey all, especially those working on the k8s stuff. > > Currently we have 3 docker images that need to be built and provided > by the user when

[jira] [Commented] (SPARK-22903) AlreadyBeingCreatedException in stage retry caused by wrong attemptNumber

2017-12-29 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306583#comment-16306583 ] Mridul Muralidharan commented on SPARK-22903: - [~imranr] I agree that S

[jira] [Resolved] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

2017-12-24 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-22465. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request

[jira] [Updated] (SPARK-22866) Kubernetes dockerfile path needs update

2017-12-21 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-22866: Fix Version/s: (was: 3.0.0) 2.3.0 > Kubernetes dockerf

[jira] [Resolved] (SPARK-22866) Kubernetes dockerfile path needs update

2017-12-21 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-22866. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request

Re: Publishing official docker images for KubernetesSchedulerBackend

2017-11-29 Thread Mridul Muralidharan
We do support running on Apache Mesos via docker images - so this would not be restricted to k8s. But unlike mesos support, which has other modes of running, I believe k8s support more heavily depends on availability of docker images. Regards, Mridul On Wed, Nov 29, 2017 at 8:56 AM, Sean Owen

[jira] [Assigned] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-21549: --- Assignee: Sergey Zhemzhitsky > Spark fails to complete job correctly

[jira] [Resolved] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-10-06 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-21549. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue

Re: Should Flume integration be behind a profile?

2017-10-01 Thread Mridul Muralidharan
I agree, proposal 1 sounds better among the options. Regards, Mridul On Sun, Oct 1, 2017 at 3:50 PM, Reynold Xin wrote: > Probably should do 1, and then it is an easier transition in 3.0. > > On Sun, Oct 1, 2017 at 1:28 AM Sean Owen wrote: >> >> I tried and failed to do this in >> https://issu

Re: Welcoming Tejas Patil as a Spark committer

2017-09-29 Thread Mridul Muralidharan
Congratulations Tejas ! Regards, Mridul On Fri, Sep 29, 2017 at 12:58 PM, Matei Zaharia wrote: > Hi all, > > The Spark PMC recently added Tejas Patil as a committer on the > project. Tejas has been contributing across several areas of Spark for > a while, focusing especially on scalability issue

Re: Should Flume integration be behind a profile?

2017-09-26 Thread Mridul Muralidharan
Sounds good to me. +1 Regards, Mridul On Tue, Sep 26, 2017 at 2:36 AM, Sean Owen wrote: > Not a big deal, but I'm wondering whether Flume integration should at least > be opt-in and behind a profile? it still sees some use (at least on our end) > but not applicable to the majority of users. Mo

Re: [DISCUSS][bahir-flink] Drop Java 7 support

2017-09-19 Thread Mridul Muralidharan
Thanks for clarifying Robert ! Given this, I am +1 on removing support too. Regards, Mridul On Mon, Sep 18, 2017 at 11:02 AM, Robert Metzger wrote: > Thanks all for the overwhelming support to drop Java7. > > @Mridul: I don't know for sure what the schedule for Flink 1.4 is. I > believe it'll co

Re: [DISCUSS][bahir-flink] Drop Java 7 support

2017-09-16 Thread Mridul Muralidharan
Do we know when Flink 1.4 is expected to be out ? Will we be making any intermediate Bahir release for flink before then ? If we are not making any, then dropping java 7 will have no impact for our users. On spark side, jdk 7 support has already been dropped. Regards, Mridul On Sat, Sep 16, 201

[jira] [Commented] (SPARK-9213) Improve regular expression performance (via joni)

2017-08-29 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145831#comment-16145831 ] Mridul Muralidharan commented on SPARK-9213: [~rxin] Curious what happene

Re: Welcoming Saisai (Jerry) Shao as a committer

2017-08-28 Thread Mridul Muralidharan
Congratulations Jerry, well deserved ! Regards, Mridul On Mon, Aug 28, 2017 at 6:28 PM, Matei Zaharia wrote: > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has > been contributing to many areas of the project for a long time, so it’s great > to see

Re: [VOTE] Apache Bahir 2.2.0 (RC1)

2017-08-18 Thread Mridul Muralidharan
Build works fine, signatures check out, tests run successfully. +1 for release. Btw, I dont recall RAT check failing build earlier, very nice touch to enforce it as part of build ! Regards, Mridul On Wed, Aug 16, 2017 at 10:17 PM, Luciano Resende wrote: > Dear community member, > > Please vo

Re: SPIP: Spark on Kubernetes

2017-08-17 Thread Mridul Muralidharan
While I definitely support the idea of Apache Spark being able to leverage kubernetes, IMO it is better for long term evolution of spark to expose appropriate SPI such that this support need not necessarily live within Apache Spark code base. It will allow for multiple backends to evolve, decoupled

Re: Welcoming Hyukjin Kwon and Sameer Agarwal as committers

2017-08-07 Thread Mridul Muralidharan
Congratulations Hyukjin, Sameer ! Regards, Mridul On Mon, Aug 7, 2017 at 8:53 AM, Matei Zaharia wrote: > Hi everyone, > > The Spark PMC recently voted to add Hyukjin Kwon and Sameer Agarwal as > committers. Join me in congratulating both of them and thanking them for > their contributions to t

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:1

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:1

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan commented on SPARK-21549: - This affects both ma

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

2017-05-23 Thread Mridul Muralidharan
s://issues.apache.org/jira/browse/BAHIR-88 >> >> >> And I guess it is also applicable for Flink extensions >> >> >> But the issue is still open. >> >> >> On Tue, May 23, 2017 at 12:08 PM, Mridul Muralidharan < >> mridul...@apache.org>

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

2017-05-23 Thread Mridul Muralidharan
11:50 PM, Mridul Muralidharan > wrote: > >> Hi, >> >> From the release tag, clean build and tests work fine, signatures match. >> >> The source at https://dist.apache.org/repos/dist/dev/bahir/bahir-flink/1. >> 0-rc5/ >> has additional files

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

2017-05-21 Thread Mridul Muralidharan
Hi, From the release tag, clean build and tests work fine, signatures match. The source at https://dist.apache.org/repos/dist/dev/bahir/bahir-flink/1.0-rc5/ has additional files which are not in repo: Not in repo ./distribution/pom.xml.releaseBackup Not in repo ./flink-connector-activemq/pom.x

Re: PR permission to kick Jenkins?

2017-05-05 Thread Mridul Muralidharan
I had the same issue last night ... Regards, Mridul On Fri, May 5, 2017 at 12:06 PM, Tom Graves wrote: > Does anyone know how to configure Jenkins to allow committers to tell it to > test prs? I used to have this access but lately it is either not working or > only intermittently working. > >

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995550#comment-15995550 ] Mridul Muralidharan commented on SPARK-20589: - coalasce with shuffle=f

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985575#comment-15985575 ] Mridul Muralidharan commented on SPARK-20480: - Shouldn't fix for SP

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC3)

2017-04-11 Thread Mridul Muralidharan
+1 for release. Tag checks out fine - full build and tests go through fine. Signatures validated. Regards, Mridul On Thu, Apr 6, 2017 at 4:08 PM, Luciano Resende wrote: > Please vote to approve the release of the following candidate as Apache > Bahir Flink extensions 1.0 based on Apache Flink

Re: [VOTE] Apache Spark 2.1.1 (RC2)

2017-04-04 Thread Mridul Muralidharan
Hi, https://issues.apache.org/jira/browse/SPARK-20202?jql=priority%20%3D%20Blocker%20AND%20affectedVersion%20%3D%20%222.1.1%22%20and%20project%3D%22spark%22 Indicates there is another blocker (SPARK-20197 should have come in the list too, but was marked major). Regards, Mridul On Tue, Apr 4,

[jira] [Comment Edited] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

2017-04-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954387#comment-15954387 ] Mridul Muralidharan edited comment on SPARK-20205 at 4/4/17 12:1

[jira] [Commented] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

2017-04-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954387#comment-15954387 ] Mridul Muralidharan commented on SPARK-20205: - For history server that

[jira] [Commented] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

2017-04-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954348#comment-15954348 ] Mridul Muralidharan commented on SPARK-20205: - bq. I wouldn't say

[jira] [Commented] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

2017-04-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954333#comment-15954333 ] Mridul Muralidharan commented on SPARK-20205: - This is nasty ! This m

[jira] [Commented] (SPARK-19796) taskScheduler fails serializing long statements received by thrift server

2017-03-02 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893584#comment-15893584 ] Mridul Muralidharan commented on SPARK-19796: - I would not prefer (b)

[jira] [Commented] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2017-02-26 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884979#comment-15884979 ] Mridul Muralidharan commented on SPARK-5159: For 1.6.x and earlier, we f

[jira] [Commented] (SPARK-19698) Race condition in stale attempt task completion vs current attempt task completion when task is doing persistent state changes

2017-02-23 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881991#comment-15881991 ] Mridul Muralidharan commented on SPARK-19698: - Depending on ordering

[jira] [Comment Edited] (SPARK-19698) Race condition in stale attempt task completion vs current attempt task completion when task is doing persistent state changes

2017-02-23 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881991#comment-15881991 ] Mridul Muralidharan edited comment on SPARK-19698 at 2/24/17 5:2

Re: [VOTE] Apache Bahir 2.1.0 (RC1)

2017-02-07 Thread Mridul Muralidharan
ir source distributions contained these release > build artifacts. > > I created https://issues.apache.org/jira/browse/BAHIR-88 to track it. > > Thanks, Christian > >> ---- >> From: Mridul Muralidharan &g

Re: [VOTE] Apache Bahir 2.1.0 (RC1)

2017-02-07 Thread Mridul Muralidharan
The tests pass, and the signatures check out. I noticed that there are a few 'releaseBackup' files and 'release.properties' in the 'source distribution' which are not from our repo (probably some artifact of the build process ?). I am +1 assuming this is not an issue : have not noticed (or probabl

[jira] [Assigned] (SPARK-19244) Sort MemoryConsumers according to their memory usage when spilling

2017-02-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-19244: --- Assignee: Liang-Chi Hsieh > Sort MemoryConsumers according to their mem

[jira] [Resolved] (SPARK-19244) Sort MemoryConsumers according to their memory usage when spilling

2017-02-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-19244. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request

Re: welcoming Burak and Holden as committers

2017-01-25 Thread Mridul Muralidharan
Congratulations and welcome Holden and Burak ! Regards, Mridul On Tue, Jan 24, 2017 at 10:13 AM, Reynold Xin wrote: > Hi all, > > Burak and Holden have recently been elected as Apache Spark committers. > > Burak has been very active in a large number of areas in Spark, including > linear algebr

Re: [VOTE] Apache Bahir 2.0.2 (RC3)

2017-01-23 Thread Mridul Muralidharan
+1 Build succeeds and tests run fine from the tag. Signature's check out fine. Regards, Mridul On Fri, Jan 20, 2017 at 6:31 PM, Christian Kadner wrote: > Please vote to approve the third release candidate for Apache Bahir > version 2.0.2 based on Apache Spark 2.0.2. > > The difference between

Re: What about removing TaskContext#getPartitionId?

2017-01-14 Thread Mridul Muralidharan
Since TaskContext.getPartitionId is part of the public api, it cant be removed as user code can be depending on it (unless we go through a deprecation process for it). Regards, Mridul On Sat, Jan 14, 2017 at 2:02 AM, Jacek Laskowski wrote: > Hi, > > Just noticed that TaskContext#getPartitionId

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816026#comment-15816026 ] Mridul Muralidharan commented on SPARK-19143: - bq. You say "we ad

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814408#comment-15814408 ] Mridul Muralidharan commented on SPARK-19143: - As part of an unrelated

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-20 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15764876#comment-15764876 ] Mridul Muralidharan commented on SPARK-18886: - I am not sure wha

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756179#comment-15756179 ] Mridul Muralidharan commented on SPARK-18886: - - Delay 'usin

[jira] [Comment Edited] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752579#comment-15752579 ] Mridul Muralidharan edited comment on SPARK-18886 at 12/15/16 9:3

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752579#comment-15752579 ] Mridul Muralidharan commented on SPARK-18886: - [~imranr] For almost

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2016-12-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752426#comment-15752426 ] Mridul Muralidharan commented on SPARK-18886: - Spark scheduler ca

Re: [ANNOUNCE] New Apache Bahir Committer : Christian Kadner

2016-11-30 Thread Mridul Muralidharan
Congratulations Christian ! Regards, Mridul On Wed, Nov 30, 2016 at 5:17 PM, Luciano Resende wrote: > The Project Management Committee (PMC) for Apache Bahir > has invited Christian Kadner to become a committer and we are pleased > to announce that he has accepted. > > Christian has been a key c

[jira] [Commented] (SPARK-16554) Spark should kill executors when they are blacklisted

2016-11-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703851#comment-15703851 ] Mridul Muralidharan commented on SPARK-16554: - It would also be good i

[jira] [Updated] (SPARK-16759) Spark expose an API to pass in Caller Context into it

2016-11-11 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-16759: Assignee: Weiqing Yang > Spark expose an API to pass in Caller Context into

[jira] [Resolved] (SPARK-16759) Spark expose an API to pass in Caller Context into it

2016-11-11 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-16759. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request

[jira] [Closed] (SPARK-18250) Minor issues with utf8string observed

2016-11-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-18250. --- Resolution: Later > Minor issues with utf8string obser

[jira] [Updated] (SPARK-18250) Minor issues with utf8string observed

2016-11-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-18250: Description: A few minor issues observed with utf8 which did not match the

[jira] [Created] (SPARK-18250) Minor issues with utf8string observed

2016-11-03 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-18250: --- Summary: Minor issues with utf8string observed Key: SPARK-18250 URL: https://issues.apache.org/jira/browse/SPARK-18250 Project: Spark Issue

[jira] [Closed] (SPARK-18008) Support skipping test compilation

2016-11-01 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-18008. --- Resolution: Won't Fix Closing jira due to pushback in commiting it. See p

Re: [VOTE] Apache Bahir 2.0.1 (RC1)

2016-10-21 Thread Mridul Muralidharan
+1 for release. Tests pass. Signatures checkout. Great work everybody ! Regards, Mridul On Sat, Oct 15, 2016 at 11:16 AM, Luciano Resende wrote: > Please vote to approve the release of the following candidate as Apache > Bahir version 2.0.1 based on Apache Spark 2.0.1. > > The tag to be vote

Re: [ANNOUNCE] New Apache Bahir Committer : Robert Metzger

2016-10-19 Thread Mridul Muralidharan
Congratulations Robert ! Regards Mridul On Oct 19, 2016 2:54 PM, "Luciano Resende" wrote: > The Project Management Committee (PMC) for Apache Bahirhas invited > Robert Metzger to become a committer and we are pleased to announce > that he has accepted. > > Robert has been a key community member

[jira] [Updated] (SPARK-18008) Support skipping test compilation

2016-10-19 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-18008: Description: Add support for skipping compilation of test code through

[jira] [Updated] (SPARK-18008) Support skipping test compilation

2016-10-19 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-18008: Summary: Support skipping test compilation (was: Support test compilation and

[jira] [Created] (SPARK-18008) Support test compilation and skipping javadoc generation

2016-10-19 Thread Mridul Muralidharan (JIRA)
Mridul Muralidharan created SPARK-18008: --- Summary: Support test compilation and skipping javadoc generation Key: SPARK-18008 URL: https://issues.apache.org/jira/browse/SPARK-18008 Project: Spark

[jira] [Updated] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-17637: Affects Version/s: 2.1.0 > Packed scheduling for Spark tasks across execut

[jira] [Resolved] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-17637. - Resolution: Fixed Assignee: Zhan Zhang Target Version/s

[jira] [Updated] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-17637: Fix Version/s: 2.1.0 > Packed scheduling for Spark tasks across execut

Re: Reading from and writing to different S3 buckets in spark

2016-10-12 Thread Mridul Muralidharan
If using RDD's, you can use saveAsHadoopFile or saveAsNewAPIHadoopFile with the conf passed in which overrides the keys you need. For example, you can do : val saveConf = new Configuration(sc.hadoopConfiguration) // configure saveConf with overridden s3 config rdd.saveAsNewAPIHadoopFile(..., conf

[jira] [Commented] (SPARK-17064) Reconsider spark.job.interruptOnCancel

2016-10-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547073#comment-15547073 ] Mridul Muralidharan commented on SPARK-17064: - I agree, interrupt&#

[jira] [Closed] (SPARK-7313) Allow for configuring max_samples in range partitioner.

2016-10-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-7313. -- Resolution: Won't Fix Not sure if still relevant, I opened it for an earlier pr

Edit access for spark confluence wiki

2016-10-04 Thread Mridul Muralidharan
Can someone add me to edit list for the spark wiki please ? Thanks, Mridul - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.0.1 (RC4)

2016-09-29 Thread Mridul Muralidharan
+1 Regards, Mridul On Wed, Sep 28, 2016 at 7:14 PM, Reynold Xin wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.0.1. The vote is open until Sat, Oct 1, 2016 at 20:00 PDT and passes if a > majority of at least 3+1 PMC votes are cast. > > [ ] +1 Release this p

Re: What's the meaning when the partitions is zero?

2016-09-16 Thread Mridul Muralidharan
When numPartitions is 0, there is no data in the rdd: so getPartition is never invoked. - Mridul On Friday, September 16, 2016, WangJianfei wrote: > if so, we will get exception when the numPartitions is 0. > def getPartition(key: Any): Int = key match { > case null => 0 > //case None

Re: Removing published kinesis, ganglia artifacts due to license issues?

2016-09-07 Thread Mridul Muralidharan
It is good to get clarification, but the way I read it, the issue is whether we publish it as official Apache artifacts (in maven, etc). Users can of course build it directly (and we can make it easy to do so) - as they are explicitly agreeing to additional licenses. Regards Mridul On Wednesday

Re: Removing published kinesis, ganglia artifacts due to license issues?

2016-09-07 Thread Mridul Muralidharan
I agree, we should not be publishing both of them. Thanks for bringing this up ! Regards, Mridul On Wed, Sep 7, 2016 at 1:29 AM, Sean Owen wrote: > It's worth calling attention to: > > https://issues.apache.org/jira/browse/SPARK-17418 > https://issues.apache.org/jira/browse/SPARK-17422 > > It l

Re: Documentation for Bahir Flink extensions

2016-08-26 Thread Mridul Muralidharan
+1 sounds good. Regards Mridul On Friday, August 26, 2016, Luciano Resende wrote: > For the Spark extensions we currently have a README.md on each extension > project which helps users accessing github to easily find out how to use > the extension, and we also use that to "copy" to the website

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards, Mridul On Mon, Aug 15, 2016 at 2:19 PM, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > A

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards, Mridul On Mon, Aug 15, 2016 at 2:19 PM, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > A

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards, Mridul On Mon, Aug 15, 2016 at 2:19 PM, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > A

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards, Mridul On Mon, Aug 15, 2016 at 2:19 PM, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > A

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards Mridul On Monday, August 15, 2016, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > Akka S

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards Mridul On Monday, August 15, 2016, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > Akka S

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards Mridul On Monday, August 15, 2016, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > Akka S

Re: [ANNOUNCE] Apache Bahir 2.0.0

2016-08-15 Thread Mridul Muralidharan
Congratulations, great job everyone ! Regards Mridul On Monday, August 15, 2016, Luciano Resende wrote: > The Apache Bahir PMC is pleased to announce the release of Apache Bahir > 2.0.0 which is our first major release and provides the following > extensions for Apache Spark 2.0.0 : > > Akka S

Re: [VOTE] Apache Bahir 2.0.0 (RC1)

2016-08-10 Thread Mridul Muralidharan
Continuing with my earlier mail : Signatures check out +1 for release. If we do cut another RC, would be great to include Prashant's doc changes - but agree that we dont want to block release on that. Regards, Mridul On Tue, Aug 9, 2016 at 9:47 PM, Mridul Muralidharan wrote: > > &

Re: [VOTE] Apache Bahir 2.0.0 (RC1)

2016-08-09 Thread Mridul Muralidharan
On Tuesday, August 9, 2016, Luciano Resende wrote: > On Tuesday, August 9, 2016, Mridul Muralidharan > wrote: > > > - Built from source successfully. > > - Tests run successfully. > > - README.md still has "Coming Soon." for sections 'Online > &g

Re: [VOTE] Apache Bahir 2.0.0 (RC1)

2016-08-09 Thread Mridul Muralidharan
- Built from source successfully. - Tests run successfully. - README.md still has "Coming Soon." for sections 'Online Documentation' and 'A Note About Apache Spark Integration'. - md5 signatures validated - I did not find the key used to sign artifacts, so did not validate that. Queries: a) Do w

Re: rdd.distinct with Partitioner

2016-06-08 Thread Mridul Muralidharan
The example violates the basic contract of a Partitioner. It does make sense to take Partitioner as a param to distinct - though it is fairly trivial to simulate that in user code as well ... Regards Mridul On Wednesday, June 8, 2016, 汪洋 wrote: > Hi Alexander, > > I think it does not guarantee

Re: Welcoming Yanbo Liang as a committer

2016-06-03 Thread Mridul Muralidharan
Congratulations Yanbo ! Regards Mridul On Friday, June 3, 2016, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add Yanbo Liang as a committer. Yanbo has been a > super active contributor in many areas of MLlib. Please join me in > welcoming Yanbo! > > Matei > -

Re: Importing source code from Apache Spark

2016-06-01 Thread Mridul Muralidharan
of o.a.b code 'clean' ... This of course assumes a reasonable shim can be built. Definitely option two compared to clean separation Regards Mridul On Wednesday, June 1, 2016, Marcelo Vanzin wrote: > On Wed, Jun 1, 2016 at 10:44 AM, Mridul Muralidharan > wrote: > > Tha

Re: Importing source code from Apache Spark

2016-06-01 Thread Mridul Muralidharan
n move to folders you want >> >> Now, what kind of code layout do we want, so as to encourage a contrib/ >> tree? >> >> >> >> On 1 June 2016 at 07:27, Mridul Muralidharan wrote: >> >> > I agree with importing just the source >> >

Re: Importing source code from Apache Spark

2016-05-31 Thread Mridul Muralidharan
I agree with importing just the source Regards Mridul On Tuesday, May 31, 2016, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > I think source is fine by me for the first import - is there > a specific reason to have all the artifacts? > >

<    2   3   4   5   6   7   8   9   10   11   >