[jira] [Resolved] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

[ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32663. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue

[jira] [Resolved] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars

[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32119. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request

Re: LiveListenerBus is occupying most of the Driver Memory and frequent GC is degrading the performance

Hi, 50% of driver time being spent in gc just for listenerbus sounds very high in a 30G heap. Did you try to take a heap dump and see what is occupying so much memory ? This will help us eliminate if the memory usage is due to some user code/library holding references to large objects/graph of

Re: [VOTE] Update the committer guidelines to clarify when to commit changes.

+1 Thanks, Mridul On Thu, Jul 30, 2020 at 4:49 PM Holden Karau wrote: > Hi Spark Developers, > > After the discussion of the proposal to amend Spark committer guidelines, > it appears folks are generally in agreement on policy clarifications. (See >

Re: [DISCUSS] Apache Spark 3.0.1 Release

I agree, that would be a new feature; and unless compelling reason (like security concerns) would not qualify. Regards, Mridul On Wed, Jul 15, 2020 at 11:46 AM Wenchen Fan wrote: > Supporting Python 3.8.0 sounds like a new feature, and doesn't qualify a > backport. But I'm open to other

Re: [DISCUSS] Amend the commiter guidelines on the subject of -1s & how we expect PR discussion to be treated.

Thanks Holden, this version looks good to me. +1 Regards, Mridul On Thu, Jul 23, 2020 at 3:56 PM Imran Rashid wrote: > Sure, that sounds good to me. +1 > > On Wed, Jul 22, 2020 at 1:50 PM Holden Karau wrote: > >> >> >> On Wed, Jul 22, 2020 at 7:39 AM Imran Rashid < iras...@apache.org > >>

Re: Welcoming some new Apache Spark committers

Congratulations ! Regards, Mridul On Tue, Jul 14, 2020 at 12:37 PM Matei Zaharia wrote: > Hi all, > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > - Huaxin Gao > - Jungtaek Lim > - Dilip Biswal >

[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150729#comment-17150729 ] Mridul Muralidharan commented on SPARK-25594: - Given regression in functionality

[jira] [Resolved] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-25594. - Resolution: Won't Fix > OOM in long running applications even with UI disab

Re: [VOTE] Decommissioning SPIP

+1 Thanks, Mridul On Wed, Jul 1, 2020 at 6:36 PM Hyukjin Kwon wrote: > +1 > > 2020년 7월 2일 (목) 오전 10:08, Marcelo Vanzin 님이 작성: > >> I reviewed the docs and PRs from way before an SPIP was explicitly >> asked, so I'm comfortable with giving a +1 even if I haven't really >> fully read the new

Re: [DISCUSS][SPIP] Graceful Decommissioning

Thanks for shepherding this Holden ! I left a few comments, but overall it looks good to me. Regards, Mridul On Sat, Jun 27, 2020 at 9:34 PM Holden Karau wrote: > There’s been some comments & a few additions in the doc, but it seems like > the folks taking a look generally agree on the

Re: [ANNOUNCE] Apache Spark 3.0.0

Great job everyone ! Congratulations :-) Regards, Mridul On Thu, Jun 18, 2020 at 10:21 AM Reynold Xin wrote: > Hi all, > > Apache Spark 3.0.0 is the first release of the 3.x line. It builds on many > of the innovations from Spark 2.x, bringing new ideas as well as continuing > long-term

Re: [ANNOUNCE] Apache Spark 3.0.0

Great job everyone ! Congratulations :-) Regards, Mridul On Thu, Jun 18, 2020 at 10:21 AM Reynold Xin wrote: > Hi all, > > Apache Spark 3.0.0 is the first release of the 3.x line. It builds on many > of the innovations from Spark 2.x, bringing new ideas as well as continuing > long-term

Re: [vote] Apache Spark 3.0 RC3

+1 Regards, Mridul On Sat, Jun 6, 2020 at 1:20 PM Reynold Xin wrote: > Apologies for the mistake. The vote is open till 11:59pm Pacific time on > Mon June 9th. > > On Sat, Jun 6, 2020 at 1:08 PM Reynold Xin wrote: > >> Please vote on releasing the following candidate as Apache Spark version

Re: [VOTE] Release Spark 2.4.6 (RC8)

Is this a behavior change in 2.4.x from earlier version ? Or are we proposing to introduce a functionality to help with adoption ? Regards, Mridul On Wed, Jun 3, 2020 at 10:32 AM Xiao Li wrote: > Yes. Spark 3.0 RC2 works well. > > I think the current behavior in Spark 2.4 affects the

Re: [VOTE] Release Spark 2.4.6 (RC8)

+1 (binding) Thanks, Mridul On Sun, May 31, 2020 at 4:47 PM Holden Karau wrote: > Please vote on releasing the following candidate as Apache Spark > version 2.4.6. > > The vote is open until June 5th at 9AM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. >

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086124#comment-17086124 ] Mridul Muralidharan commented on SPARK-29302: - I agree with [~feiwang], it looks like

[jira] [Issue Comment Deleted] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-29302: Comment: was deleted (was: Drive by observations: * Speculative execution does

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086117#comment-17086117 ] Mridul Muralidharan commented on SPARK-29302: - Drive by observations: * Speculative

Re: [DISCUSS] filling affected versions on JIRA issue

I agree with what Sean detailed. The only place where I can see some amount of investigation being required would be for security issues or correctness issues. Knowing the affected versions, particularly if an earlier supported version does not have the bug, will help users understand the

Re: [VOTE] Amend Spark's Semantic Versioning Policy

I am in broad agreement with the prposal, as any developer, I prefer stable well designed API's :-) Can we tie the proposal to stability guarantees given by spark and reasonable expectation from users ? In my opinion, an unstable or evolving could change - while an experimental api which has been

Re: Is RDD thread safe?

Very well put Imran. This is a variant of executor failure after an RDD has been computed (including caching). In general, non determinism in spark is going to lead to inconsistency. The only reasonable solution for us, at that time, was to make pseudo-randomness repeatable and checkpoint after so

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Just for completeness sake, spark is not version neutral to hadoop; particularly in yarn mode, there is a minimum version requirement (though fairly generous I believe). I agree with Steve, it is a long standing pain that we are bundling a positively ancient version of hive. Having said that, we

Re: [DISCUSS] Preferred approach on dealing with SPARK-29322

Makes more sense to drop support for zstd assuming the fix is not something at spark end (configuration, etc). Does not make sense to try to detect deadlock in codec. Regards, Mridul On Tue, Oct 1, 2019 at 8:39 PM Jungtaek Lim wrote: > > Hi devs, > > I've discovered an issue with event logger,

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

Add a +1 from me as well. Just managed to finish going over it. Thanks Bobby for leading this effort ! Regards, Mridul On Wed, May 29, 2019 at 2:51 PM Tom Graves wrote: > > Ok, I'm going to call this vote and send the result email. We had 9 +1's (4 > binding) and 1 +0 and no -1's. > > Tom > >

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

Unfortunately I do not have bandwidth to do a detailed review, but a few things come to mind after a quick read: - While it might be tactically beneficial to align with existing implementation, a clean design which does not tie into existing shuffle implementation would be preferable (if it can

Re: [VOTE] Functional DataSourceV2 in Spark 3.0

I am -1 on this vote for pretty much all the reasons that Mark mentioned. A major version change gives us an opportunity to remove deprecated interfaces, stabilize experimental/developer api, drop support for outdated functionality/platforms and evolve the project with a vision for foreseeable

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750477#comment-16750477 ] Mridul Muralidharan commented on SPARK-26688: - If this is a legitimate usecase, we should

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748915#comment-16748915 ] Mridul Muralidharan commented on SPARK-26688: - What is the usecase for this ? As others have

Re: Automated formatting

Is this handling only scala or java as well ? Regards, Mridul On Thu, Nov 22, 2018 at 9:11 AM Cody Koeninger wrote: > Plugin invocation is ./build/mvn mvn-scalafmt_2.12:format > > It takes about 5 seconds, and errors out on the first different file > that doesn't match formatting. > > I made a

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651301#comment-16651301 ] Mridul Muralidharan commented on SPARK-25732: - [~vanzin] With long running applications

[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled

[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635122#comment-16635122 ] Mridul Muralidharan commented on SPARK-25594: - Task level information is required only when

[jira] [Created] (SPARK-25594) OOM in long running applications even with UI disabled

Mridul Muralidharan created SPARK-25594: --- Summary: OOM in long running applications even with UI disabled Key: SPARK-25594 URL: https://issues.apache.org/jira/browse/SPARK-25594 Project: Spark

Re: data source api v2 refactoring

Is it only me or are all others getting Wenchen’s mails ? (Obviously Ryan did :-) ) I did not see it in the mail thread I received or in archives ... [1] Wondering which othersenderswere getting dropped (if yes). Regards Mridul [1]

Re: SPIP: Executor Plugin (SPARK-24918)

+1 I left a couple of comments in NiharS's PR, but this is very useful to have in spark ! Regards, Mridul On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid wrote: > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging >

[jira] [Resolved] (SPARK-24948) SHS filters wrongly some applications due to permission check

[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24948. - Resolution: Fixed > SHS filters wrongly some applications due to permiss

[jira] [Updated] (SPARK-24948) SHS filters wrongly some applications due to permission check

[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24948: Fix Version/s: 2.4.0 > SHS filters wrongly some applications due to permiss

Re: Set up Scala 2.12 test build in Jenkins

o non-serializable objects etc. > In all these cases you know you are adding references you shouldn't. > If users were used to another UX we can try fix it, not sure how well this > worked in the past though and if covered all cases. > > Regards, > Stavros > > On Mon, Aug 6,

Re: Set up Scala 2.12 test build in Jenkins

I agree, we should not work around the testcase but rather understand and fix the root cause. Closure cleaner should have null'ed out the references and allowed it to be serialized. Regards, Mridul On Sun, Aug 5, 2018 at 8:38 PM Wenchen Fan wrote: > > It seems to me that the closure cleaner

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363 ] Mridul Muralidharan edited comment on SPARK-24375 at 8/5/18 4:42 AM

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363 ] Mridul Muralidharan commented on SPARK-24375: - {quote} We've thought hard on the issue

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568664#comment-16568664 ] Mridul Muralidharan commented on SPARK-24375: - {quote} It's not desired behavior to catch

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551198#comment-16551198 ] Mridul Muralidharan commented on SPARK-24615: - [~tgraves] This was indeed a recurring issue

[jira] [Commented] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536016#comment-16536016 ] Mridul Muralidharan commented on SPARK-24755: - Go for it - thanks [~hthuynh2] ! > Execu

[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost

[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Summary: Executor loss can cause task to not be resubmitted (was: Executor loss

[jira] [Updated] (SPARK-24755) Executor loss can cause task to be not resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost

[jira] [Created] (SPARK-24755) Executor loss can cause task to be not resubmitted

Mridul Muralidharan created SPARK-24755: --- Summary: Executor loss can cause task to be not resubmitted Key: SPARK-24755 URL: https://issues.apache.org/jira/browse/SPARK-24755 Project: Spark

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411 ] Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:17 PM

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411 ] Mridul Muralidharan commented on SPARK-24375: - [~jiangxb1987] A couple of comments based

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411 ] Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:15 PM

Re: time for Apache Spark 3.0?

I agree, I dont see pressing need for major version bump as well. Regards, Mridul On Fri, Jun 15, 2018 at 10:25 AM Mark Hamstra wrote: > > Changing major version numbers is not about new features or a vague notion > that it is time to do something that will be seen to be a significant >

Re: Hadoop 3 support

Specifically to run spark with hadoop 3 docker support, I have filed a few jira's tracked under [1]. Regards, Mridul [1] https://issues.apache.org/jira/browse/SPARK-23717 On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin wrote: > Does anybody know what needs to be done in order

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418029#comment-16418029 ] Mridul Muralidharan commented on YARN-7935: --- [~eyang] I think there is some confusion here. Spark

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415223#comment-16415223 ] Mridul Muralidharan commented on YARN-7935: --- [~eyang] Using YARN service to run spark AM

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machine hostname

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host machine

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machie hostname

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host machie

[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host name

[ https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23721: Summary: Enhance BlockManagerId to include container's underlying host name

[jira] [Created] (SPARK-23721) Use actual node's hostname for host and rack locality computation

Mridul Muralidharan created SPARK-23721: --- Summary: Use actual node's hostname for host and rack locality computation Key: SPARK-23721 URL: https://issues.apache.org/jira/browse/SPARK-23721

[jira] [Created] (SPARK-23720) Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support

Mridul Muralidharan created SPARK-23720: --- Summary: Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support Key: SPARK-23720 URL: https://issues.apache.org/jira/browse

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Documentation (was: Task) > Document using docker in host network

[jira] [Created] (SPARK-23719) Use correct hostname in non-host networking mode in hadoop 3 docker support

Mridul Muralidharan created SPARK-23719: --- Summary: Use correct hostname in non-host networking mode in hadoop 3 docker support Key: SPARK-23719 URL: https://issues.apache.org/jira/browse/SPARK-23719

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Improvement (was: Documentation) > Document using docker in h

[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Task (was: Improvement) > Document using docker in host network

[jira] [Created] (SPARK-23718) Document using docker in host networking mode in hadoop 3

Mridul Muralidharan created SPARK-23718: --- Summary: Document using docker in host networking mode in hadoop 3 Key: SPARK-23718 URL: https://issues.apache.org/jira/browse/SPARK-23718 Project

[jira] [Created] (SPARK-23717) Leverage docker support in Hadoop 3

Mridul Muralidharan created SPARK-23717: --- Summary: Leverage docker support in Hadoop 3 Key: SPARK-23717 URL: https://issues.apache.org/jira/browse/SPARK-23717 Project: Spark Issue Type

Re: Welcoming some new committers

Congratulations ! Regards, Mridul On Fri, Mar 2, 2018 at 2:41 PM, Matei Zaharia wrote: > Hi everyone, > > The Spark PMC has recently voted to add several new committers to the > project, based on their contributions to Spark 2.3 and other past work: > > - Anirudh

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373202#comment-16373202 ] Mridul Muralidharan commented on YARN-7935: --- Hi [~tgraves],    Initial expectation

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372320#comment-16372320 ] Mridul Muralidharan commented on YARN-7935: --- The hostname executor's bind port(s) to and share

Re: [Core][Suggestion] sortWithinPartitions and aggregateWithinPartitions for RDD

On Wed, Jan 31, 2018 at 1:15 AM, Ruifeng Zheng wrote: > HI all: > > > >1, Dataset API supports operation “sortWithinPartitions”, but in RDD > API there is no counterpart (I know there is > “repartitionAndSortWithinPartitions”, but I don’t want to repartition the >

Re: Kubernetes backend and docker images

We should definitely clean this up and make it the default, nicely done Marcelo ! Thanks, Mridul On Fri, Jan 5, 2018 at 5:06 PM Marcelo Vanzin wrote: > Hey all, especially those working on the k8s stuff. > > Currently we have 3 docker images that need to be built and

[jira] [Commented] (SPARK-22903) AlreadyBeingCreatedException in stage retry caused by wrong attemptNumber

[ https://issues.apache.org/jira/browse/SPARK-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306583#comment-16306583 ] Mridul Muralidharan commented on SPARK-22903: - [~imranr] I agree that SPARK-22162 has

[jira] [Resolved] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-22465. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request

[jira] [Updated] (SPARK-22866) Kubernetes dockerfile path needs update

[ https://issues.apache.org/jira/browse/SPARK-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-22866: Fix Version/s: (was: 3.0.0) 2.3.0 > Kubernetes dockerf

[jira] [Resolved] (SPARK-22866) Kubernetes dockerfile path needs update

[ https://issues.apache.org/jira/browse/SPARK-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-22866. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request

Re: Publishing official docker images for KubernetesSchedulerBackend

We do support running on Apache Mesos via docker images - so this would not be restricted to k8s. But unlike mesos support, which has other modes of running, I believe k8s support more heavily depends on availability of docker images. Regards, Mridul On Wed, Nov 29, 2017 at 8:56 AM, Sean Owen

[jira] [Assigned] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-21549: --- Assignee: Sergey Zhemzhitsky > Spark fails to complete job correc

[jira] [Resolved] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-21549. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue

Re: Should Flume integration be behind a profile?

I agree, proposal 1 sounds better among the options. Regards, Mridul On Sun, Oct 1, 2017 at 3:50 PM, Reynold Xin wrote: > Probably should do 1, and then it is an easier transition in 3.0. > > On Sun, Oct 1, 2017 at 1:28 AM Sean Owen wrote: >> >> I

Re: Welcoming Tejas Patil as a Spark committer

Congratulations Tejas ! Regards, Mridul On Fri, Sep 29, 2017 at 12:58 PM, Matei Zaharia wrote: > Hi all, > > The Spark PMC recently added Tejas Patil as a committer on the > project. Tejas has been contributing across several areas of Spark for > a while, focusing

Re: Should Flume integration be behind a profile?

Sounds good to me. +1 Regards, Mridul On Tue, Sep 26, 2017 at 2:36 AM, Sean Owen wrote: > Not a big deal, but I'm wondering whether Flume integration should at least > be opt-in and behind a profile? it still sees some use (at least on our end) > but not applicable to the

Re: [DISCUSS][bahir-flink] Drop Java 7 support

Thanks for clarifying Robert ! Given this, I am +1 on removing support too. Regards, Mridul On Mon, Sep 18, 2017 at 11:02 AM, Robert Metzger wrote: > Thanks all for the overwhelming support to drop Java7. > > @Mridul: I don't know for sure what the schedule for Flink 1.4

Re: [DISCUSS][bahir-flink] Drop Java 7 support

Do we know when Flink 1.4 is expected to be out ? Will we be making any intermediate Bahir release for flink before then ? If we are not making any, then dropping java 7 will have no impact for our users. On spark side, jdk 7 support has already been dropped. Regards, Mridul On Sat, Sep 16,

[jira] [Commented] (SPARK-9213) Improve regular expression performance (via joni)

[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145831#comment-16145831 ] Mridul Muralidharan commented on SPARK-9213: [~rxin] Curious what happened to this effort

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations Jerry, well deserved ! Regards, Mridul On Mon, Aug 28, 2017 at 6:28 PM, Matei Zaharia wrote: > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has > been contributing to many areas of the project for a long

Re: [VOTE] Apache Bahir 2.2.0 (RC1)

Build works fine, signatures check out, tests run successfully. +1 for release. Btw, I dont recall RAT check failing build earlier, very nice touch to enforce it as part of build ! Regards, Mridul On Wed, Aug 16, 2017 at 10:17 PM, Luciano Resende wrote: > Dear

Re: SPIP: Spark on Kubernetes

While I definitely support the idea of Apache Spark being able to leverage kubernetes, IMO it is better for long term evolution of spark to expose appropriate SPI such that this support need not necessarily live within Apache Spark code base. It will allow for multiple backends to evolve,

Re: Welcoming Hyukjin Kwon and Sameer Agarwal as committers

Congratulations Hyukjin, Sameer ! Regards, Mridul On Mon, Aug 7, 2017 at 8:53 AM, Matei Zaharia wrote: > Hi everyone, > > The Spark PMC recently voted to add Hyukjin Kwon and Sameer Agarwal as > committers. Join me in congratulating both of them and thanking them for

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:16 PM

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:14 PM

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105630#comment-16105630 ] Mridul Muralidharan commented on SPARK-21549: - This affects both mapred ("mapred.outpu

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

; The issue reported was for Spark extensions >> https://issues.apache.org/jira/browse/BAHIR-88 >> >> >> And I guess it is also applicable for Flink extensions >> >> >> But the issue is still open. >> >> >> On Tue, May 23, 2017 at 12:08 PM, Mrid

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

n, May 21, 2017 at 11:50 PM, Mridul Muralidharan <mridul...@apache.org> > wrote: > >> Hi, >> >> From the release tag, clean build and tests work fine, signatures match. >> >> The source at https://dist.apache.org/repos/dist/dev/bahir/bahir-flink/1. >

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC5)

Hi, From the release tag, clean build and tests work fine, signatures match. The source at https://dist.apache.org/repos/dist/dev/bahir/bahir-flink/1.0-rc5/ has additional files which are not in repo: Not in repo ./distribution/pom.xml.releaseBackup Not in repo

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995550#comment-15995550 ] Mridul Muralidharan commented on SPARK-20589: - coalasce with shuffle=false might

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985575#comment-15985575 ] Mridul Muralidharan commented on SPARK-20480: - Shouldn't fix for SPARK-19276 by [~imranr

Re: [VOTE] Apache Bahir Flink extensions 1.0 (RC3)

+1 for release. Tag checks out fine - full build and tests go through fine. Signatures validated. Regards, Mridul On Thu, Apr 6, 2017 at 4:08 PM, Luciano Resende wrote: > Please vote to approve the release of the following candidate as Apache > Bahir Flink extensions 1.0

Re: [VOTE] Apache Spark 2.1.1 (RC2)

Hi, https://issues.apache.org/jira/browse/SPARK-20202?jql=priority%20%3D%20Blocker%20AND%20affectedVersion%20%3D%20%222.1.1%22%20and%20project%3D%22spark%22 Indicates there is another blocker (SPARK-20197 should have come in the list too, but was marked major). Regards, Mridul On Tue, Apr 4,

[jira] [Comment Edited] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954387#comment-15954387 ] Mridul Muralidharan edited comment on SPARK-20205 at 4/4/17 12:15 AM

[jira] [Commented] (SPARK-20205) DAGScheduler posts SparkListenerStageSubmitted before updating stage

[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954387#comment-15954387 ] Mridul Muralidharan commented on SPARK-20205: - For history server that will fail - good point

<    1   2   3   4   5   6   7   8   9   10   >