Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations  On Wed, 4 Oct, 2023, 8:52 pm huaxin gao, wrote: > Congratulations! > > On Wed, Oct 4, 2023 at 7:39 AM Chao Sun wrote: > >> Congratulations! >> >> On Wed, Oct 4, 2023 at 5:11 AM Jungtaek Lim >> wrote: >> >>> Congrats! >>> >>> 2023년 10월 4일 (수) 오후 5:04, yangjie01 님이 작성: >>>

Re: Resolves too old JIRAs as incomplete

+1 On Thu, May 20, 2021 at 7:08 PM Wenchen Fan wrote: > +1 > > On Thu, May 20, 2021 at 11:59 AM Dongjoon Hyun > wrote: > >> +1. >> >> Thank you, Takeshi. >> >> On Wed, May 19, 2021 at 7:49 PM Hyukjin Kwon wrote: >> >>> Yeah, I wanted to discuss this. I agree since 2.4.x became EOL >>> >>>

Re: Welcoming six new Apache Spark committers

Congratulations  all!! On Sat, Mar 27, 2021, 5:10 AM huaxin gao wrote: > Congratulations to you all!! > > On Fri, Mar 26, 2021 at 4:22 PM Yuming Wang wrote: > >> Congrats! >> >> On Sat, Mar 27, 2021 at 7:13 AM Takeshi Yamamuro >> wrote: >> >>> Congrats, all~ >>> >>> On Sat, Mar 27, 2021 at

Re: [VOTE] Release Spark 3.0.2 (RC1)

+1 On Tue, Feb 16, 2021 at 1:22 PM Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.0.2. > > The vote is open until February 19th 9AM (PST) and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this

Re: [VOTE] Release Spark 3.1.1 (RC1)

+1 On Tue, Jan 19, 2021 at 4:38 PM Yang,Jie(INF) wrote: > +1 > > > > *发件人**: *Gengliang Wang > *日期**: *2021年1月19日 星期二 下午3:04 > *收件人**: *Jungtaek Lim > *抄送**: *Yuming Wang , Hyukjin Kwon , > dev > *主题**: *Re: [VOTE] Release Spark 3.1.1 (RC1) > > > > +1 (non-binding) > > > > > > On Tue, Jan

Automatically mount the user-specific configurations on K8s cluster, using a config map.

to know, how the users of spark on k8s, propagate these configurations to the spark driver in cluster mode? and what if some configuration needs to be picked by executors (e.g. HDFS core-site.xml etc...) Please take a look at: https://github.com/apache/spark/pull/27735 Thanks, Prashant Sharma

Re: [VOTE][RESULT] Release Spark 2.4.7 (RC3)

You are welcome, Dongjoon - just did my part. Thanks to everyone who contributed in various ways. Thanks, On Sat, Sep 12, 2020 at 9:29 AM Dongjoon Hyun wrote: > Thank you, Prashant! > > Bests, > Dongjoon. > > On Fri, Sep 11, 2020 at 7:02 PM Prashant Sharma > wrote:

[ANNOUNCE] Announcing Apache Spark 2.4.7

to acknowledge all community members for contributing to this release. This release would not have been possible without you. Enjoy the release ! Thanks, Prashant Sharma

[VOTE][RESULT] Release Spark 2.4.7 (RC3)

The vote passes. Thanks to all who helped with the release! (* = binding) +1: - Sean Owen * - Wenchan Fan * - Dongjoon Hyun * - Mridul * +0: None -1: None

Re: [VOTE] Release Spark 2.4.7 (RC3)

Thanks again, looks like it works now. Please take a look. On Thu, Sep 10, 2020 at 11:42 AM Prashant Sharma wrote: > Hi Wenchen and Sean, > > Thanks for looking into this and all the details. > > I have now updated the key in those keyservers. Now, how do I refresh >

Re: [VOTE] Release Spark 2.4.7 (RC3)

tu.com:11371 >>>> >>>> >>>> On Wed, Sep 9, 2020 at 6:12 AM Mridul Muralidharan >>>> wrote: >>>> >>>>> >>>>> +1 >>>>> >>>>> Signatures, digests, etc check out fine. >>>>

[VOTE] Release Spark 2.4.7 (RC3)

Please vote on releasing the following candidate as Apache Spark version 2.4.7. The vote is open until Sep 11th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.7 [ ] -1 Do not release this package because

Re: Running K8s integration tests for changes in core?

> wrote: > >> Sounds good. In the meantime would folks committing things in core run >> the K8s PRB or run it locally? A second change this morning was committed >> that broke the K8s PR tests. >> >> On Tue, Aug 18, 2020 at 9:53 PM Prashant Sharma >> wr

Re: Running K8s integration tests for changes in core?

+1, we should enable. On Wed, Aug 19, 2020 at 9:18 AM Holden Karau wrote: > Hi Dev Folks, > > I was wondering how people feel about enabling the K8s PRB automatically > for all core changes? Sometimes I forget that a change might impact one of > the K8s integration tests since a bunch of them

Re: [VOTE] Release Spark 2.4.7 (RC1)

https://github.com/apache/spark/pull/29125 for on-going >>>> discussion for both 3.0/2.4. >>>> >>>> [SPARK-32018][SQL][3.0] UnsafeRow.setDecimal should set null with >>>> overflowed value >>>> >>>> I also confirmed that

[VOTE] Release Spark 2.4.7 (RC1)

Please vote on releasing the following candidate as Apache Spark version 2.4.7. The vote is open until Aug 9th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.7 [ ] -1 Do not release this package because ...

Re: spark-on-k8s is still experimental?

My thoughts are, External shuffle service is not a blocker for spark on k8s to be a production ready. Others may think otherwise, but there are other ways to enable auto scaling. External shuffle service feature will be useful for all the ways of deployments be it yarn, standalone, k8s and not

Re: [VOTE] Update the committer guidelines to clarify when to commit changes.

+1 On Fri, Jul 31, 2020 at 10:18 PM Xiao Li wrote: > +1 > > Xiao > > On Fri, Jul 31, 2020 at 9:32 AM Mridul Muralidharan > wrote: > >> >> +1 >> >> Thanks, >> Mridul >> >> On Thu, Jul 30, 2020 at 4:49 PM Holden Karau >> wrote: >> >>> Hi Spark Developers, >>> >>> After the discussion of the

Re: [DISCUSS] Apache Spark 3.0.1 Release

Hi Folks, So, I am back, and searched the JIRAS with target version as "2.4.7" and Resolved, found only 2 jiras. So, are we good to go, with just a couple of jiras fixed ? Shall I proceed with making a RC? Thanks, Prashant On Thu, Jul 2, 2020 at 5:23 PM Prashant Sharma wrote: &

Re: Welcoming some new Apache Spark committers

Congratulations all ! It's great to have such committed folks as committers. :) On Wed, Jul 15, 2020 at 9:24 AM Yi Wu wrote: > Congrats!! > > On Wed, Jul 15, 2020 at 8:02 AM Hyukjin Kwon wrote: > >> Congrats! >> >> 2020년 7월 15일 (수) 오전 7:56, Takeshi Yamamuro 님이 작성: >> >>> Congrats, all! >>> >>>

Re: [DISCUSS] Apache Spark 3.0.1 Release

gt; Could I get some input on the severity of this one that I found >>>> yesterday? If that’s a correctness issue, should it block this patch? Let >>>> me know under the ticket if there’s more info that I can provide to help. >>>> >> >>>> >>

Re: [DISCUSS] Apache Spark 3.0.1 Release

+1 for 3.0.1 release. I too can help out as release manager. On Thu, Jun 25, 2020 at 4:58 AM 郑瑞峰 wrote: > I volunteer to be a release manager of 3.0.1, if nobody is working on this. > > > -- 原始邮件 -- > *发件人:* "Gengliang Wang"; > *发送时间:* 2020年6月24日(星期三) 下午4:15 >

Re: [vote] Apache Spark 3.0 RC3

+1 On Sun, Jun 7, 2020 at 1:50 AM Reynold Xin wrote: > Apologies for the mistake. The vote is open till 11:59pm Pacific time on > Mon June 9th. > > On Sat, Jun 6, 2020 at 1:08 PM Reynold Xin wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 3.0.0. >> >>

Re: [VOTE] Release Spark 2.4.6 (RC8)

+1 Thanks! On Mon, Jun 1, 2020 at 10:50 AM Holden Karau wrote: > Yes thats correct, the release script needs a bit of work and it's > diverged a bit from 3.0 as well. I'll follow up with some more PRs in > addition to the current one I have. > > On Sun, May 31, 2020 at 10:08 PM Sean Owen

Re: [VOTE] Release Spark 2.4.6 (RC3)

+1, Looks good. Thank you for putting this together. On Sat, May 16, 2020 at 10:38 AM Holden Karau wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.6. > > The vote is open until May 22nd at 9AM PST and passes if a majority +1 PMC > votes are cast, with a

Re: [VOTE] Release Spark 2.4.6 (RC1)

Hi Holden, I am +1 on this release, the fix for SPARK-31663 can make it to next release as well. Thanks, On Tue, May 12, 2020 at 8:09 PM Holden Karau wrote: > Thanks. The 2.4.6 RC1 vote fails because we don’t have enough binding +1s, > I’ll start a new RC once 31663 is merged or next week

[DISCUSS][K8s] Copy files securely to the pods or containers.

Hello All, The issue SPARK-23153 lets us copy any file to the pod/container, by first copying it to a hadoop supported filesystem e.g. HDFS, s3, cos etc. This is especially useful if, the files have to be copied to large number of pods/nodes.

Re: [DISCUSS] Remove multiple workers on the same host support from Standalone backend

It was by design, one could run multiple workers on his laptop for trying out or testing spark in distributed mode, one could launch multiple workers and see how resource offers and requirements work. Certainly, I have not commonly seen, starting multiple workers on the same node as a practice so

Re: [DISCUSS] Shall we mark spark streaming component as deprecated.

I may have speculated, or believed the unauthorised sources, nevertheless I am happy to be corrected. On Mon, Mar 2, 2020 at 8:05 PM Sean Owen wrote: > Er, who says it's deprecated? I have never heard anything like that. > Why would it be? > > On Mon, Mar 2, 2020 at 4:52 AM Pra

[DISCUSS] Shall we mark spark streaming component as deprecated.

Hi All, It is noticed that some of the users of Spark streaming do not immediately realise that it is a deprecated component and it would be scary, if they end up with it in production. Now that we are in a position to release about Spark 3.0.0, may be we should discuss - should the spark

Re: Kafka Spark structured streaming latency benchmark.

This issue was fixed in https://issues.apache.org/jira/browse/SPARK-18991. --Prashant On Tue, Dec 20, 2016 at 6:16 PM, Prashant Sharma <scrapco...@gmail.com> wrote: > Hi Shixiong, > > Thanks for taking a look, I am trying to run and see if making > ContextCleaner r

Re: Task not Serializable Exception

Can you minimize the code snippet with which we can get this `NotSerializableException` exception? Thanks, - Prashant Sharma Spark Technology Center http://www.spark.tc/ -- On Sun, Jan 1, 2017 at 9:36 AM, khyati <khyati.s...@guavus.com> wrote: > Getting error for the follo

Re: Kafka Spark structured streaming latency benchmark.

atabricks.com> wrote: > >> Hey, Prashant. Could you track the GC root of byte arrays in the heap? >> >> On Sat, Dec 17, 2016 at 10:04 PM, Prashant Sharma <scrapco...@gmail.com> >> wrote: >> >>> Furthermore, I ran the same thing with 26 GB as t

Re: Kafka Spark structured streaming latency benchmark.

re than 13GB, and are not garbage collected. --Prashant On Sun, Dec 18, 2016 at 8:49 AM, Prashant Sharma <scrapco...@gmail.com> wrote: > Hi, > > Goal of my benchmark is to arrive at end to end latency lower than 100ms > and sustain them over time, by consuming from a kafka topic and writing >

Re: [VOTE] Apache Spark 2.1.0 (RC2)

I am getting 404 for Link https://repository.apache.org/content/repositories/orgapachespark-1217. --Prashant On Fri, Dec 9, 2016 at 10:43 AM, Michael Allman wrote: > I believe https://github.com/apache/spark/pull/16122 needs to be included > in Spark 2.1. It's a simple

Re: Scala 2.11 default build

Yes, That should be changed to 2.11.7. Mind sending a patch ? Prashant Sharma On Thu, Feb 4, 2016 at 2:11 PM, zzc <441586...@qq.com> wrote: > hi, rxin, in pom.xml file, 'scala.version' still is 2.10.5, does it need > to > be modified to 2.11.7? > > > > -- >

Re: A proposal for Spark 2.0

1.7 I am +1 on the proposal for Spark 2.0. Thanks, Prashant Sharma On Thu, Nov 12, 2015 at 3:02 AM, Matei Zaharia <matei.zaha...@gmail.com> wrote: > I like the idea of popping out Tachyon to an optional component too to > reduce the number of dependencies. In the future, it migh

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

That is correct !, I have thought about this a lot of times. The only solution is to implement a "real" cross build for both version. I am going to think more in this. :) Prashant Sharma On Sat, Oct 10, 2015 at 2:04 AM, Patrick Wendell <pwend...@gmail.com> wrote: > I would

An alternate UI for Spark.

Hi all, TLDR; Some of my colleagues at Imaginea are interested in building an alternate UI for Spark. Basically allow people or groups to build an alternate UI for Spark. More Details: Looking at feasibility, it feels definitely possible to do. But we need a consensus on a public(can be

Re: Make off-heap store pluggable

+1 Looks like a nice idea(I do not see any harm). Would you like to work on the patch to support it ? Prashant Sharma On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk alexey.goncha...@gmail.com wrote: Hello Spark community, I was looking through the code in order to understand better how

Re: Speeding up Spark build during development

Hi Pramod, If you are using sbt as your build, then you need to do sbt assembly once and use sbt ~compile. Also export SPARK_PREPEND_CLASSES=1 this in your shell and all nodes. You can may be try this out ? Thanks, Prashant Sharma On Fri, May 1, 2015 at 2:16 PM, Pramod Biligiri pramodbilig

Re: Semantics of LGTM

Patrick's original proposal LGTM :). However until now, I have been in the impression of LGTM with special emphasis on TM part. That said, I will be okay/happy(or Responsible ) for the patch, if it goes in. Prashant Sharma On Sun, Jan 18, 2015 at 2:33 PM, Reynold Xin r...@databricks.com wrote

Re: sbt publish-local fails, missing spark-network-common

Can you update to latest master and see if this issue exists. On Nov 21, 2014 10:58 PM, pedrorodriguez ski.rodrig...@gmail.com wrote: Haven't found one yet, but work in AMPlab/at ampcamp so I will see if I can find someone who would know more about this (maybe reynold since he rolled out

Re: [NOTICE] [BUILD] Minor changes to Spark's build

For scala 2.11.4, there are minor changes needed in repl code. I can do that if that is a high priority. Prashant Sharma On Thu, Nov 13, 2014 at 11:59 AM, Prashant Sharma scrapco...@gmail.com wrote: Thanks Patrick, I have one suggestion that we should make passing -Pscala-2.10 mandatory

Re: [NOTICE] [BUILD] Minor changes to Spark's build

One thing we can do it is print a helpful error and break. I don't know about how this can be done, but since now I can write groovy inside maven build so we have more control. (Yay!!) Prashant Sharma On Thu, Nov 13, 2014 at 12:05 PM, Patrick Wendell pwend...@gmail.com wrote: Yeah Sandy

Re: [VOTE] Designating maintainers for some Spark components

+1, Sounds good. Now I know whom to ping for what, even if I did not follow the whole history of the project very carefully. Prashant Sharma On Thu, Nov 6, 2014 at 7:01 AM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, I wanted to share a discussion we've been having on the PMC

Re: [VOTE] Release Apache Spark 1.1.0 (RC3)

Easy or quicker way to build spark is sbt/sbt assembly/assembly Prashant Sharma On Mon, Sep 1, 2014 at 8:40 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: If this is not a confirmed regression from 1.0.2, I think it's better to report it in a separate thread or JIRA. I believe

Re: [SNAPSHOT] Snapshot1 of Spark 1.1.0 has been posted

Yeah this should be changed. You can change the banner in the repl, printWelcome function. Mind sending a PR ? I think this should be a one place change in the future (Not sure how feasible it is). Volunteers ? Prashant Sharma On Fri, Aug 8, 2014 at 12:48 PM, Debasish Das debasish.da

Re: Welcoming two new committers

Congratulations Andrew and Joey. Prashant Sharma On Fri, Aug 8, 2014 at 2:10 PM, Xiangrui Meng men...@gmail.com wrote: Congrats, Joey Andrew!! -Xiangrui On Fri, Aug 8, 2014 at 12:14 AM, Christopher Nguyen c...@adatao.com wrote: +1 Joey Andrew :) -- Christopher T. Nguyen Co

Re: FYI -- javax.servlet dependency issue workaround

libraryDependencies += org.mortbay.jetty % servlet-api % 3.0.20100224 *force()* force it. Thanks Prashant Sharma On Wed, May 28, 2014 at 10:38 AM, Kay Ousterhout k...@eecs.berkeley.eduwrote: Hi all, I had some trouble compiling an application (Shark) against Spark 1.0, where Shark had

Re: Apache spark on 27gb wikipedia data

I just thought may be we could put a warning whenever that error comes user can tune either memoryFraction or executor memory options. And this warning get's displayed when TaskSetManager receives task failures due to OOM. Prashant Sharma On Mon, May 5, 2014 at 2:10 PM, Ajay Nair prodig

Re: Fw: Is there any way to make a quick test on some pre-commit code?

Not sure but I use sbt/sbt ~compile instead of package. Any reason we use package instead of compile(which is slightly faster ofc.) Prashant Sharma On Thu, Apr 24, 2014 at 1:32 PM, Patrick Wendell pwend...@gmail.com wrote: This is already on the wiki: https://cwiki.apache.org/confluence

Re: Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1

I think we should upgrade sbt, I have been using sbt since 13.2-M1 and have not spotted any issues. So RC1 should be good + it has the fast incremental compilation. Prashant Sharma On Wed, Mar 26, 2014 at 10:41 AM, Will Benton wi...@redhat.com wrote: - Original Message - At last, I