Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Hi Jay As far as I am aware in Spark 2.4.4, there is no feature to enable executor decommissioning with graceful shutdown, nor is there a way to specify a timeout for forcefully killing executors. These were introduced in Spark 3.0. HTH Mich Talebzadeh, Architect | Data Engineer | Data Scienc

Re: [DISCUSS] Support spark.ml on Spark Connect

Thank you for working on this! Xiao Martin Grund 于2024年10月10日周四 03:01写道： > > Hi Bobby, > > Awesome to see the proposal! I'm very much looking forward to the > contributions! > > Martin > > On Thu, Oct 10, 2024 at 4:16 AM Ángel > wrote: > >> You have my vote (btw, great idea, ML is so sexy now

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Thank you for your prompt reply. By the way, I am utilizing Spark 2.4.4. My question is as the following: When the Spark driver tries to remove executors from Kubernetes, it invokes pods().delete() without specifying a grace period, regardless of whether it's due to job failure or success. If this

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

Didn't know about that. I'll have a look at it and check whether fix the issue or not. Thanks El jue, 10 oct 2024, 13:29, Wenchen Fan escribió: > There is a `try_to_timestamp` function but not `try_to_date`, we should > probably add it for users who don't want to get runtime errors when > proces

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

Oh sorry, I misunderstood the issue. It's not about the user-facing error, but the performance issue when the `to_date` function deals with invalid date strings. This is unfortunately not easy to fix, as Spark relies on the JDK library to parse datetime strings, and we can only know if a string is

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

There is a `try_to_timestamp` function but not `try_to_date`, we should probably add it for users who don't want to get runtime errors when processing big dataset. On Thu, Oct 10, 2024 at 11:05 AM Ángel wrote: > Hi, > > I opened a Jira ticket back in August, but it seems to have been > overlooke

Re: [DISCUSS] Support spark.ml on Spark Connect

Hi Bobby, Awesome to see the proposal! I'm very much looking forward to the contributions! Martin On Thu, Oct 10, 2024 at 4:16 AM Ángel wrote: > You have my vote (btw, great idea, ML is so sexy nowadays 😉) > > El jue, 10 oct 2024 a las 3:19, Bobby () escribió: > >> Hi, >> >> I'd like to start

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

to be clear are you referring to these spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true thanks Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD Imperial College London <

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Do you know by any chance if that config also applies to Databricks? El jue, 10 oct 2024 a las 10:02, Ángel () escribió: > Thanks a lot for the clarification. Interesting... I've never needed it, > even though I've been using Spark for over 8 years. > > El jue, 10 oct 2024 a las 9:21, Liu Cao ()

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Thanks a lot for the clarification. Interesting... I've never needed it, even though I've been using Spark for over 8 years. El jue, 10 oct 2024 a las 9:21, Liu Cao () escribió: > I’m unclear on what the exact issue the OP ran into. > > But if we are talking about decommission, just one side note

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

I’m unclear on what the exact issue the OP ran into. But if we are talking about decommission, just one side note: The decommission feature has been in spark for a while, and deco

Re: Dev list policy on posting genAI hallucinations

Thanks to the committers and PMC members who chimed in on this thread. > On Oct 10, 2024, at 3:27 AM, Jungtaek Lim > wrote: > > I'd ask people to quote the part you got from GPT and explicitly call out the > part as "GPT-generated" so that people would consider that there is > hallucination.

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

The Synapse config "spark.yarn.executor.decommission.enabled" is the closest thing to the proposed config "spark.executor.decommission.enabled" we've seen so far, I was only remarking that. On the other hand, seems like the "decomission" config came out in Spark 3.4.0: https://archive.apache.org

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Ángel, https://spark.apache.org/docs/latest/configuration.html search through `decommission` in this page. The config you may have found from Synapse is spark."yarn".executor.decommission.enabled. And the question was even "k8s" and none of the information for the vendor was mentioned from the qu

Re: [DISCUSS] Support spark.ml on Spark Connect

You have my vote (btw, great idea, ML is so sexy nowadays 😉) El jue, 10 oct 2024 a las 3:19, Bobby () escribió: > Hi, > > I'd like to start a discussion about support spark.ml on Connect. With > this feature, Users don't need to change their code to run Spark ML cases > on Connect. > > Please re

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Looks like it actually exists ... but only for the Spark Synapse implementation ... https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable Jay Han was asking for some config on k8s, so we shouldn't bring this config to the table, shoul

Re: Dev list policy on posting genAI hallucinations

minor clarification: content from GPT might be valuable - what I meant was, the value of content from GPT "in the community". The content is either something people in the thread knows, or something they can query to GPT on their own. Though I barely use GPT for the query which matters with experti

Re: Dev list policy on posting genAI hallucinations

+1 on this. I'd ask people to quote the part you got from GPT and explicitly call out the part as "GPT-generated" so that people would consider that there is hallucination. Do not ever expect that people wouldn't know the content GPT wrote. "People know." ASF requires every code contribution to e

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Mich: you can set any key-value pair you want in Spark config. It doesn't mean it is a real flag that code reads. spark.conf.set("ham", "sandwich") print(spark.conf.get("ham")) prints "sandwich" forceKillTimeout is a real config: https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

sorry, a missed line spark = SparkSession.builder \ .appName("Verifying Spark Configurations") \ .config("spark.executor.decommission.enabled", "true") \ *.config("spark.executor.decommission.gracefulShutdown", "true")* \ .config("spark.executor.decommission.forceKillTimeout", "100

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Let us take this for a ride using these so called non-existent configuration settings spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true Tested on Spark 3.4 from pyspark.sql import SparkSession # Initialize a Spark session spark = SparkSession.builder \

Re: Dev list policy on posting genAI hallucinations

Agree, I can't really explain this post except as AI hallucination, because: - those configs don't exist and it's not a simple typo away from a real one - they are kind of like unrelated real Spark config names and the kind of thing it seems an AI would 'infer' - no claim it was a typo with plausi

Re: Dev list policy on posting genAI hallucinations

FWIW - Mich - I've often found your responses "gpt" like and can often be a distraction. Now I don't know if that's your actual writing style or you were indeed using genai tools to generate the responses on your behalf. I don't think we should sanction you if that's your writing style. But if you

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Do you have a better recommendation? Or trying to waste time as usual. It is far easier to throw than catch. Do your homework and stop throwing spanners at work. Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Mich, Can you please share with the list where exactly you are citing these configs from? As far as I can tell, these two configs don’t exist and have never existed in the Spark codebase: spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true Where exactly

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

spark.executor.decommission.gracefulShutdown=true WHAT? ons. 9. okt. 2024 kl. 12:30 skrev Mich Talebzadeh : > Before responding, what configuration parameters are you using to make > this work? > > spark.executor.decommission.enabled=true > spark.executor.decommission.gracefulShutdown=true > spa

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Before responding, what configuration parameters are you using to make this work? spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true spark.executor.decommission.forceKillTimeout=100s HTH Mich Talebzadeh, Architect | Data Engineer | Data Science | Financia

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

>> >>>> On Mon, Oct 7, 2024 at 3:25 PM Russell Jurney >>>> wrote: >>>> >>>>> I volunteer to maintain GraphX to keep GraphFrames a viable project. I >>>>> don’t have a clear view on whether it works with Spark 4 or if it needs >>

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

ble project. I >>>>> don’t have a clear view on whether it works with Spark 4 or if it needs >>>>> updates? I don’t have Spark commits but I’m a committer on Apache DataFu >>>>> and mentored the Spark feature for it. >>>>> >>>&

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

have Spark commits but I’m a committer on Apache DataFu >>>> and mentored the Spark feature for it. >>>> >>>> Can someone tell me what is involved? Point me at a ticket? >>>> >>>> Russell >>>> >>>> On Mon, Oct 7, 202

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

>> >>>> Hello, >>>> We rely on GraphX for an important component of our product. And we >>>> really want it to stay a typed interface. Please keep GraphX. >>>> >>>> >>>> Erik >>>> >>>>

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

t 7, 2024 at 12:11 AM Erik Eklund >> wrote: >> >>> Hello, >>> We rely on GraphX for an important component of our product. And we >>> really want it to stay a typed interface. Please keep GraphX. >>> >>> >>> Erik >>> >>>

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

The VOTE is now closed and I believe it passes*, albeit with notable dissent. The votes are: +1: Mich Talebzadeh Dongjoon Hyun LC Hsieh Sean Owen Jungtaek Lim Mridul Muralidharan Yang Jie beliefer Wenchen Fan Denny Lee Hyukjin Kwon Herman van Hovell 0s: -1: Mark Hamstra Ángel While not on th

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

face. Please keep GraphX. >> >> >> Erik >> >> >> >> *From: *Holden Karau >> *Date: *Sunday, October 6, 2024 at 06:22 >> *To: *Ángel >> *Cc: *Russell Jurney , Mich Talebzadeh < >> mich.talebza...@gmail.com>, Spark dev list , us

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

; *Date: *Sunday, October 6, 2024 at 06:22 > *To: *Ángel > *Cc: *Russell Jurney , Mich Talebzadeh < > mich.talebza...@gmail.com>, Spark dev list , user > @spark > *Subject: *Re: [DISCUSS] Deprecate GraphX OR Find new maintainers > interested in GraphX OR leave it as is? >

Re: How to run spark connect in kubernetes?

https://isitdns.com/ On Wed, 2 Oct 2024 at 22:45, kant kodali wrote: > please ignore this. it was a dns issue > > On Wed, Oct 2, 2024 at 11:16 AM kant kodali wrote: > >> Here >>

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 YouTube Live Streams: https://www.yo

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

So are there companies using it? And are they willing to contribute to maintaining it? Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ Books (Learning Spark, High Performance Spark, etc.): htt

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

That would definitely affect companies using GraphX, but at least they’d have the choice to migrate their code. I think that’s probably the way to go. El dom, 6 oct 2024 a las 6:09, Holden Karau () escribió: > So removing GraphX from Spark would not prevent GraphFrames from > continuing, they co

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

So removing GraphX from Spark would not prevent GraphFrames from continuing, they could pick up the GraphX source and incorporate it into their project. Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

A lot of people like me use GraphFrames for its connected components implementation and its motif matching feature. I am willing to work on it to keep it alive. They did a 0.8.3 release not too long ago. Please keep GraphX alive. On Sat, Oct 5, 2024 at 3:44 PM Mich Talebzadeh wrote: > I added th

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I added the user list as they may have vested interest here and and hopefully can contribute Few suggestions: 1. Data-Driven Decision Making: Return to the core metrics—analyze usage trends, performance benchmarks, and the actual impact on businesses that rely on GraphX. Objectivity can

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I'm not saying that deprecation necessarily precludes further contributions to the deprecated code. Without explicit and visible encouragement of further contributions, though, a deprecation label does actively discourage further contributions. That, then, raises the question of whether we do want

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I completely agree with everyone here. I don’t think the issue is deprecating it; to me, the problem lies in not providing a new and better solution for handling graphs in Spark. In the past, I used GraphX via GraphFrames for record linkage, and I found it both useful and effective. Is there any di

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Deprecation doesn't stop any of that though, if you want to encourage people to do something with GraphX. We can un-deprecate things. We don't have to remove deprecated things. But, why would we not encourage people to work on GraphFrames if interested in this domain? Nobody has been willing to c

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

As I wrote to Holden privately, I might well change my vote to be in favor of a deprecation label combined with some effective means of communicating that this doesn't mean the end for GraphX if interested contributors come forward to rescue it. I don't like either the idea of keeping unmaintained

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

This is a reasonable discussion, but maybe the more practical point is: are you sure you want to block this unilaterally? This effectively makes a decision that GraphX cannot be removed for a long while. I'd understand it more if we had an active maintainer and/or active user proposing to veto, but

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Personally, I would promote Best Practices given fair bit of opinions either way. - Share best practices for using GraphX effectively, including tips for optimizing performance and avoiding common pitfalls. - Encourage the use of alternative libraries when appropriate, highlighting the

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

"You can't say nothing is removable until there are no users." That is not what I am saying. Rather, I am countering what others seem to be suggesting: There are no users and no interest, therefore we can and should deprecate. On Fri, Oct 4, 2024 at 3:10 PM Sean Owen wrote: > > I could flip this

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Interesting, personally there are many use cases where I would recommend RDDs — definitely to more advanced users — and I think that RDDs and GraphX are in pretty different boats (RDDs are very actively used). Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealth

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I could flip this argument around. More strongly, *not* being deprecated means "won't be removed" and likewise implies support and development. I don't think either of the latter have been true for years. What suggests this will change? A todo list is not going to do anything, IMHO. I'm also conce

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

No, I wouldn't encourage anyone to base a new production deployment on GraphX, but neither would I encourage new production deployments based on the RDD API without deep study and understanding of the implications and limitations. What I would be most comfortable with is documenting the current sta

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Personally I think people should not depend on it — there’s literally no one working on it, and not being up front about that I think draws everything else into question. Would anyone here feel comfortable using GraphX for a new production deployment today? Twitter: https://twitter.com/holdenkar

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

-1(*) reasoning posted in the DISCUSS thread On Mon, Sep 30, 2024 at 12:40 PM Holden Karau wrote: > > I think it has been de-facto deprecated, we haven’t updated it meaningfully > in several years. I think removing the API would be excessive but deprecating > it would give us the flexibility to

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I'm -1(*) because, while it technically means "might be removed in the future", I think developers and users are more prone to interpret something being marked as deprecated as "very likely will be removed in the future, so don't depend on this or waste your time contributing to its further develop

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

The graphframes library depends on GraphX and has changed recently (3 months ago). https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/graphframes/GraphFrame.scala El vie, 4 oct 2024, 11:35, Nimrod Ofek escribió: > Hi, > > Did anyone do any search about the GraphX API i

Re: [VOTE][RESULT] Single-pass Analyzer for Catalyst

Looks like a missed a vote from Mich Talebzadeh Updated list, 15 +1s (8 bindings, * = binding): +1: Reynold Xin (*) Herman van Hovell (*) Dongjoon Hyun (*) Xiao Li (*) Jungtaek Lim Gengliang Wang (*) John Zhuge Yang Jie Mridul Muralidharan (*) Peter Toth Wenchen Fan (*) L. C. Hsieh (*) Angel Alva

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

FWIW, this is a Google Search trend in last 5 years: [image: Screenshot 2024-10-04 at 6.54.42 PM.png] I think it's fine to deprecate it On Fri, 4 Oct 2024 at 18:40, Nimrod Ofek wrote: > Hi, > > Did anyone do any search about the GraphX API in Gitlab/Github and > different search engines to see

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

Hi, Did anyone do any search about the GraphX API in Gitlab/Github and different search engines to see if they are searched and actually used - or we are considering it not used because the API wasn't changed? Thanks! Nimrod On Mon, Sep 30, 2024 at 9:02 PM Holden Karau wrote: > I think it has

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 On Fri, 4 Oct 2024 at 07:14, Ángel wrote: > -1 Don’t deprecate GraphX because may be useful for some people and ... > would there be any replacement for that API? Anyway, I don't think > deprecating an API only because it hasn't been updated in ages is a good > practice (but I could be perfec

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 on the assumption that we should phase this release on an incremental basis. Probably will take us to end of release 5. HTH Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD Imperial College London

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 El jue, 3 oct 2024, 20:06, Wenchen Fan escribió: > +1 > > On Wed, Oct 2, 2024 at 7:50 AM Peter Toth wrote: > >> +1 >> >> On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: >> >>> +1, Thanks >>> >>> Jie Yang >>> >>> On 2024/10/01 03:26:40 John Zhuge wrote: >>> > +1 (non-binding) >>> > >>> > On Mon,

Re: [VOTE] Single-pass Analyzer for Catalyst

Hi Mich! Thank you for this input. Yes, this is exactly the approach I would propose too. Putting the new analyzer under a flag and making the tests pass for both implementations is crucial. We need to compare the logical (analyzed) plans and ensure that they are identical. On Thu, Oct 3, 2024 at

Re: [VOTE] Single-pass Analyzer for Catalyst

With a project manager hat on and having read the SPIP This proposed single-pass Analyzer framework does potentially offer significant long-term benefits in terms of efficiency, maintenance, and stability, especially for large or complex queries.

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

-1 Don’t deprecate GraphX because may be useful for some people and ... would there be any replacement for that API? Anyway, I don't think deprecating an API only because it hasn't been updated in ages is a good practice (but I could be perfectly wrong). El jue, 3 oct 2024, 16:31, Wenchen Fan esc

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Thu, Oct 3, 2024 at 7:31 AM Wenchen Fan wrote: > > +1 > > On Wed, Oct 2, 2024 at 7:50 AM Peter Toth wrote: >> >> +1 >> >> >> On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: >>> >>> +1, Thanks >>> >>> Jie Yang >>> >>> On 2024/10/01 03:26:40 John Zhuge wrote: >>> > +1 (non-binding) >>> > >>> > O

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 (non-binding) On Thu, Oct 3, 2024 at 7:40 AM Wenchen Fan wrote: > +1 > > On Tue, Oct 1, 2024 at 4:20 PM beliefer wrote: > >> +1. >> >> I didn't hear users need it. >> >> >> At 2024-10-01 02:01:17, "Holden Karau" wrote: >> >> I think it has been de-facto deprecated, we haven’t updated it >>

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 On Tue, Oct 1, 2024 at 4:20 PM beliefer wrote: > +1. > > I didn't hear users need it. > > > At 2024-10-01 02:01:17, "Holden Karau" wrote: > > I think it has been de-facto deprecated, we haven’t updated it > meaningfully in several years. I think removing the API would be excessive > but depr

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Wed, Oct 2, 2024 at 7:50 AM Peter Toth wrote: > +1 > > On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: > >> +1, Thanks >> >> Jie Yang >> >> On 2024/10/01 03:26:40 John Zhuge wrote: >> > +1 (non-binding) >> > >> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang >> > wrote: >> > >> > > +1 >> > >

Re: How to run spark connect in kubernetes?

please ignore this. it was a dns issue On Wed, Oct 2, 2024 at 11:16 AM kant kodali wrote: > Here > > are more details about my question that I posted in SO > > On Tue, Oc

Re: How to run spark connect in kubernetes?

Here are more details about my question that I posted in SO On Tue, Oct 1, 2024 at 11:32 PM kant kodali wrote: > Hi All, > > Is it possible to run a Spark Connect server

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: > +1, Thanks > > Jie Yang > > On 2024/10/01 03:26:40 John Zhuge wrote: > > +1 (non-binding) > > > > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang > > wrote: > > > > > +1 > > > > > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim < > kabhwan.opensou..

Re:[VOTE] Officialy Deprecate GraphX in Spark 4

+1. I didn't hear users need it. At 2024-10-01 02:01:17, "Holden Karau" wrote: I think it has been de-facto deprecated, we haven’t updated it meaningfully in several years. I think removing the API would be excessive but deprecating it would give us the flexibility to remove it in the not

Re: [VOTE] Single-pass Analyzer for Catalyst

+1, Thanks Jie Yang On 2024/10/01 03:26:40 John Zhuge wrote: > +1 (non-binding) > > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang > wrote: > > > +1 > > > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim > > wrote: > > > >> +1 (non-binding), promising proposal! > >> > >> 2024년 10월 1일 (화) 오전 8:0

Re: [VOTE] Single-pass Analyzer for Catalyst

+1, Thanks Jie Yang On 2024/10/01 03:26:40 John Zhuge wrote: > +1 (non-binding) > > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang > wrote: > > > +1 > > > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim > > wrote: > > > >> +1 (non-binding), promising proposal! > >> > >> 2024년 10월 1일 (화) 오전 8:0

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1, Thanks Jie Yang On 2024/10/01 02:53:11 Mridul Muralidharan wrote: > +1. > > Regards, > Mridul > > PS: In the past, I did have fun with graphx ... unfortunate that it has > come to this :-( > > > On Mon, Sep 30, 2024 at 6:23 PM Sean Owen wrote: > > > For reasons in the previous thread, y

Re: [VOTE] Single-pass Analyzer for Catalyst

Folks, thanks for voting! Yes, the target version was set by my mistake, Dongjoon. Sorry for that. Vladimir. On Tue, Oct 1, 2024, 05:28 John Zhuge wrote: > +1 (non-binding) > > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang > wrote: > >> +1 >> >> On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim <

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 (non-binding) On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang wrote: > +1 > > On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim > wrote: > >> +1 (non-binding), promising proposal! >> >> 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun 님이 작성: >> >>> Thank you for the swift clarification, Reynold and Xiao. >

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1. Regards, Mridul PS: In the past, I did have fun with graphx ... unfortunate that it has come to this :-( On Mon, Sep 30, 2024 at 6:23 PM Sean Owen wrote: > For reasons in the previous thread, yes +1 to deprecation > > On Mon, Sep 30, 2024 at 1:02 PM Holden Karau > wrote: > >> I think it

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 Regards, Mridul On Mon, Sep 30, 2024 at 5:39 PM Reynold Xin wrote: > I don't actually "lead" this. But I don't think this needs to target a > specific Spark version given it should not have any user facing > consequences? > > > On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun wrote: > >> Thank

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Mon, Sep 30, 2024 at 6:22 PM Jungtaek Lim wrote: > +1 (non-binding), promising proposal! > > 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun 님이 작성: > >> Thank you for the swift clarification, Reynold and Xiao. >> >> It seems that the Target Version was set mistakenly initially. >> >> I removed the

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 (non-binding) 2024년 10월 1일 (화) 오전 8:19, Sean Owen 님이 작성: > For reasons in the previous thread, yes +1 to deprecation > > On Mon, Sep 30, 2024 at 1:02 PM Holden Karau > wrote: > >> I think it has been de-facto deprecated, we haven’t updated it >> meaningfully in several years. I think removing

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 (non-binding), promising proposal! 2024년 10월 1일 (화) 오전 8:04, Dongjoon Hyun 님이 작성: > Thank you for the swift clarification, Reynold and Xiao. > > It seems that the Target Version was set mistakenly initially. > > I removed the `Target Version` from the SPIP JIRA. > > https://issues.apache.org/j

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

For reasons in the previous thread, yes +1 to deprecation On Mon, Sep 30, 2024 at 1:02 PM Holden Karau wrote: > I think it has been de-facto deprecated, we haven’t updated it > meaningfully in several years. I think removing the API would be excessive > but deprecating it would give us the flexi

Re: [VOTE] Single-pass Analyzer for Catalyst

Thank you for the swift clarification, Reynold and Xiao. It seems that the Target Version was set mistakenly initially. I removed the `Target Version` from the SPIP JIRA. https://issues.apache.org/jira/browse/SPARK-49834 I'm switching my cast to +1 for this SPIP vote. Thanks, Dongjoon. On 202

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 in support of the direction of the Single-pass Analyzer for Catalyst. I think we should not have a target version for the new Catalyst SPARK-49834 . It should not be a blocker for Spark 4.0. When implementing the new analyzer, the code changes

Re: [VOTE] Single-pass Analyzer for Catalyst

I don't actually "lead" this. But I don't think this needs to target a specific Spark version given it should not have any user facing consequences? On Mon, Sep 30, 2024 at 3:36 PM Dongjoon Hyun wrote: > Thank you for leading this, Vladimir, Reynold, Herman. > > I'm wondering if this is really

Re: [VOTE] Single-pass Analyzer for Catalyst

Thank you for leading this, Vladimir, Reynold, Herman. I'm wondering if this is really achievable goal for Apache Spark 4.0.0. If it's expected that we are unable to deliver it, shall we postpone this vote until 4.1.0 planning? Anyway, since SPARK-49834 has a target version 4.0.0 explicitly,

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 On Mon, Sep 30, 2024 at 1:25 PM Herman van Hovell wrote: > > +1 > > On Mon, Sep 30, 2024 at 12:21 PM Dongjoon Hyun wrote: >> >> +1 >> >> Thank you, Holden. >> >> Dongjoon. >> >> On 2024/09/30 18:01:17 Holden Karau wrote: >> > I think it has been de-facto deprecated, we haven’t updated it mean

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 On Mon, Sep 30, 2024 at 12:21 PM Dongjoon Hyun wrote: > +1 > > Thank you, Holden. > > Dongjoon. > > On 2024/09/30 18:01:17 Holden Karau wrote: > > I think it has been de-facto deprecated, we haven’t updated it > meaningfully > > in several years. I think removing the API would be excessive bu

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD Imperial College London London, United Kingdom view my Linkedin profile

Re: [VOTE] Officialy Deprecate GraphX in Spark 4

+1 Thank you, Holden. Dongjoon. On 2024/09/30 18:01:17 Holden Karau wrote: > I think it has been de-facto deprecated, we haven’t updated it meaningfully > in several years. I think removing the API would be excessive but > deprecating it would give us the flexibility to remove it in the not too

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin wrote: > +1 > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev > wrote: > >> Hi all, >> >> I’d like to start a vote for a single-pass Analyzer for the Catalyst >> project. This project will introduce a new analysis framework to the >> Catalyst, w

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

I support deprecating GraphX because: - GraphFrames supersedes it, really - No maintainers and no reason to believe there will be - we can take the last 5+ years as thorough evidence - Low (but not trivial) docs hits compared to other modules: https://analytics.apache.org/index.php

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev wrote: > Hi all, > > I’d like to start a vote for a single-pass Analyzer for the Catalyst > project. This project will introduce a new analysis framework to the > Catalyst, which will eventually replace the fixed-point one. > > Please refer to

Re: [DISCUSS] Deprecate GraphX OR Find new maintainers interested in GraphX OR leave it as is?

Hi, These are my Views: 1. Deprecation Consideration: I lean towards the idea of officially deprecating GraphX, given the lack of active development and community engagement over the past few years as you alluded. This would set clear expectations for users about its future and encourage them to

Re: [DISCUSS] Creating `branch-4.0` and Feature Freeze for Apache Spark 4.0

Thank you for the detailed proposal with dates, Hyukjin. I guess it's aligned with Herman already, too. I'm fine if we have a pre-defined and predictably achievable schedule. :-) We know the well-known goal of `Spark Connect` and I love to have. I've been monitoring the progress, but I'm not su

Re: [DISCUSS] Creating `branch-4.0` and Feature Freeze for Apache Spark 4.0

I meant 2025 :-). On Fri, Sep 27, 2024 at 11:15 AM Hyukjin Kwon wrote: > We're basically working on making Scala Spark Connect ready. > For example, I am working on having a parent class for both Spark Classic > and Spark Connect so users would face less breaking changes, and they can > run thei

Re: [DISCUSS] Creating `branch-4.0` and Feature Freeze for Apache Spark 4.0

We're basically working on making Scala Spark Connect ready. For example, I am working on having a parent class for both Spark Classic and Spark Connect so users would face less breaking changes, and they can run their application without changing anything. In addition, I am also working on sharing

Re: [DISCUSS] Creating `branch-4.0` and Feature Freeze for Apache Spark 4.0

Hi, Can we push back the dates by at least 2 months? We are working on unifying the Connect and Classic Scala interface, and I would like to avoid rushing things. Kind regards, Herman On Thu, Sep 26, 2024 at 3:19 PM Dongjoon Hyun wrote: > Hi, All. > > We've delivered two preview releases for

  1   2   3   4   5   6   7   8   9   10   >