Hi, Herman, I inlined my thoughts. > 1. You seem to be favoring speed over quality here. Even if my vote were > erroneous, you should give me more than two hours to respond.
This is completely not true again. I prolonged Apache Spark RC3 to clarify and evaluate your claim (and PR technically) over two days. In addition, after you made your follow-up PR, I reviewed and gave you my feedback swiftly (within 3 hours). However, although I asked you to change your follow-up PR twice (19 hours ago and 16 hours ago), I didn't get your reply. https://github.com/apache/spark/pull/53480#pullrequestreview-3581246081 (19 hours ago) https://github.com/apache/spark/pull/53480#pullrequestreview-3582441426 (16 hours ago) While waiting your action, I spent more time around this area and noticed that your claim (regression) is not true. > 2. The problem itself is serious since it can cause driver crashes. > 3. The offending change was backported to a maintenance release I'm not disagreeing with your points, but it doesn't mean that it's a release blocker we pre-defined. > 4. I am not sure I follow the PR argument. You merged my initial PR > without even checking in with me. You wrote "A PR has been open since Friday: https://github.com/apache/spark/pull/53452. I hope that I can get it merged today." and removed `[WIP]` 4 days ago and left your PR unchanged over 3 days. Technically, your PR has been waiting for reviews over 3 days and every committer can merge the open PR after reviewing. Since nobody took a look until then, I reviewed and merged to unblock Apache Spark 4.1.0 release as the release manager. I'm not sure what was your expectation there after removing `[WIP]` by yourself. > That PR fixed the issue, it just needed proper tests and some touch-ups > (again quality is important). We agree that the quality is always crucial. Let's see your SPARK-54696. Especially, the follow-up PR. Does it pass the basic CI test? Does the follow-up contain only test-change which you claims in the first PR? As I commented about the CI result on your follow-up, it turns out that the original PR and the follow-up is still **unfinished**. To simplify put, SPARK-54696 didn't meet the Apache Spark qualification yet for all branches (master/branch-4.1/branch-4.0). That's the reason why I decided to give more time to SPARK-54696 (including longevity tests) in Apache Spark 4.1.1 timeframe. The situation seems to become more complex because both of us are confused about the root cause of the issues due to the misleading description (about regression). I hope you can finish to deliver the high quality patch of SPARK-54696 (including passing CI at least and the longevity integration test). Warmly, Dongjoon > Kind regards, > Herman > > On Tue, Dec 16, 2025 at 7:27 AM Dongjoon Hyun <[email protected]> wrote: > > > After rechecking, this vote passed. > > > > I'll send a vote result email. > > > > Dongjoon. > > > > On 2025/12/16 11:03:39 Dongjoon Hyun wrote: > > > Hi, All. > > > > > > I've been working with Herman's PRs so far. > > > > > > As a kind of fact checking, I need to correct two things in RC3 thread. > > > > > > First, Herman claimed that he found a regression of Apache Spark 4.1.0, > > but actually it's not true because Apache Spark 4.0.1 also has SPARK-53342 > > since 2025-09-06. > > > > > > Second, although Herman shared us a patch since last Friday, Herman also > > made another PR containing the main code change 9 hours ago. In addition, > > unfortunately, it also didn't pass our CIs yet. It simply means that there > > is no complete patch yet in the community for both Apache Spark 4.1.0 and > > 4.0.2. > > > > > > https://github.com/apache/spark/pull/53480 > > > ([SPARK-54696][CONNECT] Clean-up Arrow Buffers - follow-up) > > > > > > In short, he seems to block RC3 as a mistake. I'm re-checking the > > situation around RC3 vote and `branch-4.1` situation. > > > > > > Dongjoon. > > > > > > > > > > > > > > > On 2025/12/15 14:59:32 Herman van Hovell via dev wrote: > > > > > > > I pasted a non-existing link for the root cause. The actual link > > is here: > > > > > > > https://issues.apache.org/jira/browse/SPARK-53342 > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 15, 2025 at 10:47 AM Herman van Hovell < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hey Dongjoon, > > > > > > > > > > > > > > > > Regarding your questions. > > > > > > > > > > > > > > > > 1. If you define a large-ish local relation (which makes us > > cache it > > > > > > > > on the serverside) and keep using it, then leak off-heap > > memory > > > > > > every time > > > > > > > > it is being used. At some point the OS will OOM kill the > > driver. > > > > > > While I > > > > > > > > have a repro, testing it like this in CI is not a good > > idea. As an > > > > > > > > alternative I am working on a test that checks buffer > > clean-up.For > > > > > > the > > > > > > > > record I don't appreciate the term `claim` here; I am not > > blocking a > > > > > > > > release without genuine concern. > > > > > > > > 2. The root cause is > > > > > > > > https://databricks.atlassian.net/browse/SPARK-53342 and > > not the > > > > > > large > > > > > > > > local relations work. > > > > > > > > 3. A PR has been open since Friday: > > > > > > > > https://github.com/apache/spark/pull/53452. I hope that I > > can get > > > > > > it > > > > > > > > merged today. > > > > > > > > 4. I don't see a reason why. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Herman > > > > > > > > > > > > > > > > On Mon, Dec 15, 2025 at 5:47 AM Dongjoon Hyun < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > >> How can we verify the regression, Herman? > > > > > > > >> > > > > > > > >> It's a little difficult for me to evaluate your claim so far > > due to > > > > > > the > > > > > > > >> lack of the shared information. Specifically, there is no > > update for > > > > > > last 3 > > > > > > > >> days on "SPARK-54696 (Spark Connect LocalRelation support leak > > > > > > off-heap > > > > > > > >> memory)" after you created it. > > > > > > > >> > > > > > > > >> Could you provide us more technical information about your > > Spark > > > > > > Connect > > > > > > > >> issue? > > > > > > > >> > > > > > > > >> 1. How can we reproduce your claim? Do you have a test case? > > > > > > > >> > > > > > > > >> 2. For the root cause, I'm wondering if you are saying > > literally > > > > > > > >> SPARK-53917 (Support large local relations) or another JIRA > > issue. > > > > > > Which > > > > > > > >> commit is the root cause? > > > > > > > >> > > > > > > > >> 3. Since you assigned SPARK-54696 to yourself for last 3 > > days, do you > > > > > > > >> want to provide a PR soon? > > > > > > > >> > > > > > > > >> 4. If you need more time, shall we simply revert the root > > cause from > > > > > > > >> Apache Spark 4.1.0 ? > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> Dongjoon > > > > > > > >> > > > > > > > >> On 2025/12/14 23:29:59 Herman van Hovell via dev wrote: > > > > > > > >> > Yes. It is a regression in Spark 4.1. The root cause is a > > change > > > > > > where > > > > > > > >> we > > > > > > > >> > fail to clean-up allocated (off-heap) buffers. > > > > > > > >> > > > > > > > > >> > On Sun, Dec 14, 2025 at 4:25 AM Dongjoon Hyun < > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > > Hi, Herman. > > > > > > > >> > > > > > > > > > >> > > Do you mean that is a regression at Apache Spark 4.1.0? > > > > > > > >> > > > > > > > > > >> > > If then, do you know what was the root cause? > > > > > > > >> > > > > > > > > > >> > > Dongjoon. > > > > > > > >> > > > > > > > > > >> > > On 2025/12/13 23:09:02 Herman van Hovell via dev wrote: > > > > > > > >> > > > -1. We need to get > > > > > > > >> https://issues.apache.org/jira/browse/SPARK-54696 > > > > > > > >> > > fixed. > > > > > > > >> > > > > > > > > > > >> > > > On Sat, Dec 13, 2025 at 11:07 AM Jules Damji < > > > > > > [email protected] > > > > > > > >> > > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > > +1 non-binding > > > > > > > >> > > > > — > > > > > > > >> > > > > Sent from my iPhone > > > > > > > >> > > > > Pardon the dumb thumb typos :) > > > > > > > >> > > > > > > > > > > > >> > > > > > On Dec 11, 2025, at 8:34 AM, [email protected] > > wrote: > > > > > > > >> > > > > > > > > > > > > >> > > > > > Please vote on releasing the following candidate > > as Apache > > > > > > > >> Spark > > > > > > > >> > > > > version 4.1.0. > > > > > > > >> > > > > > > > > > > > > >> > > > > > The vote is open until Sun, 14 Dec 2025 09:34:31 > > PST and > > > > > > passes > > > > > > > >> if a > > > > > > > >> > > > > majority +1 PMC votes are cast, with > > > > > > > >> > > > > > a minimum of 3 +1 votes. > > > > > > > >> > > > > > > > > > > > > >> > > > > > [ ] +1 Release this package as Apache Spark 4.1.0 > > > > > > > >> > > > > > [ ] -1 Do not release this package because ... > > > > > > > >> > > > > > > > > > > > > >> > > > > > To learn more about Apache Spark, please see > > > > > > > >> > > https://spark.apache.org/ > > > > > > > >> > > > > > > > > > > > > >> > > > > > The tag to be voted on is v4.1.0-rc3 (commit > > e221b56be7b): > > > > > > > >> > > > > > https://github.com/apache/spark/tree/v4.1.0-rc3 > > > > > > > >> > > > > > > > > > > > > >> > > > > > The release files, including signatures, digests, > > etc. can > > > > > > be > > > > > > > >> found > > > > > > > >> > > at: > > > > > > > >> > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc3-bin/ > > > > > > > >> > > > > > > > > > > > > >> > > > > > Signatures used for Spark RCs can be found in this > > file: > > > > > > > >> > > > > > https://downloads.apache.org/spark/KEYS > > > > > > > >> > > > > > > > > > > > > >> > > > > > The staging repository for this release can be > > found at: > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > https://repository.apache.org/content/repositories/orgapachespark-1508/ > > > > > > > >> > > > > > > > > > > > > >> > > > > > The documentation corresponding to this release can > > be > > > > > > found at: > > > > > > > >> > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc3-docs/ > > > > > > > >> > > > > > > > > > > > > >> > > > > > The list of bug fixes going into 4.1.0 can be found > > at the > > > > > > > >> following > > > > > > > >> > > URL: > > > > > > > >> > > > > > > > > > > > https://issues.apache.org/jira/projects/SPARK/versions/12355581 > > > > > > > >> > > > > > > > > > > > > >> > > > > > FAQ > > > > > > > >> > > > > > > > > > > > > >> > > > > > ========================= > > > > > > > >> > > > > > How can I help test this release? > > > > > > > >> > > > > > ========================= > > > > > > > >> > > > > > > > > > > > > >> > > > > > If you are a Spark user, you can help us test this > > release > > > > > > by > > > > > > > >> taking > > > > > > > >> > > > > > an existing Spark workload and running on this > > release > > > > > > > >> candidate, > > > > > > > >> > > then > > > > > > > >> > > > > > reporting any regressions. > > > > > > > >> > > > > > > > > > > > > >> > > > > > If you're working in PySpark you can set up a > > virtual env > > > > > > and > > > > > > > >> install > > > > > > > >> > > > > > the current RC via "pip install > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc3-bin/pyspark-4.1.0.tar.gz > > > > > > > >> > > > > " > > > > > > > >> > > > > > and see if anything important breaks. > > > > > > > >> > > > > > In the Java/Scala, you can add the staging > > repository to > > > > > > your > > > > > > > >> > > project's > > > > > > > >> > > > > resolvers and test > > > > > > > >> > > > > > with the RC (make sure to clean up the artifact > > cache > > > > > > > >> before/after so > > > > > > > >> > > > > > you don't end up building with an out of date RC > > going > > > > > > forward). > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > --------------------------------------------------------------------- > > > > > > > >> > > > > > To unsubscribe e-mail: > > [email protected] > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > --------------------------------------------------------------------- > > > > > > > >> > > > > To unsubscribe e-mail: > > [email protected] > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > >> > > To unsubscribe e-mail: [email protected] > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > --------------------------------------------------------------------- > > > > > > > >> To unsubscribe e-mail: [email protected] > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe e-mail: [email protected] > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
