Now wait... we created a regression in 2.4.0. Arguably, we should have blocked that release until we had a fix; but the issue came up late in the release process and it looks to me like there wasn't an adequate fix immediately available, so we did something bad and released 2.4.0 with a known regression. Saying that there is now no regression from 2.4 is tautological and no excuse for not taking in a fix -- and it looks like that fix has been waiting for months.
On Sun, Mar 10, 2019 at 3:42 PM DB Tsai <dbt...@dbtsai.com.invalid> wrote: > As we have many important fixes in 2.4 branch which we want to release > asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1 > will be not blocked by this. > > Sincerely, > > DB Tsai > ---------------------------------------------------------- > Web: https://www.dbtsai.com > PGP Key ID: 0359BC9965359766 > > > On Sun, Mar 10, 2019 at 3:08 PM Michael Heuer <heue...@gmail.com> wrote: > >> Any chance we could get some movement on this for 2.4.1? >> >> https://issues.apache.org/jira/browse/SPARK-25588 >> https://github.com/apache/parquet-mr/pull/560 >> >> It would require a new Parquet release, which would then need to be >> picked up by Spark. We're dead in the water on 2.4.0 without a large >> refactoring (remove all the RDD code paths for reading Avro stored in >> Parquet). >> >> michael >> >> >> On Mar 8, 2019, at 6:22 PM, Sean Owen <sro...@gmail.com> wrote: >> >> FWIW RC6 looked fine to me. Passed all tests, etc. >> >> On Fri, Mar 8, 2019 at 6:09 PM DB Tsai <dbt...@dbtsai.com> wrote: >> >>> Sounds fair to me. I'll cut another rc7 when the PR is merged. >>> Hopefully, this is the final rc. Thanks. >>> >>> Sincerely, >>> >>> DB Tsai >>> ---------------------------------------------------------- >>> Web: https://www.dbtsai.com >>> PGP Key ID: 42E5B25A8F7A82C1 >>> >>> >>> On Fri, Mar 8, 2019 at 3:23 PM Xiao Li <lix...@databricks.com> wrote: >>> >>>> It is common to hit this issue when driver and executors are different >>>> object layout, but Spark might not return a wrong answer. It is very hard >>>> to find out the root cause. Thus, I would suggest to include it in Spark >>>> 2.4.1. >>>> >>>> On Fri, Mar 8, 2019 at 3:13 PM DB Tsai <dbt...@dbtsai.com> wrote: >>>> >>>>> BTW, practically, is it common for users running into this bug when >>>>> the driver and executors have different object layout? >>>>> >>>>> Sincerely, >>>>> >>>>> DB Tsai >>>>> ---------------------------------------------------------- >>>>> Web: https://www.dbtsai.com >>>>> PGP Key ID: 42E5B25A8F7A82C1 >>>>> >>>>> >>>>> On Fri, Mar 8, 2019 at 3:00 PM DB Tsai <dbt...@dbtsai.com> wrote: >>>>> >>>>>> Hi Xiao, >>>>>> >>>>>> I already cut rc7 and start the build process. If we definitely need >>>>>> this fix, I can cut rc8. Let me know what you think. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Fri, Mar 8, 2019 at 1:46 PM Xiao Li <lix...@databricks.com> wrote: >>>>>> >>>>>>> Hi, DB, >>>>>>> >>>>>>> Since this RC will fail, could you hold it until we fix >>>>>>> https://issues.apache.org/jira/browse/SPARK-27097? Either Kris or I >>>>>>> will submit a PR today. The PR is small and the risk is low. This is a >>>>>>> correctness bug. It would be good to have it. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Xiao >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 8, 2019 at 12:17 PM DB Tsai <d_t...@apple.com.invalid> >>>>>>> wrote: >>>>>>> >>>>>>>> Since I can not find the commit of `Preparing development version >>>>>>>> 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and >>>>>>>> do a >>>>>>>> force-push. I'll follow Marcelo's suggestion to have another rc7 cut. >>>>>>>> Thus, >>>>>>>> this vote fails. >>>>>>>> >>>>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | >>>>>>>> Apple, Inc >>>>>>>> >>>>>>>> > On Mar 8, 2019, at 11:45 AM, DB Tsai <d_t...@apple.com.INVALID> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's >>>>>>>> very weird. It must be overwritten by a force push. >>>>>>>> > >>>>>>>> > DB Tsai | Siri Open Source Technologies [not a contribution] >>>>>>>> | Apple, Inc >>>>>>>> > >>>>>>>> >> On Mar 8, 2019, at 11:39 AM, DB Tsai <d_t...@apple.com.INVALID> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> I was using `./do-release-docker.sh` to create a release. But >>>>>>>> since the gpg validation fails couple times when the script tried to >>>>>>>> publish the jars into Nexus, I re-ran the scripts multiple times >>>>>>>> without >>>>>>>> creating a new rc. I was wondering if the script will overwrite the >>>>>>>> v.2.4.1-rc6 tag instead of using the same commit causing this issue. >>>>>>>> >> >>>>>>>> >> Should we create a new rc7? >>>>>>>> >> >>>>>>>> >> DB Tsai | Siri Open Source Technologies [not a contribution] >>>>>>>> | Apple, Inc >>>>>>>> >> >>>>>>>> >>> On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin < >>>>>>>> van...@cloudera.com.INVALID> wrote: >>>>>>>> >>> >>>>>>>> >>> I personally find it a little weird to not have the commit in >>>>>>>> branch-2.4. >>>>>>>> >>> >>>>>>>> >>> Not that this would happen, but if the v2.4.1-rc6 tag is >>>>>>>> overwritten >>>>>>>> >>> (e.g. accidentally) then you lose the reference to that commit, >>>>>>>> and >>>>>>>> >>> then the exact commit from which the rc was generated is lost. >>>>>>>> >>> >>>>>>>> >>> On Fri, Mar 8, 2019 at 7:49 AM Sean Owen <sro...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> That's weird. I see the commit but can't find it in the >>>>>>>> branch. Was it pushed, or lost in a force push of 2.4 along the way? >>>>>>>> The >>>>>>>> change is there, just under a different commit in the 2.4 branch. >>>>>>>> >>>> >>>>>>>> >>>> It doesn't necessarily invalidate the RC as it is a valid >>>>>>>> public tagged commit and all that. I just want to be sure we do have >>>>>>>> the >>>>>>>> code from that commit in these tatballs. It looks like it. >>>>>>>> >>>> >>>>>>>> >>>> On Fri, Mar 8, 2019, 4:14 AM Mihály Tóth <misut...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> Hi, >>>>>>>> >>>>> >>>>>>>> >>>>> I am not sure how problematic it is but v2.4.1-rc6 is not on >>>>>>>> branch-2.4. Release related commits I have seen so far were also part >>>>>>>> of >>>>>>>> the branch. >>>>>>>> >>>>> >>>>>>>> >>>>> I guess the "Preparing Spark release v2.4.1-rc6" and >>>>>>>> "Preparing development version 2.4.2-SNAPSHOT" commits were simply not >>>>>>>> pushed to spark-2.4 just the tag itself was pushed. I dont know what >>>>>>>> is the >>>>>>>> practice in such cases but one solution is to rebase branch-2.4 changes >>>>>>>> after 3336a21 onto these commits and do a (sorry) force push. In this >>>>>>>> case >>>>>>>> there is no impact on this RC. >>>>>>>> >>>>> >>>>>>>> >>>>> Best Regards, >>>>>>>> >>>>> >>>>>>>> >>>>> Misi >>>>>>>> >>>>> >>>>>>>> >>>>> DB Tsai <d_t...@apple.com.invalid> ezt írta (időpont: 2019. >>>>>>>> márc. 8., P, 1:15): >>>>>>>> >>>>>> >>>>>>>> >>>>>> Please vote on releasing the following candidate as Apache >>>>>>>> Spark version 2.4.1. >>>>>>>> >>>>>> >>>>>>>> >>>>>> The vote is open until March 11 PST and passes if a majority >>>>>>>> +1 PMC votes are cast, with >>>>>>>> >>>>>> a minimum of 3 +1 votes. >>>>>>>> >>>>>> >>>>>>>> >>>>>> [ ] +1 Release this package as Apache Spark 2.4.1 >>>>>>>> >>>>>> [ ] -1 Do not release this package because ... >>>>>>>> >>>>>> >>>>>>>> >>>>>> To learn more about Apache Spark, please see >>>>>>>> http://spark.apache.org/ >>>>>>>> >>>>>> >>>>>>>> >>>>>> The tag to be voted on is v2.4.1-rc6 (commit >>>>>>>> 201ec8c9b46f9d037cc2e3a5d9c896b9840ca1bc): >>>>>>>> >>>>>> https://github.com/apache/spark/tree/v2.4.1-rc6 >>>>>>>> >>>>>> >>>>>>>> >>>>>> The release files, including signatures, digests, etc. can >>>>>>>> be found at: >>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-bin/ >>>>>>>> >>>>>> >>>>>>>> >>>>>> Signatures used for Spark RCs can be found in this file: >>>>>>>> >>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>>> >>>>>> >>>>>>>> >>>>>> The staging repository for this release can be found at: >>>>>>>> >>>>>> >>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1308/ >>>>>>>> >>>>>> >>>>>>>> >>>>>> The documentation corresponding to this release can be found >>>>>>>> at: >>>>>>>> >>>>>> >>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.1-rc6-docs/ >>>>>>>> >>>>>> >>>>>>>> >>>>>> The list of bug fixes going into 2.4.1 can be found at the >>>>>>>> following URL: >>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.1 >>>>>>>> >>>>>> >>>>>>>> >>>>>> FAQ >>>>>>>> >>>>>> >>>>>>>> >>>>>> ========================= >>>>>>>> >>>>>> How can I help test this release? >>>>>>>> >>>>>> ========================= >>>>>>>> >>>>>> >>>>>>>> >>>>>> If you are a Spark user, you can help us test this release >>>>>>>> by taking >>>>>>>> >>>>>> an existing Spark workload and running on this release >>>>>>>> candidate, then >>>>>>>> >>>>>> reporting any regressions. >>>>>>>> >>>>>> >>>>>>>> >>>>>> If you're working in PySpark you can set up a virtual env >>>>>>>> and install >>>>>>>> >>>>>> the current RC and see if anything important breaks, in the >>>>>>>> Java/Scala >>>>>>>> >>>>>> you can add the staging repository to your projects >>>>>>>> resolvers and test >>>>>>>> >>>>>> with the RC (make sure to clean up the artifact cache >>>>>>>> before/after so >>>>>>>> >>>>>> you don't end up building with a out of date RC going >>>>>>>> forward). >>>>>>>> >>>>>> >>>>>>>> >>>>>> =========================================== >>>>>>>> >>>>>> What should happen to JIRA tickets still targeting 2.4.1? >>>>>>>> >>>>>> =========================================== >>>>>>>> >>>>>> >>>>>>>> >>>>>> The current list of open tickets targeted at 2.4.1 can be >>>>>>>> found at: >>>>>>>> >>>>>> https://issues.apache.org/jira/projects/SPARK and search >>>>>>>> for "Target Version/s" = 2.4.1 >>>>>>>> >>>>>> >>>>>>>> >>>>>> Committers should look at those and triage. Extremely >>>>>>>> important bug >>>>>>>> >>>>>> fixes, documentation, and API tweaks that impact >>>>>>>> compatibility should >>>>>>>> >>>>>> be worked on immediately. Everything else please retarget to >>>>>>>> an >>>>>>>> >>>>>> appropriate release. >>>>>>>> >>>>>> >>>>>>>> >>>>>> ================== >>>>>>>> >>>>>> But my bug isn't fixed? >>>>>>>> >>>>>> ================== >>>>>>>> >>>>>> >>>>>>>> >>>>>> In order to make timely releases, we will typically not hold >>>>>>>> the >>>>>>>> >>>>>> release unless the bug in question is a regression from the >>>>>>>> previous >>>>>>>> >>>>>> release. That being said, if there is something which is a >>>>>>>> regression >>>>>>>> >>>>>> that has not been correctly targeted please ping me or a >>>>>>>> committer to >>>>>>>> >>>>>> help target the issue. >>>>>>>> >>>>>> >>>>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a >>>>>>>> contribution] | Apple, Inc >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> >>>>>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> -- >>>>>>>> >>> Marcelo >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> >>> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> >> >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> --------------------------------------------------------------------- >>>>>>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> <sais19-emailsig-keynote...@2x.jpg> >>>>>>> >>>>>> -- >>>>>> - DB Sent from my iPhone >>>>>> >>>>> >>>> >>>> -- >>>> <sais19-emailsig-keynote...@2x.jpg> >>>> >>> >>
