"*As far as I can tell this makes both 3.3.5 and 3.3.6 unusable with s3 without providing an alternative committer code.*"
https://github.com/apache/hadoop/pull/5706#issuecomment-1619927992 ons. 2. aug. 2023 kl. 08:05 skrev Emil Ejbyfeldt <eejbyfe...@liveintent.com.invalid>: > > Apache Spark is not affected by HADOOP-18757 because it is not a part of > > both Apache Hadoop 3.3.5 and 3.3.6. > > I am not sure I am following what you are trying to say here. Is that > the jira is saying that only 3.3.5 is affected? Here I think the Jira is > just incorrect. The jira was created (and the PR with the fix) was > created before 3.3.6 was released and I just think the jira has not been > updated to reflect the fact that 3.3.6 is also affected. > > > HADOOP-18757 seems to be merged just two weeks ago and there is no > > Apache Hadoop release with it, isn't it? > > That is correct, there is no hadoop release containing the fix. So > therefore 3.3.6 would also be affected by the regression. > > Best, > Emil > > On 02/08/2023 07:51, Dongjoon Hyun wrote: > > It's still invalid information, Emil. > > > > Apache Spark is not affected by HADOOP-18757 because it is not a part of > > both Apache Hadoop 3.3.5 and 3.3.6. > > > > HADOOP-18757 seems to be merged just two weeks ago and there is no > > Apache Hadoop release with it, isn't it? > > > > Could you check your local branch once more, please? > > > > Dongjoon. > > > > > > > > On Tue, Aug 1, 2023 at 9:46 PM Emil Ejbyfeldt <eejbyfe...@liveintent.com > > <mailto:eejbyfe...@liveintent.com>> wrote: > > > > Hi, > > > > Yes, sorry about that seem to have messed up the link. Should have > been > > https://issues.apache.org/jira/browse/HADOOP-18757 > > <https://issues.apache.org/jira/browse/HADOOP-18757> > > > > Best, > > Emil > > > > On 01/08/2023 19:08, Dongjoon Hyun wrote: > > > Hi, Emil. > > > > > > HADOOP-18568 is still open and it seems to be never a part of the > > Hadoop > > > trunk branch. > > > > > > Do you mean another JIRA? > > > > > > Dongjoon. > > > > > > > > > > > > On Tue, Aug 1, 2023 at 2:59 AM Emil Ejbyfeldt > > > <eejbyfe...@liveintent.com > > <mailto:eejbyfe...@liveintent.com>.invalid> wrote: > > > > > > Hi, > > > > > > We previously ran some experiments on builds from the 3.5 > > branch and > > > noticed that Hadoop had a regression > > > (https://issues.apache.org/jira/browse/HADOOP-18568 > > <https://issues.apache.org/jira/browse/HADOOP-18568> > > > <https://issues.apache.org/jira/browse/HADOOP-18568 > > <https://issues.apache.org/jira/browse/HADOOP-18568>>) in their s3a > > > committer affecting 3.3.5 and 3.3.6 (Spark 3.4 uses hadoop > > 3.3.4). This > > > fix has been merged into Hadoop and will be part the next > > release of > > > Hadoop. > > > > > > From our testing the regression when writing data to S3 > > with large > > > number of tasks S3 is severe enough that we would need to > > revert to > > > hadoop 3.3.4 in order to use spark 3.5 release. > > > > > > Since it only for S3 I am not sure it warrants action changes > > in Spark > > > (e.g rolling back hadoop to 3.3.4). But it probably something > > people > > > testing the rc against s3 should be aware of. > > > > > > Best, > > > Emil > > > > > > On 29/07/2023 10:29, Yuanjian Li wrote: > > > > Hi everyone, > > > > > > > > Following the release timeline, I will cut the RC > > on*Tuesday, Aug > > > 1st at > > > > 1 pm PST* as scheduled. > > > > > > > > Date Event > > > > July 17th 2023 > > > > Late July > > > > 2023 Code freeze. Release branch cut. > > > > QA period. Focus on bug fixes, tests, stability and docs. > > > > Generally, no new features merged. > > > > > > > > > > > > August 2023 Release candidates (RC), voting, etc. until > > final > > > release passes > > > > > > > > > > > > Best, > > > > Yuanjian > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > <mailto:dev-unsubscr...@spark.apache.org> > > > <mailto:dev-unsubscr...@spark.apache.org > > <mailto:dev-unsubscr...@spark.apache.org>> > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
