> Apache Spark is not affected by HADOOP-18757 because it is not a part of
> both Apache Hadoop 3.3.5 and 3.3.6.
I am not sure I am following what you are trying to say here. Is that
the jira is saying that only 3.3.5 is affected? Here I think the Jira is
just incorrect. The jira was created (and the PR with the fix) was
created before 3.3.6 was released and I just think the jira has not been
updated to reflect the fact that 3.3.6 is also affected.
> HADOOP-18757 seems to be merged just two weeks ago and there is no
> Apache Hadoop release with it, isn't it?
That is correct, there is no hadoop release containing the fix. So
therefore 3.3.6 would also be affected by the regression.
Best,
Emil
On 02/08/2023 07:51, Dongjoon Hyun wrote:
It's still invalid information, Emil.
Apache Spark is not affected by HADOOP-18757 because it is not a part of
both Apache Hadoop 3.3.5 and 3.3.6.
HADOOP-18757 seems to be merged just two weeks ago and there is no
Apache Hadoop release with it, isn't it?
Could you check your local branch once more, please?
Dongjoon.
On Tue, Aug 1, 2023 at 9:46 PM Emil Ejbyfeldt <eejbyfe...@liveintent.com
<mailto:eejbyfe...@liveintent.com>> wrote:
Hi,
Yes, sorry about that seem to have messed up the link. Should have been
https://issues.apache.org/jira/browse/HADOOP-18757
<https://issues.apache.org/jira/browse/HADOOP-18757>
Best,
Emil
On 01/08/2023 19:08, Dongjoon Hyun wrote:
> Hi, Emil.
>
> HADOOP-18568 is still open and it seems to be never a part of the
Hadoop
> trunk branch.
>
> Do you mean another JIRA?
>
> Dongjoon.
>
>
>
> On Tue, Aug 1, 2023 at 2:59 AM Emil Ejbyfeldt
> <eejbyfe...@liveintent.com
<mailto:eejbyfe...@liveintent.com>.invalid> wrote:
>
> Hi,
>
> We previously ran some experiments on builds from the 3.5
branch and
> noticed that Hadoop had a regression
> (https://issues.apache.org/jira/browse/HADOOP-18568
<https://issues.apache.org/jira/browse/HADOOP-18568>
> <https://issues.apache.org/jira/browse/HADOOP-18568
<https://issues.apache.org/jira/browse/HADOOP-18568>>) in their s3a
> committer affecting 3.3.5 and 3.3.6 (Spark 3.4 uses hadoop
3.3.4). This
> fix has been merged into Hadoop and will be part the next
release of
> Hadoop.
>
> From our testing the regression when writing data to S3
with large
> number of tasks S3 is severe enough that we would need to
revert to
> hadoop 3.3.4 in order to use spark 3.5 release.
>
> Since it only for S3 I am not sure it warrants action changes
in Spark
> (e.g rolling back hadoop to 3.3.4). But it probably something
people
> testing the rc against s3 should be aware of.
>
> Best,
> Emil
>
> On 29/07/2023 10:29, Yuanjian Li wrote:
> > Hi everyone,
> >
> > Following the release timeline, I will cut the RC
on*Tuesday, Aug
> 1st at
> > 1 pm PST* as scheduled.
> >
> > Date Event
> > July 17th 2023
> > Late July
> > 2023 Code freeze. Release branch cut.
> > QA period. Focus on bug fixes, tests, stability and docs.
> > Generally, no new features merged.
> >
> >
> > August 2023 Release candidates (RC), voting, etc. until
final
> release passes
> >
> >
> > Best,
> > Yuanjian
>
>
---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
<mailto:dev-unsubscr...@spark.apache.org>
> <mailto:dev-unsubscr...@spark.apache.org
<mailto:dev-unsubscr...@spark.apache.org>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
