Re: Call for Presentations, Community Over Code 2023

[Wei-Chiu Chuang]

There's also a call for presentation for Community over Code Asia 2023

https://www.bagevent.com/event/cocasia-2023-EN
Happening Aug 18-20. CfP due by 6/6


On Tue, May 9, 2023 at 8:39 PM Ayush Saxena  wrote:

> Forwarding from dev@hadoop to the dev ML which we use.
>
> The actual mail lies here:
> https://www.mail-archive.com/dev@hadoop.apache.org/msg00160.html
>
> -Ayush
>
> On 2023/05/09 21:24:09 Rich Bowen wrote:
> > (Note: You are receiving this because you are subscribed to the dev@
> > list for one or more Apache Software Foundation projects.)
> >
> > The Call for Presentations (CFP) for Community Over Code (formerly
> > Apachecon) 2023 is open at
> > https://communityovercode.org/call-for-presentations/, and will close
> > Thu, 13 Jul 2023 23:59:59 GMT.
> >
> > The event will be held in Halifax, Canada, October 7-10, 2023.
> >
> > We welcome submissions on any topic related to the Apache Software
> > Foundation, Apache projects, or the communities around those projects.
> > We are specifically looking for presentations in the following
> > catetegories:
> >
> > Fintech
> > Search
> > Big Data, Storage
> > Big Data, Compute
> > Internet of Things
> > Groovy
> > Incubator
> > Community
> > Data Engineering
> > Performance Engineering
> > Geospatial
> > API/Microservices
> > Frameworks
> > Content Wrangling
> > Tomcat and httpd
> > Cloud and Runtime
> > Streaming
> > Sustainability
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: dev-h...@hadoop.apache.org
> >
> >
>
>
> Sent from my iPhone
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

RE: Call for Presentations, Community Over Code 2023

[Ayush Saxena]

Forwarding from dev@hadoop to the dev ML which we use.

The actual mail lies here:
https://www.mail-archive.com/dev@hadoop.apache.org/msg00160.html

-Ayush

On 2023/05/09 21:24:09 Rich Bowen wrote:
> (Note: You are receiving this because you are subscribed to the dev@
> list for one or more Apache Software Foundation projects.)
>
> The Call for Presentations (CFP) for Community Over Code (formerly
> Apachecon) 2023 is open at
> https://communityovercode.org/call-for-presentations/, and will close
> Thu, 13 Jul 2023 23:59:59 GMT.
>
> The event will be held in Halifax, Canada, October 7-10, 2023.
>
> We welcome submissions on any topic related to the Apache Software
> Foundation, Apache projects, or the communities around those projects.
> We are specifically looking for presentations in the following
> catetegories:
>
> Fintech
> Search
> Big Data, Storage
> Big Data, Compute
> Internet of Things
> Groovy
> Incubator
> Community
> Data Engineering
> Performance Engineering
> Geospatial
> API/Microservices
> Frameworks
> Content Wrangling
> Tomcat and httpd
> Cloud and Runtime
> Streaming
> Sustainability
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: dev-h...@hadoop.apache.org
>
>


Sent from my iPhone

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

One thing I forgot to mention. If you are planning to run tests also for
Windows OS then github actions isn’t something to be chased. It has 6 hrs
timeout.

So, in case of multi module changes or root level changes, we will get
screwed, the tests take more than 20 hrs in those cases, so we will hit the
timeout

-Ayush

On Tue, 9 May 2023 at 11:49 PM, Ayush Saxena  wrote:

> Hi Gautham,
> The only advantage I can think of is your limitation to nodes shouldn’t
> bother much. Though there are limited resources given to Apache but still
> never observed any starvation in any project for the resources using Github
> Actions.
>
> If Precommit CI for windows is there and it can work in parallel, that
> should be good as well
>
> Good Luck!!!
>
> -Ayush
>
> On Tue, 9 May 2023 at 8:47 PM, Gautham Banasandra 
> wrote:
>
>> Hi Ayush,
>>
>> I was planning to set up the precommit CI job for Windows 10, so that
>> we validate each PR against Windows 10. The precommit CI can actually
>> run in parallel to the Linux precommit CI. But I think we've got very few
>> Windows nodes (about 12 I think) compared to the number of Linux nodes
>> (about 40 in number).
>> However, I think the setting up the precommit CI is quite worth
>> the effort.
>> It was quite hard to get to this point and we certainly don't want to
>> regress
>> in this regard. I think I'll pursue this once I stabilize the nightly CI
>> and after
>> I sort out the rest of the failing parts (mvnsite, javadoc etc).
>>
>> Also, does Github Actions provide any advantage over precommit CI?
>>
>> Thanks,
>> --Gautham
>>
>> On Tue, 9 May 2023 at 02:44, Ayush Saxena  wrote:
>>
>>> Awesome!!!
>>>
>>> Do you plan to run some builds per PR to ensure no commits breaks it
>>> in the future? Worth exploring GithubActions in the later stages if we
>>> can have an action for maven install on windows, it won't affect the
>>> normal build time as well. can run in parallel with the existing
>>> builds.
>>>
>>> -Ayush
>>>
>>> On Mon, 8 May 2023 at 23:39, Gautham Banasandra 
>>> wrote:
>>> >
>>> > Dear Hadoop community,
>>> >
>>> > It is my pleasure to announce that I've set up the Nightly Jenkins CI
>>> for
>>> > Hadoop on the Windows 10 platform[1]. The effort mainly involved
>>> getting
>>> > Yetus to run on Windows against Hadoop.
>>> > The nightly CI will run every 36 hours and send out the build report
>>> to the
>>> > same recipients as this email, upon completion.
>>> > There are still quite a few things that need to be sorted out.
>>> Currently,
>>> > mvninstall has a +1. Other phases like mvnsite, javadoc etc still need
>>> to be
>>> > fixed.
>>> >
>>> > [1]
>>> >
>>> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>>> >
>>> > Thanks,
>>> > --Gautham
>>>
>>

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/486/

[May 8, 2023, 4:55:15 PM] (github) HADOOP-18734. Create qbt.sh symlink on 
Windows (#5626)
[May 8, 2023, 5:13:09 PM] (github) YARN-11340. [Federation] Improve 
SQLFederationStateStore DataSource Config. (#5403)
[May 9, 2023, 2:05:28 AM] (github) HDFS-16999. Fix wrong use of 
processFirstBlockReport(). (#5622). Contributed by Shuyan Zhang.




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory))
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory))
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
  doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState)) Redundant null check at 
Re

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/

[May 8, 2023, 4:55:15 PM] (github) HADOOP-18734. Create qbt.sh symlink on 
Windows (#5626)
[May 8, 2023, 5:13:09 PM] (github) YARN-11340. [Federation] Improve 
SQLFederationStateStore DataSource Config. (#5403)




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1221/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

Hi Gautham,
The only advantage I can think of is your limitation to nodes shouldn’t
bother much. Though there are limited resources given to Apache but still
never observed any starvation in any project for the resources using Github
Actions.

If Precommit CI for windows is there and it can work in parallel, that
should be good as well

Good Luck!!!

-Ayush

On Tue, 9 May 2023 at 8:47 PM, Gautham Banasandra 
wrote:

> Hi Ayush,
>
> I was planning to set up the precommit CI job for Windows 10, so that
> we validate each PR against Windows 10. The precommit CI can actually
> run in parallel to the Linux precommit CI. But I think we've got very few
> Windows nodes (about 12 I think) compared to the number of Linux nodes
> (about 40 in number).
> However, I think the setting up the precommit CI is quite worth the effort.
> It was quite hard to get to this point and we certainly don't want to
> regress
> in this regard. I think I'll pursue this once I stabilize the nightly CI
> and after
> I sort out the rest of the failing parts (mvnsite, javadoc etc).
>
> Also, does Github Actions provide any advantage over precommit CI?
>
> Thanks,
> --Gautham
>
> On Tue, 9 May 2023 at 02:44, Ayush Saxena  wrote:
>
>> Awesome!!!
>>
>> Do you plan to run some builds per PR to ensure no commits breaks it
>> in the future? Worth exploring GithubActions in the later stages if we
>> can have an action for maven install on windows, it won't affect the
>> normal build time as well. can run in parallel with the existing
>> builds.
>>
>> -Ayush
>>
>> On Mon, 8 May 2023 at 23:39, Gautham Banasandra 
>> wrote:
>> >
>> > Dear Hadoop community,
>> >
>> > It is my pleasure to announce that I've set up the Nightly Jenkins CI
>> for
>> > Hadoop on the Windows 10 platform[1]. The effort mainly involved getting
>> > Yetus to run on Windows against Hadoop.
>> > The nightly CI will run every 36 hours and send out the build report to
>> the
>> > same recipients as this email, upon completion.
>> > There are still quite a few things that need to be sorted out.
>> Currently,
>> > mvninstall has a +1. Other phases like mvnsite, javadoc etc still need
>> to be
>> > fixed.
>> >
>> > [1]
>> >
>> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>> >
>> > Thanks,
>> > --Gautham
>>
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1220/

No changes

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/

[May 6, 2023, 3:20:21 AM] (github) HDFS-16865. The source path is always / 
after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. 
(#5200). Contributed by ZanderXu.




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
   
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 [500K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1219/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
 [96K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1020/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

Awesome!!!

Do you plan to run some builds per PR to ensure no commits breaks it
in the future? Worth exploring GithubActions in the later stages if we
can have an action for maven install on windows, it won't affect the
normal build time as well. can run in parallel with the existing
builds.

-Ayush

On Mon, 8 May 2023 at 23:39, Gautham Banasandra  wrote:
>
> Dear Hadoop community,
>
> It is my pleasure to announce that I've set up the Nightly Jenkins CI for
> Hadoop on the Windows 10 platform[1]. The effort mainly involved getting
> Yetus to run on Windows against Hadoop.
> The nightly CI will run every 36 hours and send out the build report to the
> same recipients as this email, upon completion.
> There are still quite a few things that need to be sorted out. Currently,
> mvninstall has a +1. Other phases like mvnsite, javadoc etc still need to be
> fixed.
>
> [1]
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>
> Thanks,
> --Gautham

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/485/

[May 5, 2023, 10:57:42 AM] (github) HADOOP-18706. Improve S3ABlockOutputStream 
recovery (#5563)
[May 5, 2023, 8:08:58 PM] (github) HADOOP-18729. Fix mvnsite on Windows 10 
(#5618)
[May 5, 2023, 8:41:38 PM] (github) YARN-11470. FederationStateStoreFacade Cache 
Support Guava Cache. (#5609)
[May 5, 2023, 8:43:40 PM] (github) YARN-11477. [Federation] 
MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616)
[May 6, 2023, 3:20:21 AM] (github) HDFS-16865. The source path is always / 
after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. 
(#5200). Contributed by ZanderXu.




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory))
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory))
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
  doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullc

Re: [DISCUSS] Hadoop 3.3.6 release planning

[Ayush Saxena]

That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
stuff got sorted out, Steve knew a way out

On Sat, 6 May 2023 at 00:51, Ayush Saxena  wrote:
>
> Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>
> With my Hive developer hat on, I would like to bring some stuff up for
> consideration(feel free to say no, if it is beyond scope or feels even
> a bit unsafe, don't want to mess up the release)
>
> * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
> compaction in Hive, bothers only with HDFS though. There is a
> workaround to that, if it doesn't feel safe. no issues, or if some
> improvements suggested. I can quickly do that :)
>
> * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
> but if it works and is safe then to 2.2.5. I got flagged today that
> this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
> think(need to dig in more),
>
> Now for the dependency upgrades:
>
> A big NO to Jackson, that ain't safe and the wounds are still fresh,
> it screwed the 3.3.3 release for many projects. So, let's not get into
> that. Infact anything that touches those shaded jars is risky, some
> package-json exclusion also created a mess recently. So, Lets not
> touch only and that too when we have less time.
>
> Avoid anything around Jetty upgrade, I have selfish reasons for that.
> Jetty messes something up with Hbase and Hive has a dependency on
> Hbase, and it is crazy, in case interested [1]. So, any upgrade to
> Jetty will block hive from upgrading Hadoop as of today. But that is a
> selfish reason and just up for consideration. Go ahead if necessary. I
> just wanted to let folks know
>
>
> Apart from the Jackson stuff, everything is suggestive in nature, your
> call feel free to ignore.
>
> @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
> considering the timelines, but if we find a few fancy safe tickets,
> maybe if you have identified any already, can put them up on this
> thread and if folks are convinced. We can get them in? Juzz my
> thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>
>
> God Luck
>
> -Ayush
>
> [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>
> On Fri, 5 May 2023 at 16:13, Steve Loughran  
> wrote:
> >
> > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very broken,
> > api change for ozone and any critical jar updates"
> >
> > so much lower risk/easier to qualify and ship.
> >
> > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in here;
> > maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> > remove json-smart...)
> >
> > I'd also like to downgrade protobuf 2.5 from required to optional; even
> > though hadoop uses the shaded one, to support hbase etc the IPC code still
> > has direct use of the 2.5 classes. that coud be optional
> >
> > if anyone wants to take up this PR, I would be very happy
> > https://github.com/apache/hadoop/pull/4996
> >
> > On Fri, 5 May 2023 at 04:27, Xiaoqiao He  wrote:
> >
> > > Thanks Wei-Chiu for driving this release.
> > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed version.
> > >
> > > so far only 8 jiras were resolved in the branch-3.3 line.
> > >
> > >
> > > If we should consider both 3.3.6 and 3.3.9 (which is from release-3.3.5
> > > discuss)[1] for this release line?
> > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > > fixVersion in (3.3.6, 3.3.9)`[2],
> > > there are more than hundred jiras now.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > > [2]
> > >
> > > https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> > >
> > >
> > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang  wrote:
> > >
> > > > Hi community,
> > > >
> > > > I'd like to kick off the discussion around Hadoop 3.3.6 release plan.
> > > >
> > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs in
> > > > HADOOP-18671  added
> > > so
> > > > we can have HBase to adopt this new API. Other than that, perhaps
> > > > thirdparty dependency updates.
> > > >
> > > > If you have open items to be added in the coming weeks, please add 3.3.6
> > > to
> > > > the target release version. Right now I am only seeing three open jiras
> > > > targeting 3.3.6.
> > > >
> > > > I imagine this is going to be a small release as 3.3.5 (hat tip to 
> > > > Steve)
> > > > was only made two months back, and so far only 8 jiras were resolved in
> > > the
> > > > branch-3.3 line.
> > > >
> > > > Best,
> > > > Weichiu
> > > >
> > >

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h..

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1021/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Gautham Banasandra]

Hi Ayush,

I was planning to set up the precommit CI job for Windows 10, so that
we validate each PR against Windows 10. The precommit CI can actually
run in parallel to the Linux precommit CI. But I think we've got very few
Windows nodes (about 12 I think) compared to the number of Linux nodes
(about 40 in number).
However, I think the setting up the precommit CI is quite worth the effort.
It was quite hard to get to this point and we certainly don't want to
regress
in this regard. I think I'll pursue this once I stabilize the nightly CI
and after
I sort out the rest of the failing parts (mvnsite, javadoc etc).

Also, does Github Actions provide any advantage over precommit CI?

Thanks,
--Gautham

On Tue, 9 May 2023 at 02:44, Ayush Saxena  wrote:

> Awesome!!!
>
> Do you plan to run some builds per PR to ensure no commits breaks it
> in the future? Worth exploring GithubActions in the later stages if we
> can have an action for maven install on windows, it won't affect the
> normal build time as well. can run in parallel with the existing
> builds.
>
> -Ayush
>
> On Mon, 8 May 2023 at 23:39, Gautham Banasandra 
> wrote:
> >
> > Dear Hadoop community,
> >
> > It is my pleasure to announce that I've set up the Nightly Jenkins CI for
> > Hadoop on the Windows 10 platform[1]. The effort mainly involved getting
> > Yetus to run on Windows against Hadoop.
> > The nightly CI will run every 36 hours and send out the build report to
> the
> > same recipients as this email, upon completion.
> > There are still quite a few things that need to be sorted out. Currently,
> > mvninstall has a +1. Other phases like mvnsite, javadoc etc still need
> to be
> > fixed.
> >
> > [1]
> >
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
> >
> > Thanks,
> > --Gautham
>

[jira] [Created] (HDFS-17005) NameJournalStatus JMX is not updated with new JN IP address on JN host change

[Prateek Agarwal (Jira)]

Prateek Agarwal created HDFS-17005:
--

 Summary: NameJournalStatus JMX is not updated with new JN IP 
address on JN host change
 Key: HDFS-17005
 URL: https://issues.apache.org/jira/browse/HDFS-17005
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: journal-node
Affects Versions: 3.3.4, 2.10.2, 2.8.2
Reporter: Prateek Agarwal


Whenever a JournalNode host gets replaced, 'org.apache.hadoop.ipc.Client' is 
able to refresh the new JournalNode IP by re-resolving the JN DNS or host.

However, the JMX metrics exposed by Namenode still point to the stale JN IP 
address. For example:
{code}
"NameJournalStatus" : "[{\"manager\":\"QJM to [10.92.29.151:8485, 
10.94.17.167:8485, 10.92.59.158:8485]\",\"stream\":\"Writing segment beginning 
at txid 93886612. \\n10.92.29.151:8485 (Written txid 93894676), 
10.94.17.167:8485 (Written txid 93894674 (2 txns/1ms behind)), 
10.92.59.158:8485 (Written txid 
93894676)\",\"disabled\":\"false\",\"required\":\"true\"},{\"manager\":\"FileJournalManager(root=/data/1/dfs/name-data)\",\"stream\":\"EditLogFileOutputStream(/data/1/dfs/name-data/current/edits_inprogress_00093886612)\",\"disabled\":\"false\",\"required\":\"false\"}]",
{code}

The IP address '10.92.29.151' isn't updated even if the JN node now points to 
an updated IP address, say, '10.36.72.221'.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org