[VOTE] Release Apache Hadoop 3.3.2 - RC0

[Chao Sun]

Hi all,

Sorry for the long delay. I've prepared RC0 for Hadoop 3.3.2 below:

The RC is available at: http://people.apache.org/~sunchao/hadoop-3.3.2-RC0/
The RC tag is at:
https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC0
The Maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1330/

You can find my public key at: https://people.apache.org/~sunchao/KEYS

Please evaluate the RC and vote.

Thanks,
Chao

[jira] [Created] (HDFS-16376) Expose metrics of NodeNotChosenReason to JMX

[tomscut (Jira)]

tomscut created HDFS-16376:
--

 Summary: Expose metrics of NodeNotChosenReason to JMX
 Key: HDFS-16376
 URL: https://issues.apache.org/jira/browse/HDFS-16376
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: tomscut
Assignee: tomscut
 Attachments: image-2021-12-09-23-48-42-865.png

In our cluster, we can see logs for nodes that are not chosen. But it's hard to 
see the percentages in each reason from the logs. It is best to add relevant 
metrics to monitor the entire cluster.

!image-2021-12-09-23-48-42-865.png|width=517,height=187!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/

[Dec 7, 2021 1:56:55 PM] (Szilard Nemeth) YARN-10757. 
jsonschema2pojo-maven-plugin version is not defined. Contributed by Tamas Domok
[Dec 7, 2021 2:26:07 PM] (Szilard Nemeth) YARN-11023. Extend the root QueueInfo 
with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
[Dec 7, 2021 2:43:55 PM] (Szilard Nemeth) YARN-11006. Allow overriding user 
limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by 
Benjamin Teke
[Dec 7, 2021 2:55:16 PM] (Szilard Nemeth) YARN-11016. Queue weight is 
incorrectly reset to zero. Contributed by Andras Gyori
[Dec 8, 2021 7:52:22 AM] (noreply) HADOOP-18033. Upgrade fasterxml Jackson to 
2.13.0 (#3749)
[Dec 8, 2021 9:48:43 AM] (noreply) HADOOP-18024. SocketChannel is not closed 
when IOException happens in Server$Listener.doAccept (#3719)
[Dec 8, 2021 11:04:54 AM] (noreply) HDFS-16345. Fix test case fail in 
TestBlockStoragePolicy  (#3696). Contributed by guophilipse.
[Dec 8, 2021 11:10:11 AM] (noreply) MAPREDUCE-7368. 
DBOutputFormat.DBRecordWriter#write must throw exception when it fails. 
(#3671). Contributed by  Stamatis Zampetakis.
[Dec 8, 2021 11:44:18 AM] (noreply) HDFS-15862. Make 
TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724)




-1 overall


The following subsystems voted -1:
blanks compile golang mvninstall pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead 
   hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure 
   hadoop.yarn.csi.client.TestCsiClient 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
  

   mvninstall:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-mvninstall-root.txt
 [1.9M]

   compile:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-compile-root.txt
 [1.2M]

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-compile-root.txt
 [1.2M]

   golang:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-compile-root.txt
 [1.2M]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-compile-root.txt
 [1.2M]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/results-checkstyle-root.txt
 [14M]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/results-javadoc-javadoc-root.txt
 [408K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 [528K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/713/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
 [64K]
  

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator 
   hadoop.tools.TestDistCpSystem 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-compile-javac-root.txt
  [500K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-mvnsite-root.txt
  [584K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-patch-pylint.txt
  [48K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/diff-patch-shelldocs.txt
  [48K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-javadoc-root.txt
  [32K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [232K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [428K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [40K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [128K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/506/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [40K]
   

Re: Next Hadoop community online meetup (Mandarin)

[Wei-Chiu Chuang]

Thanks for participating in today's session. We have nearly a hundred
participants so I am quite impressed!

Yiyang graciously agreed to share the presentation slides, which can be
downloaded from here:
https://drive.google.com/file/d/1aAplPKU2frkKZoLeeHxbAzxLeJcL3bsf/view?usp=sharing
(it's in English so the wider audience can read)

Please find today's recording here:

https://cloudera.zoom.us/rec/share/MOwRDPP50eL7gRfNazfjqAXdl_7HwAaPH460sONmmlx_tLQrftVjjjCBkVvE-9sf.GRb9Kp_16xOqgzNu
Passcode:
L^n30&%d

On Thu, Dec 9, 2021 at 2:10 PM Wei-Chiu Chuang  wrote:

> We are having a little technical difficulties.
> For those who are late, we are using this link instead:
> https://cloudera.zoom.us/j/93901387554
>
> On Thu, Dec 9, 2021 at 12:55 PM Takanobu Asanuma <
> takanobu.asan...@gmail.com> wrote:
>
>> Thanks for the information!
>>
>> 2021年12月9日(木) 13:52 Wei-Chiu Chuang :
>>
>>> Hello, this talk will start in about an hour.
>>>
>>> (I'm sorry forgot to add this is 2PM Beijing time)
>>>
>>> On Thu, Dec 9, 2021 at 11:00 AM Takanobu Asanuma 
>>> wrote:
>>>
 What is the timezone?

 2021年11月29日(月) 14:50 Wei-Chiu Chuang :

> Hello HDFS devs
>
> After a long hiatus, I'm happy to share that we'll have Yiyang from
> Shopee
> who is going to talk about their experience of using HDFS at Shopee.
>
> This is a Mandarin talk.
>
> 题目：《HDFS在Shopee的演进》
>
> 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时，如何借鉴社区成果，并结合内部具体需求，解决性能和稳定性上的问题。
> 关键词： Observer, RBF, slownode
> 演讲者：Yiyang Zhou (Apache id: Symious)
>
> 12月9日周四下午2点
>
> Zoom Meeting
>
> https://cloudera.zoom.us/j/880548968
>
> One tap mobile
>
> +16465588656,,880548968# US (New York)
>
> +17207072699,,880548968# US
>
> Download Center 
>
> Dial by your location
>
> +1 646 558 8656 US (New York)
>
> +1 720 707 2699 US
>
> 877 853 5257 US Toll-free
>
> 888 475 4499 US Toll-free
>
> Meeting ID: 880 548 968
>
> Find your local number: https://zoom.us/u/acaGRDfMVl
>