Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

2022-03-17 Thread Masatake Iwasaki 

I found that INFO level log message is emitted on every CLI invocation.

 $ bin/hdfs dfs -mkdir /tmp
 2022-03-17 23:38:55,141 INFO Configuration.deprecation: No unit for 
dfs.client.datanode-restart.timeout(30) assuming SECONDS

The time unit was intentionally removed from hdfs-default.xml by DFS-12920,
There seems to be no quick fix though it is noisy.


HDFS-14630 is the fix for this.
I will backport this to branch-3.2.3 if I need to cut RC1.

On 2022/03/18 0:54, Masatake Iwasaki wrote:

+1 from myself.

* built the source tarball on CentOS 7 (x86_64) and OpenJDK 8 by `mvn install 
-DskipTests -Pnative -Pdist`.

* launched pseudo distributed cluster.
   * ran some MR example jobs.
   * launched `spark-shell --master yarn` with spark-3.2.1-bin-without-hadoop 
and ran some tutorial code.
* launched pseudo distributed cluster with Kerberos security enabled.
   * launched KMS and created encryption zone.
   * ran sample MR wordcount job over encryption zone.

* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.

* built site documentation by `mvn site site:stage` and skimmed the contents.

* built HBase 2.4.8, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.3 RC0
   on Rocky Linux 8 (x86_64) by using Bigtop and ran smoke-tests.
   https://issues.apache.org/jira/browse/BIGTOP-3606


I found that INFO level log message is emitted on every CLI invocation.

     $ bin/hdfs dfs -mkdir /tmp
     2022-03-17 23:38:55,141 INFO Configuration.deprecation: No unit for 
dfs.client.datanode-restart.timeout(30) assuming SECONDS

The time unit was intentionally removed from hdfs-default.xml by DFS-12920,
There seems to be no quick fix though it is noisy.


Masatake Iwasaki


On 2022/03/16 18:36, Viraj Jasani wrote:

+1 (non-binding)

Using hadoop-vote.sh

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
  - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
  - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Tested HDFS on pseudo-distributed mode with HBase 2.4 pseudo-distributed 
cluster (1M rows ingested), all good.

Test PR to run full build and track UT failures 
https://github.com/apache/hadoop/pull/4073 
, few tests are flaky but they are 
passing locally.


On Mon, Mar 14, 2022 at 12:45 PM Masatake Iwasaki mailto:iwasak...@oss.nttdata.co.jp>> wrote:

    Hi all,

    Here's Hadoop 3.2.3 release candidate #0:

    The RC is available at:
    https://home.apache.org/~iwasakims/hadoop-3.2.3-RC0/ 


    The RC tag is at:
    https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC0 


    The Maven artifacts are staged at:
    https://repository.apache.org/content/repositories/orgapachehadoop-1339 


    You can find my public key at:
    https://downloads.apache.org/hadoop/common/KEYS 


    Please evaluate the RC and vote.

    Thanks,
    Masatake Iwasaki

    -
    To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 

    For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2022-03-17 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/

[Mar 16, 2022 12:43:38 PM] (Xiaoqiao He) HDFS-16494.Removed reuse of 
AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by 
JiangHua Zhu.




-1 overall


The following subsystems voted -1:
blanks pathlen xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-compile-javac-root.txt
 [340K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-checkstyle-root.txt
 [14M]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/811/artifact/out/results-javadoc-javadoc-root.txt
 [404K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

2022-03-17 Thread Masatake Iwasaki 

+1 from myself.

* built the source tarball on CentOS 7 (x86_64) and OpenJDK 8 by `mvn install 
-DskipTests -Pnative -Pdist`.

* launched pseudo distributed cluster.
  * ran some MR example jobs.
  * launched `spark-shell --master yarn` with spark-3.2.1-bin-without-hadoop 
and ran some tutorial code.
  
* launched pseudo distributed cluster with Kerberos security enabled.

  * launched KMS and created encryption zone.
  * ran sample MR wordcount job over encryption zone.

* launched HA enabled 3-nodes docker cluster and ran sample MR jobs.

* built site documentation by `mvn site site:stage` and skimmed the contents.

* built HBase 2.4.8, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.3 RC0
  on Rocky Linux 8 (x86_64) by using Bigtop and ran smoke-tests.
  https://issues.apache.org/jira/browse/BIGTOP-3606


I found that INFO level log message is emitted on every CLI invocation.

$ bin/hdfs dfs -mkdir /tmp
2022-03-17 23:38:55,141 INFO Configuration.deprecation: No unit for 
dfs.client.datanode-restart.timeout(30) assuming SECONDS

The time unit was intentionally removed from hdfs-default.xml by DFS-12920,
There seems to be no quick fix though it is noisy.


Masatake Iwasaki


On 2022/03/16 18:36, Viraj Jasani wrote:

+1 (non-binding)

Using hadoop-vote.sh

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
  - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
  - mvn clean install  -DskipTests
* Built tar from source (1.8.0_301): ok
  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Tested HDFS on pseudo-distributed mode with HBase 2.4 pseudo-distributed 
cluster (1M rows ingested), all good.

Test PR to run full build and track UT failures 
https://github.com/apache/hadoop/pull/4073 
, few tests are flaky but they are 
passing locally.


On Mon, Mar 14, 2022 at 12:45 PM Masatake Iwasaki mailto:iwasak...@oss.nttdata.co.jp>> wrote:

Hi all,

Here's Hadoop 3.2.3 release candidate #0:

The RC is available at:
https://home.apache.org/~iwasakims/hadoop-3.2.3-RC0/ 


The RC tag is at:
https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC0 


The Maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1339 


You can find my public key at:
https://downloads.apache.org/hadoop/common/KEYS 


Please evaluate the RC and vote.

Thanks,
Masatake Iwasaki

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 

For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2022-03-17 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/273/

[Mar 16, 2022 1:32:29 AM] (noreply) HDFS-16502. Reconfigure Block Invalidate 
limit (#4064)
[Mar 16, 2022 12:43:38 PM] (Xiaoqiao He) HDFS-16494.Removed reuse of 
AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by 
JiangHua Zhu.
[Mar 17, 2022 4:04:48 AM] (noreply) HDFS-16499. [SPS]: Should not start 
indefinitely while another SPS process is running (#4058)


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16510) Fix EC decommission when rack is not enough

2022-03-17 Thread daimin (Jira) 
daimin created HDFS-16510:
-

 Summary: Fix EC decommission when rack is not enough
 Key: HDFS-16510
 URL: https://issues.apache.org/jira/browse/HDFS-16510
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: block placement, ec
Affects Versions: 3.3.2, 3.3.1
Reporter: daimin
Assignee: daimin


The decommission always fail when we start decommission multiple nodes on a 
cluster whose racks is not enough, a cluster with 6 racks to deploy RS-6-3, for 
example.

We find that those decommission nodes cover at least a rack, it's actulaly like 
we are decommission one or more racks. And rack decommission is not well 
supported currently, especially for cluster whose racks is not enough already.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2022-03-17 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/

[Mar 15, 2022 10:40:17 PM] (Owen O'Malley) HADOOP-13722. Code cleanup -- 
ViewFileSystem and InodeTree. Contributed by Manoj Govindassamy.
[Mar 15, 2022 10:40:17 PM] (Owen O'Malley) HADOOP-12077. Provide a multi-URI 
replication Inode for ViewFs. Contributed by Gera Shegalov
[Mar 16, 2022 12:04:50 AM] (Owen O'Malley) HADOOP-13055. Implement 
linkMergeSlash and linkFallback for ViewFileSystem




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestTrash 
   hadoop.io.compress.snappy.TestSnappyCompressorDecompressor 
   hadoop.fs.TestFileUtil 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.tools.TestDistCpSystem 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-compile-javac-root.txt
  [476K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-mvnsite-root.txt
  [556K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-javadoc-root.txt
  [40K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [224K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [428K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [112K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/603/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC0

2022-03-17 Thread Masatake Iwasaki 

Hadoop 3.2.x depends on Protocol Buffers 2.5.0 which does not support aarch64.

If you want to build 3.2.3 RC0 on aarch64,
you can use the patch backporting aarch64 support.
https://github.com/apache/bigtop/blob/master/bigtop_toolchain/files/0001-Backport-atomic-operations-with-support-of-arm64-and.patch

You will be able to build Hadoop 3.2.3 package on aarch64 by using Bigtop after 
BIGTOP-3606 is fixed.
https://issues.apache.org/jira/browse/BIGTOP-3606

On 2022/03/17 15:40, Wei-Chiu Chuang wrote:

aarch64 support is only introduced in/after 3.3.0

On Thu, Mar 17, 2022 at 2:27 PM Emil Ejbyfeldt
 wrote:


Hi,


There is no aarch64 artifact in the release candidate. Is this something
that is intended?

Best,
Emil Ejbyfeldt

On 14/03/2022 08:14, Masatake Iwasaki wrote:

Hi all,

Here's Hadoop 3.2.3 release candidate #0:

The RC is available at:
https://home.apache.org/~iwasakims/hadoop-3.2.3-RC0/

The RC tag is at:
https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC0

The Maven artifacts are staged at:


https://repository.apache.org/content/repositories/orgapachehadoop-1339


You can find my public key at:
https://downloads.apache.org/hadoop/common/KEYS

Please evaluate the RC and vote.

Thanks,
Masatake Iwasaki

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org






-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org