[jira] [Resolved] (HDFS-16433) The synchronized lock of IncrementalBlockReportManager will slow down the speed of creating/deleting/finalizing/opening block

2022-01-28 Thread Yuanbo Liu (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuanbo Liu resolved HDFS-16433.
---
Resolution: Not A Problem

> The synchronized lock of IncrementalBlockReportManager will slow down the 
> speed of creating/deleting/finalizing/opening block
> -
>
> Key: HDFS-16433
> URL: https://issues.apache.org/jira/browse/HDFS-16433
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Critical
> Attachments: image-2022-01-21-22-52-52-912.png
>
>
> The code in IncrementalBlockReportManager.java
> {code:java}
> synchronized void waitTillNextIBR(long waitTime) {
>   if (waitTime > 0 && !sendImmediately()) {
> try {
>   wait(ibrInterval > 0 && ibrInterval < waitTime? ibrInterval: waitTime);
> } catch (InterruptedException ie) {
>   LOG.warn(getClass().getSimpleName() + " interrupted");
> }
>   }
> } {code}
> We can see that wait(waitime) will hold synchronized, if ibr interval is 
> enabled or heartbeat time is not reached. The lock will block 
> notifyNamenodeBlock function which is widely used in 
> deleting/creating/finalizing/opening blocks, then the speed of DataNode IO 
> will slow down. Here is the graph of blocking relationship
> !image-2022-01-21-22-52-52-912.png|width=976,height=299!
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16444) Show start time of JournalNode on Web

2022-01-28 Thread tomscut (Jira) 
tomscut created HDFS-16444:
--

 Summary: Show start time of JournalNode on Web
 Key: HDFS-16444
 URL: https://issues.apache.org/jira/browse/HDFS-16444
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: tomscut
Assignee: tomscut
 Attachments: image-2022-01-29-08-09-42-544.png, 
image-2022-01-29-08-09-53-734.png

Show start time of JournalNode on Web.

Before:

!image-2022-01-29-08-09-42-544.png|width=379,height=98!

After:

!image-2022-01-29-08-09-53-734.png|width=378,height=118!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2022-01-28 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/

[Jan 27, 2022 4:42:44 AM] (noreply) HDFS-16427. Add debug log for 
BlockManager#chooseExcessRedundancyStriped (#3888)
[Jan 27, 2022 8:53:21 AM] (noreply) HDFS-16429. Add DataSetLockManager to 
manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang.
[Jan 27, 2022 3:04:01 PM] (noreply) YARN-11068. Exclude transitive log4j2 
dependency coming from solr 8. (#3936)
[Jan 27, 2022 3:36:25 PM] (Akira Ajisaka) Revert "YARN-11068. Exclude 
transitive log4j2 dependency coming from solr 8. (#3936)"




-1 overall


The following subsystems voted -1:
blanks compile golang mvninstall mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.ipc.TestIPC 
   hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure 
   hadoop.yarn.csi.client.TestCsiClient 
  

   mvninstall:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-mvninstall-root.txt
 [1.6M]

   compile:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-compile-root.txt
 [1.2M]

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-compile-root.txt
 [1.2M]

   golang:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-compile-root.txt
 [1.2M]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-compile-root.txt
 [1.2M]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/blanks-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/results-checkstyle-root.txt
 [14M]

   mvnsite:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/patch-mvnsite-root.txt
 [392K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/results-shellcheck.txt
 [28K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/results-javadoc-javadoc-root.txt
 [404K]

   spotbugs:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn.txt
 [344K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications.txt
 [28K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog.txt
 [8.0K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt
 [8.0K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-spotbugs-hadoop-yarn-project.txt
 [344K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/764/artifact/out/branch-s

[jira] [Created] (HDFS-16443) Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception

2022-01-28 Thread Kevin Wikant (Jira) 
Kevin Wikant created HDFS-16443:
---

 Summary: Fix edge case where DatanodeAdminDefaultMonitor doubly 
enqueues a DatanodeDescriptor on exception
 Key: HDFS-16443
 URL: https://issues.apache.org/jira/browse/HDFS-16443
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Reporter: Kevin Wikant


As part of the fix merged in: https://issues.apache.org/jira/browse/HDFS-16303

There was a rare edge case noticed in DatanodeAdminDefaultMonitor which causes 
a DatanodeDescriptor to be added twice to the pendingNodes queue. 
 * a [datanode is unhealthy so it gets added to 
"unhealthyDns"]([https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L227)]
 * an exception is thrown which causes [this catch 
block](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L271)
 to execute
 * the [datanode is added to 
"pendingNodes"]([https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L276)]
 * under certain conditions the [datanode can be added again from 
"unhealthyDns" to "pendingNodes" 
here]([https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L296)]

This Jira is to track the 1 line fix for this bug



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2022-01-28 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.fs.TestTrash 
   hadoop.io.compress.snappy.TestSnappyCompressorDecompressor 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-compile-javac-root.txt
  [500K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-mvnsite-root.txt
  [608K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-javadoc-root.txt
  [76K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [248K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [428K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [28K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/556/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applica

[jira] [Resolved] (HDFS-16441) The following error occurs when accessing webhdfs in Kerberos security mode:Failed to obtain user group information: java.io.IOException: Security enabled but user not a

2022-01-28 Thread Akira Ajisaka (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HDFS-16441.
--
Resolution: Invalid

Hi [~xiaoqiuqiu] - In Apache Hadoop community, JIRA is used for the development 
and not for end-user questions. Please use 
[u...@hadoop.apache.org|mailto:u...@hadoop.apache.org] mailing list for 
end-user questions.

> The following error occurs when accessing webhdfs in Kerberos security 
> mode:Failed to obtain user group information: java.io.IOException: Security 
> enabled but user not authenticated by filter
> ---
>
> Key: HDFS-16441
> URL: https://issues.apache.org/jira/browse/HDFS-16441
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: 3.1.1
>Affects Versions: 3.3.1
>Reporter: xiaoqiuqiu
>Priority: Major
> Attachments: 1.png, 2.png, 3.png, 4.png
>
>
> The following error occurs when accessing webhdfs in Kerberos security 
> mode:Failed to obtain user group information: java.io.IOException: Security 
> enabled but user not authenticated by filter；
> When I use the browser to access, I still get the same error when the machine 
> has Kerberos authentication；
>  
> The code is the first in the comment area
> How to solve it？



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

2022-01-28 Thread Akira Ajisaka 
Thank you Masatake and Chao!

On Fri, Jan 28, 2022 at 5:11 PM Chao Sun  wrote:

> Thanks Masatake and Akira for discovering the issue. I used
> `dev-support/bin/create-release` which runs `mvn deploy -DskipTests
> -Pnative -Pdist ...` in a separate container and somehow it didn't hit this
> issue.
>
> Let me cherry-pick https://issues.apache.org/jira/browse/YARN-10561 to
> branch-3.3.2 and start another RC then.
>
> Thanks,
> Chao
>
> On Fri, Jan 28, 2022 at 12:01 AM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
>> Thanks, Akira.
>>
>> I confirmed that the issue is fixed in current branch-3.3 containing
>> YARN-10561.
>>
>> On 2022/01/28 14:25, Akira Ajisaka wrote:
>> > Hi Masatake,
>> >
>> > I faced the same error in a clean environment and
>> https://issues.apache.org/jira/browse/YARN-10561 <
>> https://issues.apache.org/jira/browse/YARN-10561> should fix this issue.
>> I'll rebase the patch shortly.
>> >
>> > By the way, I'm afraid there is no active maintainer in
>> hadoop-yarn-applications-catalog module. The module is for a sample
>> application catalog, so I think we can move the module to a separate
>> repository. Of course, it should be discussed separately.
>> >
>> > Thanks and regards,
>> > Akira
>> >
>> > On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp > wrote:
>> >
>> > Thanks for putting this up, Chao Sun.
>> >
>> > I got following error on building the RC3 source tarball.
>> > It is reproducible even in the container launched by
>> `./start-build-env.sh`.
>> > There seems to be no relevant diff between release-3.3.2-RC0 and
>> release-3.3.2-RC3 (and trunk)
>> > under hadoop-yarn-applications-catalog-webapp.
>> >
>> > I guess developers having caches of related artifacts under ~/.m2
>> did not see this?
>> >
>> > ```
>> > $ mvn clean install -DskipTests -Pnative -Pdist
>> > ...
>> > [INFO] Installing node version v8.11.3
>> > [INFO] Downloading
>> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz <
>> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz> to
>> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
>> > [INFO] No proxies configured
>> > [INFO] No proxy was configured, downloading directly
>> > [INFO] Unpacking
>> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
>> into
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
>> > [INFO] Copying node binary from
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
>> to
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
>> > [INFO] Installed node locally.
>> > [INFO] Installing Yarn version v1.7.0
>> > [INFO] Downloading
>> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
>> <
>> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz>
>> to
>> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
>> > [INFO] No proxies configured
>> > [INFO] No proxy was configured, downloading directly
>> > [INFO] Unpacking
>> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
>> into
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
>> > [INFO] Installed Yarn locally.
>> > [INFO]
>> > [INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @
>> hadoop-yarn-applications-catalog-webapp ---
>> > [INFO] testFailureIgnore property is ignored in non test phases
>> > [INFO] Running 'yarn ' in
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target
>> > [INFO] yarn install v1.7.0
>> > [INFO] info No lockfile found.
>> > [INFO] [1/4] Resolving packages...
>> > [INFO] [2/4] Fetching packages...
>> > [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
>> incompatible with this module. Expected version ">=10".
>> > [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
>> incompatible with this module. Expected version ">=10".info Visit
>> https://yarnpkg.com/en/docs/cli/install <
>> https://yarnpkg.com/en/docs/cli/install> for documentation about this
>> command.
>> > [INFO] error Found incompatible module
>> > [INFO]
>> --

[jira] [Resolved] (HDFS-16169) Fix TestBlockTokenWithDFSStriped#testEnd2End failure

2022-01-28 Thread Akira Ajisaka (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HDFS-16169.
--
Fix Version/s: 3.4.0
   3.3.3
   Resolution: Fixed

Committed to trunk and branch-3.3. Thank you [~secfree.teng] for your 
contribution! Nice catch.

> Fix TestBlockTokenWithDFSStriped#testEnd2End failure
> 
>
> Key: HDFS-16169
> URL: https://issues.apache.org/jira/browse/HDFS-16169
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Affects Versions: 3.4.0
>Reporter: Hui Fei
>Assignee: secfree
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.3
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 141.936 s <<< FAILURE! - in 
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
> [ERROR] 
> testEnd2End(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
>  Time elapsed: 28.325 s <<< FAILURE! java.lang.AssertionError: expected:<9> 
> but was:<10> at org.junit.Assert.fail(Assert.java:89) at 
> org.junit.Assert.failNotEquals(Assert.java:835) at 
> org.junit.Assert.assertEquals(Assert.java:647) at 
> org.junit.Assert.assertEquals(Assert.java:633) at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLocatedStripedBlocks(StripedFileTestUtil.java:344)
>  at 
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTestBalancerWithStripedFile(TestBalancer.java:1666)
>  at 
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.integrationTestWithStripedFile(TestBalancer.java:1601)
>  at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testEnd2End(TestBlockTokenWithDFSStriped.java:119)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.lang.Thread.run(Thread.java:748)
>  
> CI result is 
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3296/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

2022-01-28 Thread Chao Sun 
Thanks Masatake and Akira for discovering the issue. I used
`dev-support/bin/create-release` which runs `mvn deploy -DskipTests
-Pnative -Pdist ...` in a separate container and somehow it didn't hit this
issue.

Let me cherry-pick https://issues.apache.org/jira/browse/YARN-10561 to
branch-3.3.2 and start another RC then.

Thanks,
Chao

On Fri, Jan 28, 2022 at 12:01 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Thanks, Akira.
>
> I confirmed that the issue is fixed in current branch-3.3 containing
> YARN-10561.
>
> On 2022/01/28 14:25, Akira Ajisaka wrote:
> > Hi Masatake,
> >
> > I faced the same error in a clean environment and
> https://issues.apache.org/jira/browse/YARN-10561 <
> https://issues.apache.org/jira/browse/YARN-10561> should fix this issue.
> I'll rebase the patch shortly.
> >
> > By the way, I'm afraid there is no active maintainer in
> hadoop-yarn-applications-catalog module. The module is for a sample
> application catalog, so I think we can move the module to a separate
> repository. Of course, it should be discussed separately.
> >
> > Thanks and regards,
> > Akira
> >
> > On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp > wrote:
> >
> > Thanks for putting this up, Chao Sun.
> >
> > I got following error on building the RC3 source tarball.
> > It is reproducible even in the container launched by
> `./start-build-env.sh`.
> > There seems to be no relevant diff between release-3.3.2-RC0 and
> release-3.3.2-RC3 (and trunk)
> > under hadoop-yarn-applications-catalog-webapp.
> >
> > I guess developers having caches of related artifacts under ~/.m2
> did not see this?
> >
> > ```
> > $ mvn clean install -DskipTests -Pnative -Pdist
> > ...
> > [INFO] Installing node version v8.11.3
> > [INFO] Downloading
> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz <
> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz> to
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> > [INFO] No proxies configured
> > [INFO] No proxy was configured, downloading directly
> > [INFO] Unpacking
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> into
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
> > [INFO] Copying node binary from
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
> to
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
> > [INFO] Installed node locally.
> > [INFO] Installing Yarn version v1.7.0
> > [INFO] Downloading
> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
> <
> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz>
> to
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> > [INFO] No proxies configured
> > [INFO] No proxy was configured, downloading directly
> > [INFO] Unpacking
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> into
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
> > [INFO] Installed Yarn locally.
> > [INFO]
> > [INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @
> hadoop-yarn-applications-catalog-webapp ---
> > [INFO] testFailureIgnore property is ignored in non test phases
> > [INFO] Running 'yarn ' in
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target
> > [INFO] yarn install v1.7.0
> > [INFO] info No lockfile found.
> > [INFO] [1/4] Resolving packages...
> > [INFO] [2/4] Fetching packages...
> > [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
> incompatible with this module. Expected version ">=10".
> > [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
> incompatible with this module. Expected version ">=10".info Visit
> https://yarnpkg.com/en/docs/cli/install <
> https://yarnpkg.com/en/docs/cli/install> for documentation about this
> command.
> > [INFO] error Found incompatible module
> > [INFO]
> 
> > ```
> >
> > Masatake Iwasaki
> >
> >
> > On 2022/01/27 4:16, Chao Sun wrote:
> >  > Hi all,
> >  >
> >  > I've put together Hadoop 3.3

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

2022-01-28 Thread Masatake Iwasaki 

Thanks, Akira.

I confirmed that the issue is fixed in current branch-3.3 containing YARN-10561.

On 2022/01/28 14:25, Akira Ajisaka wrote:

Hi Masatake,

I faced the same error in a clean environment and 
https://issues.apache.org/jira/browse/YARN-10561 
 should fix this issue. I'll 
rebase the patch shortly.

By the way, I'm afraid there is no active maintainer in 
hadoop-yarn-applications-catalog module. The module is for a sample application 
catalog, so I think we can move the module to a separate repository. Of course, 
it should be discussed separately.

Thanks and regards,
Akira

On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki mailto:iwasak...@oss.nttdata.co.jp>> wrote:

Thanks for putting this up, Chao Sun.

I got following error on building the RC3 source tarball.
It is reproducible even in the container launched by `./start-build-env.sh`.
There seems to be no relevant diff between release-3.3.2-RC0 and 
release-3.3.2-RC3 (and trunk)
under hadoop-yarn-applications-catalog-webapp.

I guess developers having caches of related artifacts under ~/.m2 did not 
see this?

```
$ mvn clean install -DskipTests -Pnative -Pdist
...
[INFO] Installing node version v8.11.3
[INFO] Downloading https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz 
 to 
/home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
[INFO] No proxies configured
[INFO] No proxy was configured, downloading directly
[INFO] Unpacking 
/home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
 into 
/home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
[INFO] Copying node binary from 
/home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
 to 
/home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
[INFO] Installed node locally.
[INFO] Installing Yarn version v1.7.0
[INFO] Downloading 
https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz 
 
to /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
[INFO] No proxies configured
[INFO] No proxy was configured, downloading directly
[INFO] Unpacking 
/home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz 
into 
/home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
[INFO] Installed Yarn locally.
[INFO]
[INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @ 
hadoop-yarn-applications-catalog-webapp ---
[INFO] testFailureIgnore property is ignored in non test phases
[INFO] Running 'yarn ' in 
/home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target
[INFO] yarn install v1.7.0
[INFO] info No lockfile found.
[INFO] [1/4] Resolving packages...
[INFO] [2/4] Fetching packages...
[INFO] error safe-stable-stringify@2.3.1: The engine "node" is incompatible with this 
module. Expected version ">=10".
[INFO] error safe-stable-stringify@2.3.1: The engine "node" is incompatible with this module. 
Expected version ">=10".info Visit https://yarnpkg.com/en/docs/cli/install 
 for documentation about this command.
[INFO] error Found incompatible module
[INFO] 

```

Masatake Iwasaki


On 2022/01/27 4:16, Chao Sun wrote:
 > Hi all,
 >
 > I've put together Hadoop 3.3.2 RC3 below:
 >
 > The RC is available at: http://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ 

 > The RC tag is at:
 > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC3 

 > The Maven artifacts are staged at:
 > https://repository.apache.org/content/repositories/orgapachehadoop-1333 

 >
 > You can find my public key at:
 > https://downloads.apache.org/hadoop/common/KEYS 

 >
 > The only delta between this and RC2 is the add