Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2020-10-05 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/

No changes




-1 overall


The following subsystems voted -1:
asflicense pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.TestFileChecksum 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.TestDFSShell 
   hadoop.hdfs.TestFileChecksumCompositeCrc 
   hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier 
   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks 
   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics 
   hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.tools.TestDistCpSystem 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.yarn.sls.TestReservationSystemInvariants 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-compile-cc-root.txt
  [48K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-compile-javac-root.txt
  [568K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-checkstyle-root.txt
  [16M]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-patch-shelldocs.txt
  [44K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/whitespace-eol.txt
  [13M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/whitespace-tabs.txt
  [1.9M]

   xml:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/xml.txt
  [24K]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/diff-javadoc-javadoc-root.txt
  [1.3M]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [560K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt
  [32K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [12K]

   asflicense:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/286/artifact/out/patch-asflicens

Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-05 Thread Wei-Chiu Chuang 
We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and
3.1.x.
Hermanth experienced the same issue on Hadoop 3.1.1 as well (HDFS-15569
)

On Mon, Oct 5, 2020 at 11:03 AM Igor Dvorzhak  wrote:

> What Hadoop 3 version do you use?
>
> On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang 
> wrote:
>
>> I have anecdotally learned of multiple data points where during the
>> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point
>> where it goes OOM.
>>
>> Don't have much logs for this issue, but I suspect it's caused by the
>> layout change added in Hadoop 2.8.0.
>>
>> Does anyone else observe the same issue and how do you mitigate this? For
>> now we suggested increasing DN heap size prior to upgrade as part of
>> pre-upgrade checklist.
>>
>> Thanks,
>> Wei-Chiu
>>
>

[jira] [Created] (HDFS-15615) HDFS user permission does not work correctly

2020-10-05 Thread Heba Aboulmagd (Jira) 
Heba Aboulmagd created HDFS-15615:
-

 Summary: HDFS user permission does not work correctly
 Key: HDFS-15615
 URL: https://issues.apache.org/jira/browse/HDFS-15615
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Heba Aboulmagd


HDFS users can create/delete folders for each other, while the permission is 
set to {{745}} on {{/user/username}}.

I used the following commands to create users and set their permissions
 {{useradd -m -g hdfs username}}
{{su hdfs}}
{{hdfs dfs -mkdir -p /user/username}}
{{hdfs dfs -chown username:hdfs /user/username}}
{{hdfs dfs -chmod 745 /user/username}}
when {{user1}} executes: {{hdfs dfs -mkdir /user/user2/from_user1}}
 {{The directory is created successfully}}
The same happens when {{user1}} tries to delete files from {{/user/user2}} 
while the permission is set to read only for the users in the same group.g



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-05 Thread Igor Dvorzhak 
What Hadoop 3 version do you use?

On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang  wrote:

> I have anecdotally learned of multiple data points where during the
> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point
> where it goes OOM.
>
> Don't have much logs for this issue, but I suspect it's caused by the
> layout change added in Hadoop 2.8.0.
>
> Does anyone else observe the same issue and how do you mitigate this? For
> now we suggested increasing DN heap size prior to upgrade as part of
> pre-upgrade checklist.
>
> Thanks,
> Wei-Chiu
>


smime.p7s
Description: S/MIME Cryptographic Signature

Increased DN heap usage during Hadoop 3 upgrade

2020-10-05 Thread Wei-Chiu Chuang 
I have anecdotally learned of multiple data points where during the
upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point
where it goes OOM.

Don't have much logs for this issue, but I suspect it's caused by the
layout change added in Hadoop 2.8.0.

Does anyone else observe the same issue and how do you mitigate this? For
now we suggested increasing DN heap size prior to upgrade as part of
pre-upgrade checklist.

Thanks,
Wei-Chiu

[jira] [Created] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled

2020-10-05 Thread Siyao Meng (Jira) 
Siyao Meng created HDFS-15614:
-

 Summary: Initialize snapshot trash root during NameNode startup if 
enabled
 Key: HDFS-15614
 URL: https://issues.apache.org/jira/browse/HDFS-15614
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Siyao Meng


This is a follow-up to HDFS-15607.

Goal:

Initialize (create) snapshot trash root for all existing snapshottable 
directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to {{true}}. 
So admins won't have to run {{dfsadmin -provisionTrash}} manually on all those 
existing snapshottable directories.

The change is expected to land in {{FSNamesystem}}.

Discussion:
1. Currently in HDFS-15607, the snapshot trash root creation logic is on the 
client side. But in order for NN to create it at startup, the logic must (also) 
be implemented on the server side as well. -- which is also a requirement by 
WebHDFS (HDFS-15612).
2. Alternatively, we can provide an extra parameter to the {{-provisionTrash}} 
command like: {{dfsadmin -provisionTrash -all}} to initialize/provision trash 
root on all existing snapshottable dirs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2020-10-05 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint jshint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml 
   hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-tools/hadoop-azure/src/config/checkstyle.xml 
   hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

Failed junit tests :

   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.TestRollingUpgrade 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.tools.TestDistCpSystem 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   jshint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-patch-jshint.txt
  [208K]

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-compile-javac-root.txt
  [456K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/whitespace-tabs.txt
  [1.3M]

   xml:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/diff-javadoc-javadoc-root.txt
  [20K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [216K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [280K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [116K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/77/artifact/out/pat

[jira] [Created] (HDFS-15613) RBF: Router FSCK fails after HDFS-15223

2020-10-05 Thread Akira Ajisaka (Jira) 
Akira Ajisaka created HDFS-15613:


 Summary: RBF: Router FSCK fails after HDFS-15223
 Key: HDFS-15613
 URL: https://issues.apache.org/jira/browse/HDFS-15613
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: rbf
Affects Versions: 3.3.0
Reporter: Akira Ajisaka


After HDFS-15223 fsck uses getHAServiceState operation to detect Active 
NameNode, however, DFSRouter does not support the operation.
{noformat}
20/10/05 16:41:30 DEBUG hdfs.HAUtil: Error while connecting to namenode
org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException): 
Operation "getHAServiceState" is not supported
at 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:488)
at 
org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getHAServiceState(RouterClientProtocol.java:1773)
at 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getHAServiceState(RouterRpcServer.java:1333)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getHAServiceState(ClientNamenodeProtocolServerSideTranslatorPB.java:2011)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
at com.sun.proxy.$Proxy12.getHAServiceState(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getHAServiceState(ClientNamenodeProtocolTranslatorPB.java:2055)
at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:281)
at 
org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:271)
at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:339)
at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:75)
at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:164)
at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:160)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:409)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org