Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

2019-05-05 Thread Xiaoyu Yao 
+1 Binding. Thanks all who contributed to the release. 

+ Download sources and verify signature.
+ Build from source and ran docker-based ad-hot security tests.
++ From 1 datanode scale to 3 datanodes, verify certificates were correctly 
issued when security enabled
++ Smoke test for both non-secure and secure mode.
++ Put/Get/Delete/Rename Key with 
+++ Kerberos testing
+++ Delegation token testing with DTUtil CLI and MR jobs.
+++ S3 token.

Just have one minor question for the expanded source code which points to 
hadoop-3.3.0-SNAPSHOT-src-with-hdds/hadoop-ozone. But in hadoop-ozone/pom.xml, 
we explicitly declare dependency on Hadoop 3.2.0.
I understand we just take the trunk source code(3.3.0-SNAPSHOT up to the 
ozone-0.4 RC) here, should we fix this by giving the git hash of the trunk or 
clarify it to avoid confusion? 
This might be done by just updating the name of the binaries without reset the 
release itself. 

-Xiaoyu
 

On 5/3/19, 4:07 PM, "Dinesh Chitlangia"  wrote:

+1 (non-binding)

- Built from sources and ran smoke test
- Verified all checksums
- Toggled audit log and verified audit parser tool

Thanks Ajay for organizing the release.

Cheers,
Dinesh



On 5/3/19, 5:42 PM, "Eric Yang"  wrote:

+1

On 4/29/19, 9:05 PM, "Ajay Kumar"  wrote:

Hi All,



We have created the third release candidate (RC2) for Apache Hadoop 
Ozone 0.4.0-alpha.



This release contains security payload for Ozone. Below are some 
important features in it:



  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
  *   Transparent Data Encryption (TDE) Support - Allows data 
blocks to be encrypted-at-rest.
  *   Kerberos support for Ozone.
  *   Certificate Infrastructure for Ozone  - Tokens use PKI 
instead of shared secrets.
  *   Datanode to Datanode communication secured via mutual TLS.
  *   Ability secure ozone cluster that works with Yarn, Hive, and 
Spark.
  *   Skaffold support to deploy Ozone clusters on K8s.
  *   Support S3 Authentication Mechanisms like - S3 v4 
Authentication protocol.
  *   S3 Gateway supports Multipart upload.
  *   S3A file system is tested and supported.
  *   Support for Tracing and Profiling for all Ozone components.
  *   Audit Support - including Audit Parser tools.
  *   Apache Ranger Support in Ozone.
  *   Extensive failure testing for Ozone.

The RC artifacts are available at 
https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc2/



The RC tag in git is ozone-0.4.0-alpha-RC2 (git hash 
4ea602c1ee7b5e1a5560c6cbd096de4b140f776b)



Please try 
out,
 vote, or just give us feedback.



The vote will run for 5 days, ending on May 4, 2019, 04:00 UTC.



Thank you very much,

Ajay



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

2019-05-05 Thread Arpit Agarwal 
Thanks for building this RC Ajay.

+1 binding.

- verified signatures and checksums
- built from source 
- deployed to 3 node cluster, tried out basic operations
- ran smoke tests
- ran unit tests
- LICENSE/NOTICE files look ok

There is an extra file in the source root named JenkinsFile.



> On Apr 29, 2019, at 9:04 PM, Ajay Kumar  wrote:
> 
> Hi All,
> 
> 
> 
> We have created the third release candidate (RC2) for Apache Hadoop Ozone 
> 0.4.0-alpha.
> 
> 
> 
> This release contains security payload for Ozone. Below are some important 
> features in it:
> 
> 
> 
>  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
>  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be 
> encrypted-at-rest.
>  *   Kerberos support for Ozone.
>  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of shared 
> secrets.
>  *   Datanode to Datanode communication secured via mutual TLS.
>  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
>  *   Skaffold support to deploy Ozone clusters on K8s.
>  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication 
> protocol.
>  *   S3 Gateway supports Multipart upload.
>  *   S3A file system is tested and supported.
>  *   Support for Tracing and Profiling for all Ozone components.
>  *   Audit Support - including Audit Parser tools.
>  *   Apache Ranger Support in Ozone.
>  *   Extensive failure testing for Ozone.
> 
> The RC artifacts are available at 
> https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc2/
> 
> 
> 
> The RC tag in git is ozone-0.4.0-alpha-RC2 (git hash 
> 4ea602c1ee7b5e1a5560c6cbd096de4b140f776b)
> 
> 
> 
> Please try 
> out,
>  vote, or just give us feedback.
> 
> 
> 
> The vote will run for 5 days, ending on May 4, 2019, 04:00 UTC.
> 
> 
> 
> Thank you very much,
> 
> Ajay


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2019-05-05 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore
 
   Unread field:TimelineEventSubDoc.java:[line 56] 
   Unread field:TimelineMetricSubDoc.java:[line 44] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core
 
   Class org.apache.hadoop.applications.mawo.server.common.TaskStatus 
implements Cloneable but does not define or use clone method At 
TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 
39-346] 
   Equals method for 
org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument 
is of type WorkerId At WorkerId.java:the argument is of type WorkerId At 
WorkerId.java:[line 114] 
   
org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does 
not check for null argument At WorkerId.java:null argument At 
WorkerId.java:[lines 114-115] 

Failed junit tests :

   hadoop.ha.TestZKFailoverController 
   hadoop.util.TestReadWriteDiskValidator 
   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.yarn.client.cli.TestLogsCLI 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapreduce.v2.app.TestRuntimeEstimators 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.hdds.scm.pipeline.TestNodeFailure 
   hadoop.ozone.scm.TestSCMNodeManagerMXBean 
   hadoop.ozone.TestMiniOzoneCluster 
   hadoop.ozone.om.TestOMDbCheckpointServlet 
   hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient 
   hadoop.ozone.web.client.TestBuckets 
   hadoop.ozone.om.TestOmMetrics 
   hadoop.hdds.scm.pipeline.TestPipelineClose 
   hadoop.ozone.scm.TestSCMMXBean 
   hadoop.hdds.scm.pipeline.TestSCMRestart 
   
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler
 
   hadoop.ozone.client.rpc.TestCommitWatcher 
   hadoop.ozone.om.TestOmAcls 
   hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException 
   hadoop.ozone.client.rpc.TestOzoneRpcClient 
   hadoop.ozone.ozShell.TestOzoneShell 
   hadoop.ozone.scm.TestAllocateContainer 
   hadoop.ozone.om.TestOmBlockVersioning 
   hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics 
   hadoop.ozone.om.TestScmSafeMode 
   
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
 
   hadoop.ozone.web.client.TestVolume 
   hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean 
   hadoop.ozone.scm.TestXceiverClientManager 
   hadoop.ozone.client.rpc.TestReadRetries 
   hadoop.fs.ozone.contract.ITestOzoneContractGetFileStatus 
   hadoop.fs.ozone.contract.ITestOzoneContractDelete 
   hadoop.fs.ozone.contract.ITestOzoneContractSeek 
   hadoop.fs.ozone.contract.ITestOzoneContractMkdir 
   hadoop.fs.ozone.contract.ITestOzoneContractCreate 
   hadoop.fs.ozone.contract.ITestOzoneContractRootDir 
   hadoop.fs.ozone.contract.ITestOzoneContractDistCp 
   hadoop.fs.ozone.contract.ITestOzoneContractOpen 
   hadoop.ozone.freon.TestFreonWithDatanodeFastRestart 
   hadoop.ozone.freon.TestFreonWithPipelineDestroy 
   hadoop.ozone.freon.TestFreonWithDatanodeRestart 
   hadoop.ozone.freon.TestDataValidateWithSafeByteOperations 
   hadoop.ozone.freon.TestRandomKeyGenerator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1127/ar

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2019-05-05 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient 
non-serializable instance field map In GlobalStorageStatistics.java:instance 
field map In GlobalStorageStatistics.java 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.server.namenode.TestNameNodeMXBean 
   hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-compile-cc-root-jdk1.8.0_191.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-compile-javac-root-jdk1.8.0_191.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_191.txt
  [1.1M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/312/artifact/out/p