[jira] [Created] (HDDS-1404) Fix typo in HDDS

[bianqi (JIRA)]

bianqi created HDDS-1404:


 Summary: Fix typo in HDDS
 Key: HDDS-1404
 URL: https://issues.apache.org/jira/browse/HDDS-1404
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: documentation
Affects Versions: 0.3.0
Reporter: bianqi


[https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto#L465]

[https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/hdds.proto#L172]

[https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto#L37]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10585) Incorrect offset/length calculation in pipeline recovery causes block corruption

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-10585.

Resolution: Duplicate

> Incorrect offset/length calculation in pipeline recovery causes block 
> corruption
> 
>
> Key: HDFS-10585
> URL: https://issues.apache.org/jira/browse/HDFS-10585
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
>
> We found incorrect offset and length calculation in pipeline recovery may 
> cause block corruption and results in missing blocks under a very unfortunate 
> scenario. 
> (1) A client established pipeline and started writing data to the pipeline.
> (2) One of the data node in the pipeline restarted, closing the socket, and 
> some written data were unacknowledged.
> (3) Client replaced the failed data node with a new one, initiating block 
> transfer to copy existing data in the block to the new datanode.
> (4) The block is transferred to the new node. Crucially, the entire block, 
> including the unacknowledged data, was transferred.
> (5) The last chunk (512 bytes) was not a full chunk, but the destination 
> still reserved the whole chunk in its buffer, and wrote the entire buffer to 
> disk, therefore some written data is garbage.
> (6) When the transfer was done, the destination data node converted the 
> replica from temporary to rbw, which made its visible length as the length of 
> bytes on disk. That is to say, it thought whatever was transferred was 
> acknowledged. However, the visible length of the replica is different (round 
> up to the next multiple of 512) than the source of transfer.
> (7) Client then truncated the block in the attempt to remove unacknowledged 
> data. However, because the visible length is equivalent of the bytes on disk, 
> it did not truncate unacknowledged data.
> (8) When new data was appended to the destination, it skipped the bytes 
> already on disk. Therefore, whatever was written as garbage was not replaced.
> (9) the volume scanner detected corrupt replica, but due to HDFS-10512, it 
> wouldn't tell NameNode to mark the replica as corrupt, so the client 
> continued to form a pipeline using the corrupt replica.
> (10) Finally the DN that had the only healthy replica was restarted. NameNode 
> then update the pipeline to only contain the corrupt replica.
> (11) Client continue to write to the corrupt replica, because neither client 
> nor the data node itself knows the replica is corrupt. When the restarted 
> datanodes comes back, their replica are stale, despite they are not corrupt. 
> Therefore, none of the replica is good and up to date.
> The sequence of events was reconstructed based on DataNode/NameNode log and 
> my understanding of code.
> Incidentally, we have observed the same sequence of events on two independent 
> clusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE]: Hadoop Ozone 0.4.0-alpha RC0 release

[Ajay Kumar]

Hi all,



We have created the first release candidate (RC0) for Apache Hadoop Ozone 
0.4.0-alpha.



This release contains security payload for Ozone. Below are some important 
features in it:

  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be 
encrypted-at-rest.
  *   Kerberos support for Ozone.
  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of shared 
secrets.
  *   Datanode to Datanode communication secured via mutual TLS.
  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
  *   Skaffold support to deploy Ozone clusters on K8s.
  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication protocol.
  *   S3 Gateway supports Multipart upload.
  *   S3A file system is tested and supported.
  *   Support for Tracing and Profiling for all Ozone components.
  *   Audit Support - including Audit Parser tools.
  *   Apache Ranger Support in Ozone.
  *   Extensive failure testing for Ozone.


The RC artifacts are available at 
https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc0



The RC tag in git is ozone-0.4.0-alpha-RC0 (git hash 
e5df6e94a5137b84cf7a6ee2e0ef7a8dae1b11a1)



Please try 
out,
 vote, or just give us feedback.



The vote will run for 5 days, ending on April 13, 2019, 19:00 UTC.



Thank you very much,

Ajay

Re: [DISCUSS] Use of AssertJ for testing

[Anu Engineer]

+1, on AssertJ usage, thanks for getting this done.
--Anu


On 3/31/19, 9:37 PM, "Akira Ajisaka"  wrote:

Hi folks,

Now I'm going to upgrade the JUnit version from 4 to 5 for Java 11 support.
I wanted to start with the small module, so I uploaded a patch to upgrade
the API in hadoop-yarn-api module at first (YARN-8943), and in this JIRA,
Szilard Nemeth suggested using AssertJ with JUnit 5. (Thanks Szilard
for the suggestion!)

I think the JUnit upgrade and the use of AssertJ are separated, but
related tasks.
Therefore, I'd like to decide:
- Use AssertJ or not
- If we are going to use AssertJ, when to use AssertJ (before
upgrading JUnit or after?)

My opinion is:
- JUnit migration is required for Java 11, so upgrading JUnit as soon
as possible.
- After the migration, we may use AssertJ for existing tests.
- We may use AssertJ for new tests. (not must)

Any thoughts?

Thanks,
Akira

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1403) KeyOutputStream writes fails after max retries while writing to a closed container

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDDS-1403:
---

 Summary: KeyOutputStream writes fails after max retries while 
writing to a closed container
 Key: HDDS-1403
 URL: https://issues.apache.org/jira/browse/HDDS-1403
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: SCM
Affects Versions: 0.4.0
Reporter: Mukul Kumar Singh


Currently a Ozone Client retries a write operation 5 times. It is possible that 
the container being written to is already closed by the time it is written to. 
The key write will fail after retrying multiple times with this error. This 
needs to be fixed as this is an internal error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler

[Elek, Marton (JIRA)]

Elek, Marton created HDDS-1402:
--

 Summary: Remove unused ScmBlockLocationProtocol from 
ObjectStoreHandler
 Key: HDDS-1402
 URL: https://issues.apache.org/jira/browse/HDDS-1402
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Elek, Marton
Assignee: Elek, Marton


When I analyzed the usages of the available RPC protocols in Ozone I found that 
the ScmBlockLocationProtocol is not used in ObjectStore at all.

I would propose to remove it...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1100/

[Apr 8, 2019 1:02:34 AM] (aajisaka) HADOOP-10848. Cleanup calling of 
sun.security.krb5.Config.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-common-project/hadoop-kms 
   Null passed for non-null parameter of 
com.google.common.base.Strings.isNullOrEmpty(String) in 
org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus, 
Object, UserGroupInformation, String, String, String) Method invoked at 
KMSAudit.java:of com.google.common.base.Strings.isNullOrEmpty(String) in 
org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus, 
Object, UserGroupInformation, String, String, String) Method invoked at 
KMSAudit.java:[line 195] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Null passed for non-null parameter of 
com.google.common.base.Strings.emptyToNull(String) in 
org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport()
 Method invoked at NodeHealthCheckerService.java:of 
com.google.common.base.Strings.emptyToNull(String) in 
org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport()
 Method invoked at NodeHealthCheckerService.java:[line 66] 
   Null passed for non-null parameter of 
com.google.common.base.Strings.emptyToNull(String) in 
org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport()
 Method invoked at NodeHealthCheckerService.java:of 
com.google.common.base.Strings.emptyToNull(String) in 
org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport()
 Method invoked at NodeHealthCheckerService.java:[line 72] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 
   Null passed for non-null parameter of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore,
 RMStateStoreEvent) At RMStateStore.java:of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore,
 RMStateStoreEvent) At RMStateStore.java:[line 291] 
   Null passed for non-null parameter of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority,
 ApplicationId, SettableFuture, UserGroupInformation) At 
CapacityScheduler.java:of 
com.google.common.util.concurrent.SettableFuture.set(Object) in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority,
 ApplicationId, SettableFuture, UserGroupInformation) At 
CapacityScheduler.java:[line 2647] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore
 
   
org.apache.hadoop.yarn.server.timelineservice.documentstore.collection.document.entity.TimelineEntityDocument.setEvents(Map)
 makes inefficient use of keySet iterator instead of entrySet iterator At 
TimelineEntityDocument.java:keySet iterator instead of entrySet iterator At 
TimelineEntityDocument.java:[line 159] 
   
org.apache.hadoop.yarn.server.timelineservice.documentstore.collection.document.entity.TimelineEntityDocument.setMetrics(Map)
 makes inefficient use of keySet iterator instead of entrySet iterator At 
TimelineEntityDocument.java:keySet iterator instead of entrySet iterator At 
TimelineEntityDocument.java:[line 142] 
   Unread field:TimelineEventSubDoc.java:[line 56] 
   Unread field:TimelineMetricSubDoc.java:[line 44] 
   Switch statement found in 
org.apache.hadoop.yarn.server.timelineservice.documentstore.collection.document.flowrun.FlowRunDocument.aggregate(TimelineMetric,
 TimelineMetric) where default case is missing At 
FlowRunDocument.java:TimelineMetric) where default case is missing At 
FlowRunDocument.java:[lines 121-136] 
   
org.apache.hadoop.yarn.server.timelineservice.documentstore.collection.document.flowrun.FlowRunDocument.aggregateMetrics(Map)
 makes inefficient use of keySet iterator instead of entrySet iterator At 
FlowRunDocument.java:keySet iterator instead of entrySet iterator At 
FlowRunDocument.java:[line 103] 
   Possible doublecheck on 
org.apache.hadoop.yarn.server.timelineservice.documentstore.reader.cosmosdb.CosmosDBDocumentStoreReade

[jira] [Created] (HDFS-14418) Remove redundant super user priveledge checks from namenode.

[Ayush Saxena (JIRA)]

Ayush Saxena created HDFS-14418:
---

 Summary: Remove redundant super user priveledge checks from 
namenode.
 Key: HDFS-14418
 URL: https://issues.apache.org/jira/browse/HDFS-14418
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena


There are couple of methods that unnecessarily double checks super user 
privileged at namenode, which can reduced to single.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/

No changes




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml 
   hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle.xml 
   hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient 
non-serializable instance field map In GlobalStorageStatistics.java:instance 
field map In GlobalStorageStatistics.java 

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Dead store to state in 
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream,
 INodeSymlink) At 
FSImageFormatPBINode.java:org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream,
 INodeSymlink) At FSImageFormatPBINode.java:[line 623] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities 
   hadoop.mapred.TestShuffleHandler 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.yarn.client.api.impl.TestAMRMProxy 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.webapp.TestWebApp 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-compile-cc-root-jdk1.8.0_191.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-compile-javac-root-jdk1.8.0_191.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/xml.txt
  [20K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/285/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
  [8.0K]