[jira] [Updated] (HDDS-1490) Support configurable container placement policy through "ozone.scm.container.placement.classname"
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1490: - Attachment: (was: HDDS-1490.02.patch) > Support configurable container placement policy through > "ozone.scm.container.placement.classname" > - > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch > > > Support system property "ozone.scm.container.placement.classname" in > ozone-site.xml. User can specify the implementation class name as the value > of the property. Here is an example, > > ozone.scm.container.placement.classname > > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß > > If this property is not set, then default > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware > will be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1568) Add RocksDB metrics to OM
[ https://issues.apache.org/jira/browse/HDDS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850554#comment-16850554 ] Aravindan Vijayan commented on HDDS-1568: - Sure [~jnp]. I can work on Datanode RocksDB metrics on another JIRA. > Add RocksDB metrics to OM > - > > Key: HDDS-1568 > URL: https://issues.apache.org/jira/browse/HDDS-1568 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to > understand how OM behaves under heavy load. > Example: "rocksdb.bytes.written" > https://github.com/facebook/rocksdb/wiki/Statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1490) Support configurable container placement policy through "ozone.scm.container.placement.classname"
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850551#comment-16850551 ] Sammi Chen edited comment on HDDS-1490 at 5/29/19 6:54 AM: --- 02.patch, change list, 1. fix TestKeyManagerImpl failed issue. The root cause is failed to load the network-topology-default.xml. Refact the way NodeSchemaManager load schema file. And add test resource reference in hadoop-ozone-ozone-manager pom. 2. add testDatanodeWithDefaultNetworkLocation unit test to verify that default network location works when allocating datanodes 3. fix check style issues. 4. add property definition in ozone-default.xml was (Author: sammi): 02.patch, change list, 1. fix TestKeyManagerImpl failed issue. The root cause is failed to load the network-topology-default.xml. Refact the way NodeSchemaManager load schema file. And add test resource reference in hadoop-ozone-ozone-manager pom. 2. add testDatanodeWithDefaultNetworkLocation unit test to verify that default network location works when allocating datanodes 3. fix check style issues. > Support configurable container placement policy through > "ozone.scm.container.placement.classname" > - > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch > > > Support system property "ozone.scm.container.placement.classname" in > ozone-site.xml. User can specify the implementation class name as the value > of the property. Here is an example, > > ozone.scm.container.placement.classname > > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß > > If this property is not set, then default > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware > will be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1490) Support configurable containerPlacement policy
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1490: - Description: Support system property "ozone.scm.container.placement.classname" in ozone-site.xml. User can specify the implementation class name as the value of the property. Here is an example, ozone.scm.container.placement.classname org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß If this property is not set, then default org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware will be used. was:Support configurable containerPlacement policy to meet different requirements > Support configurable containerPlacement policy > -- > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch, HDDS-1490.02.patch > > > Support system property "ozone.scm.container.placement.classname" in > ozone-site.xml. User can specify the implementation class name as the value > of the property. Here is an example, > > ozone.scm.container.placement.classname > > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß > > If this property is not set, then default > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware > will be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1568) Add RocksDB metrics to OM
[ https://issues.apache.org/jira/browse/HDDS-1568?focusedWorklogId=249891&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249891 ] ASF GitHub Bot logged work on HDDS-1568: Author: ASF GitHub Bot Created on: 29/May/19 06:53 Start Date: 29/May/19 06:53 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #868: HDDS-1568 : Add RocksDB metrics to OM. URL: https://github.com/apache/hadoop/pull/868#issuecomment-496809403 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249891) Time Spent: 20m (was: 10m) > Add RocksDB metrics to OM > - > > Key: HDDS-1568 > URL: https://issues.apache.org/jira/browse/HDDS-1568 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to > understand how OM behaves under heavy load. > Example: "rocksdb.bytes.written" > https://github.com/facebook/rocksdb/wiki/Statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1490) Support configurable container placement policy through "ozone.scm.container.placement.classname"
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1490: - Summary: Support configurable container placement policy through "ozone.scm.container.placement.classname" (was: Support configurable container placement policy through "") > Support configurable container placement policy through > "ozone.scm.container.placement.classname" > - > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch, HDDS-1490.02.patch > > > Support system property "ozone.scm.container.placement.classname" in > ozone-site.xml. User can specify the implementation class name as the value > of the property. Here is an example, > > ozone.scm.container.placement.classname > > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß > > If this property is not set, then default > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware > will be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1490) Support configurable container placement policy through ""
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1490: - Summary: Support configurable container placement policy through "" (was: Support configurable containerPlacement policy) > Support configurable container placement policy through "" > -- > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch, HDDS-1490.02.patch > > > Support system property "ozone.scm.container.placement.classname" in > ozone-site.xml. User can specify the implementation class name as the value > of the property. Here is an example, > > ozone.scm.container.placement.classname > > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAwareß > > If this property is not set, then default > org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackAware > will be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1568) Add RocksDB metrics to OM
[ https://issues.apache.org/jira/browse/HDDS-1568?focusedWorklogId=249889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249889 ] ASF GitHub Bot logged work on HDDS-1568: Author: ASF GitHub Bot Created on: 29/May/19 06:53 Start Date: 29/May/19 06:53 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #868: HDDS-1568 : Add RocksDB metrics to OM. URL: https://github.com/apache/hadoop/pull/868 RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to understand how OM behaves under heavy load. Example: "rocksdb.bytes.written" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249889) Time Spent: 10m Remaining Estimate: 0h > Add RocksDB metrics to OM > - > > Key: HDDS-1568 > URL: https://issues.apache.org/jira/browse/HDDS-1568 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to > understand how OM behaves under heavy load. > Example: "rocksdb.bytes.written" > https://github.com/facebook/rocksdb/wiki/Statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1568) Add RocksDB metrics to OM
[ https://issues.apache.org/jira/browse/HDDS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1568: - Labels: pull-request-available (was: ) > Add RocksDB metrics to OM > - > > Key: HDDS-1568 > URL: https://issues.apache.org/jira/browse/HDDS-1568 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > > RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to > understand how OM behaves under heavy load. > Example: "rocksdb.bytes.written" > https://github.com/facebook/rocksdb/wiki/Statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1568) Add RocksDB metrics to OM
[ https://issues.apache.org/jira/browse/HDDS-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1568: Status: Patch Available (was: Open) > Add RocksDB metrics to OM > - > > Key: HDDS-1568 > URL: https://issues.apache.org/jira/browse/HDDS-1568 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Reporter: Siddharth Wagle >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > RocksDB statistics need to sinked to hadoop-metrics2 for Ozone Manager to > understand how OM behaves under heavy load. > Example: "rocksdb.bytes.written" > https://github.com/facebook/rocksdb/wiki/Statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1490) Support configurable containerPlacement policy
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850551#comment-16850551 ] Sammi Chen commented on HDDS-1490: -- 02.patch, change list, 1. fix TestKeyManagerImpl failed issue. The root cause is failed to load the network-topology-default.xml. Refact the way NodeSchemaManager load schema file. And add test resource reference in hadoop-ozone-ozone-manager pom. 2. add testDatanodeWithDefaultNetworkLocation unit test to verify that default network location works when allocating datanodes 3. fix check style issues. > Support configurable containerPlacement policy > -- > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch, HDDS-1490.02.patch > > > Support configurable containerPlacement policy to meet different requirements -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-13123 stopped by hemanthboyina. > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-13123 started by hemanthboyina. > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1490) Support configurable containerPlacement policy
[ https://issues.apache.org/jira/browse/HDDS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-1490: - Attachment: HDDS-1490.02.patch > Support configurable containerPlacement policy > -- > > Key: HDDS-1490 > URL: https://issues.apache.org/jira/browse/HDDS-1490 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-1490.01.patch, HDDS-1490.02.patch > > > Support configurable containerPlacement policy to meet different requirements -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina reassigned HDFS-13123: Assignee: hemanthboyina (was: Wei Yan) > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1559) Include committedBytes to determine Out of Space in VolumeChoosingPolicy
[ https://issues.apache.org/jira/browse/HDDS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850517#comment-16850517 ] Hudson commented on HDDS-1559: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16620 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16620/]) HDDS-1559. Include committedBytes to determine Out of Space in (arp7: rev 346c2b798080cc1f22d6ba85e584141e7dee2c08) * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/RoundRobinVolumeChoosingPolicy.java > Include committedBytes to determine Out of Space in VolumeChoosingPolicy > > > Key: HDDS-1559 > URL: https://issues.apache.org/jira/browse/HDDS-1559 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is a follow-up from HDDS-1511 and HDDS-1535 > Currently when creating a new Container, the DN invokes > RoundRobinVolumeChoosingPolicy:chooseVolume(). This routine checks for > (volume available space > container max size). If no eligible volume is > found, the policy throws a DiskOutOfSpaceException. This is the current > behaviour. > However, the computation of available space does not take into consideration > the space > that is going to be consumed by writes to existing containers which are still > Open and accepting chunk writes. > This Jira proposes to enhance the space availability check in chooseVolume by > inclusion of committed space(committedBytes in HddsVolume) in the equation. > The handling/management of the exception in Ratis will not be modified in > this Jira. That will be scoped separately as part of Datanode IO Failure > handling work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14455) Fix typo in HAState.java
[ https://issues.apache.org/jira/browse/HDFS-14455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina reassigned HDFS-14455: Assignee: (was: hemanthboyina) > Fix typo in HAState.java > > > Key: HDFS-14455 > URL: https://issues.apache.org/jira/browse/HDFS-14455 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: hunshenshi >Priority: Major > > There are some typo in HAState > destructuve -> destructive > Aleady -> Already > Transtion -> Transition -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1555) Disable install snapshot for ContainerStateMachine
[
https://issues.apache.org/jira/browse/HDDS-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-1555:
Status: Patch Available (was: Open)
> Disable install snapshot for ContainerStateMachine
> --
>
> Key: HDDS-1555
> URL: https://issues.apache.org/jira/browse/HDDS-1555
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In case a follower lags behind the leader by a large number, the leader tries
> to send the snapshot to the follower. For ContainerStateMachine, the
> information in the snapshot it not the entire state machine data.
> InstallSnapshot for ContainerStateMachine should be disabled.
> {code}
> 2019-05-19 10:58:22,198 WARN server.GrpcLogAppender
> (GrpcLogAppender.java:installSnapshot(423)) -
> GrpcLogAppender(e3e19760-1340-4acd-b50d-f8a796a97254->28d9bd2f-3fe2-4a69-8120-757a00fa2f20):
> failed to install snapshot
> [/Users/msingh/code/apache/ozone/github/git_oz_bugs_fixes/hadoop-ozone/integration-test/target/test/data/MiniOzoneClusterImpl-c2a863ef-8be9-445c-886f-57cad3a7b12e/datanode-6/data/ratis/fb88b749-3e75-4381-8973-6e0cb4904c7e/sm/snapshot.2_190]:
> {}
> java.lang.NullPointerException
> at
> org.apache.ratis.server.impl.LogAppender.readFileChunk(LogAppender.java:369)
> at
> org.apache.ratis.server.impl.LogAppender.access$1100(LogAppender.java:54)
> at
> org.apache.ratis.server.impl.LogAppender$SnapshotRequestIter$1.next(LogAppender.java:318)
> at
> org.apache.ratis.server.impl.LogAppender$SnapshotRequestIter$1.next(LogAppender.java:303)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.installSnapshot(GrpcLogAppender.java:412)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:101)
> at
> org.apache.ratis.server.impl.LogAppender$AppenderDaemon.run(LogAppender.java:80)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850500#comment-16850500
]
Hadoop QA commented on HDFS-12703:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
36s{color} | {color:red} The patch generated 17 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12703 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970095/HDFS-12703.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ac1c1b5a3d2f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f2e87a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26855/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26855/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.o
[jira] [Commented] (HDFS-14516) RBF: Create hdfs-rbf-site.xml for RBF specific properties
[
https://issues.apache.org/jira/browse/HDFS-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850488#comment-16850488
]
Takanobu Asanuma commented on HDFS-14516:
-
BTW, HADOOP-16331 is addressing the license warnings.
> RBF: Create hdfs-rbf-site.xml for RBF specific properties
> -
>
> Key: HDFS-14516
> URL: https://issues.apache.org/jira/browse/HDFS-14516
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-14516.1.patch, HDFS-14516.2.patch
>
>
> Currently, users write rbf properties in {{hdfs-site.xml}} though the
> definitions are in {{hdfs-rbf-default.xml}}. Like other modules, it would be
> better if there is a specific configuration file, {{hdfs-rbf-site.xml}}.
> {{hdfs-rbf-default.xml}} also should be loaded when it exists in the
> configuration directory. It is just a document at the moment.
> There is an early discussion in HDFS-13215.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14516) RBF: Create hdfs-rbf-site.xml for RBF specific properties
[
https://issues.apache.org/jira/browse/HDFS-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850486#comment-16850486
]
Takanobu Asanuma commented on HDFS-14516:
-
Thanks for the review [~elgoiri]. And thanks for your comment [~ayushtkn]. I
agree with your opinion.
Actually, the 1st patch has a problem. I thought {{hdfs-site.xml}} is still
effective for rbf properties but actually it isn't.
The 1st patch loads configuration files in the following order. The latter
overwrites the same properties.
# {{core-site.xml}} and {{hdfs-site.xml}}
# {{hdfs-rbf-default.xml}} (this overwrite all rbf properties by the default
values.)
# {{hdfs-rbf-site.xml}}
The 2nd patch doesn't load {{hdfs-rbf-default.xml}}. It won't affect current
users who use {{hdfs-site.xml}} for rbf properties. So it keeps compatibility.
> RBF: Create hdfs-rbf-site.xml for RBF specific properties
> -
>
> Key: HDFS-14516
> URL: https://issues.apache.org/jira/browse/HDFS-14516
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-14516.1.patch, HDFS-14516.2.patch
>
>
> Currently, users write rbf properties in {{hdfs-site.xml}} though the
> definitions are in {{hdfs-rbf-default.xml}}. Like other modules, it would be
> better if there is a specific configuration file, {{hdfs-rbf-site.xml}}.
> {{hdfs-rbf-default.xml}} also should be loaded when it exists in the
> configuration directory. It is just a document at the moment.
> There is an early discussion in HDFS-13215.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850485#comment-16850485
]
Ayush Saxena commented on HDFS-13480:
-
I had a version to current, forgot to upload when I moved it to branch
yesterday, Have uploaded it. Give a check if something is required.
> RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
> ---
>
> Key: HDFS-13480
> URL: https://issues.apache.org/jira/browse/HDFS-13480
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13480-HDFS-13891-05.patch, HDFS-13480.001.patch,
> HDFS-13480.002.patch, HDFS-13480.002.patch, HDFS-13480.003.patch,
> HDFS-13480.004.patch
>
>
> Now, if i enable the heartbeat.enable, but i do not want to monitor any
> namenode, i get an ERROR log like:
> {code:java}
> [2018-04-19T14:00:03.057+08:00] [ERROR]
> federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is
> enabled but there are no namenodes to monitor
> {code}
> and if i disable the heartbeat.enable, we cannot get any mounttable update,
> because the following logic in Router.java：
> {code:java}
> if (conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) {
> // Create status updater for each monitored Namenode
> this.namenodeHeartbeatServices = createNamenodeHeartbeatServices();
> for (NamenodeHeartbeatService hearbeatService :
> this.namenodeHeartbeatServices) {
> addService(hearbeatService);
> }
> if (this.namenodeHeartbeatServices.isEmpty()) {
> LOG.error("Heartbeat is enabled but there are no namenodes to
> monitor");
> }
> // Periodically update the router state
> this.routerHeartbeatService = new RouterHeartbeatService(this);
> addService(this.routerHeartbeatService);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14516) RBF: Create hdfs-rbf-site.xml for RBF specific properties
[
https://issues.apache.org/jira/browse/HDFS-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14516:
Attachment: HDFS-14516.2.patch
> RBF: Create hdfs-rbf-site.xml for RBF specific properties
> -
>
> Key: HDFS-14516
> URL: https://issues.apache.org/jira/browse/HDFS-14516
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-14516.1.patch, HDFS-14516.2.patch
>
>
> Currently, users write rbf properties in {{hdfs-site.xml}} though the
> definitions are in {{hdfs-rbf-default.xml}}. Like other modules, it would be
> better if there is a specific configuration file, {{hdfs-rbf-site.xml}}.
> {{hdfs-rbf-default.xml}} also should be loaded when it exists in the
> configuration directory. It is just a document at the moment.
> There is an early discussion in HDFS-13215.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850477#comment-16850477
]
Hadoop QA commented on HDFS-13480:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
48s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
25s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-13480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970106/HDFS-13480-HDFS-13891-05.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux ab79cf678932 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 4c0c9ff |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26854/testReport/ |
| Max. process+thread count | 1015 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26854/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache
[jira] [Assigned] (HDDS-1595) Handling IO Failures on the Datanode
[ https://issues.apache.org/jira/browse/HDDS-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-1595: --- Assignee: Supratim Deka > Handling IO Failures on the Datanode > > > Key: HDDS-1595 > URL: https://issues.apache.org/jira/browse/HDDS-1595 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: Handling IO Failures on the Datanode.pdf, Raft IO v2.svg > > > This Jira covers all the changes required to handle IO Failures on the > Datanode. Handling an IO failure on the Datanode involves detecting failures > as they happen and propagating the failure to the appropriate component in > the system - possibly the Client and/or the SCM based on the type of failure. > At a high-level, IO Failure handling has the following goals: > 1. Prevent Inconsistencies and corruption - due to non-handling or > mishandling of failures. > 2. Prevent any data loss - timely detection of failure and propagate correct > error back to the initiator instead of silently dropping the data while the > client assumes the operation is committed. > 3. Contain the disruption in the system - if a disk volume fails on a DN, > operations to the other nodes and volumes should not be affected. > Details pertaining to design and changes required are covered in the attached > pdf document. > A sequence diagram used to analyse the Datanode IO Path is also attached, in > svg format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1559) Include committedBytes to determine Out of Space in VolumeChoosingPolicy
[ https://issues.apache.org/jira/browse/HDDS-1559?focusedWorklogId=249795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249795 ] ASF GitHub Bot logged work on HDDS-1559: Author: ASF GitHub Bot Created on: 29/May/19 03:48 Start Date: 29/May/19 03:48 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #841: HDDS-1559. Include committedBytes to determine Out of Space in VolumeChoosingPolicy. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/841 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249795) Time Spent: 1h 50m (was: 1h 40m) > Include committedBytes to determine Out of Space in VolumeChoosingPolicy > > > Key: HDDS-1559 > URL: https://issues.apache.org/jira/browse/HDDS-1559 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is a follow-up from HDDS-1511 and HDDS-1535 > Currently when creating a new Container, the DN invokes > RoundRobinVolumeChoosingPolicy:chooseVolume(). This routine checks for > (volume available space > container max size). If no eligible volume is > found, the policy throws a DiskOutOfSpaceException. This is the current > behaviour. > However, the computation of available space does not take into consideration > the space > that is going to be consumed by writes to existing containers which are still > Open and accepting chunk writes. > This Jira proposes to enhance the space availability check in chooseVolume by > inclusion of committed space(committedBytes in HddsVolume) in the equation. > The handling/management of the exception in Ratis will not be modified in > this Jira. That will be scoped separately as part of Datanode IO Failure > handling work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1559) Include committedBytes to determine Out of Space in VolumeChoosingPolicy
[ https://issues.apache.org/jira/browse/HDDS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDDS-1559. - Resolution: Fixed Fix Version/s: 0.5.0 Target Version/s: (was: 0.5.0) I've committed this to trunk. Thanks for the contribution [~sdeka]. > Include committedBytes to determine Out of Space in VolumeChoosingPolicy > > > Key: HDDS-1559 > URL: https://issues.apache.org/jira/browse/HDDS-1559 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is a follow-up from HDDS-1511 and HDDS-1535 > Currently when creating a new Container, the DN invokes > RoundRobinVolumeChoosingPolicy:chooseVolume(). This routine checks for > (volume available space > container max size). If no eligible volume is > found, the policy throws a DiskOutOfSpaceException. This is the current > behaviour. > However, the computation of available space does not take into consideration > the space > that is going to be consumed by writes to existing containers which are still > Open and accepting chunk writes. > This Jira proposes to enhance the space availability check in chooseVolume by > inclusion of committed space(committedBytes in HddsVolume) in the equation. > The handling/management of the exception in Ratis will not be modified in > this Jira. That will be scoped separately as part of Datanode IO Failure > handling work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14508) RBF: Clean-up and refactor UI components
[
https://issues.apache.org/jira/browse/HDFS-14508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850393#comment-16850393
]
Takanobu Asanuma commented on HDFS-14508:
-
Thanks for the discussion, [~elgoiri] and [~crh].
I agree. Will upload a patch based on that.
> RBF: Clean-up and refactor UI components
>
>
> Key: HDFS-14508
> URL: https://issues.apache.org/jira/browse/HDFS-14508
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> Router UI has tags that are not used or incorrectly set. The code should be
> cleaned-up.
> One such example is
> Path :
> (\hadoop-hdfs-project\hadoop-hdfs-rbf\src\main\webapps\router\federationhealth.js)
> {code:java}
> {"name": "routerstat", "url":
> "/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus"},{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13909) RBF: Add Cache pools and directives related ClientProtocol APIs
[ https://issues.apache.org/jira/browse/HDFS-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-13909: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-13891 Status: Resolved (was: Patch Available) Committed. Thanx [~elgoiri] for the review and [~dibyendu_hadoop] for the report. > RBF: Add Cache pools and directives related ClientProtocol APIs > --- > > Key: HDFS-13909 > URL: https://issues.apache.org/jira/browse/HDFS-13909 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Ayush Saxena >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-13909-HDFS-13891-01.patch, > HDFS-13909-HDFS-13891-02.patch, HDFS-13909-HDFS-13891-03.patch, > HDFS-13909-HDFS-13891-04.patch, HDFS-13909-HDFS-13891-05.patch > > > Currently addCachePool, modifyCachePool, removeCachePool, listCachePools, > addCacheDirective, modifyCacheDirective, removeCacheDirective, > listCacheDirectives these APIs are not implemented in Router. > This JIRA is intend to implement above mentioned APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13909) RBF: Add Cache pools and directives related ClientProtocol APIs
[ https://issues.apache.org/jira/browse/HDFS-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-13909: Summary: RBF: Add Cache pools and directives related ClientProtocol APIs (was: RBF: Add Cache pools and directives related ClientProtocol apis) > RBF: Add Cache pools and directives related ClientProtocol APIs > --- > > Key: HDFS-13909 > URL: https://issues.apache.org/jira/browse/HDFS-13909 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13909-HDFS-13891-01.patch, > HDFS-13909-HDFS-13891-02.patch, HDFS-13909-HDFS-13891-03.patch, > HDFS-13909-HDFS-13891-04.patch, HDFS-13909-HDFS-13891-05.patch > > > Currently addCachePool, modifyCachePool, removeCachePool, listCachePools, > addCacheDirective, modifyCacheDirective, removeCacheDirective, > listCacheDirectives these APIs are not implemented in Router. > This JIRA is intend to implement above mentioned APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850319#comment-16850319 ] xuzq commented on HDFS-14431: - Yes, if we rename one file from /mntsrc to /mntdst, it is ok. But if we rename one directory from /mntsrc to /mntdst, it will be incomplete. Like this: * we will rename the directory from /mntsrc to /mntdst * /mntsrc has file1 in ns1 and has file2 in ns2 * /mntdst mount to ns2 and ns3, but dos't have files. After rename, file2 renamed to /mntdst/file2, but file1 will still in /mntsrc/file1. > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850315#comment-16850315 ] Ayush Saxena commented on HDFS-14431: - May be that would be the last thing to do, if we don't get a good way to handle this, Well I also don't have any good solution to handle. Will try to think, if I get any non hacky solution > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850314#comment-16850314 ] Íñigo Goiri commented on HDFS-14431: In the particular you mentioned before like: * /mntsrc in ns1 and ns2 * /mntdst in ns2 and ns3 If we are renaming a file that is originally in ns1, we will fail (RenameNotAllowedException), but if the file is originally in ns2, we can do the rename, right? Then if the file already exists in ns3, then we fail for the src being in ns1 but delete it from ns3 for the file being in ns2. Does it sound right? > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850312#comment-16850312
]
Eric Yang commented on HDDS-1458:
-
The license errors are false positive, and caused by HADOOP-16323.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch,
> HDDS-1458.009.patch, HDDS-1458.010.patch, HDDS-1458.011.patch,
> HDDS-1458.012.patch, HDDS-1458.013.patch, HDDS-1458.014.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14512) ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)
[ https://issues.apache.org/jira/browse/HDFS-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HDFS-14512: --- Assignee: Ayush Saxena > ONE_SSD policy will be violated while write data with > DistributedFileSystem.create(favoredNodes) > > > Key: HDFS-14512 > URL: https://issues.apache.org/jira/browse/HDFS-14512 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shen Yinjie >Assignee: Ayush Saxena >Priority: Major > Attachments: TestToRepro.patch > > > Reproduce steps: > 1.setStoragePolicy ONE_SSD for a path A; > 2. client write data to path A by > DistributedFileSystem.create(...favoredNodes) and Passing parameter > favoredNodes > then, three replicas of file in this path will be located in 2 SSD and > 1DISK,which is violating the ONE_SSD policy. > Not sure am I clear? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850311#comment-16850311 ] xuzq commented on HDFS-14431: - Like the above case, may be we can return RenameNotAllowedException. May be we can only support the rename process when the mount points is the same between src and dst. > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-13480:
Attachment: HDFS-13480-HDFS-13891-05.patch
> RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
> ---
>
> Key: HDFS-13480
> URL: https://issues.apache.org/jira/browse/HDFS-13480
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13480-HDFS-13891-05.patch, HDFS-13480.001.patch,
> HDFS-13480.002.patch, HDFS-13480.002.patch, HDFS-13480.003.patch,
> HDFS-13480.004.patch
>
>
> Now, if i enable the heartbeat.enable, but i do not want to monitor any
> namenode, i get an ERROR log like:
> {code:java}
> [2018-04-19T14:00:03.057+08:00] [ERROR]
> federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is
> enabled but there are no namenodes to monitor
> {code}
> and if i disable the heartbeat.enable, we cannot get any mounttable update,
> because the following logic in Router.java：
> {code:java}
> if (conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) {
> // Create status updater for each monitored Namenode
> this.namenodeHeartbeatServices = createNamenodeHeartbeatServices();
> for (NamenodeHeartbeatService hearbeatService :
> this.namenodeHeartbeatServices) {
> addService(hearbeatService);
> }
> if (this.namenodeHeartbeatServices.isEmpty()) {
> LOG.error("Heartbeat is enabled but there are no namenodes to
> monitor");
> }
> // Periodically update the router state
> this.routerHeartbeatService = new RouterHeartbeatService(this);
> addService(this.routerHeartbeatService);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850295#comment-16850295 ] Hudson commented on HDDS-1604: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16619 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16619/]) HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… (github: rev 7f2e87a419cf87d1b1aa2b3b56f0f23504baa110) * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/BlockUtils.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ReferenceCountedDB.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManagerHelper.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java * (edit) hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueBlockIterator.java > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting from > HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from > ContainerReader#initializeUsedBytes. > I will post a patch shortly, which will fix some UT failures exposed by > -HDDS-1449,- such as TestBCSID#testBCSID, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850293#comment-16850293
]
Hadoop QA commented on HDDS-1458:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:blue}0{color} | {color:blue} yamllint {color} | {color:blue} 0m
0s{color} | {color:blue} yamllint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 8m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 23m
4s{color} | {color:green} trunk passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
5s{color} | {color:orange} Error running pylint. Please check pylint stderr
files. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 36m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 20m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
9s{color} | {color:orange} Error running pylint. Please check pylint stderr
files. {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
10s{color} | {color:green} There were no new pylint issues. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}180m 17s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
13s{color} | {color:red} The patch generated 17 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}400m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdds.scm.block.TestBlockManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apac
[jira] [Work logged] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?focusedWorklogId=249771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249771 ] ASF GitHub Bot logged work on HDDS-1604: Author: ASF GitHub Bot Created on: 29/May/19 01:44 Start Date: 29/May/19 01:44 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #866: HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… URL: https://github.com/apache/hadoop/pull/866#issuecomment-496751909 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 546 | trunk passed | | +1 | compile | 282 | trunk passed | | +1 | checkstyle | 86 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 892 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 155 | trunk passed | | 0 | spotbugs | 293 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 476 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 66 | Maven dependency ordering for patch | | +1 | mvninstall | 470 | the patch passed | | +1 | compile | 261 | the patch passed | | +1 | javac | 261 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 639 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 158 | the patch passed | | +1 | findbugs | 496 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 228 | hadoop-hdds in the patch passed. | | -1 | unit | 1397 | hadoop-ozone in the patch failed. | | -1 | asflicense | 74 | The patch generated 17 ASF License warnings. | | | | 6546 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestContainerStateMachineIdempotency | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-866/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/866 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ebcea3b25f41 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0c73dba | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/6/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/6/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4319 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249771) Time Spent: 50m (was: 40m) > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850287#comment-16850287
]
Íñigo Goiri commented on HDFS-12703:
Thanks [~xuel1], is there any unit test we can add for this?
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12703:
---
Status: Patch Available (was: Open)
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850286#comment-16850286
]
Íñigo Goiri commented on HDFS-13480:
[~maobaolong], I can rebase to HDFS-13891 if you are not available.
> RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
> ---
>
> Key: HDFS-13480
> URL: https://issues.apache.org/jira/browse/HDFS-13480
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13480.001.patch, HDFS-13480.002.patch,
> HDFS-13480.002.patch, HDFS-13480.003.patch, HDFS-13480.004.patch
>
>
> Now, if i enable the heartbeat.enable, but i do not want to monitor any
> namenode, i get an ERROR log like:
> {code:java}
> [2018-04-19T14:00:03.057+08:00] [ERROR]
> federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is
> enabled but there are no namenodes to monitor
> {code}
> and if i disable the heartbeat.enable, we cannot get any mounttable update,
> because the following logic in Router.java：
> {code:java}
> if (conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) {
> // Create status updater for each monitored Namenode
> this.namenodeHeartbeatServices = createNamenodeHeartbeatServices();
> for (NamenodeHeartbeatService hearbeatService :
> this.namenodeHeartbeatServices) {
> addService(hearbeatService);
> }
> if (this.namenodeHeartbeatServices.isEmpty()) {
> LOG.error("Heartbeat is enabled but there are no namenodes to
> monitor");
> }
> // Periodically update the router state
> this.routerHeartbeatService = new RouterHeartbeatService(this);
> addService(this.routerHeartbeatService);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1604. -- Resolution: Fixed Fix Version/s: 0.4.1 Thanks all for the reviews. I've committed the patch to trunk. > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 40m > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting from > HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from > ContainerReader#initializeUsedBytes. > I will post a patch shortly, which will fix some UT failures exposed by > -HDDS-1449,- such as TestBCSID#testBCSID, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?focusedWorklogId=249768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249768 ] ASF GitHub Bot logged work on HDDS-1604: Author: ASF GitHub Bot Created on: 29/May/19 01:39 Start Date: 29/May/19 01:39 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on pull request #866: HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… URL: https://github.com/apache/hadoop/pull/866 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249768) Time Spent: 40m (was: 0.5h) > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting from > HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from > ContainerReader#initializeUsedBytes. > I will post a patch shortly, which will fix some UT failures exposed by > -HDDS-1449,- such as TestBCSID#testBCSID, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14508) RBF: Clean-up and refactor UI components
[
https://issues.apache.org/jira/browse/HDFS-14508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850284#comment-16850284
]
Íñigo Goiri commented on HDFS-14508:
Thanks [~crh] for the clarification.
That works for me.
> RBF: Clean-up and refactor UI components
>
>
> Key: HDFS-14508
> URL: https://issues.apache.org/jira/browse/HDFS-14508
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> Router UI has tags that are not used or incorrectly set. The code should be
> cleaned-up.
> One such example is
> Path :
> (\hadoop-hdfs-project\hadoop-hdfs-rbf\src\main\webapps\router\federationhealth.js)
> {code:java}
> {"name": "routerstat", "url":
> "/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus"},{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13909) RBF: Add Cache pools and directives related ClientProtocol apis
[ https://issues.apache.org/jira/browse/HDFS-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850283#comment-16850283 ] Íñigo Goiri commented on HDFS-13909: +1 on [^HDFS-13909-HDFS-13891-05.patch]. > RBF: Add Cache pools and directives related ClientProtocol apis > --- > > Key: HDFS-13909 > URL: https://issues.apache.org/jira/browse/HDFS-13909 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13909-HDFS-13891-01.patch, > HDFS-13909-HDFS-13891-02.patch, HDFS-13909-HDFS-13891-03.patch, > HDFS-13909-HDFS-13891-04.patch, HDFS-13909-HDFS-13891-05.patch > > > Currently addCachePool, modifyCachePool, removeCachePool, listCachePools, > addCacheDirective, modifyCacheDirective, removeCacheDirective, > listCacheDirectives these APIs are not implemented in Router. > This JIRA is intend to implement above mentioned APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850261#comment-16850261
]
Xue Liu commented on HDFS-12703:
Hi the exception I found come from check():
{code:java}
Preconditions.checkState(false, "A node is in an invalid state!")
{code}
This will cause exception and the way executor#scheduleAtFixedRate works, the
thread's further execution will be suppressed.
So I just add exception handle to prevent exception killing the decommission,
and add a few logs to help with debugging.
It is not clear why sometimes DNs will be in a erroneous state, but this change
will unblock the thread and allow us dig further.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850259#comment-16850259
]
Akira Ajisaka commented on HDFS-13255:
--
Thank you, [~wuweiwei], [~ayushtkn], and [~elgoiri].
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-003.patch, HDFS-13255-HDFS-13891-004.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xue Liu updated HDFS-12703:
---
Attachment: HDFS-12703.001.patch
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849940#comment-16849940 ] Íñigo Goiri edited comment on HDFS-14431 at 5/29/19 12:02 AM: -- I have to say I'm stuck with this. I'm covering it in a non-transactional way and it is kind of dangerous... I'll give it another try this week but started to run out of ideas. was (Author: elgoiri): I have to say I'm stuck with this. I'm covering it in a non-transactional way and it is kind of dangerous... I'll give it a nother tyr this week but started to run out of ideas. > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1224) Restructure code to validate the response from server in the Read path
[
https://issues.apache.org/jira/browse/HDDS-1224?focusedWorklogId=249721&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249721
]
ASF GitHub Bot logged work on HDDS-1224:
Author: ASF GitHub Bot
Created on: 28/May/19 23:52
Start Date: 28/May/19 23:52
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #806:
HDDS-1224. Restructure code to validate the response from server in the Read
path
URL: https://github.com/apache/hadoop/pull/806#discussion_r288345352
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##
@@ -403,6 +360,34 @@ protected ByteString readChunk(final ChunkInfo chunkInfo,
return xceiverClient.getPipeline().getNodes();
}
+ private CheckedFunction
+ validator = (response) -> {
+ReadChunkResponseProto readChunkResponse;
+final ChunkInfo chunkInfo = chunks.get(chunkIndex);
+ByteString byteString;
+try {
+ ContainerProtocolCalls.validateContainerResponse(response);
Review comment:
Instead of calling the another validate function inside a validator, we can
pass a list of validators to XceiverClientSpi#sendCommand().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249721)
Time Spent: 3h 10m (was: 3h)
> Restructure code to validate the response from server in the Read path
> --
>
> Key: HDDS-1224
> URL: https://issues.apache.org/jira/browse/HDDS-1224
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1224.000.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> In the read path, the validation of the response while reading the data from
> the datanodes happen in XceiverClientGrpc as well as additional Checksum
> verification happens in Ozone client to verify the read chunk response. The
> aim of this Jira is to modify the function call to take a validator function
> as a part of reading data so all validation can happen in a single unified
> place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1224) Restructure code to validate the response from server in the Read path
[
https://issues.apache.org/jira/browse/HDDS-1224?focusedWorklogId=249724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249724
]
ASF GitHub Bot logged work on HDDS-1224:
Author: ASF GitHub Bot
Created on: 28/May/19 23:52
Start Date: 28/May/19 23:52
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #806:
HDDS-1224. Restructure code to validate the response from server in the Read
path
URL: https://github.com/apache/hadoop/pull/806#discussion_r288344456
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
##
@@ -403,6 +360,34 @@ protected ByteString readChunk(final ChunkInfo chunkInfo,
return xceiverClient.getPipeline().getNodes();
}
+ private CheckedFunction
+ validator = (response) -> {
+ReadChunkResponseProto readChunkResponse;
+final ChunkInfo chunkInfo = chunks.get(chunkIndex);
Review comment:
The chunkIndex might get updated between the readChunk call and the
validator call.
Also, with HDDS-1496, we are going to support partial chunk reads. So we
would have to somehow let the validator know the chunkIndex, offset and length
which was used in the readChunk call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249724)
Time Spent: 3.5h (was: 3h 20m)
> Restructure code to validate the response from server in the Read path
> --
>
> Key: HDDS-1224
> URL: https://issues.apache.org/jira/browse/HDDS-1224
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1224.000.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> In the read path, the validation of the response while reading the data from
> the datanodes happen in XceiverClientGrpc as well as additional Checksum
> verification happens in Ozone client to verify the read chunk response. The
> aim of this Jira is to modify the function call to take a validator function
> as a part of reading data so all validation can happen in a single unified
> place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1224) Restructure code to validate the response from server in the Read path
[ https://issues.apache.org/jira/browse/HDDS-1224?focusedWorklogId=249723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249723 ] ASF GitHub Bot logged work on HDDS-1224: Author: ASF GitHub Bot Created on: 28/May/19 23:52 Start Date: 28/May/19 23:52 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #806: HDDS-1224. Restructure code to validate the response from server in the Read path URL: https://github.com/apache/hadoop/pull/806#discussion_r288346335 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java ## @@ -228,35 +227,31 @@ public static XceiverClientReply putBlockAsync( * @param chunk information about chunk to read * @param blockID ID of the block * @param traceID container protocol call args - * @param excludeDns datamode to exclude while executing the command Review comment: Why are we removing the exclude datanodes functionality from here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249723) Time Spent: 3h 20m (was: 3h 10m) > Restructure code to validate the response from server in the Read path > -- > > Key: HDDS-1224 > URL: https://issues.apache.org/jira/browse/HDDS-1224 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Attachments: HDDS-1224.000.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > In the read path, the validation of the response while reading the data from > the datanodes happen in XceiverClientGrpc as well as additional Checksum > verification happens in Ozone client to verify the read chunk response. The > aim of this Jira is to modify the function call to take a validator function > as a part of reading data so all validation can happen in a single unified > place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1224) Restructure code to validate the response from server in the Read path
[ https://issues.apache.org/jira/browse/HDDS-1224?focusedWorklogId=249722&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249722 ] ASF GitHub Bot logged work on HDDS-1224: Author: ASF GitHub Bot Created on: 28/May/19 23:52 Start Date: 28/May/19 23:52 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #806: HDDS-1224. Restructure code to validate the response from server in the Read path URL: https://github.com/apache/hadoop/pull/806#discussion_r288336393 ## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java ## @@ -22,6 +22,7 @@ import org.apache.hadoop.hdds.HddsUtils; import org.apache.hadoop.hdds.protocol.DatanodeDetails; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; +import org.apache.hadoop.hdds.scm.storage.CheckedFunction; Review comment: Unused import This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249722) Time Spent: 3h 20m (was: 3h 10m) > Restructure code to validate the response from server in the Read path > -- > > Key: HDDS-1224 > URL: https://issues.apache.org/jira/browse/HDDS-1224 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Attachments: HDDS-1224.000.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > In the read path, the validation of the response while reading the data from > the datanodes happen in XceiverClientGrpc as well as additional Checksum > verification happens in Ozone client to verify the read chunk response. The > aim of this Jira is to modify the function call to take a validator function > as a part of reading data so all validation can happen in a single unified > place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850234#comment-16850234
]
Hadoop QA commented on HDFS-12979:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 366 unchanged - 6 fixed = 366 total (was 372) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} | {color:red} The patch generated 17 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12979 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970063/HDFS-12979.010.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d54f465357cc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d8b18e8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26852/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26852/testReport/ |
| asflicense |
https://buil
[jira] [Commented] (HDFS-14512) ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)
[ https://issues.apache.org/jira/browse/HDFS-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850231#comment-16850231 ] Wei-Chiu Chuang commented on HDFS-14512: Thank you [~ayushtkn] that is a great repro and the reasoning makes sense to me. Would you please assign the Jira to yourself and proceed with the fix? > ONE_SSD policy will be violated while write data with > DistributedFileSystem.create(favoredNodes) > > > Key: HDFS-14512 > URL: https://issues.apache.org/jira/browse/HDFS-14512 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shen Yinjie >Priority: Major > Attachments: TestToRepro.patch > > > Reproduce steps: > 1.setStoragePolicy ONE_SSD for a path A; > 2. client write data to path A by > DistributedFileSystem.create(...favoredNodes) and Passing parameter > favoredNodes > then, three replicas of file in this path will be located in 2 SSD and > 1DISK,which is violating the ONE_SSD policy. > Not sure am I clear? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1341) TestContainerReplication#testContainerReplication fails intermittently
[
https://issues.apache.org/jira/browse/HDDS-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850228#comment-16850228
]
Hudson commented on HDDS-1341:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16618 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16618/])
HDDS-1341. TestContainerReplication#testContainerReplication fails (bharat: rev
79d14d0d421d20c2147990707238435e7808d73d)
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestContainerReplication.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
> TestContainerReplication#testContainerReplication fails intermittently
> --
>
> Key: HDDS-1341
> URL: https://issues.apache.org/jira/browse/HDDS-1341
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test fails intermittently. The link to the test report can be found below.
> https://builds.apache.org/job/PreCommit-HDDS-Build/2582/testReport/
> {code:java}
> java.lang.AssertionError: Container is not replicated to the destination
> datanode
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.ozone.container.TestContainerReplication.testContainerReplication(TestContainerReplication.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently
[
https://issues.apache.org/jira/browse/HDDS-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850225#comment-16850225
]
Hudson commented on HDDS-1536:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16618 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16618/])
HDDS-1536. testSCMSafeModeRestrictedOp is failing consistently. (github: rev
fb0b39f4bfa6d97ca758e617cb9566242813c1a3)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java
> testSCMSafeModeRestrictedOp is failing consistently
> ---
>
> Key: HDDS-1536
> URL: https://issues.apache.org/jira/browse/HDDS-1536
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Mukul Kumar Singh
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The test is failing with the following stack trace.
> {code}
> [ERROR]
> testSCMSafeModeRestrictedOp(org.apache.hadoop.ozone.om.TestScmSafeMode) Time
> elapsed: 9.79 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeModeRestrictedOp(TestScmSafeMode.java:304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter
[
https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850226#comment-16850226
]
Hudson commented on HDFS-14434:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16618 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16618/])
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
(eyang: rev d78854b928bb877f26b11b5b212a100a79941f35)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
> webhdfs that connect secure hdfs should not use user.name parameter
> ---
>
> Key: HDFS-14434
> URL: https://issues.apache.org/jira/browse/HDFS-14434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14434.001.patch, HDFS-14434.002.patch,
> HDFS-14434.003.patch, HDFS-14434.004.patch, HDFS-14434.005.patch,
> HDFS-14434.006.patch, HDFS-14434.007.patch, HDFS-14434.008.patch
>
>
> I have two secure hadoop cluster. Both cluster use cross-realm
> authentication.
> [use...@a.com|mailto:use...@a.com] can access to HDFS of B.COM realm
> by the way, hadoop username of use...@a.com in B.COM realm is
> cross_realm_a_com_user_a.
> hdfs dfs command of use...@a.com using B.COM webhdfs failed.
> root cause is webhdfs that connect secure hdfs use user.name parameter.
> according to webhdfs spec, insecure webhdfs use user.name, secure webhdfs
> use SPNEGO for authentication.
> I think webhdfs that connect secure hdfs should not use user.name parameter.
> I will attach patch.
> below is error log
>
> {noformat}
> $ hdfs dfs -ls webhdfs://b.com:50070/
> ls: Usernames not matched: name=user_a != expected=cross_realm_a_com_user_a
>
> # user.name in cross realm webhdfs
> $ curl -u : --negotiate
> 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=user_a'
> {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed
> to obtain user group information: java.io.IOException: Usernames not
> matched: name=user_a != expected=cross_realm_a_com_user_a"}}
> # USE SPNEGO
> $ curl -u : --negotiate 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN'
> {"Token"{"urlString":"XgA."}}
>
> {noformat}
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14514) Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2
[
https://issues.apache.org/jira/browse/HDFS-14514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850220#comment-16850220
]
Hadoop QA commented on HDFS-14514:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} branch-2 passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_212 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
119 unchanged - 0 fixed = 120 total (was 119) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed with JDK v1.8.0_212 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 12 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.balancer.TestBalan
[jira] [Updated] (HDFS-14456) HAState#prepareToEnterState needn't a lock
[
https://issues.apache.org/jira/browse/HDFS-14456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14456:
---
Status: Patch Available (was: Open)
> HAState#prepareToEnterState needn't a lock
> --
>
> Key: HDFS-14456
> URL: https://issues.apache.org/jira/browse/HDFS-14456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
>
> prepareToEnterState in HAState is called without the context being locked.
> But in NameNode#NameNode, prepareToEnterState is after haContext.writeLock()
>
> {code:java}
> try {
> haContext.writeLock();
> state.prepareToEnterState(haContext);
> state.enterState(haContext);
> } finally {
> haContext.writeUnlock();
> }
> {code}
>
> Is it OK?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?focusedWorklogId=249683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249683 ] ASF GitHub Bot logged work on HDDS-1604: Author: ASF GitHub Bot Created on: 28/May/19 22:30 Start Date: 28/May/19 22:30 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on issue #866: HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… URL: https://github.com/apache/hadoop/pull/866#issuecomment-496714197 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249683) Time Spent: 0.5h (was: 20m) > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This was caught by the New ContainerCache with reference counting from > HDDS-1449. The root cause is an unclosed KeyValueBlockIterator from > ContainerReader#initializeUsedBytes. > I will post a patch shortly, which will fix some UT failures exposed by > -HDDS-1449,- such as TestBCSID#testBCSID, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1580) Obtain Handler reference in ContainerScrubber
[ https://issues.apache.org/jira/browse/HDDS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1580: --- Fix Version/s: 0.5.0 > Obtain Handler reference in ContainerScrubber > - > > Key: HDDS-1580 > URL: https://issues.apache.org/jira/browse/HDDS-1580 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Shweta >Assignee: Shweta >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Obtain reference to Handler based on containerType in scrub() in > ContainerScrubber.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1341) TestContainerReplication#testContainerReplication fails intermittently
[
https://issues.apache.org/jira/browse/HDDS-1341?focusedWorklogId=249662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249662
]
ASF GitHub Bot logged work on HDDS-1341:
Author: ASF GitHub Bot
Created on: 28/May/19 21:40
Start Date: 28/May/19 21:40
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #862:
HDDS-1341. TestContainerReplication#testContainerReplication fails
intermittently
URL: https://github.com/apache/hadoop/pull/862
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249662)
Time Spent: 1h (was: 50m)
> TestContainerReplication#testContainerReplication fails intermittently
> --
>
> Key: HDDS-1341
> URL: https://issues.apache.org/jira/browse/HDDS-1341
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test fails intermittently. The link to the test report can be found below.
> https://builds.apache.org/job/PreCommit-HDDS-Build/2582/testReport/
> {code:java}
> java.lang.AssertionError: Container is not replicated to the destination
> datanode
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.ozone.container.TestContainerReplication.testContainerReplication(TestContainerReplication.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1341) TestContainerReplication#testContainerReplication fails intermittently
[
https://issues.apache.org/jira/browse/HDDS-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1341:
-
Resolution: Fixed
Fix Version/s: 0.4.1
Status: Resolved (was: Patch Available)
> TestContainerReplication#testContainerReplication fails intermittently
> --
>
> Key: HDDS-1341
> URL: https://issues.apache.org/jira/browse/HDDS-1341
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test fails intermittently. The link to the test report can be found below.
> https://builds.apache.org/job/PreCommit-HDDS-Build/2582/testReport/
> {code:java}
> java.lang.AssertionError: Container is not replicated to the destination
> datanode
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.ozone.container.TestContainerReplication.testContainerReplication(TestContainerReplication.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1341) TestContainerReplication#testContainerReplication fails intermittently
[
https://issues.apache.org/jira/browse/HDDS-1341?focusedWorklogId=249661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249661
]
ASF GitHub Bot logged work on HDDS-1341:
Author: ASF GitHub Bot
Created on: 28/May/19 21:39
Start Date: 28/May/19 21:39
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #862: HDDS-1341.
TestContainerReplication#testContainerReplication fails intermittently
URL: https://github.com/apache/hadoop/pull/862#issuecomment-496700043
+1 LGTM.
Thank You @elek for the fix, now in the Jenkins run I don't see it failing.
I will commit this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249661)
Time Spent: 50m (was: 40m)
> TestContainerReplication#testContainerReplication fails intermittently
> --
>
> Key: HDDS-1341
> URL: https://issues.apache.org/jira/browse/HDDS-1341
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The test fails intermittently. The link to the test report can be found below.
> https://builds.apache.org/job/PreCommit-HDDS-Build/2582/testReport/
> {code:java}
> java.lang.AssertionError: Container is not replicated to the destination
> datanode
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.ozone.container.TestContainerReplication.testContainerReplication(TestContainerReplication.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1341) TestContainerReplication#testContainerReplication fails intermittently
[
https://issues.apache.org/jira/browse/HDDS-1341?focusedWorklogId=249660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249660
]
ASF GitHub Bot logged work on HDDS-1341:
Author: ASF GitHub Bot
Created on: 28/May/19 21:39
Start Date: 28/May/19 21:39
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #862: HDDS-1341.
TestContainerReplication#testContainerReplication fails intermittently
URL: https://github.com/apache/hadoop/pull/862#issuecomment-496700043
+1 LGTM.
Thank You @elek for the fix.
I will commit this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249660)
Time Spent: 40m (was: 0.5h)
> TestContainerReplication#testContainerReplication fails intermittently
> --
>
> Key: HDDS-1341
> URL: https://issues.apache.org/jira/browse/HDDS-1341
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The test fails intermittently. The link to the test report can be found below.
> https://builds.apache.org/job/PreCommit-HDDS-Build/2582/testReport/
> {code:java}
> java.lang.AssertionError: Container is not replicated to the destination
> datanode
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.ozone.container.TestContainerReplication.testContainerReplication(TestContainerReplication.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter
[
https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850167#comment-16850167
]
Eric Yang edited comment on HDFS-14434 at 5/28/19 9:36 PM:
---
+1 on patch 008. I just committed this to trunk.
Thank you [~magnum] for the patch.
Thank you [~kihwal] for reviews.
was (Author: eyang):
+1 on patch 008. I just committed this to trunk.
> webhdfs that connect secure hdfs should not use user.name parameter
> ---
>
> Key: HDFS-14434
> URL: https://issues.apache.org/jira/browse/HDFS-14434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14434.001.patch, HDFS-14434.002.patch,
> HDFS-14434.003.patch, HDFS-14434.004.patch, HDFS-14434.005.patch,
> HDFS-14434.006.patch, HDFS-14434.007.patch, HDFS-14434.008.patch
>
>
> I have two secure hadoop cluster. Both cluster use cross-realm
> authentication.
> [use...@a.com|mailto:use...@a.com] can access to HDFS of B.COM realm
> by the way, hadoop username of use...@a.com in B.COM realm is
> cross_realm_a_com_user_a.
> hdfs dfs command of use...@a.com using B.COM webhdfs failed.
> root cause is webhdfs that connect secure hdfs use user.name parameter.
> according to webhdfs spec, insecure webhdfs use user.name, secure webhdfs
> use SPNEGO for authentication.
> I think webhdfs that connect secure hdfs should not use user.name parameter.
> I will attach patch.
> below is error log
>
> {noformat}
> $ hdfs dfs -ls webhdfs://b.com:50070/
> ls: Usernames not matched: name=user_a != expected=cross_realm_a_com_user_a
>
> # user.name in cross realm webhdfs
> $ curl -u : --negotiate
> 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=user_a'
> {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed
> to obtain user group information: java.io.IOException: Usernames not
> matched: name=user_a != expected=cross_realm_a_com_user_a"}}
> # USE SPNEGO
> $ curl -u : --negotiate 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN'
> {"Token"{"urlString":"XgA."}}
>
> {noformat}
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14434) webhdfs that connect secure hdfs should not use user.name parameter
[
https://issues.apache.org/jira/browse/HDFS-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-14434:
-
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
+1 on patch 008. I just committed this to trunk.
> webhdfs that connect secure hdfs should not use user.name parameter
> ---
>
> Key: HDFS-14434
> URL: https://issues.apache.org/jira/browse/HDFS-14434
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14434.001.patch, HDFS-14434.002.patch,
> HDFS-14434.003.patch, HDFS-14434.004.patch, HDFS-14434.005.patch,
> HDFS-14434.006.patch, HDFS-14434.007.patch, HDFS-14434.008.patch
>
>
> I have two secure hadoop cluster. Both cluster use cross-realm
> authentication.
> [use...@a.com|mailto:use...@a.com] can access to HDFS of B.COM realm
> by the way, hadoop username of use...@a.com in B.COM realm is
> cross_realm_a_com_user_a.
> hdfs dfs command of use...@a.com using B.COM webhdfs failed.
> root cause is webhdfs that connect secure hdfs use user.name parameter.
> according to webhdfs spec, insecure webhdfs use user.name, secure webhdfs
> use SPNEGO for authentication.
> I think webhdfs that connect secure hdfs should not use user.name parameter.
> I will attach patch.
> below is error log
>
> {noformat}
> $ hdfs dfs -ls webhdfs://b.com:50070/
> ls: Usernames not matched: name=user_a != expected=cross_realm_a_com_user_a
>
> # user.name in cross realm webhdfs
> $ curl -u : --negotiate
> 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=user_a'
> {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed
> to obtain user group information: java.io.IOException: Usernames not
> matched: name=user_a != expected=cross_realm_a_com_user_a"}}
> # USE SPNEGO
> $ curl -u : --negotiate 'http://b.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN'
> {"Token"{"urlString":"XgA."}}
>
> {noformat}
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1597) Remove hdds-server-scm dependency from ozone-common
[ https://issues.apache.org/jira/browse/HDDS-1597?focusedWorklogId=249654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249654 ] ASF GitHub Bot logged work on HDDS-1597: Author: ASF GitHub Bot Created on: 28/May/19 21:31 Start Date: 28/May/19 21:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #860: HDDS-1597. Remove hdds-server-scm dependency from ozone-common URL: https://github.com/apache/hadoop/pull/860#discussion_r288310911 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java ## @@ -52,6 +51,7 @@ import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.test.LambdaTestUtils; +import org.apache.commons.lang3.RandomStringUtils; Review comment: Same in this, I don't see any code changes only re-ordering of imports, I think we don't need this change in this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249654) Time Spent: 1h (was: 50m) > Remove hdds-server-scm dependency from ozone-common > --- > > Key: HDDS-1597 > URL: https://issues.apache.org/jira/browse/HDDS-1597 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: ozone-dependency.png > > Time Spent: 1h > Remaining Estimate: 0h > > I noticed that the hadoop-ozone/common project depends on > hadoop-hdds-server-scm project. > The common projects are designed to be a shared artifacts between client and > server side. Adding additional dependency to the common pom means that the > dependency will be available for all the clients as well. > (See the attached artifact about the current, desired structure). > We definitely don't need scm server dependency on the client side. > The code dependency is just one class (ScmUtils) and the shared code can be > easily moved to the common. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1597) Remove hdds-server-scm dependency from ozone-common
[
https://issues.apache.org/jira/browse/HDDS-1597?focusedWorklogId=249652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249652
]
ASF GitHub Bot logged work on HDDS-1597:
Author: ASF GitHub Bot
Created on: 28/May/19 21:31
Start Date: 28/May/19 21:31
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #860:
HDDS-1597. Remove hdds-server-scm dependency from ozone-common
URL: https://github.com/apache/hadoop/pull/860#discussion_r288309680
##
File path:
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
##
@@ -203,4 +203,16 @@ public static void setOzoneMetaDirPath(OzoneConfiguration
conf,
conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, path);
}
+ public static File getDBPath(Configuration conf, String dbDirectory) {
+final File dbDirPath =
+getDirectoryFromConfig(conf, dbDirectory, "OM");
+if (dbDirPath != null) {
+ return dbDirPath;
Review comment:
Add java doc for this method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249652)
Time Spent: 40m (was: 0.5h)
> Remove hdds-server-scm dependency from ozone-common
> ---
>
> Key: HDDS-1597
> URL: https://issues.apache.org/jira/browse/HDDS-1597
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Attachments: ozone-dependency.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I noticed that the hadoop-ozone/common project depends on
> hadoop-hdds-server-scm project.
> The common projects are designed to be a shared artifacts between client and
> server side. Adding additional dependency to the common pom means that the
> dependency will be available for all the clients as well.
> (See the attached artifact about the current, desired structure).
> We definitely don't need scm server dependency on the client side.
> The code dependency is just one class (ScmUtils) and the shared code can be
> easily moved to the common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1597) Remove hdds-server-scm dependency from ozone-common
[ https://issues.apache.org/jira/browse/HDDS-1597?focusedWorklogId=249655&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249655 ] ASF GitHub Bot logged work on HDDS-1597: Author: ASF GitHub Bot Created on: 28/May/19 21:31 Start Date: 28/May/19 21:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #860: HDDS-1597. Remove hdds-server-scm dependency from ozone-common URL: https://github.com/apache/hadoop/pull/860#discussion_r288310658 ## File path: hadoop-ozone/integration-test/pom.xml ## @@ -60,6 +64,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd";> org.apache.hadoop hadoop-ozone-client + + org.apache.commons + commons-lang3 Review comment: Understood, this is because of the below test needs some classes from this jar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249655) > Remove hdds-server-scm dependency from ozone-common > --- > > Key: HDDS-1597 > URL: https://issues.apache.org/jira/browse/HDDS-1597 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: ozone-dependency.png > > Time Spent: 1h > Remaining Estimate: 0h > > I noticed that the hadoop-ozone/common project depends on > hadoop-hdds-server-scm project. > The common projects are designed to be a shared artifacts between client and > server side. Adding additional dependency to the common pom means that the > dependency will be available for all the clients as well. > (See the attached artifact about the current, desired structure). > We definitely don't need scm server dependency on the client side. > The code dependency is just one class (ScmUtils) and the shared code can be > easily moved to the common. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1597) Remove hdds-server-scm dependency from ozone-common
[ https://issues.apache.org/jira/browse/HDDS-1597?focusedWorklogId=249653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249653 ] ASF GitHub Bot logged work on HDDS-1597: Author: ASF GitHub Bot Created on: 28/May/19 21:31 Start Date: 28/May/19 21:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #860: HDDS-1597. Remove hdds-server-scm dependency from ozone-common URL: https://github.com/apache/hadoop/pull/860#discussion_r288310005 ## File path: hadoop-ozone/integration-test/pom.xml ## @@ -60,6 +64,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd";> org.apache.hadoop hadoop-ozone-client + + org.apache.commons + commons-lang3 Review comment: Why is this additional dependency brought in? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249653) Time Spent: 50m (was: 40m) > Remove hdds-server-scm dependency from ozone-common > --- > > Key: HDDS-1597 > URL: https://issues.apache.org/jira/browse/HDDS-1597 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: ozone-dependency.png > > Time Spent: 50m > Remaining Estimate: 0h > > I noticed that the hadoop-ozone/common project depends on > hadoop-hdds-server-scm project. > The common projects are designed to be a shared artifacts between client and > server side. Adding additional dependency to the common pom means that the > dependency will be available for all the clients as well. > (See the attached artifact about the current, desired structure). > We definitely don't need scm server dependency on the client side. > The code dependency is just one class (ScmUtils) and the shared code can be > easily moved to the common. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1597) Remove hdds-server-scm dependency from ozone-common
[ https://issues.apache.org/jira/browse/HDDS-1597?focusedWorklogId=249651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249651 ] ASF GitHub Bot logged work on HDDS-1597: Author: ASF GitHub Bot Created on: 28/May/19 21:31 Start Date: 28/May/19 21:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #860: HDDS-1597. Remove hdds-server-scm dependency from ozone-common URL: https://github.com/apache/hadoop/pull/860#discussion_r288309094 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java ## @@ -35,28 +35,27 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos; import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB; import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolPB; import org.apache.hadoop.hdds.scm.ScmConfigKeys; -import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol; import org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolPB; import org.apache.hadoop.ipc.Client; import org.apache.hadoop.ipc.ProtobufRpcEngine; import org.apache.hadoop.ipc.RPC; import org.apache.hadoop.metrics2.util.MBeans; import org.apache.hadoop.net.DNS; import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.security.UserGroupInformation; Review comment: I see no changes in this file, just re-ordering of imports. Why is this change needed here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249651) Time Spent: 0.5h (was: 20m) > Remove hdds-server-scm dependency from ozone-common > --- > > Key: HDDS-1597 > URL: https://issues.apache.org/jira/browse/HDDS-1597 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Attachments: ozone-dependency.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > I noticed that the hadoop-ozone/common project depends on > hadoop-hdds-server-scm project. > The common projects are designed to be a shared artifacts between client and > server side. Adding additional dependency to the common pom means that the > dependency will be available for all the clients as well. > (See the attached artifact about the current, desired structure). > We definitely don't need scm server dependency on the client side. > The code dependency is just one class (ScmUtils) and the shared code can be > easily moved to the common. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850164#comment-16850164
]
Hadoop QA commented on HDFS-13480:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-13480 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12924067/HDFS-13480.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26853/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
> ---
>
> Key: HDFS-13480
> URL: https://issues.apache.org/jira/browse/HDFS-13480
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13480.001.patch, HDFS-13480.002.patch,
> HDFS-13480.002.patch, HDFS-13480.003.patch, HDFS-13480.004.patch
>
>
> Now, if i enable the heartbeat.enable, but i do not want to monitor any
> namenode, i get an ERROR log like:
> {code:java}
> [2018-04-19T14:00:03.057+08:00] [ERROR]
> federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is
> enabled but there are no namenodes to monitor
> {code}
> and if i disable the heartbeat.enable, we cannot get any mounttable update,
> because the following logic in Router.java：
> {code:java}
> if (conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) {
> // Create status updater for each monitored Namenode
> this.namenodeHeartbeatServices = createNamenodeHeartbeatServices();
> for (NamenodeHeartbeatService hearbeatService :
> this.namenodeHeartbeatServices) {
> addService(hearbeatService);
> }
> if (this.namenodeHeartbeatServices.isEmpty()) {
> LOG.error("Heartbeat is enabled but there are no namenodes to
> monitor");
> }
> // Periodically update the router state
> this.routerHeartbeatService = new RouterHeartbeatService(this);
> addService(this.routerHeartbeatService);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1600) Add userName and IPAddress as part of OMRequest.
[ https://issues.apache.org/jira/browse/HDDS-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850152#comment-16850152 ] Eric Yang commented on HDDS-1600: - [~bharatviswa] can hostname be used as part of OM request? For running in docker container, virtual private network address may not be routable or exposed to outside world. Using IP to identify the source client location may not be enough. It would be nice to have ability support hostname based request too. > Add userName and IPAddress as part of OMRequest. > > > Key: HDDS-1600 > URL: https://issues.apache.org/jira/browse/HDDS-1600 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In OM HA, the actual execution of request happens under GRPC context, so UGI > object which we retrieve from ProtobufRpcEngine.Server.getRemoteUser(); will > not be available. > In similar manner ProtobufRpcEngine.Server.getRemoteIp(). > > So, during preExecute(which happens under RPC context) extract userName and > IPAddress and add it to the OMRequest, and then send the request to ratis > server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently
[
https://issues.apache.org/jira/browse/HDDS-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-1536.
--
Resolution: Fixed
Thanks [~msingh] for reporting the issue and [~bharatviswa] for the review.
I've committed the patch to trunk.
> testSCMSafeModeRestrictedOp is failing consistently
> ---
>
> Key: HDDS-1536
> URL: https://issues.apache.org/jira/browse/HDDS-1536
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Mukul Kumar Singh
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The test is failing with the following stack trace.
> {code}
> [ERROR]
> testSCMSafeModeRestrictedOp(org.apache.hadoop.ozone.om.TestScmSafeMode) Time
> elapsed: 9.79 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeModeRestrictedOp(TestScmSafeMode.java:304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently
[
https://issues.apache.org/jira/browse/HDDS-1536?focusedWorklogId=249639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249639
]
ASF GitHub Bot logged work on HDDS-1536:
Author: ASF GitHub Bot
Created on: 28/May/19 21:02
Start Date: 28/May/19 21:02
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #865: HDDS-1536.
testSCMSafeModeRestrictedOp is failing consistently. Contr…
URL: https://github.com/apache/hadoop/pull/865
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 249639)
Time Spent: 0.5h (was: 20m)
> testSCMSafeModeRestrictedOp is failing consistently
> ---
>
> Key: HDDS-1536
> URL: https://issues.apache.org/jira/browse/HDDS-1536
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Mukul Kumar Singh
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The test is failing with the following stack trace.
> {code}
> [ERROR]
> testSCMSafeModeRestrictedOp(org.apache.hadoop.ozone.om.TestScmSafeMode) Time
> elapsed: 9.79 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.ozone.om.TestScmSafeMode.testSCMSafeModeRestrictedOp(TestScmSafeMode.java:304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1536) testSCMSafeModeRestrictedOp is failing consistently
[ https://issues.apache.org/jira/browse/HDDS-1536?focusedWorklogId=249637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249637 ] ASF GitHub Bot logged work on HDDS-1536: Author: ASF GitHub Bot Created on: 28/May/19 20:58 Start Date: 28/May/19 20:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #865: HDDS-1536. testSCMSafeModeRestrictedOp is failing consistently. Contr… URL: https://github.com/apache/hadoop/pull/865#issuecomment-496687313 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 588 | trunk passed | | +1 | compile | 320 | trunk passed | | +1 | checkstyle | 80 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 982 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 173 | trunk passed | | 0 | spotbugs | 361 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 610 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 557 | the patch passed | | +1 | compile | 327 | the patch passed | | +1 | javac | 327 | the patch passed | | +1 | checkstyle | 91 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 801 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 161 | the patch passed | | +1 | findbugs | 646 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 271 | hadoop-hdds in the patch passed. | | -1 | unit | 2382 | hadoop-ozone in the patch failed. | | -1 | asflicense | 45 | The patch generated 17 ASF License warnings. | | | | 15807 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.client.rpc.TestKeyInputStream | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-865/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/865 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1ab2e1cc9449 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d8b18e8 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-865/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-865/1/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-865/1/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4442 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-865/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249637) Time Spent: 20m (was: 10m) > testSCMSafeModeRestrictedOp is failing consistently > --- > > Key: HDDS-1536 > URL: https://issues.apache.org/jira/browse/HDDS-1536 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Xiaoyu Yao >Priority: Major >
[jira] [Commented] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850133#comment-16850133 ] Eric Yang commented on HDDS-1554: - [~elek] The approach to use docker for integration test is heavily influenced by decision to use docker-compose heavily in existing blockade tests. It is better than mini cluster in some situations because: # Each process is running with independent network address # Docker containers can run production code instead of a simulated cluster # Docker containers can isolate visibility of data disks # It will be possible to test SELinux and Hadoop compatibilities # It will be possible to test Kernel security capabilities # It is easier to verify multiple users file permission related issues # Simulation for resource constrained environment (out of disk space) > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13480) RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
[
https://issues.apache.org/jira/browse/HDFS-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-13480:
Parent Issue: HDFS-13891 (was: HDFS-13575)
> RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key
> ---
>
> Key: HDFS-13480
> URL: https://issues.apache.org/jira/browse/HDFS-13480
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13480.001.patch, HDFS-13480.002.patch,
> HDFS-13480.002.patch, HDFS-13480.003.patch, HDFS-13480.004.patch
>
>
> Now, if i enable the heartbeat.enable, but i do not want to monitor any
> namenode, i get an ERROR log like:
> {code:java}
> [2018-04-19T14:00:03.057+08:00] [ERROR]
> federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is
> enabled but there are no namenodes to monitor
> {code}
> and if i disable the heartbeat.enable, we cannot get any mounttable update,
> because the following logic in Router.java：
> {code:java}
> if (conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) {
> // Create status updater for each monitored Namenode
> this.namenodeHeartbeatServices = createNamenodeHeartbeatServices();
> for (NamenodeHeartbeatService hearbeatService :
> this.namenodeHeartbeatServices) {
> addService(hearbeatService);
> }
> if (this.namenodeHeartbeatServices.isEmpty()) {
> LOG.error("Heartbeat is enabled but there are no namenodes to
> monitor");
> }
> // Periodically update the router state
> this.routerHeartbeatService = new RouterHeartbeatService(this);
> addService(this.routerHeartbeatService);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1542) Create Radix tree from ozone objects,getAcl ACLs on OM startup
[ https://issues.apache.org/jira/browse/HDDS-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDDS-1542: Assignee: Xiaoyu Yao > Create Radix tree from ozone objects,getAcl ACLs on OM startup > --- > > Key: HDDS-1542 > URL: https://issues.apache.org/jira/browse/HDDS-1542 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Xiaoyu Yao >Priority: Major > > Create Radix tree from ozone objects,getAcl ACLs on OM startup -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14508) RBF: Clean-up and refactor UI components
[
https://issues.apache.org/jira/browse/HDFS-14508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850107#comment-16850107
]
CR Hota commented on HDFS-14508:
[~elgoiri] Thanks for the comments.
Another way this could be thought about is to define a common interface (that
will have methods definitely for router and namenodes ex security, status,
version etc) and let NamenodeMetrics and RouterBeanmetrics extend that
interface. In future new common additions can continue to be added in commons
bean and anything specific to either namenode or router. Federation metrics
should continue to strictly have metrics only around federation.
> RBF: Clean-up and refactor UI components
>
>
> Key: HDFS-14508
> URL: https://issues.apache.org/jira/browse/HDFS-14508
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> Router UI has tags that are not used or incorrectly set. The code should be
> cleaned-up.
> One such example is
> Path :
> (\hadoop-hdfs-project\hadoop-hdfs-rbf\src\main\webapps\router\federationhealth.js)
> {code:java}
> {"name": "routerstat", "url":
> "/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus"},{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1605) Implement AuditLogging for OM HA Bucket write requests
[ https://issues.apache.org/jira/browse/HDDS-1605?focusedWorklogId=249612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249612 ] ASF GitHub Bot logged work on HDDS-1605: Author: ASF GitHub Bot Created on: 28/May/19 20:16 Start Date: 28/May/19 20:16 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #867: HDDS-1605. Implement AuditLogging for OM HA Bucket write requests. URL: https://github.com/apache/hadoop/pull/867 This patch is dependent on HDDS-1551 and HDDS-1600. First 2 commits are of HDDS-1551 and HDDS-1600. The last commit is the changes which are needed in this PR. Opened this PR to get UT's and CI run. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249612) Time Spent: 10m Remaining Estimate: 0h > Implement AuditLogging for OM HA Bucket write requests > -- > > Key: HDDS-1605 > URL: https://issues.apache.org/jira/browse/HDDS-1605 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In this Jira, we shall implement audit logging for OM HA Bucket write > requests. > As now we cannot use userName, IpAddress from Server API's as these will be > null, because the requests are executed under GRPC context. So, in our > AuditLogger API's we need to pass username and remoteAddress which we can get > from OMRequest after HDDS-1600 and use these during audit logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1604) ContainerReader#initializeUsedBytes leaks DB reference
[ https://issues.apache.org/jira/browse/HDDS-1604?focusedWorklogId=249610&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249610 ] ASF GitHub Bot logged work on HDDS-1604: Author: ASF GitHub Bot Created on: 28/May/19 20:16 Start Date: 28/May/19 20:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #866: HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… URL: https://github.com/apache/hadoop/pull/866#issuecomment-496672252 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 527 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 559 | trunk passed | | +1 | compile | 264 | trunk passed | | +1 | checkstyle | 88 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 884 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | trunk passed | | 0 | spotbugs | 287 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 488 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 471 | the patch passed | | +1 | compile | 272 | the patch passed | | +1 | javac | 272 | the patch passed | | -0 | checkstyle | 41 | hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 676 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | the patch passed | | +1 | findbugs | 502 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 233 | hadoop-hdds in the patch failed. | | -1 | unit | 1302 | hadoop-ozone in the patch failed. | | -1 | asflicense | 57 | The patch generated 17 ASF License warnings. | | | | 6911 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.block.TestBlockManager | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/866 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e0d1e6b804bb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d8b18e8 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4374 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-866/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 249610) Time Spent: 20m (was: 10m) > ContainerReader#initializeUsedBytes leaks DB reference > -- > > Key: HDDS-1604 > URL: https://issues.apache.org/jira/browse/HDDS-1604 > Project: Hadoop Distributed Data Store >
[jira] [Commented] (HDFS-14516) RBF: Create hdfs-rbf-site.xml for RBF specific properties
[
https://issues.apache.org/jira/browse/HDFS-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850106#comment-16850106
]
Ayush Saxena commented on HDFS-14516:
-
if someone following the present trend, if he doesn't need to change his
configuration files and can still go ahead using Router without mandatorilly
using {{hdfs-rbf-site.xml}} . Then that is compatible, if he has to change,
then I think that would be pretty much an incompatible change.
> RBF: Create hdfs-rbf-site.xml for RBF specific properties
> -
>
> Key: HDFS-14516
> URL: https://issues.apache.org/jira/browse/HDFS-14516
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-14516.1.patch
>
>
> Currently, users write rbf properties in {{hdfs-site.xml}} though the
> definitions are in {{hdfs-rbf-default.xml}}. Like other modules, it would be
> better if there is a specific configuration file, {{hdfs-rbf-site.xml}}.
> {{hdfs-rbf-default.xml}} also should be loaded when it exists in the
> configuration directory. It is just a document at the moment.
> There is an early discussion in HDFS-13215.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1605) Implement AuditLogging for OM HA Bucket write requests
[ https://issues.apache.org/jira/browse/HDDS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1605: - Labels: pull-request-available (was: ) > Implement AuditLogging for OM HA Bucket write requests > -- > > Key: HDDS-1605 > URL: https://issues.apache.org/jira/browse/HDDS-1605 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > In this Jira, we shall implement audit logging for OM HA Bucket write > requests. > As now we cannot use userName, IpAddress from Server API's as these will be > null, because the requests are executed under GRPC context. So, in our > AuditLogger API's we need to pass username and remoteAddress which we can get > from OMRequest after HDDS-1600 and use these during audit logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14486) The exception classes in some throw statements do not accurately describe why they are thrown
[
https://issues.apache.org/jira/browse/HDFS-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850100#comment-16850100
]
Ayush Saxena commented on HDFS-14486:
-
Thanx [~elgoiri] . I guess we need to close all the JIRA, may be the ones not
being fixed we can mark as Not a problem and close.
> The exception classes in some throw statements do not accurately describe why
> they are thrown
> -
>
> Key: HDFS-14486
> URL: https://issues.apache.org/jira/browse/HDFS-14486
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Assignee: Ayush Saxena
>Priority: Minor
> Attachments: HDFS-14486-01.patch, HDFS-14486-02.patch,
> HDFS-14486-03.patch, HDFS-14486-04.patch
>
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted a few {{throw}} statements whose
> exception class does not accurately describe why they are thrown. This can be
> dangerous since it makes correctly handling them challenging. For example, in
> an old bug, HDFS-8224, throwing a general {{IOException}} makes it difficult
> to perform data recovery specifically when a metadata file is corrupted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13909) RBF: Add Cache pools and directives related ClientProtocol apis
[ https://issues.apache.org/jira/browse/HDFS-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850099#comment-16850099 ] Ayush Saxena commented on HDFS-13909: - Test Failure doesn't seem related. [~elgoiri] can you give a check to the latest patch, have made changes as suggested. > RBF: Add Cache pools and directives related ClientProtocol apis > --- > > Key: HDFS-13909 > URL: https://issues.apache.org/jira/browse/HDFS-13909 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Dibyendu Karmakar >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13909-HDFS-13891-01.patch, > HDFS-13909-HDFS-13891-02.patch, HDFS-13909-HDFS-13891-03.patch, > HDFS-13909-HDFS-13891-04.patch, HDFS-13909-HDFS-13891-05.patch > > > Currently addCachePool, modifyCachePool, removeCachePool, listCachePools, > addCacheDirective, modifyCacheDirective, removeCacheDirective, > listCacheDirectives these APIs are not implemented in Router. > This JIRA is intend to implement above mentioned APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-13255:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-13891
Status: Resolved (was: Patch Available)
Committed.
Thanx [~aajisaka] for the contribution [~wuweiwei] for the report and
[~elgoiri] for the review.
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-003.patch, HDFS-13255-HDFS-13891-004.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1602) Fix TestContainerPersistence#testDeleteBlockTwice
[ https://issues.apache.org/jira/browse/HDDS-1602?focusedWorklogId=249604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249604 ] ASF GitHub Bot logged work on HDDS-1602: Author: ASF GitHub Bot Created on: 28/May/19 19:56 Start Date: 28/May/19 19:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #858: HDDS-1602. Fix TestContainerPersistence#testDeleteBlockTwice. URL: https://github.com/apache/hadoop/pull/858#issuecomment-496665066 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 584 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 23 | hadoop-ozone in trunk failed. | | -1 | compile | 18 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | -0 | checkstyle | 30 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 826 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 155 | trunk passed | | 0 | spotbugs | 370 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 548 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 529 | the patch passed | | +1 | compile | 279 | the patch passed | | -1 | javac | 98 | hadoop-hdds generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0) | | +1 | checkstyle | 84 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | -1 | whitespace | 0 | The patch has 65 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 | whitespace | 0 | The patch 600 line(s) with tabs. | | +1 | shadedclient | 666 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 157 | the patch passed | | +1 | findbugs | 520 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 222 | hadoop-hdds in the patch passed. | | -1 | unit | 1045 | hadoop-ozone in the patch failed. | | -1 | asflicense | 49 | The patch generated 17 ASF License warnings. | | | | 6073 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBCSID | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/858 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 76e0a1bb6af0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d1ec1c5 | | Default Java | 1.8.0_212 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/branch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-858/out/maven-branch-checkstyle-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/diff-compile-javac-hadoop-hdds.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/whitespace-tabs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-858/3/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4681 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service | | Console output | https://bui
[jira] [Updated] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12979: -- Attachment: HDFS-12979.010.patch > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch, > HDFS-12979.003.patch, HDFS-12979.004.patch, HDFS-12979.005.patch, > HDFS-12979.006.patch, HDFS-12979.007.patch, HDFS-12979.008.patch, > HDFS-12979.009.patch, HDFS-12979.010.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14514) Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2
[
https://issues.apache.org/jira/browse/HDFS-14514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-14514:
-
Attachment: HDFS-14514.branch-2.002.patch
> Actual read size of open file in encryption zone still larger than listing
> size even after enabling HDFS-11402 in Hadoop 2
> --
>
> Key: HDFS-14514
> URL: https://issues.apache.org/jira/browse/HDFS-14514
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs, snapshots
>Affects Versions: 2.6.5, 2.9.2, 2.8.5, 2.7.7
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14514.branch-2.001.patch,
> HDFS-14514.branch-2.002.patch
>
>
> In Hadoop 2, when a file is opened for write in *encryption zone*, taken a
> snapshot and appended, the read out file size in the snapshot is larger than
> the listing size. This happens even when immutable snapshot HDFS-11402 is
> enabled.
> Note: The refactor HDFS-8905 happened in Hadoop 3.0 and later fixed the bug
> silently (probably incidentally). Hadoop 2.x are still suffering from this
> issue.
> Thanks [~sodonnell] for locating the root cause in the codebase.
> Repro:
> 1. Set dfs.namenode.snapshot.capture.openfiles to true in hdfs-site.xml,
> start HDFS cluster
> 2. Create an empty directory /dataenc, create encryption zone and allow
> snapshot on it
> {code:bash}
> hadoop key create reprokey
> sudo -u hdfs hdfs dfs -mkdir /dataenc
> sudo -u hdfs hdfs crypto -createZone -keyName reprokey -path /dataenc
> sudo -u hdfs hdfs dfsadmin -allowSnapshot /dataenc
> {code}
> 3. Use a client that keeps a file open for write under /dataenc. For example,
> I'm using Flume HDFS sink to tail a local file.
> 4. Append the file several times using the client, keep the file open.
> 5. Create a snapshot
> {code:bash}
> sudo -u hdfs hdfs dfs -createSnapshot /dataenc snap1
> {code}
> 6. Append the file one or more times, but don't let the file size exceed the
> block size limit. Wait for several seconds for the append to be flushed to DN.
> 7. Do a -ls on the file inside the snapshot, then try to read the file using
> -get, you should see the actual file size read is larger than the listing
> size from -ls.
> The patch and an updated unit test will be uploaded later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14514) Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2
[
https://issues.apache.org/jira/browse/HDFS-14514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850065#comment-16850065
]
Stephen O'Donnell commented on HDFS-14514:
--
I can reproduce those test failures locally. Looking at the refactored code in
HDFS-8905 (HADOOP 3.0 and later) where this issue does not occur, the code
becomes:
{code}
public int readFromBlock(BlockReader blockReader,
int length) throws IOException {
ByteBuffer tmpBuf = readBuf.duplicate();
tmpBuf.limit(tmpBuf.position() + length);
int nRead = blockReader.read(tmpBuf);
// Only when data are read, update the position
if (nRead > 0) {
readBuf.position(readBuf.position() + nRead);
}
return nRead;
}
{code}
Compared to the same code in the 2.x branch:
{code}
public int doRead(BlockReader blockReader, int off, int len)
throws ChecksumException, IOException {
int oldpos = buf.position();
int oldlimit = buf.limit();
boolean success = false;
try {
int ret = blockReader.read(buf);
success = true;
updateReadStatistics(readStatistics, ret, blockReader);
return ret;
} finally {
if (!success) {
// Reset to original state so that retries work correctly.
buf.position(oldpos);
buf.limit(oldlimit);
}
}
}
{code}
Note that:
1. In the refactored code, the 'off' parameter has been dropped completely,
indicating it is not needed but 'len' is now used.
2. In the 2.x code, off and len are parameters to the method but never used,
and I believe "len" needs to be used or it causes this bug.
In the v001 patch, if I remove the line setting the offset, it fixes the test
failures I checked locally. I suspect with a ByteBufferReader, the offset is
controlled elsewhere.
To start with, I will push up a new patch with that change to see if all tests
pass. Then we can look further into why non-encrypted reads are not affected.
> Actual read size of open file in encryption zone still larger than listing
> size even after enabling HDFS-11402 in Hadoop 2
> --
>
> Key: HDFS-14514
> URL: https://issues.apache.org/jira/browse/HDFS-14514
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs, snapshots
>Affects Versions: 2.6.5, 2.9.2, 2.8.5, 2.7.7
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14514.branch-2.001.patch
>
>
> In Hadoop 2, when a file is opened for write in *encryption zone*, taken a
> snapshot and appended, the read out file size in the snapshot is larger than
> the listing size. This happens even when immutable snapshot HDFS-11402 is
> enabled.
> Note: The refactor HDFS-8905 happened in Hadoop 3.0 and later fixed the bug
> silently (probably incidentally). Hadoop 2.x are still suffering from this
> issue.
> Thanks [~sodonnell] for locating the root cause in the codebase.
> Repro:
> 1. Set dfs.namenode.snapshot.capture.openfiles to true in hdfs-site.xml,
> start HDFS cluster
> 2. Create an empty directory /dataenc, create encryption zone and allow
> snapshot on it
> {code:bash}
> hadoop key create reprokey
> sudo -u hdfs hdfs dfs -mkdir /dataenc
> sudo -u hdfs hdfs crypto -createZone -keyName reprokey -path /dataenc
> sudo -u hdfs hdfs dfsadmin -allowSnapshot /dataenc
> {code}
> 3. Use a client that keeps a file open for write under /dataenc. For example,
> I'm using Flume HDFS sink to tail a local file.
> 4. Append the file several times using the client, keep the file open.
> 5. Create a snapshot
> {code:bash}
> sudo -u hdfs hdfs dfs -createSnapshot /dataenc snap1
> {code}
> 6. Append the file one or more times, but don't let the file size exceed the
> block size limit. Wait for several seconds for the append to be flushed to DN.
> 7. Do a -ls on the file inside the snapshot, then try to read the file using
> -get, you should see the actual file size read is larger than the listing
> size from -ls.
> The patch and an updated unit test will be uploaded later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14514) Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2
[
https://issues.apache.org/jira/browse/HDFS-14514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850047#comment-16850047
]
Stephen O'Donnell edited comment on HDFS-14514 at 5/28/19 7:09 PM:
---
I believe it only happens in snapshots, as that is the only place where the
file length recorded in the namenode matters. In the snapshot, the length
should be frozen, but outside of a snapshot, the reader will just read all
available data based on the visible length returned from the datanode.
I have not yet traced why this only impacts CryptoInputStream - the bug appears
to be in ByteBufferStrategy, which is a static class within DFSInputStream, so
it is probably related to the ReaderStrategy used by the calling class. I need
to look a bit further to see why non-encrypted data is not affected.
I agree that these test failures seem related. I will have a further look at
them too.
was (Author: sodonnell):
I believe it only happens in snapshots due to, as that is the only place where
the file length recorded in the namenode matters. In the snapshot, the length
should be frozen, but outside of a snapshot, the reader will just read all
available data based on the visible length returned from the datanode.
I have not yet traced why this only impacts CryptoInputStream - the bug appears
to be in ByteBufferStrategy, which is a static class within DFSInputStream, so
it is probably related to the ReaderStrategy used by the calling class. I need
to look a bit further to see why non-encrypted data is not affected.
I agree that these test failures seem related. I will have a further look at
them too.
> Actual read size of open file in encryption zone still larger than listing
> size even after enabling HDFS-11402 in Hadoop 2
> --
>
> Key: HDFS-14514
> URL: https://issues.apache.org/jira/browse/HDFS-14514
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs, snapshots
>Affects Versions: 2.6.5, 2.9.2, 2.8.5, 2.7.7
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14514.branch-2.001.patch
>
>
> In Hadoop 2, when a file is opened for write in *encryption zone*, taken a
> snapshot and appended, the read out file size in the snapshot is larger than
> the listing size. This happens even when immutable snapshot HDFS-11402 is
> enabled.
> Note: The refactor HDFS-8905 happened in Hadoop 3.0 and later fixed the bug
> silently (probably incidentally). Hadoop 2.x are still suffering from this
> issue.
> Thanks [~sodonnell] for locating the root cause in the codebase.
> Repro:
> 1. Set dfs.namenode.snapshot.capture.openfiles to true in hdfs-site.xml,
> start HDFS cluster
> 2. Create an empty directory /dataenc, create encryption zone and allow
> snapshot on it
> {code:bash}
> hadoop key create reprokey
> sudo -u hdfs hdfs dfs -mkdir /dataenc
> sudo -u hdfs hdfs crypto -createZone -keyName reprokey -path /dataenc
> sudo -u hdfs hdfs dfsadmin -allowSnapshot /dataenc
> {code}
> 3. Use a client that keeps a file open for write under /dataenc. For example,
> I'm using Flume HDFS sink to tail a local file.
> 4. Append the file several times using the client, keep the file open.
> 5. Create a snapshot
> {code:bash}
> sudo -u hdfs hdfs dfs -createSnapshot /dataenc snap1
> {code}
> 6. Append the file one or more times, but don't let the file size exceed the
> block size limit. Wait for several seconds for the append to be flushed to DN.
> 7. Do a -ls on the file inside the snapshot, then try to read the file using
> -get, you should see the actual file size read is larger than the listing
> size from -ls.
> The patch and an updated unit test will be uploaded later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14514) Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2
[
https://issues.apache.org/jira/browse/HDFS-14514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850047#comment-16850047
]
Stephen O'Donnell commented on HDFS-14514:
--
I believe it only happens in snapshots due to, as that is the only place where
the file length recorded in the namenode matters. In the snapshot, the length
should be frozen, but outside of a snapshot, the reader will just read all
available data based on the visible length returned from the datanode.
I have not yet traced why this only impacts CryptoInputStream - the bug appears
to be in ByteBufferStrategy, which is a static class within DFSInputStream, so
it is probably related to the ReaderStrategy used by the calling class. I need
to look a bit further to see why non-encrypted data is not affected.
I agree that these test failures seem related. I will have a further look at
them too.
> Actual read size of open file in encryption zone still larger than listing
> size even after enabling HDFS-11402 in Hadoop 2
> --
>
> Key: HDFS-14514
> URL: https://issues.apache.org/jira/browse/HDFS-14514
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs, snapshots
>Affects Versions: 2.6.5, 2.9.2, 2.8.5, 2.7.7
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14514.branch-2.001.patch
>
>
> In Hadoop 2, when a file is opened for write in *encryption zone*, taken a
> snapshot and appended, the read out file size in the snapshot is larger than
> the listing size. This happens even when immutable snapshot HDFS-11402 is
> enabled.
> Note: The refactor HDFS-8905 happened in Hadoop 3.0 and later fixed the bug
> silently (probably incidentally). Hadoop 2.x are still suffering from this
> issue.
> Thanks [~sodonnell] for locating the root cause in the codebase.
> Repro:
> 1. Set dfs.namenode.snapshot.capture.openfiles to true in hdfs-site.xml,
> start HDFS cluster
> 2. Create an empty directory /dataenc, create encryption zone and allow
> snapshot on it
> {code:bash}
> hadoop key create reprokey
> sudo -u hdfs hdfs dfs -mkdir /dataenc
> sudo -u hdfs hdfs crypto -createZone -keyName reprokey -path /dataenc
> sudo -u hdfs hdfs dfsadmin -allowSnapshot /dataenc
> {code}
> 3. Use a client that keeps a file open for write under /dataenc. For example,
> I'm using Flume HDFS sink to tail a local file.
> 4. Append the file several times using the client, keep the file open.
> 5. Create a snapshot
> {code:bash}
> sudo -u hdfs hdfs dfs -createSnapshot /dataenc snap1
> {code}
> 6. Append the file one or more times, but don't let the file size exceed the
> block size limit. Wait for several seconds for the append to be flushed to DN.
> 7. Do a -ls on the file inside the snapshot, then try to read the file using
> -get, you should see the actual file size read is larger than the listing
> size from -ls.
> The patch and an updated unit test will be uploaded later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14517) Display bug in permissions when ACL mask is defined
Istvan Fajth created HDFS-14517:
---
Summary: Display bug in permissions when ACL mask is defined
Key: HDFS-14517
URL: https://issues.apache.org/jira/browse/HDFS-14517
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Environment: Tested on latest CDH integration, and CDH5 as well with
the same result.
Reporter: Istvan Fajth
When ACLs are enabled on a folder, the following sequence of commands provide
the following result:
{{$ hdfs dfs -mkdir /tmp/acl
$ hdfs dfs -ls /tmp/acl
$ hdfs dfs -ls /tmp
Found 1 items
drwxr-xr-x - hdfs supergroup 0 2019-05-28 11:48 /tmp/acl
$ hdfs dfs -getfacl /tmp/acl
# file: /tmp/acl
# owner: hdfs
# group: supergroup
user::rwx
group::r-x
other::r-x
$ hdfs dfs -setfacl -m mask::rwx /tmp/acl
$ hdfs dfs -ls /tmp
Found 1 items
drwxrwxr-x+ - hdfs supergroup 0 2019-05-28 11:48 /tmp/acl
drwx-wx-wx - hive supergroup 0 2019-05-27 23:48 /tmp/hive
drwxrwxrwt - mapred hadoop 0 2019-05-28 01:32 /tmp/logs
$ hdfs dfs -setfacl -m mask::r-- /tmp/acl
$ hdfs dfs -ls /tmp
Found 1 items
drwxr--r-x+ - hdfs supergroup 0 2019-05-28 11:48 /tmp/acl
$ hdfs dfs -setfacl -m mask::r-x /tmp/acl
$ hdfs dfs -ls /tmp
Found 1 items
drwxr-xr-x+ - hdfs supergroup 0 2019-05-28 11:48 /tmp/acl
$ hdfs dfs -getfacl /tmp/acl
# file: /tmp/acl
# owner: hdfs
# group: supergroup
user::rwx
group::r-x
mask::r-x
other::r-x}}
So the group permission representation is changing with the defined mask ACL
instead of the group ACL or, maybe even better, the effective group ACL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=249579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249579 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 28/May/19 18:47 Start Date: 28/May/19 18:47 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #850: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuf… URL: https://github.com/apache/hadoop/pull/850#issuecomment-496641495 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 15 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 78 | Maven dependency ordering for branch | | +1 | mvninstall | 555 | trunk passed | | +1 | compile | 248 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 840 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 300 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 496 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 507 | the patch passed | | +1 | compile | 272 | the patch passed | | +1 | cc | 272 | the patch passed | | +1 | javac | 272 | the patch passed | | +1 | checkstyle | 71 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 660 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 77 | hadoop-ozone generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5) | | +1 | findbugs | 522 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 285 | hadoop-hdds in the patch passed. | | -1 | unit | 1649 | hadoop-ozone in the patch failed. | | -1 | asflicense | 49 | The patch generated 17 ASF License warnings. | | | | 6811 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.impl.TestContainerPersistence | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/850 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle cc | | uname | Linux 599bea7f4f9a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 06fa4d2 | | Default Java | 1.8.0_212 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4212 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pleas
[jira] [Created] (HDDS-1605) Implement AuditLogging for OM HA Bucket write requests
Bharat Viswanadham created HDDS-1605: Summary: Implement AuditLogging for OM HA Bucket write requests Key: HDDS-1605 URL: https://issues.apache.org/jira/browse/HDDS-1605 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham In this Jira, we shall implement audit logging for OM HA Bucket write requests. As now we cannot use userName, IpAddress from Server API's as these will be null, because the requests are executed under GRPC context. So, in our AuditLogger API's we need to pass username and remoteAddress which we can get from OMRequest after HDDS-1600 and use these during audit logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
