[jira] [Updated] (HDDS-1880) Decommissioining and maintenance mode in Ozone
[ https://issues.apache.org/jira/browse/HDDS-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-1880: Attachment: Decommission and Maintenance Overview.pdf > Decommissioining and maintenance mode in Ozone > --- > > Key: HDDS-1880 > URL: https://issues.apache.org/jira/browse/HDDS-1880 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Marton Elek >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Attachments: Decommission and Maintenance Overview.pdf > > > This is the umbrella jira for decommissioning support in Ozone. Design doc > will be attached soon. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4340) Add Operational State to the datanode list command
[
https://issues.apache.org/jira/browse/HDDS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-4340.
-
Fix Version/s: 1.1.0
Resolution: Fixed
> Add Operational State to the datanode list command
> --
>
> Key: HDDS-4340
> URL: https://issues.apache.org/jira/browse/HDDS-4340
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM Client
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> The existing CLI command `ozone admin datanode list` provides output like:
> {code}
> bash-4.2$ ozone admin datanode list
> Datanode: f2b2452a-bf7b-4c6d-b2d6-a0d9d219b21a
> (/default-rack/172.20.0.8/ozone_datanode_1.ozone_default/2 pipelines)
> Related pipelines:
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Leader
> 4e45ff9c-478b-4ab8-a66c-7bfa98c8c632/ONE/RATIS/OPEN/Leader
> Datanode: 57c7fd5f-e32c-4de9-a04a-89d8d4273431
> (/default-rack/172.20.0.6/ozone_datanode_3.ozone_default/2 pipelines)
> Related pipelines:
> 4b24bc61-28cf-471a-893c-a05cac273856/ONE/RATIS/OPEN/Leader
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
> Datanode: 6699fc6d-5c2d-4110-8d88-5ffa5b99f326
> (/default-rack/172.20.0.3/ozone_datanode_2.ozone_default/2 pipelines)
> Related pipelines:
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
> 5ce21cae-9a2d-486d-8b4b-f8ddf75efc61/ONE/RATIS/OPEN/Leader
> {code}
> We should extend this to show the "Operational State" of the node for
> decommission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4323) Add integration tests for putting nodes into maintenance and fix any issues uncovered in the tests
[ https://issues.apache.org/jira/browse/HDDS-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-4323. - Fix Version/s: 1.1.0 Resolution: Fixed > Add integration tests for putting nodes into maintenance and fix any issues > uncovered in the tests > -- > > Key: HDDS-4323 > URL: https://issues.apache.org/jira/browse/HDDS-4323 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 1.1.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Add a series of intergration tests to prove nodes can enter and leave > maintenance correctly and address any issues in the code when adding the tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4324) DatanodeAdminMonitor no longers needs maintenance end time to be passed
[
https://issues.apache.org/jira/browse/HDDS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-4324.
-
Fix Version/s: 1.1.0
Resolution: Fixed
> DatanodeAdminMonitor no longers needs maintenance end time to be passed
> ---
>
> Key: HDDS-4324
> URL: https://issues.apache.org/jira/browse/HDDS-4324
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> An earlier change moved the maintenance endtime into the NodeStatus object.
> However when adding a node to the decommission monitor the end time must
> still be passed. This value is never used.
> This Jira will remove the endInHours field from the interface:
> {code}
> public interface DatanodeAdminMonitor extends Runnable {
> void startMonitoring(DatanodeDetails dn, int endInHours);
> void stopMonitoring(DatanodeDetails dn);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4343) ReplicationManager.handleOverReplicatedContainer() does not handle unhealthyReplicas properly.
[
https://issues.apache.org/jira/browse/HDDS-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-4343.
-
Fix Version/s: 1.1.0
Resolution: Fixed
> ReplicationManager.handleOverReplicatedContainer() does not handle
> unhealthyReplicas properly.
> --
>
> Key: HDDS-4343
> URL: https://issues.apache.org/jira/browse/HDDS-4343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code:java}
> // If there are unhealthy replicas, then we should remove them even if
> it
> // makes the container violate the placement policy, as excess unhealthy
> // containers are not really useful. It will be corrected later as a
> // mis-replicated container will be seen as under-replicated.
> for (ContainerReplica r : unhealthyReplicas) {
> if (excess > 0) {
> sendDeleteCommand(container, r.getDatanodeDetails(), true);
> excess -= 1;
> }
> break;
> }
> // After removing all unhealthy replicas, if the container is still over
> // replicated then we need to check if it is already mis-replicated.
> // If it is, we do no harm by removing excess replicas. However, if it
> is
> // not mis-replicated, then we can only remove replicas if they don't
> // make the container become mis-replicated.
> {code}
> From the comment, it wants to remove all unhealthy replicas until excessÂ
> reach 0 ? It should be
> {code:java}
> for (ContainerReplica r : unhealthyReplicas) {
> if (excess > 0) {
> sendDeleteCommand(container, r.getDatanodeDetails(), true);
> excess -= 1;
> } else {
> break;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4340) Add Operational State to the datanode list command
Stephen O'Donnell created HDDS-4340:
---
Summary: Add Operational State to the datanode list command
Key: HDDS-4340
URL: https://issues.apache.org/jira/browse/HDDS-4340
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Components: SCM Client
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
The existing CLI command `ozone admin datanode list` provides output like:
{code}
bash-4.2$ ozone admin datanode list
Datanode: f2b2452a-bf7b-4c6d-b2d6-a0d9d219b21a
(/default-rack/172.20.0.8/ozone_datanode_1.ozone_default/2 pipelines)
Related pipelines:
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Leader
4e45ff9c-478b-4ab8-a66c-7bfa98c8c632/ONE/RATIS/OPEN/Leader
Datanode: 57c7fd5f-e32c-4de9-a04a-89d8d4273431
(/default-rack/172.20.0.6/ozone_datanode_3.ozone_default/2 pipelines)
Related pipelines:
4b24bc61-28cf-471a-893c-a05cac273856/ONE/RATIS/OPEN/Leader
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
Datanode: 6699fc6d-5c2d-4110-8d88-5ffa5b99f326
(/default-rack/172.20.0.3/ozone_datanode_2.ozone_default/2 pipelines)
Related pipelines:
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
5ce21cae-9a2d-486d-8b4b-f8ddf75efc61/ONE/RATIS/OPEN/Leader
{code}
We should extend this to show the "Operational State" of the node for
decommission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4336) ContainerInfo does not persist BCSID leading to failed replicas reports
[
https://issues.apache.org/jira/browse/HDDS-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-4336:
Fix Version/s: 1.1.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ContainerInfo does not persist BCSID leading to failed replicas reports
> ---
>
> Key: HDDS-4336
> URL: https://issues.apache.org/jira/browse/HDDS-4336
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> If you create a container, and then close it, the BCSID is synced on the
> datanodes and then the value is updated in SCM via setting the "sequenceID"
> field on the containerInfo object for the container.
> If you later restart just SCM, the sequenceID becomes zero, and then
> container reports for the replica fail with a stack trace like:
> {code}
> Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
> java.lang.AssertionError
> at
> org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
> at
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
> at
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The assertion here is failing, as it does not allow for the sequenceID to be
> changed on a CLOSED container:
> {code}
> public void updateSequenceId(long sequenceID) {
> assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
> sequenceId = max(sequenceID, sequenceId);
> }
> {code}
> The issue seems to be caused by the serialisation and deserialisation of the
> containerInfo object to protobuf, as sequenceId never persisted or restored.
> However, I am also confused about how this ever worked, as this is a pretty
> significant problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4336) ContainerInfo does not persist BCSID leading to failed replicas reports
[
https://issues.apache.org/jira/browse/HDDS-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-4336:
Description:
If you create a container, and then close it, the BCSID is synced on the
datanodes and then the value is updated in SCM via setting the "sequenceID"
field on the containerInfo object for the container.
If you later restart just SCM, the sequenceID becomes zero, and then container
reports for the replica fail with a stack trace like:
{code}
Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
java.lang.AssertionError
at
org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The assertion here is failing, as it does not allow for the sequenceID to be
changed on a CLOSED container:
{code}
public void updateSequenceId(long sequenceID) {
assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
sequenceId = max(sequenceID, sequenceId);
}
{code}
The issue seems to be caused by the serialisation and deserialisation of the
containerInfo object to protobuf, as sequenceId never persisted or restored.
However, I am also confused about how this ever worked, as this is a pretty
significant problem.
was:
If you create a container, and then close it, the BCSID is synced on the
datanodes and then the value is updated in SCM via setting the "sequenceID"
field on the containerInfo object for the container.
If you later restart just SCM, the sequenceID becomes zero, and then container
reports for the replica fail with a stack trace like:
{code}
Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
java.lang.AssertionError
at
org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The assertion here is what is failing, as it does not allow for the sequenceID
to be changed on a CLOSED container:
{code}
public void updateSequenceId(long sequenceID) {
assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
sequenceId = max(sequenceID, sequenceId);
}
{code}
The issue seems to be caused by the serialisation and deserialisation of the
containerInfo object to protobuf, as sequenceId never persisted or restored.
However, I am also confused about how this ever worked, as this is a pretty
significant problem.
> ContainerInfo does not persist BCSID leading to failed replicas reports
> ---
>
> Key: HDDS-4336
> URL: https://issues.apache.org/jira/browse/HDDS-4336
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> If you create a container, and
[jira] [Updated] (HDDS-4336) ContainerInfo does not persist BCSID leading to failed replicas reports
[
https://issues.apache.org/jira/browse/HDDS-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-4336:
Description:
If you create a container, and then close it, the BCSID is synced on the
datanodes and then the value is updated in SCM via setting the "sequenceID"
field on the containerInfo object for the container.
If you later restart just SCM, the sequenceID becomes zero, and then container
reports for the replica fail with a stack trace like:
{code}
Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
java.lang.AssertionError
at
org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The assertion here is what is failing, as it does not allow for the sequenceID
to be changed on a CLOSED container:
{code}
public void updateSequenceId(long sequenceID) {
assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
sequenceId = max(sequenceID, sequenceId);
}
{code}
The issue seems to be caused by the serialisation and deserialisation of the
containerInfo object to protobuf, as sequenceId never persisted or restored.
However, I am also confused about how this ever worked, as this is a pretty
significant problem.
was:
If you create a container, and then close it, the BCSID is synced on the
datanodes and then the value is updated in SCM via setting the "sequenceID"
field on the containerInfo object for the container.
If you later restart just SCM, the sequenceID becomes null, and then container
reports for the replica fail with a stack trace like:
{code}
Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
java.lang.AssertionError
at
org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The assertion here is what is failing, as it does not allow for the sequenceID
to be changed on a CLOSED container:
{code}
public void updateSequenceId(long sequenceID) {
assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
sequenceId = max(sequenceID, sequenceId);
}
{code}
The issue seems to be caused by the serialisation and deserialisation of the
containerInfo object to protobuf, as sequenceId never persisted or restored.
However, I am also confused about how this ever worked, as this is a pretty
significant problem.
> ContainerInfo does not persist BCSID leading to failed replicas reports
> ---
>
> Key: HDDS-4336
> URL: https://issues.apache.org/jira/browse/HDDS-4336
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> If you create a
[jira] [Created] (HDDS-4336) ContainerInfo does not persist BCSID leading to failed replicas reports
Stephen O'Donnell created HDDS-4336:
---
Summary: ContainerInfo does not persist BCSID leading to failed
replicas reports
Key: HDDS-4336
URL: https://issues.apache.org/jira/browse/HDDS-4336
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
If you create a container, and then close it, the BCSID is synced on the
datanodes and then the value is updated in SCM via setting the "sequenceID"
field on the containerInfo object for the container.
If you later restart just SCM, the sequenceID becomes null, and then container
reports for the replica fail with a stack trace like:
{code}
Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
java.lang.AssertionError
at
org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
at
org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
at
org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The assertion here is what is failing, as it does not allow for the sequenceID
to be changed on a CLOSED container:
{code}
public void updateSequenceId(long sequenceID) {
assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
sequenceId = max(sequenceID, sequenceId);
}
{code}
The issue seems to be caused by the serialisation and deserialisation of the
containerInfo object to protobuf, as sequenceId never persisted or restored.
However, I am also confused about how this ever worked, as this is a pretty
significant problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4324) DatanodeAdminMonitor no longers needs maintenance end time to be passed
Stephen O'Donnell created HDDS-4324:
---
Summary: DatanodeAdminMonitor no longers needs maintenance end
time to be passed
Key: HDDS-4324
URL: https://issues.apache.org/jira/browse/HDDS-4324
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Components: SCM
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
An earlier change moved the maintenance endtime into the NodeStatus object.
However when adding a node to the decommission monitor the end time must still
be passed. This value is never used.
This Jira will remove the endInHours field from the interface:
{code}
public interface DatanodeAdminMonitor extends Runnable {
void startMonitoring(DatanodeDetails dn, int endInHours);
void stopMonitoring(DatanodeDetails dn);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4323) Add integration tests for putting nodes into maintenance and fix any issues uncovered in the tests
Stephen O'Donnell created HDDS-4323: --- Summary: Add integration tests for putting nodes into maintenance and fix any issues uncovered in the tests Key: HDDS-4323 URL: https://issues.apache.org/jira/browse/HDDS-4323 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 1.1.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell Add a series of intergration tests to prove nodes can enter and leave maintenance correctly and address any issues in the code when adding the tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4322) Add integration tests for Decommission and resolve issues detected by the tests
Stephen O'Donnell created HDDS-4322: --- Summary: Add integration tests for Decommission and resolve issues detected by the tests Key: HDDS-4322 URL: https://issues.apache.org/jira/browse/HDDS-4322 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 1.1.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell Add a series of integration tests to prove decommission work, and that decommission can survive a restart of SCM. As part of adding these tests, some issues were discover that were fixed in the process of debugging the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3345) Investigate failure of TestDecommissionAndMaintenance integration test
[ https://issues.apache.org/jira/browse/HDDS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3345. - Resolution: Invalid The problem tests have been replaced with new ones, so this is no longer valid. > Investigate failure of TestDecommissionAndMaintenance integration test > -- > > Key: HDDS-3345 > URL: https://issues.apache.org/jira/browse/HDDS-3345 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 1.0.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > The test in the above class is failing and needs corrected. It is currently > ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4304) Close Container event can fail if pipeline is removed
Stephen O'Donnell created HDDS-4304:
---
Summary: Close Container event can fail if pipeline is removed
Key: HDDS-4304
URL: https://issues.apache.org/jira/browse/HDDS-4304
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
If you call `pipelineManager.finalizeAndDestroyPipeline()` with
onTimeout=false, then the finalizePipeline call will result in a closeContainer
event to be fired for every container on the pipeline. These are handled
asynchronously.
However, immediately after that, the `destroyPipeline(...)` call is made. This
will remove the pipeline details from the various maps / stores.
Then the closeContainer events get processed, and they attempt to remove the
container from the pipeline. However as the pipeline has already been
destroyed, this throws an exception and the close container events never get
sent to the DNs:
{code}
2020-10-01 15:44:18,838
[EventQueue-CloseContainerForCloseContainerEventHandler] INFO
container.CloseContainerEventHandler: Close container Event triggered for
container : #2
2020-10-01 15:44:18,842
[EventQueue-CloseContainerForCloseContainerEventHandler] ERROR
container.CloseContainerEventHandler: Failed to close the container #2.
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException:
PipelineID=59e5ae16-f1fe-45ff-9044-dd237b0e91c6 not found
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removeContainerFromPipeline(PipelineStateMap.java:372)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removeContainerFromPipeline(PipelineStateManager.java:111)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removeContainerFromPipeline(SCMPipelineManager.java:413)
at
org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:352)
at
org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerState(SCMContainerManager.java:331)
at
org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler.onMessage(CloseContainerEventHandler.java:66)
at
org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler.Onmessage(CloseContainerEventHandler.java:45)
at
org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor
{code}
The simple solution is to catch the exception and ignore it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4300) Remove no longer needed class DatanodeAdminNodeDetails
Stephen O'Donnell created HDDS-4300: --- Summary: Remove no longer needed class DatanodeAdminNodeDetails Key: HDDS-4300 URL: https://issues.apache.org/jira/browse/HDDS-4300 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 1.1.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell DatanodeAdminNodeDetails was added earlier in the decommission branch, to track metrics and, the decommission state and maintenance end time. After enhancing NodeStatus to old the Maintenance Expiry time, this class is no longer needed and it also duplicates information which is stored in other existing places. This change removes it and then metrics etc can be added later in a different way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3630) Merge rocksdb in datanode
[
https://issues.apache.org/jira/browse/HDDS-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196076#comment-17196076
]
Stephen O'Donnell commented on HDDS-3630:
-
Have a look at HDDS-4246 - it seems there is only one 8MB cache shared by all
RocksDBs related to datanode containers.
Looking at the rocksDB manual, one key memory user is the "write buffer size"
https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#write-buffer-size;
{quote}
It represents the amount of data to build up in memory (backed by an unsorted
log on disk) before converting to a sorted on-disk file. The default is 64 MB.
You need to budget for 2 x your worst case memory use. If you don't have enough
memory for this, you should reduce this value. Otherwise, it is not recommended
to change this option.
{quote}
It seems to be, this default of 64MB is setup for "high write throughput",
which is probably a usual use case for RocksDB. However for datanode
containers, I doubt rocksDB is really stressed, especially for closed
containers. What if we:
1. Reduced this value significantly - eg to 1MB?
2. Reduced it significantly for only closed containers?
There are also some other interesting Rocks DB options. You can configure a
"Write Buffer Manager" and give it a target size for all RocksDB instances /
column families related to write buffers, and then all open instances will
share this. You can also make it be part of the LRU cache:
https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager
And you can have the Index and Filter blocks cached in the LRU cache too via
the option - cache_index_and_filter_blocks.
Therefore, if we created a large shared LRU cache, use a shared Write Buffer
Manager which stored the memtables inside this LRU cache, and also cache the
Index and Filter block there - perhaps we could constrain the rocksDB memory
within reasonable bounds.
It would be good to experiment with some of these options before jumping into a
major refactor to use a single RocksDB per disk or other major changes.
> Merge rocksdb in datanode
> -
>
> Key: HDDS-3630
> URL: https://issues.apache.org/jira/browse/HDDS-3630
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: runzhiwang
>Assignee: runzhiwang
>Priority: Major
> Attachments: Merge RocksDB in Datanode-v1.pdf, Merge RocksDB in
> Datanode-v2.pdf
>
>
> Currently, one rocksdb for one container. one container has 5GB capacity.
> 10TB data need more than 2000 rocksdb in one datanode. It's difficult to
> limit the memory of 2000 rocksdb. So maybe we should limited instance of
> rocksdb for each disk.
> The design of improvement is in the follow link, but still is a draft.
> TODO:
> 1. compatibility with current logic i.e. one rocksdb for each container
> 2. measure the memory usage before and after improvement
> 3. effect on efficiency of read and write.
> https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4246) Consider increasing shared RocksDB LRU cache size on datanodes
Stephen O'Donnell created HDDS-4246:
---
Summary: Consider increasing shared RocksDB LRU cache size on
datanodes
Key: HDDS-4246
URL: https://issues.apache.org/jira/browse/HDDS-4246
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: Ozone Datanode
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
By default when a rocksDB instance is opened, a 8MB LRU cache is associated
with the instance. From the rocksDB manual, many instances in the same process
can share the same LRU cache:
https://github.com/facebook/rocksdb/wiki/Block-Cache
{quote}
A Cache object can be shared by multiple RocksDB instances in the same process,
allowing users to control the overall cache capacity.
{quote}
This is of particular interest on the datanodes, where there are potentially
thousands of small rocksDB instances.
This RocksDB PR, added a feature to the Java implementation, allowing a LRU
cache to be explicitly created and passed to different "Options" objects to
ensure the same cache is reused:
{code}
Cache cache = new LRUCache(64 * SizeUnit.MB);
BlockBasedTableConfig table_options = new BlockBasedTableConfig();
table_options.setBlockCache(cache);
Options options = new Options();
options.setCreateIfMissing(true)
.setStatistics(stats)
.setTableFormatConfig(table_options);
...
{code}
Before this feature, the way to reuse a cache across many DB instances is to
pass the exact same RocksDB Options object when creating the RocksDB instance.
This means that a possible unintended side effect of HDDS-2283 (which caches
the RocksDB options, and re-uses them across all DB containers) is that there
is now only 1 8MB RocksDB cache across all the container RocksDBs on the
datanode.
You can see this is the case, by grepping the rocksDB LOG file. Eg, with Option
caching, in two containers:
{code}
bash-4.2$ grep -A5 "block_cache:"
./hdds/hdds/2ad8eea5-b9e1-41e1-85eb-8cae745efcb6/current/containerDir0/2/metadata/2-dn-container.db/LOG
no_block_cache: 0
block_cache: 0x563ba9088bb0<=
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
bash-4.2$ grep -A5 "block_cache:"
./hdds/hdds/2ad8eea5-b9e1-41e1-85eb-8cae745efcb6/current/containerDir0/3/metadata/3-dn-container.db/LOG
no_block_cache: 0
block_cache: 0x563ba9088bb0 <=
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
{code}
Note the block cache in both containers shares the same address
"0x563ba9088bb0".
Reverting the caching change, so that a new Options object is passed into the
RocksDB instance, we can see the cache address is different:
{code}
bash-4.2$ grep -A5 "block_cache:"
./hdds/hdds/ac115132-9693-4ab9-9d73-dd4bf7e40caf/current/containerDir0/4/metadata/4-dn-container.db/LOG
no_block_cache: 0
block_cache: 0x7feec0b86270 <=
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
bash-4.2$ grep -A5 "block_cache:"
./hdds/hdds/ac115132-9693-4ab9-9d73-dd4bf7e40caf/current/containerDir0/1/metadata/1-dn-container.db/LOG
no_block_cache: 0
block_cache: 0x565360926f70 <=
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
{code}
>From this, it is very likely a single 8MB cache for all containers on a large
>node is not sufficient. We should consider if it makes sense to set a larger
>shared cache size on the DN, or have several shared caches.
Note that I have not seen any performance issues caused by this, but I came
across this when investigating RocksDB in general.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4213) Log when a datanode has become dead in the DeadNodeHandler
[ https://issues.apache.org/jira/browse/HDDS-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-4213: Summary: Log when a datanode has become dead in the DeadNodeHandler (was: Print the dead datanode when DEAD message comming) > Log when a datanode has become dead in the DeadNodeHandler > -- > > Key: HDDS-4213 > URL: https://issues.apache.org/jira/browse/HDDS-4213 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4213) Log when a datanode has become dead in the DeadNodeHandler
[ https://issues.apache.org/jira/browse/HDDS-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-4213. - Fix Version/s: 1.1.0 Resolution: Fixed > Log when a datanode has become dead in the DeadNodeHandler > -- > > Key: HDDS-4213 > URL: https://issues.apache.org/jira/browse/HDDS-4213 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4131) Container report should update container key count and bytes used if they differ in SCM
[
https://issues.apache.org/jira/browse/HDDS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-4131.
-
Fix Version/s: 1.1.0
Resolution: Fixed
> Container report should update container key count and bytes used if they
> differ in SCM
> ---
>
> Key: HDDS-4131
> URL: https://issues.apache.org/jira/browse/HDDS-4131
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> In HDDS-4037 it was noted that when blocks are deleted from closed
> containers, the bytesUsed and Key Count metrics on the SCM container are not
> updated correctly.
> These stats should be updated via the container reports issued by the DNs to
> SCM periodically. However, in
> `AbstractContainerReportHandler#updateContainerStats`, the code assumes the
> values are always increasing and it will not update them if they are
> decreasing:
> {code}
> private void updateContainerStats(final ContainerID containerId,
> final ContainerReplicaProto replicaProto)
> throws ContainerNotFoundException {
> if (isHealthy(replicaProto::getState)) {
> final ContainerInfo containerInfo = containerManager
> .getContainer(containerId);
> if (containerInfo.getSequenceId() <
> replicaProto.getBlockCommitSequenceId()) {
> containerInfo.updateSequenceId(
> replicaProto.getBlockCommitSequenceId());
> }
> if (containerInfo.getUsedBytes() < replicaProto.getUsed()) {
> containerInfo.setUsedBytes(replicaProto.getUsed());
> }
> if (containerInfo.getNumberOfKeys() < replicaProto.getKeyCount()) {
> containerInfo.setNumberOfKeys(replicaProto.getKeyCount());
> }
> }
> }
> {code}
> In HDDS-4037 a change was made to the Replication Manager, so it updates the
> stats. However I don't believe that is the correct place to perform this
> check, and the issue is caused by the logic shared above.
> In this Jira, I have removed the changes to Replication Manager in HDDS-4037
> (but retained the other changes), ensuring the problem statistics are only
> updated via the containers reports if they are different in SCM from what is
> reported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4131) Container report should update container key count and bytes used if they differ in SCM
Stephen O'Donnell created HDDS-4131:
---
Summary: Container report should update container key count and
bytes used if they differ in SCM
Key: HDDS-4131
URL: https://issues.apache.org/jira/browse/HDDS-4131
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: SCM
Affects Versions: 0.7.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
In HDDS-4037 it was noted that when blocks are deleted from closed containers,
the bytesUsed and Key Count metrics on the SCM container are not updated
correctly.
These stats should be updated via the container reports issued by the DNs to
SCM periodically. However, in
`AbstractContainerReportHandler#updateContainerStats`, the code assumes the
values are always increasing and it will not update them if they are decreasing:
{code}
private void updateContainerStats(final ContainerID containerId,
final ContainerReplicaProto replicaProto)
throws ContainerNotFoundException {
if (isHealthy(replicaProto::getState)) {
final ContainerInfo containerInfo = containerManager
.getContainer(containerId);
if (containerInfo.getSequenceId() <
replicaProto.getBlockCommitSequenceId()) {
containerInfo.updateSequenceId(
replicaProto.getBlockCommitSequenceId());
}
if (containerInfo.getUsedBytes() < replicaProto.getUsed()) {
containerInfo.setUsedBytes(replicaProto.getUsed());
}
if (containerInfo.getNumberOfKeys() < replicaProto.getKeyCount()) {
containerInfo.setNumberOfKeys(replicaProto.getKeyCount());
}
}
}
{code}
In HDDS-4037 a change was made to the Replication Manager, so it updates the
stats. However I don't believe that is the correct place to perform this check,
and the issue is caused by the logic shared above.
In this Jira, I have removed the changes to Replication Manager in HDDS-4037
(but retained the other changes), ensuring the problem statistics are only
updated via the containers reports if they are different in SCM from what is
reported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4065) Regularly close and open new pipelines
[ https://issues.apache.org/jira/browse/HDDS-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175439#comment-17175439 ] Stephen O'Donnell commented on HDDS-4065: - Attached a draft doc outlining some ideas and pros / cons of this idea. > Regularly close and open new pipelines > -- > > Key: HDDS-4065 > URL: https://issues.apache.org/jira/browse/HDDS-4065 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Attachments: Regularly_Closing_Pipelines.001.pdf > > > There are scenarios where non-rack-aware pipelines can be created on a > cluster and when that happens they should be closed and replaced with new > pipelines. > There is also a desire to regularly close piplelines and open new ones, to > provide better shuffling of data across the nodes. > This Jira will discuss ways to solve both of these problems. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4065) Regularly close and open new pipelines
[ https://issues.apache.org/jira/browse/HDDS-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-4065: Attachment: Regularly_Closing_Pipelines.001.pdf > Regularly close and open new pipelines > -- > > Key: HDDS-4065 > URL: https://issues.apache.org/jira/browse/HDDS-4065 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Attachments: Regularly_Closing_Pipelines.001.pdf > > > There are scenarios where non-rack-aware pipelines can be created on a > cluster and when that happens they should be closed and replaced with new > pipelines. > There is also a desire to regularly close piplelines and open new ones, to > provide better shuffling of data across the nodes. > This Jira will discuss ways to solve both of these problems. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4065) Regularly close and open new pipelines
Stephen O'Donnell created HDDS-4065: --- Summary: Regularly close and open new pipelines Key: HDDS-4065 URL: https://issues.apache.org/jira/browse/HDDS-4065 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell There are scenarios where non-rack-aware pipelines can be created on a cluster and when that happens they should be closed and replaced with new pipelines. There is also a desire to regularly close piplelines and open new ones, to provide better shuffling of data across the nodes. This Jira will discuss ways to solve both of these problems. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4062) Non rack aware pipelines should not be created if multiple racks are alive
Stephen O'Donnell created HDDS-4062: --- Summary: Non rack aware pipelines should not be created if multiple racks are alive Key: HDDS-4062 URL: https://issues.apache.org/jira/browse/HDDS-4062 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Affects Versions: 0.7.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell If we have a scenario where one rack has more nodes that others, it is possible for all hosts in the cluster to have reached their pipeline limit, while 3 nodes on the larger rack have not. The current fallback logic will then allow a pipeline to be created which uses only the 3 nodes on the same rack, violating the rack placement policy. There may be other ways this could happen with cluster load too, were the pipeline capacity has reached its limit on some nodes but not others. The proposal here, is that if the cluster has multiple racks AND there are healthy nodes covering at least 2 racks, where healthy is defined as a node which is registered and not stale or dead, then we should not allow "fallback" (pipelines which span only 1 rack) pipelines to be created. This means if you have a badly configured cluster - eg Rack 1 = 10 nodes; Rack 2 = 1 node, the pipeline limit will be constrained by the capacity of that 1 node on rack 2. Even a setup like Rack 1 = 10 nodes, Rack 2 = 5 would be constrained by this. This constraint is better than creating non rack aware pipelines, and the rule above will handle the case when the cluster degrades to 1 rack, as the healthy node definition will notice only 1 rack is alive. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-698) Support Topology Awareness for Ozone
[ https://issues.apache.org/jira/browse/HDDS-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159256#comment-17159256 ] Stephen O'Donnell commented on HDDS-698: [~Sammi] I think we can close this overall Jira now. Any further topology issues can be handled as normal Jiras without this umbrella Jira. > Support Topology Awareness for Ozone > > > Key: HDDS-698 > URL: https://issues.apache.org/jira/browse/HDDS-698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Blocker > Fix For: 0.6.0 > > Attachments: HDDS-698.000.patch, network-topology-default.xml, > network-topology-nodegroup.xml > > > This is an umbrella JIRA to add topology aware support for Ozone Pipelines, > Containers and Blocks. Long time since HDFS is created, we provide > rack/nodegroup awareness for reliability and high performance for data > access. Ozone need a similar mechanism and this can be more flexible for > cloud scenarios. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[ https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159255#comment-17159255 ] Stephen O'Donnell commented on HDDS-1930: - I believe this is not an issue, or no longer an issue if it previously was an issue. In the counters I posted above, we can see there are data-local and rack-local tasks being scheduled. I also enabled debug logging for the application master and you could see if making container allocation decisions based on the locality information. Therefore I am going to close this issue. > Test Topology Aware Job scheduling with Ozone Topology > -- > > Key: HDDS-1930 > URL: https://issues.apache.org/jira/browse/HDDS-1930 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 0.6.0 > > > My initial results with Terasort does not seem to report the counter > properly. Most of the requests are handled by rack local but no node local. > This ticket is opened to add more system testing to validate the feature. > Total Allocated Containers: 3778 > Each table cell represents the number of NodeLocal/RackLocal/OffSwitch > containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. > Node Local RequestRack Local Request Off Switch Request > Num Node Local Containers (satisfied by) 0 > Num Rack Local Containers (satisfied by) 0 3648 > Num Off Switch Containers (satisfied by) 0 96 34 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[ https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-1930. - Resolution: Not A Problem > Test Topology Aware Job scheduling with Ozone Topology > -- > > Key: HDDS-1930 > URL: https://issues.apache.org/jira/browse/HDDS-1930 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 0.6.0 > > > My initial results with Terasort does not seem to report the counter > properly. Most of the requests are handled by rack local but no node local. > This ticket is opened to add more system testing to validate the feature. > Total Allocated Containers: 3778 > Each table cell represents the number of NodeLocal/RackLocal/OffSwitch > containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. > Node Local RequestRack Local Request Off Switch Request > Num Node Local Containers (satisfied by) 0 > Num Rack Local Containers (satisfied by) 0 3648 > Num Off Switch Containers (satisfied by) 0 96 34 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3830) Introduce OM layout version 'v0'.
[ https://issues.apache.org/jira/browse/HDDS-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3830. - Resolution: Duplicate > Introduce OM layout version 'v0'. > - > > Key: HDDS-3830 > URL: https://issues.apache.org/jira/browse/HDDS-3830 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: upgrade-p0 > > The first layout version for OzoneManager will be '0' which will be written > to the version file. Until a future Ozone release with Upgrade & Finalize > support, this will just be a dummy number, to support backward compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3879) Introduce SCM layout version 'v0'.
[ https://issues.apache.org/jira/browse/HDDS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3879. - Fix Version/s: 0.6.0 Resolution: Fixed > Introduce SCM layout version 'v0'. > -- > > Key: HDDS-3879 > URL: https://issues.apache.org/jira/browse/HDDS-3879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available, upgrade-p0 > Fix For: 0.6.0 > > > The first layout version for SCM will be '0' which will be written to the > version file. Until a future Ozone release with Upgrade & Finalize support, > this will just be a dummy number, to support backward compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3830) Introduce OM layout version 'v0'.
[ https://issues.apache.org/jira/browse/HDDS-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154408#comment-17154408 ] Stephen O'Donnell commented on HDDS-3830: - This is covered by the change committed in HDDS-3879, so resolving this one. > Introduce OM layout version 'v0'. > - > > Key: HDDS-3830 > URL: https://issues.apache.org/jira/browse/HDDS-3830 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: upgrade-p0 > > The first layout version for OzoneManager will be '0' which will be written > to the version file. Until a future Ozone release with Upgrade & Finalize > support, this will just be a dummy number, to support backward compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[
https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153798#comment-17153798
]
Stephen O'Donnell commented on HDDS-1930:
-
On a 8 datanode cluster (not ideal for Ozone as it should be a multiple of 3
DNs), I ran teragen and then terasort:
{code}
ozone sh volume create o3://ozone1/teragen
ozone sh bucket create o3://ozone1/teragen/bucket
yarn jar
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
teragen -Dmapreduce.job.maps=8 10 o3fs://bucket.teragen.ozone1/test3
yarn jar
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
terasort -Dmapreduce.job.maps=30 o3fs://bucket.teragen.ozone1/test3
o3fs://bucket.teragen.ozone1/sort-test3
{code}
The job counts at the end said:
{code}
Killed map tasks=244
Launched map tasks=620
Launched reduce tasks=1
Other local map tasks=244
Data-local map tasks=321
Rack-local map tasks=55
{code}
So it executed 620 -244 = 376 containers successfully, where 321 were
"Data-Local" and the remaining 55 were "Rack-Local". I would like to do a few
more tests to confirm, but from that, it does seem like the locality is making
it through to YARN and it is picking local nodes for the data or at least the
same rack.
> Test Topology Aware Job scheduling with Ozone Topology
> --
>
> Key: HDDS-1930
> URL: https://issues.apache.org/jira/browse/HDDS-1930
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 0.6.0
>
>
> My initial results with Terasort does not seem to report the counter
> properly. Most of the requests are handled by rack local but no node local.
> This ticket is opened to add more system testing to validate the feature.
> Total Allocated Containers: 3778
> Each table cell represents the number of NodeLocal/RackLocal/OffSwitch
> containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests.
> Node Local RequestRack Local Request Off Switch Request
> Num Node Local Containers (satisfied by) 0
> Num Rack Local Containers (satisfied by) 0 3648
> Num Off Switch Containers (satisfied by) 0 96 34
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3907) Topology related acceptance test is flaky
[
https://issues.apache.org/jira/browse/HDDS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149339#comment-17149339
]
Stephen O'Donnell commented on HDDS-3907:
-
[~elek] Has the test been stable until recently? The last time it was unstable,
it was disk space related issues, but that is not the case this time.
> Topology related acceptance test is flaky
> -
>
> Key: HDDS-3907
> URL: https://issues.apache.org/jira/browse/HDDS-3907
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Blocker
>
> Examples:
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1318/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1321/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1334/acceptance
> Some strange errors:
> {code}
> scm_1 | 2020-06-30 19:17:50,787 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor ONE. Exception: Cannot create pipeline of factor 1 using 0 nodes. Used
> 6 nodes. Healthy nodes 6
> scm_1 | 2020-06-30 19:17:50,788 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor THREE. Exception: Pipeline creation failed because nodes are engaged
> in other pipelines and every node can only be engaged in max 2 pipelines.
> Required 3. Found 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3907) Topology related acceptance test is flaky
[
https://issues.apache.org/jira/browse/HDDS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149331#comment-17149331
]
Stephen O'Donnell commented on HDDS-3907:
-
I have seen those messages bofore. They simply mean the pipeline manager has
used all available nodes, and it cannot create any more pipelines. They are
quite mis-leading and could be improved I think, as they make it sound like
there is an error, where its a normal end state of the pipeline manager each
time it runs:
{code}
scm_1 | 2020-06-30 19:17:50,787 [RatisPipelineUtilsThread] ERROR
pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and factor
ONE. Exception: Cannot create pipeline of factor 1 using 0 nodes. Used 6 nodes.
Healthy nodes 6
scm_1 | 2020-06-30 19:17:50,788 [RatisPipelineUtilsThread] ERROR
pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and factor
THREE. Exception: Pipeline creation failed because nodes are engaged in other
pipelines and every node can only be engaged in max 2 pipelines. Required 3.
Found 0
{code}
Looking at
https://github.com/elek/ozone-build-results/blob/master/2020/06/30/1334/acceptance/docker-ozone-topology-ozone-topology-basic-scm.log
I am not too sure what is going wrong.
This is the SCM trace, and we can see it creates 2 REP3 pipelines as the nodes
check in:
{code}
scm_1 | 2020-06-30 12:26:13,328
[EventQueue-NodeRegistrationContainerReportForDataNodeSafeModeRule] INFO
safemode.SCMSafeModeManager: SCM in safe mode. 4 DataNodes registered, 4
required.
scm_1 | 2020-06-30 12:26:13,343
[EventQueue-NodeRegistrationContainerReportForDataNodeSafeModeRule] INFO
safemode.SCMSafeModeManager: DataNodeSafeModeRule rule is successfully validated
...
scm_1 | 2020-06-30 12:26:13,402 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=997168be-72df-4784-bd0d-a2d077235929 to
datanode:af137417-7801-4a96-9b21-1b6e75d3578d
scm_1 | 2020-06-30 12:26:13,402 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=997168be-72df-4784-bd0d-a2d077235929 to
datanode:5d636ad6-6c45-4d25-921b-ebdea2d0
scm_1 | 2020-06-30 12:26:13,403 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=997168be-72df-4784-bd0d-a2d077235929 to
datanode:ae41103f-5e55-496f-aac5-65fab8cc3ad2
scm_1 | 2020-06-30 12:26:13,404 [RatisPipelineUtilsThread] INFO
pipeline.PipelineStateManager: Created pipeline Pipeline[ Id:
997168be-72df-4784-bd0d-a2d077235929, Nodes:
af137417-7801-4a96-9b21-1b6e75d3578d{ip: 10.5.0.9, host:
ozone-topology_datanode_6_1.ozone-topology_net, networkLocation: /rack2,
certSerialId: null}5d636ad6-6c45-4d25-921b-ebdea2d0{ip: 10.5.0.4, host:
ozone-topology_datanode_1_1.ozone-topology_net, networkLocation: /rack1,
certSerialId: null}ae41103f-5e55-496f-aac5-65fab8cc3ad2{ip: 10.5.0.7, host:
ozone-topology_datanode_4_1.ozone-topology_net, networkLocation: /rack2,
certSerialId: null}, Type:RATIS, Factor:THREE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-06-30T12:26:13.402419Z]
...
scm_1 | 2020-06-30 12:26:14,873 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=8f9e6e0b-51d1-41f9-968d-417987bd28ff to
datanode:c7c89d62-1bd1-4840-8dc0-c863fd86ce3b
scm_1 | 2020-06-30 12:26:14,878 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=8f9e6e0b-51d1-41f9-968d-417987bd28ff to
datanode:93760607-1873-432f-a86f-159b2f1f3a3a
scm_1 | 2020-06-30 12:26:14,881 [RatisPipelineUtilsThread] INFO
pipeline.RatisPipelineProvider: Sending CreatePipelineCommand for
pipeline:PipelineID=8f9e6e0b-51d1-41f9-968d-417987bd28ff to
datanode:e2097891-5bc2-4f96-b1f2-fbe3cb4beb1f
scm_1 | 2020-06-30 12:26:14,882 [RatisPipelineUtilsThread] INFO
pipeline.PipelineStateManager: Created pipeline Pipeline[ Id:
8f9e6e0b-51d1-41f9-968d-417987bd28ff, Nodes:
c7c89d62-1bd1-4840-8dc0-c863fd86ce3b{ip: 10.5.0.5, host:
ozone-topology_datanode_2_1.ozone-topology_net, networkLocation: /rack1,
certSerialId: null}93760607-1873-432f-a86f-159b2f1f3a3a{ip: 10.5.0.8, host:
ozone-topology_datanode_5_1.ozone-topology_net, networkLocation: /rack2,
certSerialId: null}e2097891-5bc2-4f96-b1f2-fbe3cb4beb1f{ip: 10.5.0.6, host:
ozone-topology_datanode_3_1.ozone-topology_net, networkLocation: /rack1,
certSerialId: null}, Type:RATIS, Factor:THREE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-06-30T12:26:14.873487Z]
{code}
So, SCM waited until 4 nodes had checked in, then create a pipeline spanning 2
racks. Then when the remaining DNs checked in, it created another REP3
pipeline. All looks good here.
[jira] [Assigned] (HDDS-3831) Enhance Recon ContainerEndPoint to report on difference unhealty container states
[ https://issues.apache.org/jira/browse/HDDS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reassigned HDDS-3831: --- Assignee: Stephen O'Donnell > Enhance Recon ContainerEndPoint to report on difference unhealty container > states > - > > Key: HDDS-3831 > URL: https://issues.apache.org/jira/browse/HDDS-3831 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Recon >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > HDDS-3082 changed the unhealthyContainers job to store the counts of under > replicated, over replicated, missing and mis-replicated containers. > We should enhance ContainerEndPoint.java to provide API access to those > states and perhaps a summary status too, eg: > under-replicated-count: 10 > over-replication-count: 0 > mis-replicated-count: 5 > ... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3854) Fix error return value of KeyValueBlockIterator#hasNext
[ https://issues.apache.org/jira/browse/HDDS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3854. - Resolution: Fixed > Fix error return value of KeyValueBlockIterator#hasNext > --- > > Key: HDDS-3854 > URL: https://issues.apache.org/jira/browse/HDDS-3854 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3854) Fix error return value of KeyValueBlockIterator#hasNext
[ https://issues.apache.org/jira/browse/HDDS-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3854: Fix Version/s: 0.6.0 > Fix error return value of KeyValueBlockIterator#hasNext > --- > > Key: HDDS-3854 > URL: https://issues.apache.org/jira/browse/HDDS-3854 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3831) Enhance Recon ContainerEndPoint to report on difference unhealty container states
[
https://issues.apache.org/jira/browse/HDDS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142014#comment-17142014
]
Stephen O'Donnell edited comment on HDDS-3831 at 6/22/20, 12:51 PM:
Currently the container endpoint has:
{code}
getContainers - list all containers in batches (/containers)
getKeys - List all keys in a container (/containers/{id}/keys)
getReplicaHistory - History of locations (/containers/{id}/replicaHistory)
getMissing - List of missing containers (/containers/missing)
{code}
Now that we have different types of unhealthy containers it seems we need some
more flexibility around reporting on them, but it also depends on what we want
to display in the recon API.
It feels like we should have a summary report for unhealthy containers, eg:
{code}
missing: x
mis-replicated: x
under-replicated: x
over-replicated: x
{code}
Or perhaps an overall report (this could be part of the getContainers or
another new API):
{code}
totalContainers: xxx
healthy: xxx
missing: xxx
mis-replicated: xxx
under-replicated: xxx
over-replicated: xxx
{code}
We also probably want to be able to list the unhealthy containers in a given
state, and it probably should be a batch API to limit the number returned if
there are a lot of them.
I wonder if we should have a new endpoint which is:
{code}
/containers/unhealthy/{state}
{code}
By default it will return a list of all unhealthy containers
(/containers/unhealthy), but this can be filtered by adding MISSING /
UNDER-REPLICATED, OVER-REPLICATED, MIS-REPLICATED.
Then we can remove the existing "missing" endpoint.
[~avijayan] [~vivekratnavel] - what do you guys think, keeping what we will
want to display on Recon in mind?
was (Author: sodonnell):
Currently the container endpoint has:
getContainers - list all containers in batches (/containers)
getKeys - List all keys in a container (/containers/{id}/keys)
getReplicaHistory - History of locations (/containers/{id}/replicaHistory)
getMissing - List of missing containers (/containers/missing)
Now that we have different types of unhealthy containers it seems we need some
more flexibility around reporting on them, but it also depends on what we want
to display in the recon API.
It feels like we should have a summary report for unhealthy containers, eg:
{code}
missing: x
mis-replicated: x
under-replicated: x
over-replicated: x
{code}
Or perhaps an overall report (this could be part of the getContainers or
another new API):
{code}
totalContainers: xxx
healthy: xxx
missing: xxx
mis-replicated: xxx
under-replicated: xxx
over-replicated: xxx
{code}
We also probably want to be able to list the unhealthy containers in a given
state, and it probably should be a batch API to limit the number returned if
there are a lot of them.
I wonder if we should have a new endpoint which is:
{code}
/containers/unhealthy/{state}
{code}
By default it will return a list of all unhealthy containers
(/containers/unhealthy), but this can be filtered by adding MISSING /
UNDER-REPLICATED, OVER-REPLICATED, MIS-REPLICATED.
Then we can remove the existing "missing" endpoint.
[~avijayan] [~vivekratnavel] - what do you guys think, keeping what we will
want to display on Recon in mind?
> Enhance Recon ContainerEndPoint to report on difference unhealty container
> states
> -
>
> Key: HDDS-3831
> URL: https://issues.apache.org/jira/browse/HDDS-3831
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Recon
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Priority: Major
>
> HDDS-3082 changed the unhealthyContainers job to store the counts of under
> replicated, over replicated, missing and mis-replicated containers.
> We should enhance ContainerEndPoint.java to provide API access to those
> states and perhaps a summary status too, eg:
> under-replicated-count: 10
> over-replication-count: 0
> mis-replicated-count: 5
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3831) Enhance Recon ContainerEndPoint to report on difference unhealty container states
[
https://issues.apache.org/jira/browse/HDDS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142014#comment-17142014
]
Stephen O'Donnell edited comment on HDDS-3831 at 6/22/20, 12:51 PM:
Currently the container endpoint has:
getContainers - list all containers in batches (/containers)
getKeys - List all keys in a container (/containers/{id}/keys)
getReplicaHistory - History of locations (/containers/{id}/replicaHistory)
getMissing - List of missing containers (/containers/missing)
Now that we have different types of unhealthy containers it seems we need some
more flexibility around reporting on them, but it also depends on what we want
to display in the recon API.
It feels like we should have a summary report for unhealthy containers, eg:
{code}
missing: x
mis-replicated: x
under-replicated: x
over-replicated: x
{code}
Or perhaps an overall report (this could be part of the getContainers or
another new API):
{code}
totalContainers: xxx
healthy: xxx
missing: xxx
mis-replicated: xxx
under-replicated: xxx
over-replicated: xxx
{code}
We also probably want to be able to list the unhealthy containers in a given
state, and it probably should be a batch API to limit the number returned if
there are a lot of them.
I wonder if we should have a new endpoint which is:
{code}
/containers/unhealthy/{state}
{code}
By default it will return a list of all unhealthy containers
(/containers/unhealthy), but this can be filtered by adding MISSING /
UNDER-REPLICATED, OVER-REPLICATED, MIS-REPLICATED.
Then we can remove the existing "missing" endpoint.
[~avijayan] [~vivekratnavel] - what do you guys think, keeping what we will
want to display on Recon in mind?
was (Author: sodonnell):
Currently the container endpoint has:
getContainers - list all containers in batches (/containers)
getKeys - List all keys in a container (/containers/{id}/keys)
getReplicaHistory - History of locations (/containers/{id}/replicaHistory)
getMissing - List of missing containers (/containers/missing)
Now that we have different types of unhealthy containers it seems we need some
more flexibility around reporting on them, but it also depends on what we want
to display in the recon API.
It feels like we should have a summary report for unhealthy containers, eg:
missing: x
mis-replicated: x
under-replicated: x
over-replicated: x
Or perhaps an overall report (this could be part of the getContainers or
another new API):
totalContainers: xxx
healthy: xxx
missing: xxx
mis-replicated: xxx
under-replicated: xxx
over-replicated: xxx
We also probably want to be able to list the unhealthy containers in a given
state, and it probably should be a batch API to limit the number returned if
there are a lot of them.
I wonder if we should have a new endpoint which is:
/containers/unhealthy/{state}
By default it will return a list of all unhealthy containers
(/containers/unhealthy), but this can be filtered by adding MISSING /
UNDER-REPLICATED, OVER-REPLICATED, MIS-REPLICATED.
Then we can remove the existing "missing" endpoint.
[~avijayan] [~vivekratnavel] - what do you guys think, keeping what we will
want to display on Recon in mind?
> Enhance Recon ContainerEndPoint to report on difference unhealty container
> states
> -
>
> Key: HDDS-3831
> URL: https://issues.apache.org/jira/browse/HDDS-3831
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Recon
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Priority: Major
>
> HDDS-3082 changed the unhealthyContainers job to store the counts of under
> replicated, over replicated, missing and mis-replicated containers.
> We should enhance ContainerEndPoint.java to provide API access to those
> states and perhaps a summary status too, eg:
> under-replicated-count: 10
> over-replication-count: 0
> mis-replicated-count: 5
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3831) Enhance Recon ContainerEndPoint to report on difference unhealty container states
[
https://issues.apache.org/jira/browse/HDDS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142014#comment-17142014
]
Stephen O'Donnell commented on HDDS-3831:
-
Currently the container endpoint has:
getContainers - list all containers in batches (/containers)
getKeys - List all keys in a container (/containers/{id}/keys)
getReplicaHistory - History of locations (/containers/{id}/replicaHistory)
getMissing - List of missing containers (/containers/missing)
Now that we have different types of unhealthy containers it seems we need some
more flexibility around reporting on them, but it also depends on what we want
to display in the recon API.
It feels like we should have a summary report for unhealthy containers, eg:
missing: x
mis-replicated: x
under-replicated: x
over-replicated: x
Or perhaps an overall report (this could be part of the getContainers or
another new API):
totalContainers: xxx
healthy: xxx
missing: xxx
mis-replicated: xxx
under-replicated: xxx
over-replicated: xxx
We also probably want to be able to list the unhealthy containers in a given
state, and it probably should be a batch API to limit the number returned if
there are a lot of them.
I wonder if we should have a new endpoint which is:
/containers/unhealthy/{state}
By default it will return a list of all unhealthy containers
(/containers/unhealthy), but this can be filtered by adding MISSING /
UNDER-REPLICATED, OVER-REPLICATED, MIS-REPLICATED.
Then we can remove the existing "missing" endpoint.
[~avijayan] [~vivekratnavel] - what do you guys think, keeping what we will
want to display on Recon in mind?
> Enhance Recon ContainerEndPoint to report on difference unhealty container
> states
> -
>
> Key: HDDS-3831
> URL: https://issues.apache.org/jira/browse/HDDS-3831
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Recon
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Priority: Major
>
> HDDS-3082 changed the unhealthyContainers job to store the counts of under
> replicated, over replicated, missing and mis-replicated containers.
> We should enhance ContainerEndPoint.java to provide API access to those
> states and perhaps a summary status too, eg:
> under-replicated-count: 10
> over-replication-count: 0
> mis-replicated-count: 5
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3832) Enhance Recon UI to display the different unhealthy container states
Stephen O'Donnell created HDDS-3832: --- Summary: Enhance Recon UI to display the different unhealthy container states Key: HDDS-3832 URL: https://issues.apache.org/jira/browse/HDDS-3832 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Recon Affects Versions: 0.6.0 Reporter: Stephen O'Donnell When HDDS-3831 is closed, we should enhance the Recon UI to display the different unhealthy container counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3831) Enhance Recon ContainerEndPoint to report on difference unhealty container states
Stephen O'Donnell created HDDS-3831: --- Summary: Enhance Recon ContainerEndPoint to report on difference unhealty container states Key: HDDS-3831 URL: https://issues.apache.org/jira/browse/HDDS-3831 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Recon Affects Versions: 0.6.0 Reporter: Stephen O'Donnell HDDS-3082 changed the unhealthyContainers job to store the counts of under replicated, over replicated, missing and mis-replicated containers. We should enhance ContainerEndPoint.java to provide API access to those states and perhaps a summary status too, eg: under-replicated-count: 10 over-replication-count: 0 mis-replicated-count: 5 ... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[ https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reassigned HDDS-1930: --- Assignee: Stephen O'Donnell > Test Topology Aware Job scheduling with Ozone Topology > -- > > Key: HDDS-1930 > URL: https://issues.apache.org/jira/browse/HDDS-1930 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Stephen O'Donnell >Priority: Major > > My initial results with Terasort does not seem to report the counter > properly. Most of the requests are handled by rack local but no node local. > This ticket is opened to add more system testing to validate the feature. > Total Allocated Containers: 3778 > Each table cell represents the number of NodeLocal/RackLocal/OffSwitch > containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. > Node Local RequestRack Local Request Off Switch Request > Num Node Local Containers (satisfied by) 0 > Num Rack Local Containers (satisfied by) 0 3648 > Num Off Switch Containers (satisfied by) 0 96 34 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3794) Topology Aware read does not work correctly in XceiverClientGrpc
Stephen O'Donnell created HDDS-3794:
---
Summary: Topology Aware read does not work correctly in
XceiverClientGrpc
Key: HDDS-3794
URL: https://issues.apache.org/jira/browse/HDDS-3794
Project: Hadoop Distributed Data Store
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
Fix For: 0.6.0
In XceiverClientGrpc.java, the calls to read a block or chunks for a Datanode
end up in the private method sendCommandWithRetry(). In this method it decides
which datanode it should send the request to. To do that, it checks if there is
a cached DN connection for the given block and if so it uses that. If there is
no cached connection, it should take network topology into account or shuffle
the nodes:
{code}
List datanodeList = null;
DatanodeBlockID blockID = null;
if (request.getCmdType() == ContainerProtos.Type.ReadChunk) {
blockID = request.getReadChunk().getBlockID();
} else if (request.getCmdType() == ContainerProtos.Type.GetSmallFile) {
blockID = request.getGetSmallFile().getBlock().getBlockID();
}
if (blockID != null) {
LOG.info("blockid is not null");
// Check if the DN to which the GetBlock command was sent has been cached.
DatanodeDetails cachedDN = getBlockDNcache.get(blockID);
if (cachedDN != null) {
LOG.info("Cached DN is not null");
datanodeList = pipeline.getNodes();
int getBlockDNCacheIndex = datanodeList.indexOf(cachedDN);
if (getBlockDNCacheIndex > 0) {
LOG.info("pulling cached dn to top of list");
// Pull the Cached DN to the top of the DN list
Collections.swap(datanodeList, 0, getBlockDNCacheIndex);
}
} else if (topologyAwareRead) {
LOG.info("topology aware - order DNs");
datanodeList = pipeline.getNodesInOrder();
}
}
if (datanodeList == null) {
LOG.info("List is null - shuffling");
datanodeList = pipeline.getNodes();
// Shuffle datanode list so that clients do not read in the same order
// every time.
Collections.shuffle(datanodeList);
}
{code}
The normal flow for the client is to first make a getBlock() call to the DN and
then a readChunk() call.
Due to the logic at the top of the block above, blockID is always going to be
null for the getBlock() call, then it never checks the topologyAwareRead
section and shuffles the node.
Then for readChunk, it will find the blockID, find a cached DN, which was the
result of the shuffle, and then it reuses that DN.
Therefore the topologyAwareRead does not work as expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3785) Update topology.aware.read parameter in ozone-topology compose config
Stephen O'Donnell created HDDS-3785: --- Summary: Update topology.aware.read parameter in ozone-topology compose config Key: HDDS-3785 URL: https://issues.apache.org/jira/browse/HDDS-3785 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.5.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell Fix For: 0.6.0 HDDS-1865 updated the parameter dfs.network.topology.aware.read.enable to ozone.network.topology.aware.read, but in the docker-compose config for ozone-topology, the old parameter is still used. This Jira is to update it to the new value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3481) SCM ask too many datanodes to replicate the same container
[ https://issues.apache.org/jira/browse/HDDS-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130640#comment-17130640 ] Stephen O'Donnell commented on HDDS-3481: - I saw this problem a long time back in theory from reading the code. I don't think it is a good idea for SCM to hand out all the replication work immediately. After SCM passes out the commands, it loses the ability to adjust the work later. It is effectively flooding downstream workers who have no ability to provide back pressure and indicate they are overloaded. Eg, if it needs to replicate 1000 containers, and it gives 500 to node 1 and 500 to node 2. What if node one completes its work more quickly (maybe its under less read load, has faster disks, is on the same rack as the target ... ) - then we cannot just take some of the containers allocated to node 2 and give them to node 1 to complete replication faster, as the commands are fired with no easy way to see their progress or cancel them. It is better for the supervisor (SCM) to hand out the work incrementally as the workers have capacity for it. Even with a longer timeout, I reckon this bad feedback loop will happen. This is roughly how HDFS does it - there is a replication queue in the namenode, and each datanode has a limit of how many replications it can have. On each heartbeat, it gets given more work up to its maximum. The namenode holds the work back until the workers have capacity to receive it. There isn't a feedback loop for the commands in HDFS, but the limit of work + a relatively short deadline to complete that work results in it working well. > SCM ask too many datanodes to replicate the same container > -- > > Key: HDDS-3481 > URL: https://issues.apache.org/jira/browse/HDDS-3481 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Blocker > Labels: TriagePending, pull-request-available > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png > > > *What's the problem ?* > As the image shows, scm ask 31 datanodes to replicate container 2037 every > 10 minutes from 2020-04-17 23:38:51. And at 2020-04-18 08:58:52 scm find the > replicate num of container 2037 is 12, then it ask 11 datanodes to delete > container 2037. > !screenshot-1.png! > !screenshot-2.png! > *What's the reason ?* > scm check whether (container replicates num + > inflightReplication.get(containerId).size() - > inflightDeletion.get(containerId).size()) is less than 3. If less than 3, it > will ask some datanode to replicate the container, and add the action into > inflightReplication.get(containerId). The replicate action time out is 10 > minutes, if action timeout, scm will delete the action from > inflightReplication.get(containerId) as the image shows. Then (container > replicates num + inflightReplication.get(containerId).size() - > inflightDeletion.get(containerId).size()) is less than 3 again, and scm ask > another datanode to replicate the container. > Because replicate container cost a long time, sometimes it cannot finish in > 10 minutes, thus 31 datanodes has to replicate the container every 10 > minutes. 19 of 31 datanodes replicate container from the same source > datanode, it will also cause big pressure on the source datanode and > replicate container become slower. Actually it cost 4 hours to finish the > first replicate. > !screenshot-4.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3686) Insert an argument of ozone shell to accept jvm arguments
[ https://issues.apache.org/jira/browse/HDDS-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3686: Fix Version/s: 0.6.0 > Insert an argument of ozone shell to accept jvm arguments > - > > Key: HDDS-3686 > URL: https://issues.apache.org/jira/browse/HDDS-3686 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3686) Insert an argument of ozone shell to accept jvm arguments
[ https://issues.apache.org/jira/browse/HDDS-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3686. - Resolution: Fixed > Insert an argument of ozone shell to accept jvm arguments > - > > Key: HDDS-3686 > URL: https://issues.apache.org/jira/browse/HDDS-3686 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Affects Versions: 0.6.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1880) Decommissioining and maintenance mode in Ozone
[ https://issues.apache.org/jira/browse/HDDS-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124829#comment-17124829 ] Stephen O'Donnell commented on HDDS-1880: - [~arp] Nobody is actively working on this at the moment so it will not make 0.6.0. > Decommissioining and maintenance mode in Ozone > --- > > Key: HDDS-1880 > URL: https://issues.apache.org/jira/browse/HDDS-1880 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Marton Elek >Assignee: Stephen O'Donnell >Priority: Major > Labels: TriagePending > > This is the umbrella jira for decommissioning support in Ozone. Design doc > will be attached soon. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3082) Refactor recon missing containers task to detect under, over and mis-replicated containers
[ https://issues.apache.org/jira/browse/HDDS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3082: Summary: Refactor recon missing containers task to detect under, over and mis-replicated containers (was: Create Recon task to detect mis-replicated containers) > Refactor recon missing containers task to detect under, over and > mis-replicated containers > -- > > Key: HDDS-3082 > URL: https://issues.apache.org/jira/browse/HDDS-3082 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > With network topology enabled, we need a tool to check that all containers on > the cluster meet the replication policy and possibly correct them if they do > not (HDDS-3081). > Do we need a tool like "hdfs fsck" for this, or is this something we should > build into Recon? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3610) Test Recon works with MySQL and Postgres
Stephen O'Donnell created HDDS-3610: --- Summary: Test Recon works with MySQL and Postgres Key: HDDS-3610 URL: https://issues.apache.org/jira/browse/HDDS-3610 Project: Hadoop Distributed Data Store Issue Type: Test Components: Ozone Recon Affects Versions: 0.6.0 Reporter: Stephen O'Donnell Until now, Recon has only been tested with embedded DBs - Sqlite and Derby, with Derby being the default. In theory it should work with MySQL and Postgres without any changes, but we would like to verify that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3582) Update Checkstyle rule
[
https://issues.apache.org/jira/browse/HDDS-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107146#comment-17107146
]
Stephen O'Donnell commented on HDDS-3582:
-
+1 to this, especially relaxing the line length constraint.
> Update Checkstyle rule
> --
>
> Key: HDDS-3582
> URL: https://issues.apache.org/jira/browse/HDDS-3582
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
>
> Now. the rules of checkstyle is hadoop style, as we all know, hadoop repo
> started 10 years ago, so some of its rules need to be update.
> For example.
> - 120 length characters of a line make more sense, because most of all have a
> better monitor than 10 years ago.
> - We need more style rules, such as, behind and after "{" and "}" should have
> a space.
> - We should have manage the import into group and order them
> - ModifierOrder is needed
> - Remove the package javadoc rules.
> hope our community getting better and better
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3079) Extract Rack Specific code from Container and Pipeline placement policies into a common parent
[ https://issues.apache.org/jira/browse/HDDS-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3079: Description: The Ratis pipeline provider uses a class PipelinePlacementPolicy to selection datanodes for new pipelines in a rack aware fashion. This class also considers server load when selecting hosts and in the future, it may have more features. However, SCMContainerPlacementPolicyRackAware implements very similar rack aware node choosing logic and is used by the replication manager. Previous discussion indicated that we should not bring these two implementations into one, but it would be preferable to extract the common rack specific logic into a common parent in the inheritence chain. Aright now all placement policies inherit from SCMCommonPlacementPolicy, so it would make sense to have a new abstract class that extends Common, called SCMCommonRackAwarePolicy, and have the others inherit from it. Note that as part of this move, the rack specific logic in SCMCommonPlacementPolicy.validateContainerPlacement() should be extracted and moved into this common parent too. was: The Ratis pipeline provider uses a class PipelinePlacementPolicy to selection datanodes for new pipelines in a rack aware fashion. This class also considers server load when selecting hosts and in the future, it may have more features. However, SCMContainerPlacementPolicyRackAware implements very similar rack aware node choosing logic and is used by the replication manager. It is possible these two implementations could diverge and it also means we have two places to make changes if there are any improvements made. It would be worth investigating if at least the rack aware node choosing logic can be shared between these two classes, or even better the same class can be used for both. > Extract Rack Specific code from Container and Pipeline placement policies > into a common parent > -- > > Key: HDDS-3079 > URL: https://issues.apache.org/jira/browse/HDDS-3079 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Priority: Major > > The Ratis pipeline provider uses a class PipelinePlacementPolicy to selection > datanodes for new pipelines in a rack aware fashion. This class also > considers server load when selecting hosts and in the future, it may have > more features. > However, SCMContainerPlacementPolicyRackAware implements very similar rack > aware node choosing logic and is used by the replication manager. > Previous discussion indicated that we should not bring these two > implementations into one, but it would be preferable to extract the common > rack specific logic into a common parent in the inheritence chain. > Aright now all placement policies inherit from SCMCommonPlacementPolicy, so > it would make sense to have a new abstract class that extends Common, called > SCMCommonRackAwarePolicy, and have the others inherit from it. > Note that as part of this move, the rack specific logic in > SCMCommonPlacementPolicy.validateContainerPlacement() should be extracted and > moved into this common parent too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3079) Extract Rack Specific code from Container and Pipeline placement policies into a common parent
[ https://issues.apache.org/jira/browse/HDDS-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3079: Summary: Extract Rack Specific code from Container and Pipeline placement policies into a common parent (was: Consider merging PipelinePlacementPolicy and SCMContainerPlacementPolicyRackAware) > Extract Rack Specific code from Container and Pipeline placement policies > into a common parent > -- > > Key: HDDS-3079 > URL: https://issues.apache.org/jira/browse/HDDS-3079 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Priority: Major > > The Ratis pipeline provider uses a class PipelinePlacementPolicy to selection > datanodes for new pipelines in a rack aware fashion. This class also > considers server load when selecting hosts and in the future, it may have > more features. > However, SCMContainerPlacementPolicyRackAware implements very similar rack > aware node choosing logic and is used by the replication manager. > It is possible these two implementations could diverge and it also means we > have two places to make changes if there are any improvements made. > It would be worth investigating if at least the rack aware node choosing > logic can be shared between these two classes, or even better the same class > can be used for both. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-3504.
-
Fix Version/s: 0.6.0
Resolution: Fixed
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Stephen O'Donnell
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.6.0
>
> Attachments:
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz,
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
>
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:50:52.134" endtime="20200417
> 18:55:52.135" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200420 12:16:47.410" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200420 12:11:47.409" endtime="20200420
> 12:16:47.411" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200422 00:10:32.971" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200422 00:05:32.970" endtime="20200422
> 00:10:32.972" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200427 22:05:52.823" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200427 22:00:52.822" endtime="20200427
> 22:05:52.824" critical="yes">Test timeout 5 minutes exceeded.
> First failed commit: 3bb5838196536f2ea4ac1ab4dcd0bc53ae97f7e0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3082) Create Recon task to detect mis-replicated containers
[ https://issues.apache.org/jira/browse/HDDS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3082: Summary: Create Recon task to detect mis-replicated containers (was: Tool to check containers meet the replication policy) > Create Recon task to detect mis-replicated containers > - > > Key: HDDS-3082 > URL: https://issues.apache.org/jira/browse/HDDS-3082 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Priority: Major > > With network topology enabled, we need a tool to check that all containers on > the cluster meet the replication policy and possibly correct them if they do > not (HDDS-3081). > Do we need a tool like "hdfs fsck" for this, or is this something we should > build into Recon? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3082) Create Recon task to detect mis-replicated containers
[ https://issues.apache.org/jira/browse/HDDS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reassigned HDDS-3082: --- Assignee: Stephen O'Donnell > Create Recon task to detect mis-replicated containers > - > > Key: HDDS-3082 > URL: https://issues.apache.org/jira/browse/HDDS-3082 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > With network topology enabled, we need a tool to check that all containers on > the cluster meet the replication policy and possibly correct them if they do > not (HDDS-3081). > Do we need a tool like "hdfs fsck" for this, or is this something we should > build into Recon? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell reassigned HDDS-3504:
---
Assignee: Stephen O'Donnell
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Stephen O'Donnell
>Priority: Critical
> Attachments:
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz,
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
>
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:50:52.134" endtime="20200417
> 18:55:52.135" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200420 12:16:47.410" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200420 12:11:47.409" endtime="20200420
> 12:16:47.411" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200422 00:10:32.971" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200422 00:05:32.970" endtime="20200422
> 00:10:32.972" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200427 22:05:52.823" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200427 22:00:52.822" endtime="20200427
> 22:05:52.824" critical="yes">Test timeout 5 minutes exceeded.
> First failed commit: 3bb5838196536f2ea4ac1ab4dcd0bc53ae97f7e0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096732#comment-17096732
]
Stephen O'Donnell commented on HDDS-3504:
-
I wonder if these space problems are appearing in this test, because it has 6
DNs, and hence more pipelines reserving more space?
Most of the other compose tests seems to have only 1 DN, is that correct?
We might be able to reduce the cluster to 4 DNs (2 per rack) rather than 6 and
see if it improves things.
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
> Attachments:
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz,
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
>
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:50:52.134" endtime="20200417
> 18:55:52.135" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200420 12:16:47.410" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200420 12:11:47.409" endtime="20200420
> 12:16:47.411" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200422 00:10:32.971" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200422 00:05:32.970" endtime="20200422
> 00:10:32.972" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200427 22:05:52.823" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200427 22:00:52.822" endtime="20200427
> 22:05:52.824" critical="yes">Test timeout 5 minutes exceeded.
> First failed commit: 3bb5838196536f2ea4ac1ab4dcd0bc53ae97f7e0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3504:
Attachment:
docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
> Attachments:
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz,
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
>
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:50:52.134" endtime="20200417
> 18:55:52.135" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200420 12:16:47.410" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200420 12:11:47.409" endtime="20200420
> 12:16:47.411" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200422 00:10:32.971" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200422 00:05:32.970" endtime="20200422
> 00:10:32.972" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200427 22:05:52.823" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200427 22:00:52.822" endtime="20200427
> 22:05:52.824" critical="yes">Test timeout 5 minutes exceeded.
> First failed commit: 3bb5838196536f2ea4ac1ab4dcd0bc53ae97f7e0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3504:
Attachment:
docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
> Attachments:
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-20-792.log.gz,
> docker-ozone-topology-ozone-topology-basic-scm-2020-04-27-830.log.gz
>
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:50:52.134" endtime="20200417
> 18:55:52.135" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200420 12:16:47.410" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/20/792/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200420 12:11:47.409" endtime="20200420
> 12:16:47.411" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200422 00:10:32.971" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/21/803/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200422 00:05:32.970" endtime="20200422
> 00:10:32.972" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200427 22:05:52.823" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/27/830/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200427 22:00:52.822" endtime="20200427
> 22:05:52.824" critical="yes">Test timeout 5 minutes exceeded.
> First failed commit: 3bb5838196536f2ea4ac1ab4dcd0bc53ae97f7e0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3504) Topology related acceptance test is intermittent
[
https://issues.apache.org/jira/browse/HDDS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096722#comment-17096722
]
Stephen O'Donnell commented on HDDS-3504:
-
Looking at a few of the failures, it seems its the original freon step that is
failing:
{code}
2020/04/17/789:
Running command 'ozone
freon randomkeys --numOfVolumes 5 --numOfBuckets 5 --numOfKeys 5 --numOfThreads
1 --replicationType RATIS --factor THREE --validateWrites 21'.
Test timeout 5 minutes
exceeded.
2020/04/17/791:
Running command 'ozone
freon randomkeys --numOfVolumes 5 --numOfBuckets 5 --numOfKeys 5 --numOfThreads
1 --replicationType RATIS --factor THREE --validateWrites 21'.
Test timeout 5 minutes
exceeded.
2020/04/20/792:
Running command 'ozone
freon randomkeys --numOfVolumes 5 --numOfBuckets 5 --numOfKeys 5 --numOfThreads
1 --replicationType RATIS --factor THREE --validateWrites 21'.
Test timeout 5 minutes
exceeded.
{code}
Focusing on the last one above, safemode certainly exited:
{code}
scm_1 | 2020-04-20 12:11:33,974
[EventQueue-NodeRegistrationContainerReportForDataNodeSafeModeRule] INFO
safemode.SCMSafeModeManager: SCM in safe mode. 4 DataNodes registered, 4
required.
scm_1 | 2020-04-20 12:11:33,975
[EventQueue-NodeRegistrationContainerReportForDataNodeSafeModeRule] INFO
safemode.SCMSafeModeManager: DataNodeSafeModeRule rule is successfully validated
scm_1 | 2020-04-20 12:11:33,976
[EventQueue-NodeRegistrationContainerReportForDataNodeSafeModeRule] INFO
safemode.SCMSafeModeManager: All SCM safe mode pre check rules have passed
scm_1 | 2020-04-20 12:11:34,235
[EventQueue-NodeRegistrationContainerReportForContainerSafeModeRule] INFO
safemode.SCMSafeModeManager: ContainerSafeModeRule rule is successfully
validated
...
scm_1 | 2020-04-20 12:11:41,680
[EventQueue-OpenPipelineForHealthyPipelineSafeModeRule] INFO
safemode.SCMSafeModeManager: SCM in safe mode. Healthy pipelines reported count
is 1, required healthy pipeline reported count is 1
...
scm_1 | 2020-04-20 12:11:41,682
[EventQueue-OpenPipelineForHealthyPipelineSafeModeRule] INFO
safemode.SCMSafeModeManager: SCM exiting safe mode.
{code}
However the role logs are filled with disk space related errors, eg:
{code}
Caused by: org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Out of
space: The volume with the most available space (=965574656 B) is less than the
container size (=1073741824 B).
datanode_5_1 | at
org.apache.hadoop.ozone.container.common.volume.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:77)
datanode_5_1 | at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:124)
datanode_5_1 | ... 13 more
{code}
This message is appeared on various DN logs. I will attach the full log here
for reference.
The failure on 2020/04/17/791 and 2020/04/17/789 also have the space errors.
This run 2020/04/27/830 also failed at the freon step, but there are no disk
space issues:
{code}
Running command 'ozone
freon randomkeys --numOfVolumes 5 --numOfBuckets 5 --numOfKeys 5 --numOfThreads
1 --replicationType RATIS --factor THREE --validateWrites 21'.
Test timeout 5 minutes
exceeded.
{code}
I will attach the log for it too, so we have it tracked here. I am not sure why
it failed.
Based on these 3 occurrences, it seems a lot of the instability is caused by
space issues rather than the changes made to the test in HDDS-3084 / HDDS-3135
as that code is not even getting executed (test fails before it reaches that
step).
> Topology related acceptance test is intermittent
>
>
> Key: HDDS-3504
> URL: https://issues.apache.org/jira/browse/HDDS-3504
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Critical
>
> It's failed multiple times on master and PRs. See the archive of the builds
> results at: https://github.com/elek/ozone-build-results
> I am disabling it to avoid flaky runs, but we need to fix and re-enable it.
> {code}
> ./find-first.sh "Test timeout 5 minutes exceeded."
> --include="robot-ozone-topology-ozone-topology-basic-scm.xml"
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:06:32.549" level="FAIL">Test timeout 5 minutes
> exceeded.
> 2020/04/17/789/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: status="FAIL" starttime="20200417 18:01:32.547" endtime="20200417
> 18:06:32.550" critical="yes">Test timeout 5 minutes exceeded.
> 2020/04/17/791/acceptance/robot-ozone-topology-ozone-topology-basic-scm.xml: timestamp="20200417 18:55:52.135" level="FAIL">Test timeout 5 minutes
> exceeded.
>
[jira] [Updated] (HDDS-3081) Replication manager should detect and correct containers which don't meet the replication policy
[ https://issues.apache.org/jira/browse/HDDS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3081: Summary: Replication manager should detect and correct containers which don't meet the replication policy (was: Should replication manager detect and correct containers which don't meet the replication policy) > Replication manager should detect and correct containers which don't meet the > replication policy > > > Key: HDDS-3081 > URL: https://issues.apache.org/jira/browse/HDDS-3081 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > In the current implementation, replication manager does not consider the > container placement when checking if a container is healthy. Only the number > of replicas are checked. > Now we have network topology available, we should consider whether > replication manager should detect and correct mis-replicated containers. > In HDFS, the namenode will not automatically correct mis-replicated > containers automatically, except at startup when all blocks are checked. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3081) Should replication manager detect and correct containers which don't meet the replication policy
[ https://issues.apache.org/jira/browse/HDDS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reassigned HDDS-3081: --- Assignee: Stephen O'Donnell > Should replication manager detect and correct containers which don't meet the > replication policy > > > Key: HDDS-3081 > URL: https://issues.apache.org/jira/browse/HDDS-3081 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > In the current implementation, replication manager does not consider the > container placement when checking if a container is healthy. Only the number > of replicas are checked. > Now we have network topology available, we should consider whether > replication manager should detect and correct mis-replicated containers. > In HDFS, the namenode will not automatically correct mis-replicated > containers automatically, except at startup when all blocks are checked. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3139) Pipeline placement should select lowest load datanode as anchor
[ https://issues.apache.org/jira/browse/HDDS-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3139. - Target Version/s: 0.6.0 Resolution: Fixed > Pipeline placement should select lowest load datanode as anchor > --- > > Key: HDDS-3139 > URL: https://issues.apache.org/jira/browse/HDDS-3139 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > PipelinePlacementPolicy should select datanodes with lowest load as anchor. > Current logic is to random choose one, which could cause imbalance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3135) Enable test added in HDDS-3084 when blocking issues are resolved
[ https://issues.apache.org/jira/browse/HDDS-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3135. - Fix Version/s: 0.6.0 Resolution: Fixed > Enable test added in HDDS-3084 when blocking issues are resolved > > > Key: HDDS-3135 > URL: https://issues.apache.org/jira/browse/HDDS-3135 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Once the blocking issues HDDS-3107 and HDDS-3116 are resolved the test added > by HDDS-3084 show be enabled by renaming > "hadoop-ozone/dist/src/main/compose/ozone-topology/hdds-3084.sh" to "test.sh". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3361) Change replication logic to use PersistedOpState
[ https://issues.apache.org/jira/browse/HDDS-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3361. - Resolution: Fixed > Change replication logic to use PersistedOpState > > > Key: HDDS-3361 > URL: https://issues.apache.org/jira/browse/HDDS-3361 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > In an earlier change, we decided that the Datanode would change the state of > decommissioning / maintenance replicas and report the new state in its > container report. Using that assumption, the logic in replication manager was > changed to expect the ContainerReplica to contain the decom / maintenance > state. > However in HDDS-2592 it was decided that the DN would simply store the > operational state and report only the operation state in the heartbeat. This > state is reported back to SCM and stored in the datanodeDetails instances. > This means that earlier assumption (ContainerReplica will contain the decom > state) is no longer valid. > This Jira is to change the logic so replication decisions are based on the > "persistedOpState" in the datanode details object instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[
https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-3270.
-
Fix Version/s: 0.6.0
Target Version/s: 0.6.0
Resolution: Fixed
> Allow safemode listeners to be notified when some precheck rules pass
> -
>
> Key: HDDS-3270
> URL: https://issues.apache.org/jira/browse/HDDS-3270
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> As part of the SCM safemode process, there are some rules which must pass
> before safemode can be exited.
> One of these rules is the number of registered datanodes and another is that
> at least one pipeline must be created and open.
> Currently, pipeline creation is attempted each time a node registers. As soon
> as the 3rd node registers, pipelines will be created.
> There are two issue with this:
> 1. With network topology, if the first 3 nodes are all from the same rack, a
> non-rackaware pipeline will get created.
> 2. With multi-raft, it would be better if more nodes are registered to allow
> the multiple pipelines per node to be spread across all the available nodes.
> The proposal here, is to introduce a new sub-state into the safemode process,
> call "preCheckComplete". When adding rules to the Safemode Manager, some
> rules can be tagged as "preCheck" (eg number of datanodes registered). When
> all all the pre-check rules have passed a notification will be sent to all
> safemode listeners:
> {code}
> safeModeIsOn -> true
> preCheckComplete -> true
> {code}
> That will allow the listener to take action on this first stage completing.
> In the case of PipelineManager, it will then allow pipelines to be created.
> After the remaining rules have been passed, safemode will exit as normal, by
> sending a second event:
> {code}
> safeModeIsOn -> false
> preCheckComplete -> true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3361) Change replication logic to use PersistedOpState
Stephen O'Donnell created HDDS-3361: --- Summary: Change replication logic to use PersistedOpState Key: HDDS-3361 URL: https://issues.apache.org/jira/browse/HDDS-3361 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 0.6.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell In an earlier change, we decided that the Datanode would change the state of decommissioning / maintenance replicas and report the new state in its container report. Using that assumption, the logic in replication manager was changed to expect the ContainerReplica to contain the decom / maintenance state. However in HDDS-2592 it was decided that the DN would simply store the operational state and report only the operation state in the heartbeat. This state is reported back to SCM and stored in the datanodeDetails instances. This means that earlier assumption (ContainerReplica will contain the decom state) is no longer valid. This Jira is to change the logic so replication decisions are based on the "persistedOpState" in the datanode details object instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3343) Merge Master branch into decom branch
[ https://issues.apache.org/jira/browse/HDDS-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077170#comment-17077170 ] Stephen O'Donnell commented on HDDS-3343: - Test the merge by creating a PR and ensuring all tests passed. Then I closed the PR and pushed the merge to the branch manually. This is now complete to resolving this Jira. > Merge Master branch into decom branch > - > > Key: HDDS-3343 > URL: https://issues.apache.org/jira/browse/HDDS-3343 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Its been a long time since Master was merged into the decom branch. Merge the > branches and resolve the conflicts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3343) Merge Master branch into decom branch
[ https://issues.apache.org/jira/browse/HDDS-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3343. - Resolution: Fixed > Merge Master branch into decom branch > - > > Key: HDDS-3343 > URL: https://issues.apache.org/jira/browse/HDDS-3343 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Its been a long time since Master was merged into the decom branch. Merge the > branches and resolve the conflicts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3345) Investigate failure of TestDecommissionAndMaintenance integration test
Stephen O'Donnell created HDDS-3345: --- Summary: Investigate failure of TestDecommissionAndMaintenance integration test Key: HDDS-3345 URL: https://issues.apache.org/jira/browse/HDDS-3345 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 0.6.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell The test in the above class is failing and needs corrected. It is currently ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3343) Merge Master branch into decom branch
Stephen O'Donnell created HDDS-3343: --- Summary: Merge Master branch into decom branch Key: HDDS-3343 URL: https://issues.apache.org/jira/browse/HDDS-3343 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 0.6.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell Its been a long time since Master was merged into the decom branch. Merge the branches and resolve the conflicts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2592) Add Datanode command to allow the datanode to persist its admin state
[ https://issues.apache.org/jira/browse/HDDS-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-2592: Resolution: Fixed Status: Resolved (was: Patch Available) > Add Datanode command to allow the datanode to persist its admin state > -- > > Key: HDDS-2592 > URL: https://issues.apache.org/jira/browse/HDDS-2592 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode, SCM >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When the operational state of a datanode changes, an async command should be > triggered to persist the new state on the datanodes. For maintenance mode, > the datanode should also store the maintenance end time. The datanode will > then report the new state (and optional maintenance end time) back via its > heartbeat. > The purpose of the DN persisting this information and heartbeating it back to > SCM is to allow the operation state to be recovered after a SCM reboot, as > SCM does not persist any of this information. It also allows "Recon" to learn > the datanode states. > If SCM is restarted, then it will forget all knowledge of the datanodes. When > they register, their operational state will be reported and SCM can set it > correctly. > Outside of registration (ie during normal heartbeats), the SCM state is the > source of truth for the operational state and if the DN heartbeat reports a > state that is not the same as SCM, SCM should issue another command to the > datanode to set its state to the SCM value. There is a chance the state miss > match is due to an unprocessed command triggered by the SCM state change, but > the worst case is an extra command sent to the datanode. This is a very > lightweight command, so that is not an issue. > One open question is whether to persist intermediate states on the DN. Ie for > decommissioning, the DN will first persist "Decommissioning" and then > transition to "Decommissioned" when SCM is satisfied all containers are > replicated. It would be possible to persist both these states in turn on the > datanode quite easily in turn. Or, we set the end state (Decommissioned) on > the datanode and allow SCM to get the node to that state. For the latter, if > SCM is restarted, then the DN will report "Decommissioned" on registration, > but SCM will set its internal state to Decommissioning and then ensure all > containers are replicated before transitioning the node to Decommissioned. > This seems like a safer approach, but there are advantages of tracking the > intermediate states on the DNs too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3304) Consider using INFINITY in decommission and maintenance commands where not time is specified
Stephen O'Donnell created HDDS-3304: --- Summary: Consider using INFINITY in decommission and maintenance commands where not time is specified Key: HDDS-3304 URL: https://issues.apache.org/jira/browse/HDDS-3304 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Stephen O'Donnell In HDDS-2592, Anu suggested that all decommission and maintenance commands should enforce passing a time (rather than it being optional) and that we should explicitly pass a constant called "INFINITY" where no time is set. At the moment, zero is used through the code as this infinity value, but it may make the code more clear if we used the infinity constance instead. This Jira is to log that suggestion and we can consider making that change in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3303) Add some unit tests around the changes in HDDS-2592
Stephen O'Donnell created HDDS-3303: --- Summary: Add some unit tests around the changes in HDDS-2592 Key: HDDS-3303 URL: https://issues.apache.org/jira/browse/HDDS-3303 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Stephen O'Donnell The changes in HDDS-2592 were committed to branch without any unit tests added for the new command or datanode persistence changes. This Jira is here to log that and ensure that some tests are added when we pick the decommission work back up again. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3179) Pipeline placement based on Topology does not have fall back protection
[ https://issues.apache.org/jira/browse/HDDS-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3179. - Resolution: Fixed > Pipeline placement based on Topology does not have fall back protection > --- > > Key: HDDS-3179 > URL: https://issues.apache.org/jira/browse/HDDS-3179 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.4.1 >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When rack awareness and topology is enabled, pipeline placement can fail when > there is only one node on the rack. >  > Should add fall back logic to search for nodes from other racks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3278) Acceptance test ozonesecure-mr-mapreduce failing
Stephen O'Donnell created HDDS-3278:
---
Summary: Acceptance test ozonesecure-mr-mapreduce failing
Key: HDDS-3278
URL: https://issues.apache.org/jira/browse/HDDS-3278
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: docker
Affects Versions: 0.6.0
Reporter: Stephen O'Donnell
The acceptance / robot test ozonesecure-mr-mapreduce appears to be failing when
running the pi calculation:
{code}
ozonesecure-mr-mapreduce :: Execute MR jobs
==
Execute PI calculation| FAIL |
255 != 0
--
Execute WordCount | FAIL |
255 != 0
--
ozonesecure-mr-mapreduce :: Execute MR jobs | FAIL |
2 critical tests, 0 passed, 2 failed
2 tests total, 0 passed, 2 failed
{code}
The error report by the command is:
{code}
Running command 'yarn jar
./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 3 3
21'.
${rc} = 255
${output} = Number of Maps
= 3
Samples per Map = 3
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"o3fs"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281...
Logs the given message with the given level.
${output}
Number of Maps = 3
Samples per Map = 3
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"o3fs"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:275)
at
org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[
https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell reassigned HDDS-3270:
---
Assignee: Stephen O'Donnell
> Allow safemode listeners to be notified when some precheck rules pass
> -
>
> Key: HDDS-3270
> URL: https://issues.apache.org/jira/browse/HDDS-3270
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> As part of the SCM safemode process, there are some rules which must pass
> before safemode can be exited.
> One of these rules is the number of registered datanodes and another is that
> at least one pipeline must be created and open.
> Currently, pipeline creation is attempted each time a node registers. As soon
> as the 3rd node registers, pipelines will be created.
> There are two issue with this:
> 1. With network topology, if the first 3 nodes are all from the same rack, a
> non-rackaware pipeline will get created.
> 2. With multi-raft, it would be better if more nodes are registered to allow
> the multiple pipelines per node to be spread across all the available nodes.
> The proposal here, is to introduce a new sub-state into the safemode process,
> call "preCheckComplete". When adding rules to the Safemode Manager, some
> rules can be tagged as "preCheck" (eg number of datanodes registered). When
> all all the pre-check rules have passed a notification will be sent to all
> safemode listeners:
> {code}
> safeModeIsOn -> true
> preCheckComplete -> true
> {code}
> That will allow the listener to take action on this first stage completing.
> In the case of PipelineManager, it will then allow pipelines to be created.
> After the remaining rules have been passed, safemode will exit as normal, by
> sending a second event:
> {code}
> safeModeIsOn -> false
> preCheckComplete -> true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[
https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3270:
Description:
As part of the SCM safemode process, there are some rules which must pass
before safemode can be exited.
One of these rules is the number of registered datanodes and another is that at
least one pipeline must be created and open.
Currently, pipeline creation is attempted each time a node registers. As soon
as the 3rd node registers, pipelines will be created.
There are two issue with this:
1. With network topology, if the first 3 nodes are all from the same rack, a
non-rackaware pipeline will get created.
2. With multi-raft, it would be better if more nodes are registered to allow
the multiple pipelines per node to be spread across all the available nodes.
The proposal here, is to introduce a new sub-state into the safemode process,
call "preCheckComplete". When adding rules to the Safemode Manager, some rules
can be tagged as "preCheck" (eg number of datanodes registered). When all all
the pre-check rules have passed a notification will be sent to all safemode
listeners:
{code}
safeModeIsOn -> true
preCheckComplete -> true
{code}
That will allow the listener to take action on this first stage completing. In
the case of PipelineManager, it will then allow pipelines to be created.
After the remaining rules have been passed, safemode will exit as normal, by
sending a second event:
{code}
safeModeIsOn -> false
preCheckComplete -> true
{code}
> Allow safemode listeners to be notified when some precheck rules pass
> -
>
> Key: HDDS-3270
> URL: https://issues.apache.org/jira/browse/HDDS-3270
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Priority: Major
>
> As part of the SCM safemode process, there are some rules which must pass
> before safemode can be exited.
> One of these rules is the number of registered datanodes and another is that
> at least one pipeline must be created and open.
> Currently, pipeline creation is attempted each time a node registers. As soon
> as the 3rd node registers, pipelines will be created.
> There are two issue with this:
> 1. With network topology, if the first 3 nodes are all from the same rack, a
> non-rackaware pipeline will get created.
> 2. With multi-raft, it would be better if more nodes are registered to allow
> the multiple pipelines per node to be spread across all the available nodes.
> The proposal here, is to introduce a new sub-state into the safemode process,
> call "preCheckComplete". When adding rules to the Safemode Manager, some
> rules can be tagged as "preCheck" (eg number of datanodes registered). When
> all all the pre-check rules have passed a notification will be sent to all
> safemode listeners:
> {code}
> safeModeIsOn -> true
> preCheckComplete -> true
> {code}
> That will allow the listener to take action on this first stage completing.
> In the case of PipelineManager, it will then allow pipelines to be created.
> After the remaining rules have been passed, safemode will exit as normal, by
> sending a second event:
> {code}
> safeModeIsOn -> false
> preCheckComplete -> true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[ https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3270: Affects Version/s: 0.6.0 > Allow safemode listeners to be notified when some precheck rules pass > - > > Key: HDDS-3270 > URL: https://issues.apache.org/jira/browse/HDDS-3270 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[ https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3270: Component/s: SCM > Allow safemode listeners to be notified when some precheck rules pass > - > > Key: HDDS-3270 > URL: https://issues.apache.org/jira/browse/HDDS-3270 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Stephen O'Donnell >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3270) Allow safemode listeners to be notified when some precheck rules pass
[ https://issues.apache.org/jira/browse/HDDS-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3270: Summary: Allow safemode listeners to be notified when some precheck rules pass (was: Extend Safemode to have a PreCheck ) > Allow safemode listeners to be notified when some precheck rules pass > - > > Key: HDDS-3270 > URL: https://issues.apache.org/jira/browse/HDDS-3270 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Stephen O'Donnell >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3270) Extend Safemode to have a PreCheck
Stephen O'Donnell created HDDS-3270: --- Summary: Extend Safemode to have a PreCheck Key: HDDS-3270 URL: https://issues.apache.org/jira/browse/HDDS-3270 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Stephen O'Donnell -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3221) Refactor SafeModeHandler to use a Notification Interface
[
https://issues.apache.org/jira/browse/HDDS-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-3221.
-
Fix Version/s: 0.6.0
Resolution: Fixed
> Refactor SafeModeHandler to use a Notification Interface
>
>
> Key: HDDS-3221
> URL: https://issues.apache.org/jira/browse/HDDS-3221
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Affects Versions: 0.6.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The SafeModeHandler currently accepts several objects which it notifies when
> the safe mode status changes.
> Each of these object are notified using a different method (there is no
> "notification interface") and some of the logic which really belongs in those
> objects (ie what to do when safemode goes on or off) is in the safemode
> classes rather than in the receiving class.
> As we may need to extend safemode somewhat to delay pipeline creation until
> sufficient nodes have registered, I think it is worthwhile to refactor this
> area to do the following:
> 1. Introduce a new Interface "SafeModeTransition" which must be implemented
> by any object which wants to listen for safemode starting or ending.
> {code}
> public interface SafeModeTransition {
> void handleSafeModeTransition(SCMSafeModeManager.SafeModeStatus status);
> }
> {code}
> 2. Pass the SafeModeStatus object over this new interface. That way, we can
> extend SafeModeStatus to include more states in the future than just safemode
> = true / false.
> 3. Change the constructor of SafeModeHandler to allow any number of objects
> to be registered to make it more flexible going forward.
> 4. Ensure the logic of what action to take on safemode transition lives
> within the notified objects rather than in the Safemode clases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3242) Investigate use of EventQueue by SafeModeHandler instead of notifying objects directly
Stephen O'Donnell created HDDS-3242: --- Summary: Investigate use of EventQueue by SafeModeHandler instead of notifying objects directly Key: HDDS-3242 URL: https://issues.apache.org/jira/browse/HDDS-3242 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Affects Versions: 0.6.0 Reporter: Stephen O'Donnell In HDDS-3221 it was suggested we should use the the EventQueue to handle notifications from the SafeModeHandler to the objects which need to know about safe mode transitions. This Jira is to investigate that approach and ensure the idea is not lost. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3221) Refactor SafeModeHandler to use a Notification Interface
Stephen O'Donnell created HDDS-3221:
---
Summary: Refactor SafeModeHandler to use a Notification Interface
Key: HDDS-3221
URL: https://issues.apache.org/jira/browse/HDDS-3221
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Affects Versions: 0.6.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
The SafeModeHandler currently accepts several objects which it notifies when
the safe mode status changes.
Each of these object are notified using a different method (there is no
"notification interface") and some of the logic which really belongs in those
objects (ie what to do when safemode goes on or off) is in the safemode classes
rather than in the receiving class.
As we may need to extend safemode somewhat to delay pipeline creation until
sufficient nodes have registered, I think it is worthwhile to refactor this
area to do the following:
1. Introduce a new Interface "SafeModeTransition" which must be implemented by
any object which wants to listen for safemode starting or ending.
{code}
public interface SafeModeTransition {
void handleSafeModeTransition(SCMSafeModeManager.SafeModeStatus status);
}
{code}
2. Pass the SafeModeStatus object over this new interface. That way, we can
extend SafeModeStatus to include more states in the future than just safemode =
true / false.
3. Change the constructor of SafeModeHandler to allow any number of objects to
be registered to make it more flexible going forward.
4. Ensure the logic of what action to take on safemode transition lives within
the notified objects rather than in the Safemode clases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3084) Extend network topology acceptance test to read data when datanodes are stopped
[ https://issues.apache.org/jira/browse/HDDS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3084: Fix Version/s: 0.6.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Extend network topology acceptance test to read data when datanodes are > stopped > --- > > Key: HDDS-3084 > URL: https://issues.apache.org/jira/browse/HDDS-3084 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It would be good to create a smoke test which: > 1. Writes some data on a network aware cluster > 2. Stops 1 rack and ensures the data is still readable > 3. Restart the rack and stop the other rack and again check the data is > readable > That way we can have some confidence the data is being written to both racks > OK. > One issue with a test like this on a small cluster, is that there is a high > chance the data will end up on 2 racks naturally, even if no network topology > is configured. If that was the case, we would expect intermittent test > failures. > However, if network topology is working fine, then we would not expect any > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3084) Extend network topology acceptance test to read data when datanodes are stopped
[ https://issues.apache.org/jira/browse/HDDS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-3084: Summary: Extend network topology acceptance test to read data when datanodes are stopped (was: Smoketest to write data on network aware cluster) > Extend network topology acceptance test to read data when datanodes are > stopped > --- > > Key: HDDS-3084 > URL: https://issues.apache.org/jira/browse/HDDS-3084 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It would be good to create a smoke test which: > 1. Writes some data on a network aware cluster > 2. Stops 1 rack and ensures the data is still readable > 3. Restart the rack and stop the other rack and again check the data is > readable > That way we can have some confidence the data is being written to both racks > OK. > One issue with a test like this on a small cluster, is that there is a high > chance the data will end up on 2 racks naturally, even if no network topology > is configured. If that was the case, we would expect intermittent test > failures. > However, if network topology is working fine, then we would not expect any > failures. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056781#comment-17056781
]
Stephen O'Donnell commented on HDDS-3116:
-
I could reproduce this problem quite easily in docker before adding the
synchronised fix, and afterwards I have not been able to reproduce it using the
same technique.
I think long term this code should be refactored, but the synchronized blocks
seem to address the immediate issue. It would be easily for another change to
introduce more issues in this area due to these circular dependencies however.
The reason the sync blocks work, is because XceiverServerRatis ultimately tries
to get the OzoneContainer instance via the DatanodeStateMachine instance in
another thread while both those objects are constructing. It is the constructor
of OzoneContainer which starts the XceiverServerRatis, so synchronising around
the construction of OzoneContainer and its accessor in DatanodeStateMachine
forces the XceiverServer to wait until construction is finished before it can
get what it needs.
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Blocker
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at
[jira] [Updated] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3116:
Priority: Blocker (was: Major)
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Blocker
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ... 3 more
> {code}
> The DN does not recover from this automatically, although I confirmed that a
> full cluster restart fixed it (docker-compose stop; docker-compose start). I
> will try to confirm if a restart of the stuck DN would fix it or not too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3116:
Target Version/s: 0.5.0, 0.6.0 (was: 0.6.0)
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Blocker
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ... 3 more
> {code}
> The DN does not recover from this automatically, although I confirmed that a
> full cluster restart fixed it (docker-compose stop; docker-compose start). I
> will try to confirm if a restart of the stuck DN would fix it or not too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054831#comment-17054831
]
Stephen O'Donnell commented on HDDS-3116:
-
Unsure if this should be a blocker for 0.5. In my tests (docker compose
environment), this problem happens regularly on DN restart, so I am surprised
it has not come up before. I don't think this is a newly introduced issue
either - it looks to have been around for a long time and just discovered now.
The fix I have posted is pretty trivial, but we may want to consider a bigger
refactor.
It would be best if we can fix it as part of the 0.5 release, so I will mark as
a blocker and let [~dineshchitlangia] triage and downgrade if needed.
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ... 3 more
> {code}
> The
[jira] [Assigned] (HDDS-3135) Enable test added in HDDS-3084 when blocking issues are resolved
[ https://issues.apache.org/jira/browse/HDDS-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell reassigned HDDS-3135: --- Assignee: Stephen O'Donnell > Enable test added in HDDS-3084 when blocking issues are resolved > > > Key: HDDS-3135 > URL: https://issues.apache.org/jira/browse/HDDS-3135 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > Once the blocking issues HDDS-3107 and HDDS-3116 are resolved the test added > by HDDS-3084 show be enabled by renaming > "hadoop-ozone/dist/src/main/compose/ozone-topology/hdds-3084.sh" to "test.sh". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3135) Enable test added in HDDS-3084 when blocking issues are resolved
Stephen O'Donnell created HDDS-3135: --- Summary: Enable test added in HDDS-3084 when blocking issues are resolved Key: HDDS-3135 URL: https://issues.apache.org/jira/browse/HDDS-3135 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Affects Versions: 0.6.0 Reporter: Stephen O'Donnell Once the blocking issues HDDS-3107 and HDDS-3116 are resolved the test added by HDDS-3084 show be enabled by renaming "hadoop-ozone/dist/src/main/compose/ozone-topology/hdds-3084.sh" to "test.sh". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050463#comment-17050463
]
Stephen O'Donnell edited comment on HDDS-3116 at 3/3/20 10:14 PM:
--
The code gets somewhat hard to following when it gets into Ratis, but from what
I gather Ratis attempts to create a new RaftServerImpl.
This is actually a CompleteableFuture and the future is stored into the ImplMap.
The future gets this exception when it is executed, which it stores away for
later (captured with an extra debug message I added):
{code}
2020-03-03 16:12:53,098 [pool-10-thread-1] ERROR ratis.XceiverServerRatis:
Caught a NPE:
java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:766)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:742)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
at
org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
at
org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Then the Datanode State Machine Thread keeps trying over and over to start the
Impl, but as the future failed, it just repeats the same exception over and
over. The impl is never recreated, which is why once it fails it can never
recover even though the OzoneContainer inside the DataodeStateMachine is now
populated:
{code}
2020-03-03 16:12:56,813 [Datanode State Machine Thread - 0] WARN
statemachine.EndpointStateMachine: Unable to communicate to SCM server at
scm:9861 for past 0 seconds.
java.io.IOException: java.lang.NullPointerException
at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
at
org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
at
org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:421)
at
org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:237)
at
org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
at
org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:771)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:742)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
at
org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
at
org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
{code}
It feels like this is a seperate problem within ratis, but it all stems from
the same race condition.
was (Author: sodonnell):
The code gets somewhat hard to following when it gets
[jira] [Assigned] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell reassigned HDDS-3116:
---
Assignee: Stephen O'Donnell (was: Nanda kumar)
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ... 3 more
> {code}
> The DN does not recover from this automatically, although I confirmed that a
> full cluster restart fixed it (docker-compose stop; docker-compose start). I
> will try to confirm if a restart of the stuck DN would fix it or not too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-3116:
Status: Patch Available (was: Open)
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
> -
>
> Key: HDDS-3116
> URL: https://issues.apache.org/jira/browse/HDDS-3116
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.5.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Attachments: full_logs.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While working on a network Topology test (HDDS-3084) which does the following:
> 1. Start a cluster with 6 DNs and 2 racks.
> 2. Create a volume, bucket and a single key.
> 3. Stop one rack of hosts using "docker-compose down"
> 4. Read the data from the single key
> 5. Start the 3 down hosts
> 6. Stop the other 3 hosts
> 7. Attempt to read the key again.
> At step 5 I sometimes see this stack trace in one of the DNs and it fails to
> full come up:
> {code}
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Attempting to start container services.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ozoneimpl.OzoneContainer: Background container scanner has been disabled.
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] INFO
> ratis.XceiverServerRatis: Starting XceiverServerRatis
> 8c1178dd-c44d-49d1-b899-cc3e40ae8f23 at port 9858
> 2020-03-02 13:01:31,887 [Datanode State Machine Thread - 0] WARN
> statemachine.EndpointStateMachine: Unable to communicate to SCM server at
> scm:9861 for past 15000 seconds.
> java.io.IOException: java.lang.NullPointerException
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:418)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:232)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:757)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:739)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
> at
> org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ... 3 more
> {code}
> The DN does not recover from this automatically, although I confirmed that a
> full cluster restart fixed it (docker-compose stop; docker-compose start). I
> will try to confirm if a restart of the stuck DN would fix it or not too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3116) Datanode sometimes fails to start with NPE when starting Ratis xceiver server
[
https://issues.apache.org/jira/browse/HDDS-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050463#comment-17050463
]
Stephen O'Donnell commented on HDDS-3116:
-
The code gets somewhat hard to following when it gets into Ratis, but from what
I gather Ratis attempts to create a new RaftServerImpl.
This is actually a CompleteableFuture and the future is stored into the ImplMap.
The future gets this exception when it is execute, which it stores away for
later (capture with an extra debug message I added:
{code}
2020-03-03 16:12:53,098 [pool-10-thread-1] ERROR ratis.XceiverServerRatis:
Caught a NPE:
java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:766)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:742)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
at
org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
at
org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Then the Datanode State Machine Thread keeps trying over and over to start the
Impl, but as the future failed, it just repeats the same exception over and
over. The impl is never recreated, which is why once it fails it can never
recover even though the OzoneContainer inside the DataodeStateMachine is now
populated:
{code}
2020-03-03 16:12:56,813 [Datanode State Machine Thread - 0] WARN
statemachine.EndpointStateMachine: Unable to communicate to SCM server at
scm:9861 for past 0 seconds.
java.io.IOException: java.lang.NullPointerException
at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:70)
at
org.apache.ratis.server.impl.RaftServerProxy.getImpls(RaftServerProxy.java:284)
at
org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:296)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:421)
at
org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:237)
at
org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:113)
at
org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.sendPipelineReport(XceiverServerRatis.java:771)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.notifyGroupAdd(XceiverServerRatis.java:742)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.initialize(ContainerStateMachine.java:218)
at
org.apache.ratis.server.impl.ServerState.initStatemachine(ServerState.java:160)
at org.apache.ratis.server.impl.ServerState.(ServerState.java:112)
at
org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:112)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:208)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
{code}
It feels like this is a seperate problem within ratis, but it all stems from
the same race condition.
> Datanode sometimes fails to start with NPE when starting Ratis xceiver server
>
