subject:"\[GitHub\] \[hadoop\-ozone\] fapifta commented on a change in pull request #1456\: HDDS\-4172. Implement Finalize command in Ozone Manager server."

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-12 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r503559123



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMFinalizeUpgradeRequest.java
##
@@ -63,11 +64,20 @@ public OMClientResponse validateAndUpdateCache(
 
   String upgradeClientID = request.getUpgradeClientId();
 
-  UpgradeFinalizationStatus status =
+  StatusAndMessages omStatus =

Review comment:
   Nice catch, it seems I forgot to remove this one. I am pushing the 
deletion of this line.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-12 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r503556816



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,328 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.*;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private  static final OmUpgradeAction NOOP = a -> {};
+
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public synchronized StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+Status status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+  /**
+   * This class implements the finalization logic applied to every
+   * LayoutFeature that needs to be finalized.
+   *
+   * For the first approach this happens synchronously within the state machine
+   * during the FinalizeUpgrade request, but ideally this has to be moved to
+   * individual calls that are going into the

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-12 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r503552977



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,328 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.*;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private  static final OmUpgradeAction NOOP = a -> {};
+
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;

Review comment:
   yes, I did not wanted to change the overall flow of messages and 
statusas that I came up with, but technically at this point we already done in 
the current implementation.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498861508



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};

Review comment:
   Ok, it turned out that if OMUpgradeAction has just one constants, that 
qualifies as a bad practice, as if an interface does contain just constants, 
that can have some disadvantages according to seemingly more clever ppl than 
me. :)
   
   I moved it back to OMUpgradeFinalizer, as due to the logic we need to 
preserve the OMUpgradeAction-ness of the NOOP to work well there, as we want to 
push OzoneManager instances to the executeAction, and the type system is 
against having a simple UpgradeAction as a NOOP. We can thin about adding this 
to OMLayoutFeature instead, but as we do not use it elsewhere at the moment, 
and because we use Optional and we provide just a null in case the finalize 
action is a noop, we do not need it elsewhere as it seems...
   
   An other approach can be an orElse(null), but with that we can even leave 
out the Optional in the OMLayoutFeature and LayoutFeature 
implementations/definitions, and instead checking if it is a NOOP, we can check 
for null.
   
   What do you think?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498802254



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;

Review comment:
   Good point, though making clientID volatile, does not really help to 
solve a race condition here I believe, so I propose to synchronize the whole 
reportStatus method.
   The reasons is that, if there is the original client that polls together 
with a client that does a takeover, then both can step on each others toes, as 
if for the original client we are already processing the message queue when an 
other client takes over, then returning status might change as well, and also 
there might be missing messages in the responses sent to the clients.
   
   Fortunately we don't have to worry about isDone and messages at least, as 
even if isDone is being updated late, and the final message is polled from the 
msgs queue with an IN_PROGRES state, the client in the next cycle gets a DONE 
state without

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498774350



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+
+
+
+  private class Worker implements Callable {
+private OzoneManager ozoneManager;
+
+

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498766375



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+
+
+
+  private class Worker implements Callable {
+private OzoneManager ozoneManager;
+
+

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498761138



##
File path: 
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/StorageInfo.java
##
@@ -150,6 +179,16 @@ private void verifyClusterId()
 }
   }
 
+  private void verifyUpgradingToLayoutVersion()
+  throws InconsistentStorageStateException {
+int upgradeMark = getUpgradingToLayoutVersion();
+if (upgradeMark != INVALID_LAYOUT_VERSION) {
+  throw new InconsistentStorageStateException("Ozone Manager died during"
+  + "a LayoutFeature upgrade.");

Review comment:
   fixed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498760977



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+
+
+
+  private class Worker implements Callable {

Review comment:
   Added doc to the Worker class as well.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-02 Thread GitBox



fapifta commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498760785



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};

Review comment:
   I moved it to OmUpgradeAction, I believe that is the best place. Thank 
you for catching this one.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

10 matches

Site Navigation

Mail list logo

Footer information