[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=297330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297330 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 19/Aug/19 17:47 Start Date: 19/Aug/19 17:47 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297330) Time Spent: 3.5h (was: 3h 20m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=297311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297311 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 19/Aug/19 17:22 Start Date: 19/Aug/19 17:22 Worklog Time Spent: 10m Work Description: hanishakoneru commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-522674063 Thank you @avijayanhwx. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 297311) Time Spent: 3h 20m (was: 3h 10m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296708
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 17/Aug/19 03:19
Start Date: 17/Aug/19 03:19
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1259: HDDS-1105
: Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314932606
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java
##
@@ -187,5 +229,119 @@ protected DBCheckpoint getOzoneManagerDBSnapshot() {
}
return null;
}
+
+ /**
+ * Update Local OM DB with new OM DB snapshot.
+ * @throws IOException
+ */
+ @VisibleForTesting
+ void updateReconOmDBWithNewSnapshot() throws IOException {
+// Obtain the current DB snapshot from OM and
+// update the in house OM metadata managed DB instance.
+DBCheckpoint dbSnapshot = getOzoneManagerDBSnapshot();
+if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ try {
+omMetadataManager.updateOmDB(dbSnapshot.getCheckpointLocation()
+.toFile());
+ } catch (IOException e) {
+LOG.error("Unable to refresh Recon OM DB Snapshot. ", e);
+ }
+} else {
+ LOG.error("Null snapshot location got from OM.");
+}
+ }
+
+ /**
+ * Get Delta updates from OM through RPC call and apply to local OM DB as
+ * well as accumulate in a buffer.
+ * @param fromSequenceNumber from sequence number to request from.
+ * @param omdbUpdatesHandler OM DB updates handler to buffer updates.
+ * @throws IOException when OM RPC request fails.
+ * @throws RocksDBException when writing to RocksDB fails.
+ */
+ @VisibleForTesting
+ void getAndApplyDeltaUpdatesFromOM(
+ long fromSequenceNumber, OMDBUpdatesHandler omdbUpdatesHandler)
+ throws IOException, RocksDBException {
+DBUpdatesRequest dbUpdatesRequest = DBUpdatesRequest.newBuilder()
+.setSequenceNumber(fromSequenceNumber).build();
+DBUpdatesWrapper dbUpdates = ozoneManagerClient.getDBUpdates(
+dbUpdatesRequest);
+if (null != dbUpdates) {
+ RDBStore rocksDBStore = (RDBStore)omMetadataManager.getStore();
+ RocksDB rocksDB = rocksDBStore.getDb();
+ LOG.debug("Number of updates received from OM : " +
+ dbUpdates.getData().size());
+ for (byte[] data : dbUpdates.getData()) {
+WriteBatch writeBatch = new WriteBatch(data);
+writeBatch.iterate(omdbUpdatesHandler);
+RDBBatchOperation rdbBatchOperation = new
RDBBatchOperation(writeBatch);
+rdbBatchOperation.commit(rocksDB, new WriteOptions());
+ }
+}
+ }
+
+ /**
+ * Based on current state of Recon's OM DB, we either get delta updates or
+ * full snapshot from Ozone Manager.
+ */
+ @VisibleForTesting
+ void syncDataFromOM() {
+long currentSequenceNumber = getCurrentOMDBSequenceNumber();
+boolean fullSnapshot = false;
+
+if (currentSequenceNumber <= 0) {
+ fullSnapshot = true;
+} else {
Review comment:
Good question @hanishakoneru . This is handled in OM side. Please check out
org.apache.hadoop.utils.db.RDBStore#getUpdatesSince.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296708)
Time Spent: 3h 10m (was: 3h)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296707
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 17/Aug/19 03:18
Start Date: 17/Aug/19 03:18
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1259: HDDS-1105
: Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314932586
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java
##
@@ -90,7 +90,7 @@ public static File getReconDbDir(Configuration conf, String
dirConfigKey) {
* @param destPath destination path to untar to.
* @throws IOException ioException
*/
- public static void untarCheckpointFile(File tarFile, Path destPath)
+ public void untarCheckpointFile(File tarFile, Path destPath)
Review comment:
Yes I agree. But we decided to make the ReconUtils non static so that we can
inject a mock instance of it and unit test cleanly, removing the need for
Powermocks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296707)
Time Spent: 3h (was: 2h 50m)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the given sequence number (due to WAL flushing), it can error out. In that
> case the FSCK server can fall back to getting the entire checkpoint snapshot
> implemented in HDDS-1085.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296698
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 17/Aug/19 02:54
Start Date: 17/Aug/19 02:54
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #1259:
HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from
Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314915078
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java
##
@@ -90,7 +90,7 @@ public static File getReconDbDir(Configuration conf, String
dirConfigKey) {
* @param destPath destination path to untar to.
* @throws IOException ioException
*/
- public static void untarCheckpointFile(File tarFile, Path destPath)
+ public void untarCheckpointFile(File tarFile, Path destPath)
Review comment:
Any particular reason for removing the static functionality? Util classes
are generally kept stateless.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296698)
Time Spent: 2h 40m (was: 2.5h)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the given sequence number (due to WAL flushing), it can error out. In that
> case the FSCK server can fall back to getting the entire checkpoint snapshot
> implemented in HDDS-1085.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296699
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 17/Aug/19 02:54
Start Date: 17/Aug/19 02:54
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #1259:
HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from
Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314914818
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java
##
@@ -187,5 +229,119 @@ protected DBCheckpoint getOzoneManagerDBSnapshot() {
}
return null;
}
+
+ /**
+ * Update Local OM DB with new OM DB snapshot.
+ * @throws IOException
+ */
+ @VisibleForTesting
+ void updateReconOmDBWithNewSnapshot() throws IOException {
+// Obtain the current DB snapshot from OM and
+// update the in house OM metadata managed DB instance.
+DBCheckpoint dbSnapshot = getOzoneManagerDBSnapshot();
+if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ try {
+omMetadataManager.updateOmDB(dbSnapshot.getCheckpointLocation()
+.toFile());
+ } catch (IOException e) {
+LOG.error("Unable to refresh Recon OM DB Snapshot. ", e);
+ }
+} else {
+ LOG.error("Null snapshot location got from OM.");
+}
+ }
+
+ /**
+ * Get Delta updates from OM through RPC call and apply to local OM DB as
+ * well as accumulate in a buffer.
+ * @param fromSequenceNumber from sequence number to request from.
+ * @param omdbUpdatesHandler OM DB updates handler to buffer updates.
+ * @throws IOException when OM RPC request fails.
+ * @throws RocksDBException when writing to RocksDB fails.
+ */
+ @VisibleForTesting
+ void getAndApplyDeltaUpdatesFromOM(
+ long fromSequenceNumber, OMDBUpdatesHandler omdbUpdatesHandler)
+ throws IOException, RocksDBException {
+DBUpdatesRequest dbUpdatesRequest = DBUpdatesRequest.newBuilder()
+.setSequenceNumber(fromSequenceNumber).build();
+DBUpdatesWrapper dbUpdates = ozoneManagerClient.getDBUpdates(
+dbUpdatesRequest);
+if (null != dbUpdates) {
+ RDBStore rocksDBStore = (RDBStore)omMetadataManager.getStore();
+ RocksDB rocksDB = rocksDBStore.getDb();
+ LOG.debug("Number of updates received from OM : " +
+ dbUpdates.getData().size());
+ for (byte[] data : dbUpdates.getData()) {
+WriteBatch writeBatch = new WriteBatch(data);
+writeBatch.iterate(omdbUpdatesHandler);
+RDBBatchOperation rdbBatchOperation = new
RDBBatchOperation(writeBatch);
+rdbBatchOperation.commit(rocksDB, new WriteOptions());
+ }
+}
+ }
+
+ /**
+ * Based on current state of Recon's OM DB, we either get delta updates or
+ * full snapshot from Ozone Manager.
+ */
+ @VisibleForTesting
+ void syncDataFromOM() {
+long currentSequenceNumber = getCurrentOMDBSequenceNumber();
+boolean fullSnapshot = false;
+
+if (currentSequenceNumber <= 0) {
+ fullSnapshot = true;
+} else {
Review comment:
Should we check the difference between currentSequenceNum and the latest
sequence number of OM DB before deciding whether to get full snapshot or delta
updates?
Lets say currentSequenceNum = 100 and latest sequence number is 1M. Would it
not be better to just get the full snapshot and replace the old one? (Not sure
if this is applicable to Recon)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296699)
Time Spent: 2h 50m (was: 2h 40m)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296389 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 16/Aug/19 16:20 Start Date: 16/Aug/19 16:20 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-522067507 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for branch | | +1 | mvninstall | 606 | trunk passed | | +1 | compile | 351 | trunk passed | | +1 | checkstyle | 60 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 783 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 146 | trunk passed | | 0 | spotbugs | 411 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 604 | trunk passed | | -0 | patch | 441 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 538 | the patch passed | | +1 | compile | 350 | the patch passed | | +1 | javac | 349 | the patch passed | | +1 | checkstyle | 69 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 645 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 85 | hadoop-ozone generated 6 new + 20 unchanged - 0 fixed = 26 total (was 20) | | +1 | findbugs | 618 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 289 | hadoop-hdds in the patch passed. | | -1 | unit | 2194 | hadoop-ozone in the patch failed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 7678 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 864257c97278 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9b8359b | | Default Java | 1.8.0_212 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/7/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/7/testReport/ | | Max. process+thread count | 5343 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296389) Time Spent: 2.5h (was: 2h 20m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. >
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=295721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-295721 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 15/Aug/19 20:10 Start Date: 15/Aug/19 20:10 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-521779783 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 597 | trunk passed | | +1 | compile | 352 | trunk passed | | +1 | checkstyle | 66 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 855 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 152 | trunk passed | | 0 | spotbugs | 416 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 606 | trunk passed | | -0 | patch | 454 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | +1 | mvninstall | 534 | the patch passed | | +1 | compile | 358 | the patch passed | | +1 | javac | 358 | the patch passed | | +1 | checkstyle | 65 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 670 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 86 | hadoop-ozone generated 6 new + 20 unchanged - 0 fixed = 26 total (was 20) | | +1 | findbugs | 619 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 296 | hadoop-hdds in the patch passed. | | -1 | unit | 1796 | hadoop-ozone in the patch failed. | | +1 | asflicense | 37 | The patch does not generate ASF License warnings. | | | | 7423 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 12a6161526fb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 46d6191 | | Default Java | 1.8.0_222 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/6/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/6/testReport/ | | Max. process+thread count | 4448 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 295721) Time Spent: 2h 20m (was: 2h 10m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL:
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=294081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294081 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 13/Aug/19 18:17 Start Date: 13/Aug/19 18:17 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#discussion_r313542436 ## File path: hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java ## @@ -114,7 +114,7 @@ public static final String OZONE_RECON_TASK_THREAD_COUNT_KEY = "ozone.recon.task.thread.count"; - public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 1; + public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 5; Review comment: Sure, I can address this later when the number of tasks becomes sizable. Right now, we have only 2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 294081) Time Spent: 2h 10m (was: 2h) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=294020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294020 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 13/Aug/19 17:07 Start Date: 13/Aug/19 17:07 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520921949 cc @mukul1987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 294020) Time Spent: 2h (was: 1h 50m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=293647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293647 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 13/Aug/19 06:07 Start Date: 13/Aug/19 06:07 Worklog Time Spent: 10m Work Description: swagle commented on pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#discussion_r313231490 ## File path: hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java ## @@ -114,7 +114,7 @@ public static final String OZONE_RECON_TASK_THREAD_COUNT_KEY = "ozone.recon.task.thread.count"; - public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 1; + public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 5; Review comment: This should be set to a number of cores multiple instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 293647) Time Spent: 1h 50m (was: 1h 40m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=293642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293642 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 13/Aug/19 06:03 Start Date: 13/Aug/19 06:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520700669 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 66 | Maven dependency ordering for branch | | +1 | mvninstall | 599 | trunk passed | | +1 | compile | 376 | trunk passed | | +1 | checkstyle | 79 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 867 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | trunk passed | | 0 | spotbugs | 460 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 665 | trunk passed | | -0 | patch | 513 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 38 | Maven dependency ordering for patch | | +1 | mvninstall | 561 | the patch passed | | +1 | compile | 370 | the patch passed | | +1 | javac | 370 | the patch passed | | +1 | checkstyle | 67 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 683 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 86 | hadoop-ozone generated 6 new + 20 unchanged - 0 fixed = 26 total (was 20) | | +1 | findbugs | 665 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 276 | hadoop-hdds in the patch passed. | | -1 | unit | 2393 | hadoop-ozone in the patch failed. | | +1 | asflicense | 57 | The patch does not generate ASF License warnings. | | | | 8282 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 2bbc990ac71d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 454420e | | Default Java | 1.8.0_222 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/5/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/5/testReport/ | | Max. process+thread count | 4636 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=293313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293313
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 12/Aug/19 18:54
Start Date: 12/Aug/19 18:54
Worklog Time Spent: 10m
Work Description: vivekratnavel commented on pull request #1259:
HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from
Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r313076918
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java
##
@@ -101,7 +106,7 @@ private void processEvent(int cfIndex, byte[] keyBytes,
byte[]
builder.setAction(action);
OMDBUpdateEvent event = builder.build();
LOG.info("Generated OM update Event for table : " + event.getTable()
Review comment:
Can we change this to debug?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 293313)
Time Spent: 1.5h (was: 1h 20m)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the given sequence number (due to WAL flushing), it can error out. In that
> case the FSCK server can fall back to getting the entire checkpoint snapshot
> implemented in HDDS-1085.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=293303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293303
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 12/Aug/19 18:29
Start Date: 12/Aug/19 18:29
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1259: HDDS-1105 : Add
mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r313065850
##
File path:
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/spi/impl/TestOzoneManagerServiceProviderImpl.java
##
@@ -158,4 +165,162 @@ public void testGetOzoneManagerDBSnapshot() throws
Exception {
assertTrue(checkpoint.getCheckpointLocation().toFile()
.listFiles().length == 2);
}
+
+ @Test
+ public void testGetAndApplyDeltaUpdatesFromOM() throws Exception {
+
+// Writing 2 Keys into a source OM DB and collecting it in a
+// DBUpdatesWrapper.
+OMMetadataManager sourceOMMetadataMgr = initializeNewOmMetadataManager();
+writeDataToOm(sourceOMMetadataMgr, "key_one");
+writeDataToOm(sourceOMMetadataMgr, "key_two");
+
+RocksDB rocksDB = ((RDBStore)sourceOMMetadataMgr.getStore()).getDb();
+TransactionLogIterator transactionLogIterator =
rocksDB.getUpdatesSince(0L);
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+while(transactionLogIterator.isValid()) {
+ TransactionLogIterator.BatchResult result =
+ transactionLogIterator.getBatch();
+ result.writeBatch().markWalTerminationPoint();
+ WriteBatch writeBatch = result.writeBatch();
+ dbUpdatesWrapper.addWriteBatch(writeBatch.data(),
+ result.sequenceNumber());
+ transactionLogIterator.next();
+}
+
+// OM Service Provider's Metadata Manager.
+OMMetadataManager omMetadataManager = initializeNewOmMetadataManager();
+
+OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+new OzoneManagerServiceProviderImpl(configuration,
+getTestMetadataManager(omMetadataManager),
+getMockTaskController(), new ReconUtils());
+OzoneManagerProtocol ozoneManagerProtocolMock =
+mock(OzoneManagerProtocol.class);
+when(ozoneManagerProtocolMock.getDBUpdates(any(OzoneManagerProtocolProtos
+.DBUpdatesRequest.class))).thenReturn(dbUpdatesWrapper);
+injectOzoneServiceProviderField(ozoneManagerServiceProvider,
+ozoneManagerProtocolMock, "ozoneManagerClient");
+
+OMDBUpdatesHandler updatesHandler =
+new OMDBUpdatesHandler(omMetadataManager);
+ozoneManagerServiceProvider.getAndApplyDeltaUpdatesFromOM(
+0L, updatesHandler);
+
+// In this method, we have to assert the "GET" part and the "APPLY" path.
+
+// Assert GET path --> verify if the OMDBUpdatesHandler picked up the 4
+// events ( 1 Vol PUT + 1 Bucket PUT + 2 Key PUTs).
+assertEquals(4, updatesHandler.getEvents().size());
+
+// Assert APPLY path --> Verify if the OM service provider's RocksDB got
+// the changes.
+String fullKey = omMetadataManager.getOzoneKey("sampleVol",
+"bucketOne", "key_one");
+assertTrue(ozoneManagerServiceProvider.getOMMetadataManagerInstance()
+.getKeyTable().isExist(fullKey));
+fullKey = omMetadataManager.getOzoneKey("sampleVol",
+"bucketOne", "key_two");
+assertTrue(ozoneManagerServiceProvider.getOMMetadataManagerInstance()
+.getKeyTable().isExist(fullKey));
+ }
+
+ @Test
+ public void testSyncDataFromOMFullSnapshot() throws Exception {
+
+// Empty OM DB to start with.
+ReconOMMetadataManager omMetadataManager = getTestMetadataManager(
+initializeEmptyOmMetadataManager());
+ReconTaskStatusDao reconTaskStatusDaoMock =
+mock(ReconTaskStatusDao.class);
+doNothing().when(reconTaskStatusDaoMock)
+.update(any(ReconTaskStatus.class));
+
+ReconTaskController reconTaskControllerMock = getMockTaskController();
+when(reconTaskControllerMock.getReconTaskStatusDao())
+.thenReturn(reconTaskStatusDaoMock);
+doNothing().when(reconTaskControllerMock)
+.reInitializeTasks(omMetadataManager);
+
+OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+new OzoneManagerServiceProviderImpl(configuration, omMetadataManager,
+reconTaskControllerMock, new ReconUtils());
+
+//Should trigger full snapshot request.
+ozoneManagerServiceProvider.syncDataFromOM();
+
+ArgumentCaptor captor =
+ArgumentCaptor.forClass(ReconTaskStatus.class);
+verify(reconTaskStatusDaoMock, times(1))
+.update(captor.capture());
+assertTrue(captor.getValue().getTaskName().equals("OM_DB_FULL_SNAPSHOT"));
+verify(reconTaskControllerMock, times(1))
+
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=293302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293302
]
ASF GitHub Bot logged work on HDDS-1105:
Author: ASF GitHub Bot
Created on: 12/Aug/19 18:27
Start Date: 12/Aug/19 18:27
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1259: HDDS-1105 : Add
mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r313064922
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/OzoneManagerServiceProvider.java
##
@@ -28,14 +26,9 @@
public interface OzoneManagerServiceProvider {
Review comment:
We should add lifecycle methods like stop() to gracefully allow providers
and tasks to exit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 293302)
Time Spent: 1h 10m (was: 1h)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
>
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the given sequence number (due to WAL flushing), it can error out. In that
> case the FSCK server can fall back to getting the entire checkpoint snapshot
> implemented in HDDS-1085.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=292870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292870 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 12/Aug/19 05:24 Start Date: 12/Aug/19 05:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520302152 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 74 | Maven dependency ordering for branch | | +1 | mvninstall | 645 | trunk passed | | +1 | compile | 379 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 844 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 419 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 621 | trunk passed | | -0 | patch | 452 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | +1 | mvninstall | 534 | the patch passed | | +1 | compile | 365 | the patch passed | | +1 | javac | 365 | the patch passed | | +1 | checkstyle | 79 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 647 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 84 | hadoop-ozone generated 6 new + 20 unchanged - 0 fixed = 26 total (was 20) | | +1 | findbugs | 622 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 287 | hadoop-hdds in the patch passed. | | -1 | unit | 3810 | hadoop-ozone in the patch failed. | | +1 | asflicense | 39 | The patch does not generate ASF License warnings. | | | | 9564 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestKeyInputStream | | | hadoop.ozone.TestContainerOperations | | | hadoop.ozone.client.rpc.Test2WayCommitInRatis | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux efb479a9583c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf5d895 | | Default Java | 1.8.0_212 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/4/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/4/testReport/ | | Max. process+thread count | 4104 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=292793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292793 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 12/Aug/19 00:51 Start Date: 12/Aug/19 00:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520275779 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 87 | Maven dependency ordering for branch | | +1 | mvninstall | 769 | trunk passed | | +1 | compile | 480 | trunk passed | | +1 | checkstyle | 73 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 923 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | trunk passed | | 0 | spotbugs | 433 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 629 | trunk passed | | -0 | patch | 473 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 552 | the patch passed | | +1 | compile | 380 | the patch passed | | +1 | javac | 380 | the patch passed | | +1 | checkstyle | 80 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 737 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 92 | hadoop-ozone generated 6 new + 20 unchanged - 0 fixed = 26 total (was 20) | | +1 | findbugs | 643 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 328 | hadoop-hdds in the patch passed. | | -1 | unit | 342 | hadoop-ozone in the patch failed. | | +1 | asflicense | 38 | The patch does not generate ASF License warnings. | | | | 6643 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.recon.tasks.TestReconTaskControllerImpl | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux d058aaa61991 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf5d895 | | Default Java | 1.8.0_222 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/3/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/3/testReport/ | | Max. process+thread count | 1226 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292793) Time Spent: 50m (was: 40m) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL:
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=292219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292219 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 09/Aug/19 18:54 Start Date: 09/Aug/19 18:54 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520027376 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292219) Time Spent: 40m (was: 0.5h) > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=291889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291889 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 09/Aug/19 08:16 Start Date: 09/Aug/19 08:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-519827114 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 7 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 610 | trunk passed | | +1 | compile | 390 | trunk passed | | +1 | checkstyle | 74 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 923 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 183 | trunk passed | | 0 | spotbugs | 486 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 727 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 33 | Maven dependency ordering for patch | | +1 | mvninstall | 573 | the patch passed | | +1 | compile | 397 | the patch passed | | +1 | javac | 397 | the patch passed | | +1 | checkstyle | 78 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 729 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 107 | hadoop-ozone generated 6 new + 13 unchanged - 0 fixed = 19 total (was 13) | | +1 | findbugs | 749 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 231 | hadoop-hdds in the patch failed. | | -1 | unit | 2840 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 8997 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 551b2213537e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 88ed1e0 | | Default Java | 1.8.0_222 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/2/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/2/testReport/ | | Max. process+thread count | 2985 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output |
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=291883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291883 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 09/Aug/19 08:11 Start Date: 09/Aug/19 08:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-519825535 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 7 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 80 | Maven dependency ordering for branch | | +1 | mvninstall | 624 | trunk passed | | +1 | compile | 387 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 917 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 174 | trunk passed | | 0 | spotbugs | 490 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 726 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 32 | Maven dependency ordering for patch | | +1 | mvninstall | 572 | the patch passed | | +1 | compile | 393 | the patch passed | | +1 | javac | 393 | the patch passed | | +1 | checkstyle | 78 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 727 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 102 | hadoop-ozone generated 6 new + 13 unchanged - 0 fixed = 19 total (was 13) | | +1 | findbugs | 755 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 356 | hadoop-hdds in the patch passed. | | -1 | unit | 2541 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 8861 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBCSID | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1259 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux a554e500637d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 88ed1e0 | | Default Java | 1.8.0_222 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/1/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/1/testReport/ | | Max. process+thread count | 3621 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1259/1/console | | versions | git=2.7.4
[jira] [Work logged] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=291826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291826 ] ASF GitHub Bot logged work on HDDS-1105: Author: ASF GitHub Bot Created on: 09/Aug/19 05:43 Start Date: 09/Aug/19 05:43 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259 Added mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager. Recon will make RPC calls to OM to get delta updates from the latest sequence number of its own OM snapshot DB. After applying the changes to its OM DB, the updates are passed on to the set of tasks that are "listening" on OM DB updates. Other than the core logic for the above, the patch : - Cleans up the unit test code - Fixes issues in OM DB updates sender - Removes the need for powermock in recon unit tests. - Added guice injection to Task framework. - Cleans up contract of Recon task interface. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 291826) Time Spent: 10m Remaining Estimate: 0h > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
