[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248186 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 18:00 Start Date: 24/May/19 18:00 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495729519 Thank You, everyone, for the review. I will commit this shortly. @linyiqun provided an explanation for doing this way. If you have any more comments, we can open new Jira and address them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248186) Time Spent: 10h 10m (was: 10h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248176&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248176 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:51 Start Date: 24/May/19 17:51 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248176) Time Spent: 10h (was: 9h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248175&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248175 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:50 Start Date: 24/May/19 17:50 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495729519 Thank You, everyone, for the review. I will commit this shortly. @linyiqun provided my reason for doing this way. If you have any more comments, we can open new Jira and address them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248175) Time Spent: 9h 50m (was: 9h 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248169&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248169 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:44 Start Date: 24/May/19 17:44 Worklog Time Spent: 10m Work Description: arp7 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495727082 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248169) Time Spent: 9h 40m (was: 9.5h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 40m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247802 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 01:04 Start Date: 24/May/19 01:04 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495436029 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 33 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 504 | trunk passed | | +1 | compile | 274 | trunk passed | | +1 | checkstyle | 85 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 876 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | trunk passed | | 0 | spotbugs | 285 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 469 | trunk passed | | -0 | patch | 351 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 482 | the patch passed | | +1 | compile | 281 | the patch passed | | +1 | javac | 281 | the patch passed | | +1 | checkstyle | 93 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 677 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | the patch passed | | +1 | findbugs | 500 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 149 | hadoop-hdds in the patch failed. | | -1 | unit | 1405 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 10855 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestContainerOperations | | | hadoop.ozone.TestContainerStateMachineIdempotency | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.web.client.TestKeys | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/14/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 63f6212146ca 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 460ba7f | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/14/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/14/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/14/testReport/ | | Max. process+thread count | 5333 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/14/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247802) Time Spent: 9.5h (was: 9h 20m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Su
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247782 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 23/May/19 23:44 Start Date: 23/May/19 23:44 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495422900 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 504 | trunk passed | | +1 | compile | 251 | trunk passed | | +1 | checkstyle | 65 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 794 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 301 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 498 | trunk passed | | -0 | patch | 346 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 463 | the patch passed | | +1 | compile | 284 | the patch passed | | +1 | javac | 284 | the patch passed | | +1 | checkstyle | 92 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 690 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | the patch passed | | +1 | findbugs | 491 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 163 | hadoop-hdds in the patch failed. | | -1 | unit | 1162 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 6047 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/15/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 73434c5070d4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a0e7dd | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/15/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/15/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/15/testReport/ | | Max. process+thread count | 4869 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/15/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247782) Time Spent: 9h 20m (was: 9h 10m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-re
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247758 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 23/May/19 22:50 Start Date: 23/May/19 22:50 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495412132 Test failures are fixed in the last commit. Now tests are passing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247758) Time Spent: 9h 10m (was: 9h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247757 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 23/May/19 22:47 Start Date: 23/May/19 22:47 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495411571 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 527 | trunk passed | | +1 | compile | 279 | trunk passed | | +1 | checkstyle | 83 | trunk passed | | +1 | mvnsite | 1 | trunk passed | | +1 | shadedclient | 866 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 152 | trunk passed | | 0 | spotbugs | 287 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 480 | trunk passed | | -0 | patch | 327 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 464 | the patch passed | | +1 | compile | 253 | the patch passed | | +1 | javac | 253 | the patch passed | | +1 | checkstyle | 74 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 621 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 147 | the patch passed | | +1 | findbugs | 514 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 159 | hadoop-hdds in the patch failed. | | -1 | unit | 108 | hadoop-ozone in the patch failed. | | +1 | asflicense | 39 | The patch does not generate ASF License warnings. | | | | 4951 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithDummyResponse | | | hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c001d3ecddda 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6a0e7dd | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/13/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/13/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/13/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/13/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247757) Time Spent: 9h (was: 8h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spe
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247696 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 23/May/19 21:14 Start Date: 23/May/19 21:14 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495387480 Thank You @arp7 for the review. Addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247696) Time Spent: 8h 50m (was: 8h 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 8h 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247694
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 21:13
Start Date: 23/May/19 21:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287136965
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247695
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 21:13
Start Date: 23/May/19 21:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287136985
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respo
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247692&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247692
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 21:13
Start Date: 23/May/19 21:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287136934
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247693
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 21:13
Start Date: 23/May/19 21:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287136952
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247658&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247658
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 20:00
Start Date: 23/May/19 20:00
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287110363
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
Review comment:
If multiple test methods are calling setup(), it should probably be a
`@Before` method.
-
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247657
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 20:00
Start Date: 23/May/19 20:00
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287110914
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+ new
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247659
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 20:00
Start Date: 23/May/19 20:00
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287111022
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+ new
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247656
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 23/May/19 20:00
Start Date: 23/May/19 20:00
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r287111228
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,397 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Queue;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.hadoop.ozone.om.response.OMBucketCreateResponse;
+import org.apache.hadoop.ozone.om.response.OMBucketDeleteResponse;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.utils.db.BatchOperation;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.response.OMVolumeCreateResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.VolumeList;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.Time;
+
+
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.junit.Assert.fail;
+
+/**
+ * This class tests OzoneManagerDouble Buffer.
+ */
+public class TestOzoneManagerDoubleBuffer {
+
+ private OMMetadataManager omMetadataManager;
+ private OzoneManagerDoubleBuffer doubleBuffer;
+ private AtomicLong trxId = new AtomicLong(0);
+
+ @Rule
+ public TemporaryFolder folder = new TemporaryFolder();
+
+ private void setup() throws IOException {
+OzoneConfiguration configuration = new OzoneConfiguration();
+configuration.set(OZONE_METADATA_DIRS,
+folder.newFolder().getAbsolutePath());
+omMetadataManager =
+new OmMetadataManagerImpl(configuration);
+doubleBuffer = new OzoneManagerDoubleBuffer(omMetadataManager);
+ }
+
+ private void stop() {
+doubleBuffer.stop();
+ }
+
+ @Test(timeout = 300_000)
+ public void testDoubleBufferWithDummyResponse() throws Exception {
+try {
+ setup();
+ String volumeName = UUID.randomUUID().toString();
+ int bucketCount = 100;
+ for (int i=0; i < bucketCount; i++) {
+doubleBuffer.add(createDummyBucketResponse(volumeName,
+UUID.randomUUID().toString()), trxId.incrementAndGet());
+ }
+ GenericTestUtils.waitFor(() ->
+ doubleBuffer.getFlushedTransactionCount() == bucketCount, 100,
+ 12);
+ Assert.assertTrue(omMetadataManager.countRowsInTable(
+ omMetadataManager.getBucketTable()) == (bucketCount));
+ Assert.assertTrue(doubleBuffer.getFlushIterations() > 0);
+} finally {
+ stop();
+}
+ }
+
+
+ @Test(timeout = 300_000)
+ public void testDoubleBuffer() throws Exception {
+// This test checks whether count in tables are correct or not.
+testDoubleBuffer(1, 10);
+testDoubleBuffer(10, 100);
+testDoubleBuffer(100, 100);
+testDoubleBuffer(1000, 1000);
+ }
+
+
+
+ @Test
+ public void testDoubleBufferWithMixOfTransactions() throws Exception {
+// This test checks count, data in table is correct or not.
+try {
+ setup();
+
+ Queue< OMBucketCreateResponse > bucketQueue =
+ new ConcurrentLinkedQueue<>();
+ Queue< OMBucketDeleteResponse > deleteBucketQueue =
+ new
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247195 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 23/May/19 00:34 Start Date: 23/May/19 00:34 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495023686 Test failures are not related to this patch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247195) Time Spent: 7h 20m (was: 7h 10m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h 20m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=247096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247096 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 22/May/19 22:21 Start Date: 22/May/19 22:21 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494995987 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 247096) Time Spent: 7h 10m (was: 7h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246565 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 22/May/19 03:47 Start Date: 22/May/19 03:47 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494641291 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 83 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 850 | trunk passed | | +1 | compile | 354 | trunk passed | | +1 | checkstyle | 110 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1158 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 195 | trunk passed | | 0 | spotbugs | 301 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 539 | trunk passed | | -0 | patch | 356 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 497 | the patch passed | | +1 | compile | 278 | the patch passed | | +1 | javac | 278 | the patch passed | | +1 | checkstyle | 83 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 770 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 456 | the patch passed | | +1 | findbugs | 1236 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 478 | hadoop-hdds in the patch failed. | | -1 | unit | 373 | hadoop-ozone in the patch failed. | | +1 | asflicense | 205 | The patch does not generate ASF License warnings. | | | | 7876 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.security.TestOzoneDelegationTokenSecretManager | | | hadoop.ozone.om.TestOzoneManagerLock | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/11/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 33142e87d779 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ef1cc72 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/11/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/11/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/11/testReport/ | | Max. process+thread count | 1276 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/11/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246565) Time Spent: 7h (was: 6h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 7h > Remaining Es
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246457&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246457 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 23:31 Start Date: 21/May/19 23:31 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494585866 > I can see the tradeoffs of both approaches. Why don't benchmark this and see the maximum length this queue grows under load?. One suggestion based on @linyiqun thought is that it would be good to have a counter to see the max length of this queue. So we can at least identify and track this issue if it really pops up in the field. Thank You @anuengineer for the review. Created HDDS-1579 to create metrics for OMDoubleBuffer. Will address this as part of a new jira. In t this jira only added to logging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246457) Time Spent: 6h 50m (was: 6h 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246452 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 23:17 Start Date: 21/May/19 23:17 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494585866 > I can see the tradeoffs of both approaches. Why don't benchmark this and see the maximum length this queue grows under load?. One suggestion based on @linyiqun thought is that it would be good to have a counter to see the max length of this queue. So we can at least identify and track this issue if it really pops up in the field. Created HDDS-1579 to create metrics for OMDoubleBuffer. Will address this as part of a new jira. In t this jira only added to logging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246452) Time Spent: 6h 40m (was: 6.5h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246430&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246430 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 22:42 Start Date: 21/May/19 22:42 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494585866 > I can see the tradeoffs of both approaches. Why don't benchmark this and see the maximum length this queue grows under load?. One suggestion based on @linyiqun thought is that it would be good to have a counter to see the max length of this queue. So we can at least identify and track this issue if it really pops up in the field. Created HDDS-1579 to create metrics for OMDoubleBuffer. Will address this as part of a new jira. Int this jira only added to logging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246430) Time Spent: 6.5h (was: 6h 20m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6.5h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246429&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246429 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 22:38 Start Date: 21/May/19 22:38 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494584830 Thank You @arp7 for the review and offline discussion. Addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246429) Time Spent: 6h 20m (was: 6h 10m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h 20m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246369
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 20:54
Start Date: 21/May/19 20:54
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286206703
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransactionCount
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246341 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 19:58 Start Date: 21/May/19 19:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-494535515 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 64 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 572 | trunk passed | | +1 | compile | 275 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 853 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 142 | trunk passed | | 0 | spotbugs | 331 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 532 | trunk passed | | -0 | patch | 378 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 540 | the patch passed | | +1 | compile | 307 | the patch passed | | +1 | javac | 307 | the patch passed | | +1 | checkstyle | 77 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 718 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 165 | the patch passed | | +1 | findbugs | 581 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 186 | hadoop-hdds in the patch failed. | | -1 | unit | 1535 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 6851 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b87f9a979c66 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 5906268 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/6/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/6/testReport/ | | Max. process+thread count | 4718 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246341) Time Spent: 6h (was: 5h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 6h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in Oz
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246323&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246323
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 19:24
Start Date: 21/May/19 19:24
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286186378
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
Review comment:
Done. Updated the code to use wait, notify.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246323)
Time Spent: 5h 50m (was: 5h 40m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We add entry to current buffer, and we che
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246277
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:08
Start Date: 21/May/19 18:08
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286156738
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
Review comment:
We will spin in a tight loop otherwise E.g. when system is idle. That will
consume CPU and power. It will be better to use a wait and notify model.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246277)
Time Spent: 5h 40m (was: 5.5h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246271&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246271
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:03
Start Date: 21/May/19 18:03
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286154772
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
Review comment:
Updated the method name to addToDbBatch as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246271)
Time Spent: 5.5h (was: 5h 20m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instea
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246270&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246270
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:03
Start Date: 21/May/19 18:03
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286154605
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
Review comment:
Flushing to disk is happening this while loop, so during flush, this while
loop will not be iterating.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246270)
Time Spent: 5h 20m (was: 5h 10m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other i
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246260&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246260 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 21/May/19 18:02 Start Date: 21/May/19 18:02 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r286154101 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java ## @@ -0,0 +1,212 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.om.ratis; + +import java.io.IOException; +import java.util.Iterator; +import java.util.Queue; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.atomic.AtomicLong; + +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import org.apache.hadoop.ozone.om.OMMetadataManager; +import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry; +import org.apache.hadoop.ozone.om.response.OMClientResponse; +import org.apache.hadoop.util.Daemon; +import org.apache.hadoop.util.ExitUtil; +import org.apache.hadoop.utils.db.BatchOperation; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A double-buffer for OM requests. Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246260) Time Spent: 5h (was: 4h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246268&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246268
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:02
Start Date: 21/May/19 18:02
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286154308
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
Review comment:
I think we don't need this flag, as sync to DB is also happening in the same
thread.
Removed the usage of the flag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246268)
Time Spent: 5h 10m (was: 5h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We add entry to current buffer, and we check if another flush
> call is outstanding. If not, we flush to disk Otherwise we add entries to
> otherBuffer while sync is happening.
>
> In this if sync is happening, we shall add new requests to other buffer and
> when we can sync we use *RocksDB batch commit to sync to disk, instead of
> rocksdb put.*
>
> Note: If flush to disk is failed on any OM, we shall terminate the
> OzoneManager, so that OM DB’s will not diverge. Flush failure should be
> considered as catastrophic failure.
>
> Scope of this Jira is to add DoubleBuffer implementation, integrating to
> current OM will be done in further jira's.
>
--
This message was sent by Atlassian
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246259&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246259
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:01
Start Date: 21/May/19 18:01
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286153952
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact In
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=246256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246256
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 21/May/19 18:01
Start Date: 21/May/19 18:01
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r286153800
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
Review comment:
Added this try because it is running inside forEach.
Modified the logic, and moved common code to a new function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 246256)
Time Spent: 4h 40m (was: 4.5h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> T
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244790
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 20/May/19 04:04
Start Date: 20/May/19 04:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285423185
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
Review comment:
Can you add a short description of this flag?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244790)
Time Spent: 4h 20m (was: 4h 10m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We add entry to current buffer, and we check if another flush
> call is outstanding. If not, we flush to disk Otherwise we add entries to
> otherBuffer while sync is happening.
>
> In this if sync is happening, we shall add new requests to other buffer and
> when we can sync we use *RocksDB batch commit to sync to disk, instead of
> rocksdb put.*
>
> Note: If flush to disk is failed on any OM, we shall terminate the
> OzoneManager, so that OM DB’s will not diverge. Flush failure should be
> considered as catastrophic failure.
>
> Scope of this Jira is to add DoubleBuffer implementation, integrating to
> current OM will be done in further jira's.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244789&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244789
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 20/May/19 04:04
Start Date: 20/May/19 04:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285423395
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
Review comment:
We should not assume we are using RocksDB here. We may need to update the
function name to addToDbBatch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244789)
Time Spent: 4h 20m (was: 4h 10m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244791 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 20/May/19 04:04 Start Date: 20/May/19 04:04 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r285422717 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java ## @@ -0,0 +1,212 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.om.ratis; + +import java.io.IOException; +import java.util.Iterator; +import java.util.Queue; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.atomic.AtomicLong; + +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import org.apache.hadoop.ozone.om.OMMetadataManager; +import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry; +import org.apache.hadoop.ozone.om.response.OMClientResponse; +import org.apache.hadoop.util.Daemon; +import org.apache.hadoop.util.ExitUtil; +import org.apache.hadoop.utils.db.BatchOperation; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A double-buffer for OM requests. Review comment: @bharatviswa504 can you add a slightly more descriptive comment. Maybe 3-4 lines on the purpose of the double buffer and some high level implementation note. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244791) Time Spent: 4.5h (was: 4h 20m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244787
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 20/May/19 04:04
Start Date: 20/May/19 04:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285423558
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
Review comment:
Let's discuss the synchronization. It probably needs to be fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244787)
Time Spent: 4h 10m (was: 4h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is curr
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244786
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 20/May/19 04:04
Start Date: 20/May/19 04:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285419063
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
Review comment:
Agreed, this should be a debug log.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this servic
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244788&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244788
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 20/May/19 04:04
Start Date: 20/May/19 04:04
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285423337
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
Review comment:
Looks like this is spinning the CPU in a tight loop. It should wait until
the current flush completes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244788)
Time Spent: 4h 10m (was: 4h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuff
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244780 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 20/May/19 03:38 Start Date: 20/May/19 03:38 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r285420994 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketDeleteResponse.java ## @@ -0,0 +1,57 @@ +/** Review comment: Added them for testing purpose to test DoubleBuffer Implementation. Actual usage and the classes will be changed further according to needs of HA in further Jira. In this PR #827 handled bucket related operations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244780) Time Spent: 4h (was: 3h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244779 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 20/May/19 03:37 Start Date: 20/May/19 03:37 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r285420979 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketCreateResponse.java ## @@ -0,0 +1,52 @@ +/** Review comment: Added them for testing purpose to test DoubleBuffer Implementation. Actual usage and the classes will be changed further according to needs of HA in further Jira. In this PR https://github.com/apache/hadoop/pull/827 handled bucket related operations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244779) Time Spent: 3h 50m (was: 3h 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244756&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244756 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 20/May/19 03:13 Start Date: 20/May/19 03:13 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r285418524 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketCreateResponse.java ## @@ -0,0 +1,52 @@ +/** Review comment: @bharatviswa504 why is this change a part of the DoubleBuffer patch? This patch should just introduce the DoubleBuffer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244756) Time Spent: 3h 40m (was: 3.5h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244755&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244755 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 20/May/19 03:13 Start Date: 20/May/19 03:13 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#discussion_r285418551 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketDeleteResponse.java ## @@ -0,0 +1,57 @@ +/** Review comment: Same here. This should be moved to a subsequent patch right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 244755) Time Spent: 3.5h (was: 3h 20m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244524&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244524
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 18/May/19 10:29
Start Date: 18/May/19 10:29
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285337258
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
Review comment:
Nit: Should we make this a Debug instead of info. There will be many in OM.
But on the other hand, this might be ok, not sure. @arp7 What do think ? will
this flood the logs and slow us down? Based on Namenode experience what would
be your call?
-
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244521&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244521
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 18/May/19 10:28
Start Date: 18/May/19 10:28
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285337258
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
Review comment:
Nit: Should we make this a Debug instead of info. There will be many in OM.
But on the other hand, this might be ok, not sure. @arp7 What do think ? will
this flood the logs and slow us down like, based on Namenode experience what
would be your call?
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244520&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244520
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 18/May/19 10:28
Start Date: 18/May/19 10:28
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285337147
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
Review comment:
Nit: Since the Error handler is same you can put both addToRockDB and
commitBatchOperation in a single try block and follow with the exception block,
do change the error message to be more specific and leave the code as is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244520)
Time Spent: 3h (was: 2h 50m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bh
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244522
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 18/May/19 10:28
Start Date: 18/May/19 10:28
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285337361
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransactio
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244413&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244413
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:58
Start Date: 17/May/19 22:58
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285311016
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransac
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244412&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244412
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:57
Start Date: 17/May/19 22:57
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285311016
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransac
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244410
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:54
Start Date: 17/May/19 22:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285310539
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
My thought process of doing this way is, background flush will be
continuously happening if currentBuffer has any entries so that we don't wait
for the buffer to be full. So, that when a restart happens we don't need to
apply the transactions again to OM DB.
In the other way as suggested if we do, when the buffer is full, swap and
start flush, and keep further transactions adding to the currentBuffer, but if
flush is taking much time, and this buffer happened to be full, we should wait
until the first flush to be completed, and we might block all further
transactions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244410)
Time Spent: 2h 20m (was: 2h 10m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=244411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-244411
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 22:54
Start Date: 17/May/19 22:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r285310539
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
My thought process of doing this way is, background flush will be
continuously happening if currentBuffer has any entries so that we don't wait
for the buffer to be full. So, that when a restart happens we don't need to
apply the transactions again to OM DB. And in this way we keep disks busy.
In the other way as suggested if we do, when the buffer is full, swap and
start flush, and keep further transactions adding to the currentBuffer, but if
flush is taking much time, and this buffer happened to be full, we should wait
until the first flush to be completed, and we might block all further
transactions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 244411)
Time Spent: 2.5h (was: 2h 20m)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shal
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243792&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243792
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 02:21
Start Date: 17/May/19 02:21
Worklog Time Spent: 10m
Work Description: linyiqun commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284958539
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransactionCo
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243791
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 17/May/19 02:21
Start Date: 17/May/19 02:21
Worklog Time Spent: 10m
Work Description: linyiqun commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284957277
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
Actually I mean not to start flushTransactions thread in OMDoubleBuffer.
It's would be better to let the caller of function flushTransactions outside
of the class OMDoubleBuffer. How the flush transactions behavior should be
triggered by outside not by itself I think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243791)
Time Spent: 2h 10m (was: 2h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243453&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243453
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:18
Start Date: 16/May/19 16:18
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284787939
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransac
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243452
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:16
Start Date: 16/May/19 16:16
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284787939
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransac
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243450
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:15
Start Date: 16/May/19 16:15
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284788571
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
Your suggestion here is to Make Deamon thread logic in a separate class, and
instantiate that class in OMDoubleBuffer and start the thread in OMDoubleBuffer?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243450)
Time Spent: 1h 40m (was: 1.5h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We add entry to current buffer, and we check if another flush
> call is outstanding. If not, we flush to disk O
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243448
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:13
Start Date: 16/May/19 16:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #810:
HDDS-1512. Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284787939
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransac
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243444
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:10
Start Date: 16/May/19 16:10
Worklog Time Spent: 10m
Work Description: linyiqun commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284786002
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
+
+ }
+
+ /**
+ * Runs in a background thread and batches the transaction in currentBuffer
+ * and commit to DB.
+ */
+ private void flushTransactions() {
+while(true) {
+ if (canFlush()) {
+syncToDB = true;
+setReadyBuffer();
+final BatchOperation batchOperation = omMetadataManager.getStore()
+.initBatchOperation();
+
+readyBuffer.iterator().forEachRemaining((entry) -> {
+ try {
+entry.getResponse().addToRocksDBBatch(omMetadataManager,
+batchOperation);
+ } catch (IOException ex) {
+// During Adding to RocksDB batch entry got an exception.
+// We should terminate the OM.
+String message = "During flush to DB encountered error " +
+ex.getMessage();
+ExitUtil.terminate(1, message);
+ }
+});
+
+try {
+ omMetadataManager.getStore().commitBatchOperation(batchOperation);
+} catch (IOException ex) {
+ // During flush to rocksdb got an exception.
+ // We should terminate the OM.
+ String message = "During flush to DB encountered error " +
+ ex.getMessage();
+ ExitUtil.terminate(1, message);
+}
+
+int flushedTransactionsSize = readyBuffer.size();
+flushedTransactionCount.addAndGet(flushedTransactionsSize);
+flushIterations.incrementAndGet();
+
+LOG.info("Sync Iteration {} flushed transactions in this iteration{}",
+flushIterations.get(), flushedTransactionsSize);
+readyBuffer.clear();
+syncToDB = false;
+// TODO: update the last updated index in OzoneManagerStateMachine.
+ }
+}
+ }
+
+ /**
+ * Returns the flushed transaction count to OM DB.
+ * @return flushedTransactionCo
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[
https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=243443&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243443
]
ASF GitHub Bot logged work on HDDS-1512:
Author: ASF GitHub Bot
Created on: 16/May/19 16:10
Start Date: 16/May/19 16:10
Worklog Time Spent: 10m
Work Description: linyiqun commented on pull request #810: HDDS-1512.
Implement DoubleBuffer in OzoneManager.
URL: https://github.com/apache/hadoop/pull/810#discussion_r284784362
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.ratis.helpers.DoubleBufferEntry;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.util.Daemon;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A double-buffer for OM requests.
+ */
+public class OzoneManagerDoubleBuffer {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneManagerDoubleBuffer.class.getName());
+
+ private TransactionBuffer currentBuffer;
+ private TransactionBuffer readyBuffer;
+ private Daemon daemon;
+ private volatile boolean syncToDB;
+ private final OMMetadataManager omMetadataManager;
+ private AtomicLong flushedTransactionCount = new AtomicLong(0);
+ private AtomicLong flushIterations = new AtomicLong(0);
+
+ public OzoneManagerDoubleBuffer(OMMetadataManager omMetadataManager) {
+this.currentBuffer = new TransactionBuffer();
+this.readyBuffer = new TransactionBuffer();
+this.omMetadataManager = omMetadataManager;
+
+// Daemon thread which runs in back ground and flushes transactions to DB.
+daemon = new Daemon(this::flushTransactions);
+daemon.start();
Review comment:
I prefer to separate the daemon thread from the OM double buffer, so that
this class will be more flexible to use.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243443)
Time Spent: 1h 10m (was: 1h)
> Implement DoubleBuffer in OzoneManager
> --
>
> Key: HDDS-1512
> URL: https://issues.apache.org/jira/browse/HDDS-1512
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This Jira is created to implement DoubleBuffer in OzoneManager to flush
> transactions to OM DB.
>
> h2. Flushing Transactions to RocksDB:
> We propose using an implementation similar to the HDFS EditsDoubleBuffer. We
> shall flush RocksDB transactions in batches, instead of current way of using
> rocksdb.put() after every operation. At a given time only one batch will be
> outstanding for flush while newer transactions are accumulated in memory to
> be flushed later.
>
> In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is
> readyBuffer. We add entry to current buffer, and we check if another flush
> call is outstanding. If not, we flush to disk Otherwise we add entries to
> otherBuffer while syn
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=241688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241688 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 14/May/19 12:24 Start Date: 14/May/19 12:24 Worklog Time Spent: 10m Work Description: elek commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-492215253 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241688) Time Spent: 1h (was: 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=241304&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241304 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 13/May/19 20:50 Start Date: 13/May/19 20:50 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-491981350 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 31 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 425 | trunk passed | | +1 | compile | 195 | trunk passed | | +1 | checkstyle | 54 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 891 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 122 | trunk passed | | 0 | spotbugs | 239 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 425 | trunk passed | | -0 | patch | 266 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 390 | the patch passed | | +1 | compile | 198 | the patch passed | | +1 | javac | 198 | the patch passed | | +1 | checkstyle | 56 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 719 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 121 | the patch passed | | +1 | findbugs | 438 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 144 | hadoop-hdds in the patch failed. | | -1 | unit | 1160 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 6571 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 13d73a260870 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 389e640 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/5/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/5/testReport/ | | Max. process+thread count | 4791 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241304) Time Spent: 50m (was: 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation simi
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=241279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241279 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 13/May/19 20:43 Start Date: 13/May/19 20:43 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-491978630 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 103 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 515 | trunk passed | | +1 | compile | 212 | trunk passed | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 896 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 130 | trunk passed | | 0 | spotbugs | 237 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 433 | trunk passed | | -0 | patch | 264 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 408 | the patch passed | | +1 | compile | 199 | the patch passed | | +1 | javac | 199 | the patch passed | | +1 | checkstyle | 54 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 687 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 118 | the patch passed | | +1 | findbugs | 428 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 160 | hadoop-hdds in the patch failed. | | -1 | unit | 1794 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 6309 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-810/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/810 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 55df0f9aa4c8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 389e640 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/4/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/4/testReport/ | | Max. process+thread count | 5061 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-810/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241279) Time Spent: 40m (was: 0.5h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=241233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241233 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 13/May/19 18:57 Start Date: 13/May/19 18:57 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-491944348 Thank You @jiwq for the review. I have addressed your review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241233) Time Spent: 0.5h (was: 20m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=239929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239929 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 09/May/19 20:46 Start Date: 09/May/19 20:46 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-491059971 cc @anuengineer (I am not able to add you in reviewer, tagging here to notify about this) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239929) Time Spent: 20m (was: 10m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=239926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-239926 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 09/May/19 20:44 Start Date: 09/May/19 20:44 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810 1. In this Jira added double buffer implementation. (Not integrated to OM, this will be done in further jira's) 2. Added a few response classes to give an idea, how this will be integrated with OM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 239926) Time Spent: 10m Remaining Estimate: 0h > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
