[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521898 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java ## @@ -239,8 +256,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, auditLog(auditLogger, buildAuditMessage(OMAction.ALLOCATE_BLOCK, auditMap, exception, getOmRequest().getUserInfo())); - - return omClientResponse; } + Review comment: unnessary blank line. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521569
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java
##
@@ -533,6 +533,26 @@ protected FileEncryptionInfo getFileEncryptionInfo(KeyArgs
keyArgs) {
return encryptionInfo;
}
+ /**
+ * Check volume quota in bytes.
+ * @param omVolumeArgs
+ * @param allocateSize
+ * @throws IOException
+ */
+ protected void checkVolumeQuotaInBytes(OmVolumeArgs omVolumeArgs,
+ long allocateSize) throws IOException {
+long usedBytes = omVolumeArgs.getUsedBytes().sum();
+long quotaInBytes = omVolumeArgs.getQuotaInBytes();
+if (quotaInBytes - usedBytes < allocateSize) {
+ throw new OMException("The DiskSpace quota of volume:"
+ + omVolumeArgs.getVolume() + "exceeded: quotaInBytes: "
+ + quotaInBytes + " Bytes but diskspace consumed: " + (usedBytes
+ + allocateSize) + " Bytes.",
+ OMException.ResultCodes.QUOTA_EXCEEDED);
+}
+
Review comment:
unnessary empty line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521253
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##
@@ -707,9 +707,71 @@ public void testPutKey() throws IOException {
}
}
+ @Test
+ public void testCheckUsedBytesQuota() throws IOException {
+String volumeName = UUID.randomUUID().toString();
+String bucketName = UUID.randomUUID().toString();
+OzoneVolume volume = null;
+
+String value = "sample value";
+int blockSize = (int) ozoneManager.getConfiguration().getStorageSize(
+OZONE_SCM_BLOCK_SIZE, OZONE_SCM_BLOCK_SIZE_DEFAULT, StorageUnit.BYTES);
+int valueLength = value.getBytes().length;
+int countException = 0;
+
+store.createVolume(volumeName);
+volume = store.getVolume(volumeName);
+// Set quota In Bytes for a smaller value
+store.getVolume(volumeName).setQuota(
+OzoneQuota.parseQuota("1 Bytes", 100));
+volume.createBucket(bucketName);
+OzoneBucket bucket = volume.getBucket(bucketName);
+
+// Test write key.
+// The remaining quota does not satisfy a block size, so the write fails.
+try {
+ writeKey(bucket, UUID.randomUUID().toString(), ONE, value, valueLength);
+} catch (IOException ex) {
+ countException++;
+ GenericTestUtils.assertExceptionContains("QUOTA_EXCEEDED", ex);
+}
+// Write failed, volume usedBytes should be 0
+Assert.assertEquals(0L, store.getVolume(volumeName).getUsedBytes());
+
+// Test write file.
+// The remaining quota does not satisfy a block size, so the write fails.
+try {
+ writeFile(bucket, UUID.randomUUID().toString(), ONE, value, 0);
+} catch (IOException ex) {
+ countException++;
+ GenericTestUtils.assertExceptionContains("QUOTA_EXCEEDED", ex);
+}
+// Write failed, volume usedBytes should be 0
+Assert.assertEquals(0L, store.getVolume(volumeName).getUsedBytes());
+
+// Write large key, test allocateBlock fails.
+store.getVolume(volumeName).setQuota(
+OzoneQuota.parseQuota(blockSize + "Bytes", 100));
+try {
+ OzoneOutputStream out = bucket.createKey(UUID.randomUUID().toString(),
+ valueLength, STAND_ALONE, ONE, new HashMap<>());
+ for(int i=0; i <= blockSize/value.length(); i++) {
+out.write(value.getBytes());
+ }
+ out.close();
+} catch (IOException ex) {
+ countException++;
+ GenericTestUtils.assertExceptionContains("QUOTA_EXCEEDED", ex);
+}
+// AllocateBlock failed, volume usedBytes should be 0
+Assert.assertEquals(0L, store.getVolume(volumeName).getUsedBytes());
+
+Assert.assertEquals(3, countException);
Review comment:
Can we add a multiple block write case here，so the first few block will
succeed， while the later block will fail due to quota exceed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521089
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
##
@@ -227,6 +232,18 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
} catch (IOException ex) {
omMetrics.incNumBlockAllocateCallFails();
exception = ex;
+ if (exception.toString().contains(
+ OMException.ResultCodes.QUOTA_EXCEEDED.toString())) {
+long keyAllocatedSpace = openKeyInfo.getLatestVersionLocations()
+.getLocationListCount() * ozoneManager.getScmBlockSize()
+* openKeyInfo.getFactor().getNumber();
+// Update usedBytes atomically. ErrorOMResponse does not persist the
DB,
+// so we update the cache first. The next time another key is written,
+// volume Args will be persisted to the DB.
+// TODO: There is a delay in updating DB in this way, and if necessary
+// we can modify the ErrorOMResponse to avoid it.
+omVolumeArgs.getUsedBytes().add(-keyAllocatedSpace);
Review comment:
If A object has multiple blocks, and quota_exceeded exception is thrown
out at the last block, shall we only deduct the block space size at this step?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495520762
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -277,6 +277,16 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
.collect(Collectors.toList());
omKeyInfo.appendNewBlocks(newLocationList, false);
+ omVolumeArgs = getVolumeInfo(omMetadataManager, volumeName);
+ omBucketInfo = getBucketInfo(omMetadataManager, volumeName, bucketName);
+ // check volume quota
+ long preAllocatedSpace = newLocationList.size()
+ * ozoneManager.getScmBlockSize()
+ * omKeyInfo.getFactor().getNumber();
+ if (omVolumeArgs.getQuotaInBytes() > OzoneConsts.QUOTA_RESET) {
Review comment:
Can we add a isQuotaInBytesSet check in OmVolumeArgs and use this
function instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
##
@@ -229,7 +229,9 @@ public String toString() {
NOT_SUPPORTED_OPERATION,
-PARTIAL_RENAME
+PARTIAL_RENAME,
+
+QUOTA_CHECK_ERROR
Review comment:
QUOTA_CHECK_ERROR -> QUOTA_EXCEED
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
##
@@ -229,7 +229,9 @@ public String toString() {
NOT_SUPPORTED_OPERATION,
-PARTIAL_RENAME
+PARTIAL_RENAME,
+
+QUOTA_CHECK_ERROR
Review comment:
QUOTA_CHECK_ERROR -> QUOTA_EXCEEDED
##
File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
##
@@ -314,6 +314,8 @@ enum Status {
PARTIAL_RENAME = 65;
+QUOTA_CHECK_ERROR = 66;
Review comment:
same as above.
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##
@@ -707,6 +707,62 @@ public void testPutKey() throws IOException {
}
}
+ @Test
+ public void testCheckUsedBytesQuota() throws IOException {
Review comment:
Can we add used bytes check in each test case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492575126
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##
@@ -707,6 +707,62 @@ public void testPutKey() throws IOException {
}
}
+ @Test
+ public void testCheckUsedBytesQuota() throws IOException {
Review comment:
Can we add used bytes check in each test case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492533073
##
File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
##
@@ -314,6 +314,8 @@ enum Status {
PARTIAL_RENAME = 65;
+QUOTA_CHECK_ERROR = 66;
Review comment:
same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
##
@@ -229,7 +229,9 @@ public String toString() {
NOT_SUPPORTED_OPERATION,
-PARTIAL_RENAME
+PARTIAL_RENAME,
+
+QUOTA_CHECK_ERROR
Review comment:
QUOTA_CHECK_ERROR -> QUOTA_EXCEEDED
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.
ChenSammi commented on a change in pull request #1434:
URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
##
@@ -229,7 +229,9 @@ public String toString() {
NOT_SUPPORTED_OPERATION,
-PARTIAL_RENAME
+PARTIAL_RENAME,
+
+QUOTA_CHECK_ERROR
Review comment:
QUOTA_CHECK_ERROR -> QUOTA_EXCEED
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
