[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504746102
##
File path: tencent/python-client/OzoneS3.py
##
@@ -0,0 +1,89 @@
+import boto3
+from botocore.config import Config
+from botocore.exceptions import ClientError
+import xml.etree.ElementTree as ET
+from itertools import izip_longest
+from random import randrange
+
+# Ozone doesn't care access key and secret key in non-SafeMode
+AWS_ACCESS_KEY_ID = '12345678'
+AWS_SECRET_ACCESS_KEY = '12345678'
+
+# Ozone S3 gateway endpoints
+root = ET.parse('ozone_idex_config.xml').getroot()
+endpoints = root.findall('property/entry')
+SERVER_LIST = []
+for endpoint in endpoints:
+SERVER_LIST.append(endpoint.get('ozone.s3g.endpoint'))
+#endpoint = 'http://localhost:9878'
+
+# Ozone S3 bucket name
+bucket_name = 'idex_bucket'
+
+def round_robin(cur = [0]):
+length = len(SERVER_LIST)
+ret = SERVER_LIST[cur[0] % length]
+cur[0] = (cur[0] + 1) % length
+return ret
+
+def s3_resource():
+session = boto3.session.Session(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+endpoint = round_robin([randrange(len(SERVER_LIST))])
+s3 = session.resource(service_name='s3', endpoint_url=endpoint)
+print("current endpoint: " + endpoint)
+return s3
+
+class OzoneS3:
+@staticmethod
+def upload_file(file_path):
+try:
+s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+except Exception as e:
+print("retry one time. exception: " + str(e))
+s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+
+
+@staticmethod
+def upload_stream(key, byte_stream):
+try:
+s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+except Exception as e:
+s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+
+@staticmethod
+def download_file(key, output_file_path):
+try:
+s3_resource().Bucket(bucket_name).download_file(key,
output_file_path)
+except Exception as e:
+s3_resource().Bucket(bucket_name).download_file(key,
output_file_path)
+
+@staticmethod
+def download_stream(key):
+try:
+obj = s3_resource().Object(bucket_name, key)
+return obj.get()['Body'].read()
+except Exception as e:
+obj = s3_resource().Object(bucket_name, key)
+return obj.get()['Body'].read()
+
+@staticmethod
+def iter_n(iterable, n, fillvalue=None):
+args = [iter(iterable)] * n
Review comment:
Are these changes related to this PR? It has files named under directory
Tencent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504746102
##
File path: tencent/python-client/OzoneS3.py
##
@@ -0,0 +1,89 @@
+import boto3
+from botocore.config import Config
+from botocore.exceptions import ClientError
+import xml.etree.ElementTree as ET
+from itertools import izip_longest
+from random import randrange
+
+# Ozone doesn't care access key and secret key in non-SafeMode
+AWS_ACCESS_KEY_ID = '12345678'
+AWS_SECRET_ACCESS_KEY = '12345678'
+
+# Ozone S3 gateway endpoints
+root = ET.parse('ozone_idex_config.xml').getroot()
+endpoints = root.findall('property/entry')
+SERVER_LIST = []
+for endpoint in endpoints:
+SERVER_LIST.append(endpoint.get('ozone.s3g.endpoint'))
+#endpoint = 'http://localhost:9878'
+
+# Ozone S3 bucket name
+bucket_name = 'idex_bucket'
+
+def round_robin(cur = [0]):
+length = len(SERVER_LIST)
+ret = SERVER_LIST[cur[0] % length]
+cur[0] = (cur[0] + 1) % length
+return ret
+
+def s3_resource():
+session = boto3.session.Session(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+endpoint = round_robin([randrange(len(SERVER_LIST))])
+s3 = session.resource(service_name='s3', endpoint_url=endpoint)
+print("current endpoint: " + endpoint)
+return s3
+
+class OzoneS3:
+@staticmethod
+def upload_file(file_path):
+try:
+s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+except Exception as e:
+print("retry one time. exception: " + str(e))
+s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+
+
+@staticmethod
+def upload_stream(key, byte_stream):
+try:
+s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+except Exception as e:
+s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+
+@staticmethod
+def download_file(key, output_file_path):
+try:
+s3_resource().Bucket(bucket_name).download_file(key,
output_file_path)
+except Exception as e:
+s3_resource().Bucket(bucket_name).download_file(key,
output_file_path)
+
+@staticmethod
+def download_stream(key):
+try:
+obj = s3_resource().Object(bucket_name, key)
+return obj.get()['Body'].read()
+except Exception as e:
+obj = s3_resource().Object(bucket_name, key)
+return obj.get()['Body'].read()
+
+@staticmethod
+def iter_n(iterable, n, fillvalue=None):
+args = [iter(iterable)] * n
Review comment:
Are these changes related to this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504192790
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Sorry @maobaolong for the analysis, I missed it.
Yes, it makes sense. Thank You @GlenGeng and @maobaolong for the detailed
analysis.
>How about we avoid call IOUtils.closeQuietly(ozoneOutputStream); to throw
out the >NO_SUCH_MULTIPART_UPLOAD_ERROR
I am +1 for this. In this way, we can return the proper error code when
during close if it happens.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r459087101
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Thanks @maobaolong for reporting.
I will look into this, and get back to you.
From my initial analysis HDDS-3999 is caused by HDDS-3685
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458479671
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Because when No_SUCH_UPLOAD case is already handled.
```
else if (ex.getResult() == ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR) {
throw S3ErrorTable.newError(NO_SUCH_UPLOAD, uploadID);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458478856
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Do we know why omMultipartCommitUploadPartInfo is null in this case?
Because for any error, we throw exception, and that should catch the error,
and we should return an proper s3 error code from there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
