[
https://issues.apache.org/jira/browse/HDDS-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17949870#comment-17949870
]
Kohei Sugihara commented on HDDS-12123:
---------------------------------------
I will share the logs with the --debug flag. The
AWS_REQUEST_CHECKSUM_CALCULATION=when_required environment variable can be used
as another workaround for the latest awscli (v1.40.9). We will share two
patterns.
{code:java}
% pip3 install --upgrade 'awscli<2'
% export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
% aws --version
aws-cli/1.40.9 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
botocore/1.38.10
% aws s3 --endpoint ... cp ~/bin-1m.dat s3://ksugihara/bin-1m.dat
upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat {code}
Logs:
w/ the envvar
{code:java}
2025-05-07 10:20:09,850 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG -
CanonicalRequest:
PUT
/ksugihara/bin-1m.dathost:redacted
x-amz-content-sha256:620f26da9d92b8d98a038ced8ef1eca4af163216aa83e24c7a4431f55964f15f
x-amz-date:20250507T012009Zhost;x-amz-content-sha256;x-amz-date
620f26da9d92b8d98a038ced8ef1eca4af163216aa83e24c7a4431f55964f15f
...
2025-05-07 10:20:09,850 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG -
Sending http request: <AWSPreparedRequest stream_output=False, method=PUT,
url=https://redacted/ksugihara/bin-1m.dat, headers={'User-Agent':
b'aws-cli/1.40.9 md/Botocore#1.38.10 ua/2.1
os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.10.12
md/pyimpl#CPython m/N cfg/retry-mode#legacy botocore/1.38.10', 'Expect':
b'100-continue', 'X-Amz-Date': b'20250507T012009Z', 'X-Amz-Content-SHA256':
b'620f26da9d92b8d98a038ced8ef1eca4af163216aa83e24c7a4431f55964f15f',
'Authorization': b'AWS4-HMAC-SHA256
Credential=ksugihara@REDACTED/20250507/us-east-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=e96670a4399c7c218d6903e532355949ba8b142895164032598e6c6f30318456',
'amz-sdk-invocation-id': b'22d2b2fe-786b-4c29-a357-aaa123a561b7',
'amz-sdk-request': b'attempt=1', 'Content-Length': '1048576'}> {code}
w/o the envvar (default behavior after 1.38.0)
{code:java}
2025-05-07 10:20:52,465 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG -
CanonicalRequest:
PUT
/ksugihara/bin-1m.datcontent-encoding:aws-chunked
host:redacted
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250507T012052Z
x-amz-decoded-content-length:1048576
x-amz-sdk-checksum-algorithm:CRC32
x-amz-trailer:x-amz-checksum-crc32content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
STREAMING-UNSIGNED-PAYLOAD-TRAILER
...
2025-05-07 10:20:52,466 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG -
Sending http request: <AWSPreparedRequest stream_output=False, method=PUT,
url=https://redacted/ksugihara/bin-1m.dat,
headers={'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent':
b'aws-cli/1.40.9 md/Botocore#1.38.10 ua/2.1
os/linux#5.15.146.1-microsoft-standard-WSL2 md/arch#x86_64 lang/python#3.10.12
md/pyimpl#CPython m/N cfg/retry-mode#legacy botocore/1.38.10', 'Expect':
b'100-continue', 'Transfer-Encoding': b'chunked', 'Content-Encoding':
b'aws-chunked', 'X-Amz-Trailer': b'x-amz-checksum-crc32',
'X-Amz-Decoded-Content-Length': b'1048576', 'X-Amz-Date': b'20250507T012052Z',
'X-Amz-Content-SHA256': b'STREAMING-UNSIGNED-PAYLOAD-TRAILER', 'Authorization':
b'AWS4-HMAC-SHA256
Credential=ksugihara@REDACTED/20250507/us-east-1/s3/aws4_request,
SignedHeaders=content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer,
Signature=86faa91a3237e29d2d3d73e9e92fa2afbb9eacc05edf5d9f948a91a82ea250fc',
'amz-sdk-invocation-id': b'fd9a3d52-e39b-42e3-ad92-ce3dee298e57',
'amz-sdk-request': b'attempt=1'}> {code}
> PutObject using the latest awscli later than or equal to 1.37.0 is fail
> -----------------------------------------------------------------------
>
> Key: HDDS-12123
> URL: https://issues.apache.org/jira/browse/HDDS-12123
> Project: Apache Ozone
> Issue Type: Bug
> Components: s3gateway
> Affects Versions: 1.4.0, 1.4.1
> Environment: * awscli >= 1.37.0 (aws-cli/1.37.3 Python/3.10.12
> Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3)
> * Ozone 1.4.1 [compatible
> version|https://github.com/pfnet/ozone/commits/pfn-ozone-1.4.990101/] (based
> on ozone-1.4.1 tag + our internal patches)
> * Secure mode is enabled w/ Kerberos
> * w/o Ranger
> * ozone.http.policy = HTTP_AND_HTTPS
> Reporter: Kohei Sugihara
> Priority: Major
>
> S3G returns internal server errors (500) when we upload some files using
> awscli >= 1.37.0 (not AWS CLI version 2), the latest version of AWS CLI
> released in January 2025.
> {code:java}
> 2025-01-22 14:31:36,809 [qtp1327871893-36] WARN
> org.eclipse.jetty.server.HttpChannel: /ksugihara/bin-1m.dat
> javax.servlet.ServletException: javax.servlet.ServletException:
> java.lang.IllegalStateException: Expected: 0 and actual 1048622 write sizes
> do not match
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
> at
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> This happened on the S3 endpoint enabled with the secure mode. To explore
> this issue in more detail, we changed the ozone. HTTP.policy to
> HTTP_AND_HTTPS to check both endpoints. Curiously, this error does not occur
> on the HTTP endpoint but only on the HTTPS endpoint (port: 9879) in S3G. OM
> and SCM did not output failure logs in their stdout or audit logs, so I think
> this issue is closed within the S3G component.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.36.3
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
> (500) when calling the PutObject operation (reached max retries: 4): Internal
> Server Error {code}
> Once downgrading the awscli package earlier than 1.37.0, the bug seems not
> occurs.
> {code:java}
> % pip3 install --upgrade 'awscli<1.37.0'
> % aws --version
> aws-cli/1.36.40 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.35.99
> % aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat{code}
> Our current workarounds for using S3 endpoint:
> * Use earlier than awscli 1.37.0: pip3 install 'awscli<1.37.0'
> * Use the HTTP endpoint instead of the HTTPS endpoint
> I don't make sure this information is relavant to identify the root cause,
> awscli v1.37.x is also failure for file uploading to Apache Ozone 1.4{+}.0{+}
> cluster, but in this case S3G returns a conflict error when file is being
> overwritten. New upload that does not overwrite existing files seems to
> success.
> {code:java}
> % aws --version
> aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
> botocore/1.36.3
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp s3://ksugihara/bin-1m.dat
> .
> # overwrite upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
> s3://ksugihara/bin-1m.dat
> upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
> (Conflict) when calling the PutObject operation: Cannot overwrite file with
> directory
> # new upload
> % aws s3 --endpoint https://<s3g-endpoint>:9879/ --profile pfs0 cp
> ./bin-1m.dat s3://ksugihara/bin-1m.dat.1
> upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat.1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
