Kohei Sugihara created HDDS-12123:
-------------------------------------
Summary: PutObject using the latest awscli later than or equal to
1.37.0 is fail
Key: HDDS-12123
URL: https://issues.apache.org/jira/browse/HDDS-12123
Project: Apache Ozone
Issue Type: Bug
Components: s3gateway
Affects Versions: 1.4.1, 1.4.0
Environment: * awscli >= 1.37.0 (aws-cli/1.37.3 Python/3.10.12
Linux/5.15.146.1-microsoft-standard-WSL2 botocore/1.36.3)
* Ozone 1.4.1 [compatible
version|https://github.com/pfnet/ozone/commits/pfn-ozone-1.4.990101/] (based on
ozone-1.4.1 tag + our internal patches)
* Secure mode is enabled w/ Kerberos
* w/o Ranger
* ozone.http.policy = HTTP_AND_HTTPS
Reporter: Kohei Sugihara
S3G returns internal server errors (500) when we upload some files using awscli
>= 1.37.0 (not AWS CLI version 2), the latest version of AWS CLI released in
December 2024.
{code:java}
2025-01-22 14:31:36,809 [qtp1327871893-36] WARN
org.eclipse.jetty.server.HttpChannel: /ksugihara/bin-1m.dat
javax.servlet.ServletException: javax.servlet.ServletException:
java.lang.IllegalStateException: Expected: 0 and actual 1048622 write sizes do
not match
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829) {code}
This happened on the S3 endpoint enabled with the secure mode. To explore this
issue in more detail, we changed the ozone. HTTP.policy to HTTP_AND_HTTPS to
check both endpoints. Curiously, this error does not occur on the HTTP endpoint
but only on the HTTPS endpoint (port: 9879) in S3G. OM and SCM did not output
failure logs in their stdout or audit logs, so I think this issue is closed
within the S3G component.
{code:java}
% aws --version
aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
botocore/1.36.3
% aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat
upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
% aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat
upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
(500) when calling the PutObject operation (reached max retries: 4): Internal
Server Error {code}
Once downgrading the awscli package earlier than 1.37.0, the bug seems not
occurs.
{code:java}
% pip3 install --upgrade 'awscli<1.37.0'
% aws --version
aws-cli/1.36.40 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
botocore/1.35.99
% aws s3 --endpoint http://<s3g-endpoint>:9878/ cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat
upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat
% aws s3 --endpoint https://<s3g-endpoint>:9879/ cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat
upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat{code}
Our current workarounds for using S3 endpoint:
* Use earlier than awscli 1.37.0: pip3 install 'awscli<1.37.0'
* Use the HTTP endpoint instead of the HTTPS endpoint
I don't make sure this information is relavant to identify the root cause,
awscli v1.37.x is also failure for file uploading to Apache Ozone 1.4{+}.0{+}
cluster, but in this case S3G returns a conflict error when file is being
overwritten. New upload that does not overwrite existing files seems to success.
{code:java}
% aws --version
aws-cli/1.37.3 Python/3.10.12 Linux/5.15.146.1-microsoft-standard-WSL2
botocore/1.36.3
% aws s3 --endpoint https://<s3g-endpoint>.t.nf10.net/ cp
s3://ksugihara/bin-1m.dat .
# overwrite upload
% aws s3 --endpoint https://<s3g-endpoint>.t.nf10.net/ cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat
upload failed: ./bin-1m.dat to s3://ksugihara/bin-1m.dat An error occurred
(Conflict) when calling the PutObject operation: Cannot overwrite file with
directory
# new upload
% aws s3 --endpoint https://<s3g-endpoint>:9879/ --profile pfs0 cp ./bin-1m.dat
s3://ksugihara/bin-1m.dat.1
upload: ./bin-1m.dat to s3://ksugihara/bin-1m.dat.1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
