[
https://issues.apache.org/jira/browse/FLINK-40123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130955#comment-18130955
]
Mateusz Barnacki commented on FLINK-40123:
------------------------------------------
Hello [~shekharrajak] and [~martijnvisser] ,
I looked into this case. This warning originates from Hadoop's
_S3ABlockOutputStream_ and its metrics tracker _S3AInstrumentation_ . In S3A,
block writes are uploaded asynchronously using a background thread pool. When
the output stream is closed, the instrumentation checks if _bytesPendingUpload
> 0_ . It's highly likely that the thread handling the upload or the counter
update gets delayed relative to the thread calling {_}close(){_}, or the stream
is aborted while background transfers are still in flight.
Detailed analysis generated by Claude Opus 5.5:
# When a small file is closed, _S3ABlockOutputStream.putObject()_ calls
{_}statistics.blockUploadQueued(size){_}. That adds size to the
bytesPendingUpload counter.
# The upload runs on another thread, and the writer thread waits for it to
finish with {_}putObjectResult.get(){_}.
# bytesPendingUpload only goes back down in
{_}BlockUploadProgress.progressChanged(REQUEST_BYTE_TRANSFER_EVENT){_}.
# BlockUploadProgress is a plain ProgressListener, so the AWS SDK delivers its
events asynchronously. They go through SDKProgressPublisher, which runs them on
a single shared thread called java-sdk-progress-listener-callback-thread.
# Once the upload returns, the stream calls
{_}OutputStreamStatistics.close(){_}. If the callback thread hasn't processed
every byte-transfer event yet, _getBytesPendingUpload() > 0_ and the warning is
logged.
Steps to reproduce locally:
# Set up debug using _env.java.opts.sql-gateway_ config key.
# Set breakpoint in _progressChanged_ method
({_}S3ABlockOutputStream.BlockUploadProgress{_} class,
org.apache.hadoop:hadoop-aws:3.3.4 dependency).
# Run test:
{code:java}
FLINK_DIR=build-target ./flink-end-to-end-tests/run-single-test.sh
"flink-end-to-end-tests/test-scripts/test_kubernetes_materialized_table.sh"{code}
Possible fix:
# Add warning message to _default_allowed_exceptions_ in
_internal_check_logs_for_exceptions_ (common.sh), next to the existing
'INFO.*AWSErrorCode' entry.
# Lower that logger's level in the test.
# Mimic the approach used in _test_tpcds.sh_ file and create
_additional_allowed_exceptions_ array which will be passed as an argument to
i{_}nternal_check_logs_for_exceptions{_} method and executed only for this
single e2e test.
Let me know if any of the suggested solutions can be implemented. I'm looking
forward to hearing from you.
> heck_logs_for_exceptions false-positives on S3A stream_write_exceptions
> counter, flaking test_kubernetes_materialized_table.sh
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40123
> URL: https://issues.apache.org/jira/browse/FLINK-40123
> Project: Flink
> Issue Type: Improvement
> Affects Versions: 2.2.2, 2.3.1, 2.4.0
> Environment: Azure build 76427, task "Run e2e tests" (log 87).
>
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=76427
> Reporter: Shekhar Prasad Rajak
> Priority: Major
> Fix For: 2.3.1
>
>
> The e2e test Run kubernetes Materialized Table test
> (flink-end-to-end-tests/test-scripts/test_kubernetes_materialized_table.sh)
> intermittently fails even though the test logic itself
> succeeds:
> [FAIL] 'Run kubernetes Materialized Table test' failed after 4 minutes and
> 20 seconds!
> Test exited with exit code 0 but the logs contained errors, exceptions or
> non-empty .out files
> Note exit code 0 — the test passed; the failure is raised entirely by the
> post-run log scan.
>
> How to reproduce
> Run test_kubernetes_materialized_table.sh against a SeaweedFS/S3A-backed
> catalog store; whenever the S3AInstrumentation "Closing output stream
> statistics" WARN is logged, the run fails in
> the exception check.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
