Hello,
Is anyone familiar with the "blob server connection"? We have constantly
been seeing the "Error while executing Blob connection" error, which
sometimes causes a job stuck in the middle of a run if there are too many
connection errors and eventually causes a failure, though most of the time
the streaming run mode can recover from that failure in the subsequent
iterations of runs, but that slows down the entire process. We tried
adjusting the blob.fetch.num-concurrent and some other blob parameters, but
it was not very helpful, so we want to know what might be the root cause of
the issue. Are there any Flink metrics or tools to help us monitor the blob
server connections?
We use:
- Flink Kubernetes Operator
- Flink 1.15.3 and 1.16.0
- Kafka, filesystem(S3)
- Hudi 0.11.1
Full error message:
java.io.IOException: Unknown operation 71
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:116)
[flink-dist-1.15.3.jar:1.15.3]
2023-01-19 16:44:37,448 ERROR
org.apache.flink.runtime.blob.BlobServerConnection [] -
Error while executing BLOB connection.
Best regards,
Yang
