iemejia commented on issue #25991: URL: https://github.com/apache/beam/issues/25991#issuecomment-1490342178
Hi, I am less involved in Beam recently. I wonder if what is going on is that somehow the Amazon library is slow with the connections. One thing to check is if the delays are when trying to close/drain the connections from the AWS IO connector. Notice that I added this code because the library warns about not doing it, but from a quick look I have seen few connectors in other systems doing it. Maybe this manifests because the Spark runner has many many workers, is this the case? If so, this could definitely count. I cannot think of an easy fix, apart of reverting the change, but we need to check first with the current maintainers if they are ok with this @aromanenko-dev @mosche or if there are other ideas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
