ksato9700 commented on issue #2172: URL: https://github.com/apache/iceberg-rust/issues/2172#issuecomment-5068945715
Wanted to add a data point here, since #2177 (the operator-caching fix for this) got closed pending the Storage trait work: under reqsign-aws-v4's ECS credential provider, the overhead this issue describes can also cause a commit to fail outright with a non-retryable error, not just add CPU cost. Mechanism: each create_operator() call builds a fresh Signer with its own empty credential cache. When a single commit's manifest-list scan fans out to many concurrent reads (common once a table has accumulated tens of manifests), each read's freshly-built Signer independently calls the ECS task-metadata credentials endpoint at the same time. That endpoint enforces its own request-rate limit, and enough simultaneous fetches can exceed it — reqsign classifies the resulting 429 Too Many Requests as non-retryable, so the read fails, and the commit that read was part of fails with it. Understood that #2177 is parked behind the Storage trait rework rather than rejected — just flagging that the underlying issue has a correctness angle (failed reads/commits under concurrency, given a rate-limited credential provider), not only a performance one. Happy to share more reproduction detail if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
