Hi, We are running a POC to see if we can use Paimon over S3 for a very high throughput and low latency use-case. We have two flink jobs, one reads from Kafka source, converts the data and writes to a Paimon/S3 table and the other reads from Paimon/S3 table and writes to a Kafka sink. We are seeing a very large number of HEAD and LIST requests per second in S3. These requests are contributing heavily towards the AWS S3 costs and we are probably being throttled as well since we are seeing backpressure on the Kafka source. Any thoughts on what we can do to reduce the number of head/list requests to S3? Any configuration settings that we could use that will help in our situation? I would really appreciate any help you can provide.
Thanks!
