njalan opened a new issue, #9751: URL: https://github.com/apache/hudi/issues/9751
I upgrade hudi from 0.7 to 0.13.1 (actually just replace spark hudi bundle jar). In hudi 0.13.1 metadata enabled is by default. But why there are still also so many list operations and it is not reduced . I can see the metadata/files are generated and also mentioned that Listed files in partition from metadata. Did I miss any config? Why there are still so many s3 list operations <img width="774" alt="Screen Shot 2023-09-20 at 9 07 26 AM" src="https://github.com/apache/hudi/assets/8622340/d86e1e15-2c61-4605-83d6-f82f88f3890e";> <img width="1678" alt="image" src="https://github.com/apache/hudi/assets/8622340/9ac5547f-aed8-4522-a07f-301151e40eed";> **Environment Description** * Hudi version : 0.13.1 * Spark version : 3.3.2 * Hive version : 3 * Hadoop version : 3.2.2 * Storage (HDFS/S3/GCS..) : s3 * Running on Docker? (yes/no) : no **Additional context** **Stacktrace** ```Add the stacktrace of the error.``` I am using the third party's s3 storage(local deployed storage cluster) and they provided me the s3 list operations metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
