ssomuah opened a new issue #1852: URL: https://github.com/apache/hudi/issues/1852
**Describe the problem you faced** Write performance degrades over time **To Reproduce** Steps to reproduce the behavior: 1.Create an unpartitoned MOR table 2.Use it for a few days **Expected behavior** Write performance should not degrade over time **Environment Description** * Hudi version : Master @ 3b9a305 https://github.com/apache/hudi/tree/3b9a30528bd6a6369181702303f3384162b04a7f * Spark version : 2.4.4 * Hive version : N/A * Hadoop version : 2.7.3 * Storage (HDFS/S3/GCS..) : ABFSS * Running on Docker? (yes/no) : no **Additional context** The MOR table has a single partition. It's a spark streaming application with 5 minute batches. Intially it runs and completes batches in the duration. But over time the time for batches to complete increases. From the spark ui we can see that most of the time is being taken actually writing the files. <img width="1584" alt="Screen Shot 2020-07-17 at 2 03 35 PM" src="https://user-images.githubusercontent.com/2061955/87941642-7023e980-ca69-11ea-9f7a-9d801a9be131.png";> And looking at the thread dump of the executors they are almost always spending their time listing files. I think the reason for this is we have an extremely high number of files in the single partition folder. An ls on the folder is showing about 45,000 files. The other odd thing is that when we look at the write tasks in the spark ui. There are several tasks that seem to have tiny numbers of records in them. <img width="1507" alt="Screen Shot 2020-07-20 at 9 15 35 AM" src="https://user-images.githubusercontent.com/2061955/87941812-b11bfe00-ca69-11ea-9e77-32cf83f6e2e1.png";> We can see compaction taking place so it's not clear why we still have so many files. <img width="1369" alt="Screen Shot 2020-07-17 at 2 04 33 PM" src="https://user-images.githubusercontent.com/2061955/87941819-b4af8500-ca69-11ea-8f0a-9c23df08052e.png";> The table config is .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL) .option(DataSourceWriteOptions.TABLE_TYPE_OPT_KEY, HoodieTableType.MERGE_ON_READ.name) .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, s"$META_COLUMN.version") .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, s"$META_COLUMN.key") .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, s"$META_COLUMN.partition") .option(DataSourceWriteOptions.PAYLOAD_CLASS_OPT_KEY, "com.myCustompayloadClass") .option(HoodieCompactionConfig.PAYLOAD_CLASS_PROP, "com.myCustompayloadClass") .option(HoodieWriteConfig.UPSERT_PARALLELISM, 32) .option(HoodieWriteConfig.INSERT_PARALLELISM, 32) .option(HoodieCompactionConfig.CLEANER_COMMITS_RETAINED_PROP, 3) .option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP, 12 ) .option(HoodieCompactionConfig.INLINE_COMPACT_PROP, true) .option(HoodieStorageConfig.PARQUET_FILE_MAX_BYTES, String.valueOf(256 * 1024 * 1024)) .option(HoodieStorageConfig.PARQUET_BLOCK_SIZE_BYTES, String.valueOf(256 * 1024 * 1024)) .option(HoodieStorageConfig.PARQUET_COMPRESSION_CODEC, "snappy") We're using our own payload class that decides what to keep based on a timestamp in the message and not latest. **Stacktrace** StackTrace of list operation where we are spending a lot of time. sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:259) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:167) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:124) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:180) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listFiles(AzureBlobFileSystemStore.java:549) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:628) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:532) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:344) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:487) org.apache.hudi.common.fs.FSUtils.getAllLogFiles(FSUtils.java:409) org.apache.hudi.common.fs.FSUtils.getLatestLogVersion(FSUtils.java:420) org.apache.hudi.common.fs.FSUtils.computeNextLogVersion(FSUtils.java:434) org.apache.hudi.common.model.HoodieLogFile.rollOver(HoodieLogFile.java:115) org.apache.hudi.common.table.log.HoodieLogFormatWriter.<init>(HoodieLogFormatWriter.java:101) org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:249) org.apache.hudi.io.HoodieAppendHandle.createLogWriter(HoodieAppendHandle.java:291) org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:141) org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:197) org.apache.hudi.table.action.deltacommit.DeltaCommitActionExecutor.handleUpdate(DeltaCommitActionExecutor.java:77) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:246) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.lambda$execute$caffe4c4$1(BaseCommitActionExecutor.java:102) org.apache.hudi.table.action.commit.BaseCommitActionExecutor$$Lambda$192/1449069739.call(Unknown Source) org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:105) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
![https://user-images.githubusercontent.com/2061955/87941642-7023e980-ca69-11ea-9f7a-9d801a9be131.png"](https://user-images.githubusercontent.com/2061955/87941642-7023e980-ca69-11ea-9f7a-9d801a9be131.png"){kind=link}
![https://user-images.githubusercontent.com/2061955/87941812-b11bfe00-ca69-11ea-9e77-32cf83f6e2e1.png"](https://user-images.githubusercontent.com/2061955/87941812-b11bfe00-ca69-11ea-9e77-32cf83f6e2e1.png"){kind=link}
![https://user-images.githubusercontent.com/2061955/87941819-b4af8500-ca69-11ea-8f0a-9c23df08052e.png"](https://user-images.githubusercontent.com/2061955/87941819-b4af8500-ca69-11ea-8f0a-9c23df08052e.png"){kind=link}