Balaji Varadarajan created HUDI-1119: ----------------------------------------
Summary: MOR appends slow due to file listing in executor side for finding the log file Key: HUDI-1119 URL: https://issues.apache.org/jira/browse/HUDI-1119 Project: Apache Hudi Issue Type: Task Components: Writer Core Reporter: Balaji Varadarajan Fix For: 0.6.0 Another place where we do listing in executor. (Source : [https://github.com/apache/hudi/issues/1852]) : sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:259) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:167) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:124) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:180) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listFiles(AzureBlobFileSystemStore.java:549) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:628) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:532) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:344) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:487) org.apache.hudi.common.fs.FSUtils.getAllLogFiles(FSUtils.java:409) org.apache.hudi.common.fs.FSUtils.getLatestLogVersion(FSUtils.java:420) org.apache.hudi.common.fs.FSUtils.computeNextLogVersion(FSUtils.java:434) org.apache.hudi.common.model.HoodieLogFile.rollOver(HoodieLogFile.java:115) org.apache.hudi.common.table.log.HoodieLogFormatWriter.(HoodieLogFormatWriter.java:101) org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:249) org.apache.hudi.io.HoodieAppendHandle.createLogWriter(HoodieAppendHandle.java:291) org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:141) org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:197) org.apache.hudi.table.action.deltacommit.DeltaCommitActionExecutor.handleUpdate(DeltaCommitActionExecutor.java:77) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:246) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.lambda$execute$caffe4c4$1(BaseCommitActionExecutor.java:102) org.apache.hudi.table.action.commit.BaseCommitActionExecutor$$Lambda$192/1449069739.call(Unknown Source) org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:105) -- This message was sent by Atlassian Jira (v8.3.4#803005)