[ https://issues.apache.org/jira/browse/HUDI-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163018#comment-17163018 ]
Balaji Varadarajan edited comment on HUDI-1015 at 7/22/20, 7:08 PM: -------------------------------------------------------------------- Another place where we do listing in executor. Will create a separate ticket (Source : [https://github.com/apache/hudi/issues/1852]) : sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:259) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:167) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:124) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:180) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listFiles(AzureBlobFileSystemStore.java:549) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:628) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:532) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:344) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:487) org.apache.hudi.common.fs.FSUtils.getAllLogFiles(FSUtils.java:409) org.apache.hudi.common.fs.FSUtils.getLatestLogVersion(FSUtils.java:420) org.apache.hudi.common.fs.FSUtils.computeNextLogVersion(FSUtils.java:434) org.apache.hudi.common.model.HoodieLogFile.rollOver(HoodieLogFile.java:115) org.apache.hudi.common.table.log.HoodieLogFormatWriter.(HoodieLogFormatWriter.java:101) org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:249) org.apache.hudi.io.HoodieAppendHandle.createLogWriter(HoodieAppendHandle.java:291) org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:141) org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:197) org.apache.hudi.table.action.deltacommit.DeltaCommitActionExecutor.handleUpdate(DeltaCommitActionExecutor.java:77) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:246) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.lambda$execute$caffe4c4$1(BaseCommitActionExecutor.java:102) org.apache.hudi.table.action.commit.BaseCommitActionExecutor$$Lambda$192/1449069739.call(Unknown Source) org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:105) was (Author: vbalaji): Another place where we do listing in executor : (Source : [https://github.com/apache/hudi/issues/1852]) : sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:259) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:167) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:124) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:180) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listFiles(AzureBlobFileSystemStore.java:549) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:628) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:532) shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:344) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) org.apache.hudi.common.fs.HoodieWrapperFileSystem.listStatus(HoodieWrapperFileSystem.java:487) org.apache.hudi.common.fs.FSUtils.getAllLogFiles(FSUtils.java:409) org.apache.hudi.common.fs.FSUtils.getLatestLogVersion(FSUtils.java:420) org.apache.hudi.common.fs.FSUtils.computeNextLogVersion(FSUtils.java:434) org.apache.hudi.common.model.HoodieLogFile.rollOver(HoodieLogFile.java:115) org.apache.hudi.common.table.log.HoodieLogFormatWriter.(HoodieLogFormatWriter.java:101) org.apache.hudi.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:249) org.apache.hudi.io.HoodieAppendHandle.createLogWriter(HoodieAppendHandle.java:291) org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:141) org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:197) org.apache.hudi.table.action.deltacommit.DeltaCommitActionExecutor.handleUpdate(DeltaCommitActionExecutor.java:77) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:246) org.apache.hudi.table.action.commit.BaseCommitActionExecutor.lambda$execute$caffe4c4$1(BaseCommitActionExecutor.java:102) org.apache.hudi.table.action.commit.BaseCommitActionExecutor$$Lambda$192/1449069739.call(Unknown Source) org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:105) > Audit all getAllPartitionPaths() calls and keep em out of fast path > ------------------------------------------------------------------- > > Key: HUDI-1015 > URL: https://issues.apache.org/jira/browse/HUDI-1015 > Project: Apache Hudi > Issue Type: Improvement > Components: Common Core, Writer Core > Reporter: Vinoth Chandar > Assignee: Balaji Varadarajan > Priority: Blocker > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)