[jira] [Updated] (FLINK-30128) Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path
[ https://issues.apache.org/jira/browse/FLINK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30128: --- Labels: pull-request-available (was: ) > Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path > -- > > Key: FLINK-30128 > URL: https://issues.apache.org/jira/browse/FLINK-30128 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.13.1 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Labels: pull-request-available > > Currently the HadoopRecoverableWriter assumes that the underlying FS is > Hadoop and so it checks for DistributedFileSystem. It also tries to do a > truncate and ensure the lease is recovered before the 'rename' operation is > done. > In the Azure Data lake gen 2 world, the driver does not support truncate and > lease recovery API. We should be able to get the last committed size and if > it matches go for the rename. Will be back with more details here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30128) Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path
[ https://issues.apache.org/jira/browse/FLINK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated FLINK-30128: --- Attachment: (was: Flink_ABFS_support_1.pdf) > Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path > -- > > Key: FLINK-30128 > URL: https://issues.apache.org/jira/browse/FLINK-30128 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.13.1 >Reporter: ramkrishna.s.vasudevan >Priority: Major > > Currently the HadoopRecoverableWriter assumes that the underlying FS is > Hadoop and so it checks for DistributedFileSystem. It also tries to do a > truncate and ensure the lease is recovered before the 'rename' operation is > done. > In the Azure Data lake gen 2 world, the driver does not support truncate and > lease recovery API. We should be able to get the last committed size and if > it matches go for the rename. Will be back with more details here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30128) Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path
[ https://issues.apache.org/jira/browse/FLINK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated FLINK-30128: --- Attachment: Flink_ABFS_support_1.pdf > Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path > -- > > Key: FLINK-30128 > URL: https://issues.apache.org/jira/browse/FLINK-30128 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.13.1 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Attachments: Flink_ABFS_support_1.pdf > > > Currently the HadoopRecoverableWriter assumes that the underlying FS is > Hadoop and so it checks for DistributedFileSystem. It also tries to do a > truncate and ensure the lease is recovered before the 'rename' operation is > done. > In the Azure Data lake gen 2 world, the driver does not support truncate and > lease recovery API. We should be able to get the last committed size and if > it matches go for the rename. Will be back with more details here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30128) Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path
[ https://issues.apache.org/jira/browse/FLINK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated FLINK-30128: --- Attachment: (was: Flink_ABFS_support.pdf) > Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path > -- > > Key: FLINK-30128 > URL: https://issues.apache.org/jira/browse/FLINK-30128 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.13.1 >Reporter: ramkrishna.s.vasudevan >Priority: Major > > Currently the HadoopRecoverableWriter assumes that the underlying FS is > Hadoop and so it checks for DistributedFileSystem. It also tries to do a > truncate and ensure the lease is recovered before the 'rename' operation is > done. > In the Azure Data lake gen 2 world, the driver does not support truncate and > lease recovery API. We should be able to get the last committed size and if > it matches go for the rename. Will be back with more details here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30128) Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path
[ https://issues.apache.org/jira/browse/FLINK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated FLINK-30128: --- Attachment: Flink_ABFS_support.pdf > Introduce Azure Data Lake Gen2 APIs in the Hadoop Recoverable path > -- > > Key: FLINK-30128 > URL: https://issues.apache.org/jira/browse/FLINK-30128 > Project: Flink > Issue Type: Sub-task >Affects Versions: 1.13.1 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Attachments: Flink_ABFS_support.pdf > > > Currently the HadoopRecoverableWriter assumes that the underlying FS is > Hadoop and so it checks for DistributedFileSystem. It also tries to do a > truncate and ensure the lease is recovered before the 'rename' operation is > done. > In the Azure Data lake gen 2 world, the driver does not support truncate and > lease recovery API. We should be able to get the last committed size and if > it matches go for the rename. Will be back with more details here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
