[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715864#comment-17715864 ] ASF GitHub Bot commented on HADOOP-18705: - steveloughran merged PR #5560: URL: https://github.com/apache/hadoop/pull/5560 > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68) > at > org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91) > at > org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450) > at > org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.getTruststorePassword(AbfsIDBClient.java:104) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.initializeAsFullIDBClient(AbstractIDBClient.java:860) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.(AbstractIDBClient.java:139) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.(AbfsIDBClient.java:74) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.getClient(AbfsIDBIntegration.java:287) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.serviceStart(AbfsIDBIntegration.java:240) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.fromDelegationTokenManager(AbfsIDBIntegration.java:205) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBDelegationTokenManager.bind(AbfsIDBDelegationTokenManager.java:66) > at > org.apache.hadoop.fs.azurebfs.extensions.ExtensionHelper.bind(ExtensionHelper.java:54) > at > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileS
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715862#comment-17715862 ] ASF GitHub Bot commented on HADOOP-18705: - steveloughran commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1520304854 +1. tested myself against azure cardiff, threads=8, no scale. took 5 minutes, which was nice. > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68) > at > org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91) > at > org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450) > at > org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.getTruststorePassword(AbfsIDBClient.java:104) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.initializeAsFullIDBClient(AbstractIDBClient.java:860) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.(AbstractIDBClient.java:139) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.(AbfsIDBClient.java:74) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.getClient(AbfsIDBIntegration.java:287) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.serviceStart(AbfsIDBIntegration.java:240) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.fromDelegationTokenManager(AbfsIDBIntegration.java:205) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBDelegationTokenManager.bind(AbfsIDBDelegationTokenManager.java:66) > at > org.apache.hadoop.fs.azurebfs.extensions.ExtensionHelper.bind(ExtensionHelper.java:54) > at > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715655#comment-17715655 ] ASF GitHub Bot commented on HADOOP-18705: - hadoop-yetus commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1519657375 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 37s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 40m 17s | | trunk passed | | +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | checkstyle | 0m 35s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 44s | | trunk passed | | +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 19s | | trunk passed | | +1 :green_heart: | shadedclient | 20m 38s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 20m 57s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | | the patch passed | | +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javac | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | javac | 0m 29s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 19s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 33s | | the patch passed | | +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 5s | | the patch passed | | +1 :green_heart: | shadedclient | 20m 17s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. | | | | 95m 52s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/5560 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 1bb82985b3b7 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 3746dc3cd9d25f108dcc2284a40a89d2d8a7a864 | | Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/3/testReport/ | | Max. process+thread count | 556 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/3/console | | versions |
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713937#comment-17713937 ] ASF GitHub Bot commented on HADOOP-18705: - hadoop-yetus commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1514231301 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 46s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 42m 11s | | trunk passed | | +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | checkstyle | 0m 31s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 39s | | trunk passed | | +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 15s | | trunk passed | | +1 :green_heart: | shadedclient | 23m 56s | | branch has no errors when building and testing our client artifacts. | | -0 :warning: | patch | 24m 14s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | | the patch passed | | +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javac | 0m 34s | | the patch passed | | +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | javac | 0m 28s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 17s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 31s | | the patch passed | | +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 4s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 32s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 59s | | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. | | | | 103m 8s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/5560 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux db5f2deb9c92 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / efe0f9fb55bd4c520d09952f972d97000bec46e0 | | Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/2/testReport/ | | Max. process+thread count | 536 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/2/console | | versions |
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713901#comment-17713901 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1514156973 @steveloughran Thanks for the review. I just read the testing_azure.md, I haven't run the integration tests yet. > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68) > at > org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91) > at > org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450) > at > org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.getTruststorePassword(AbfsIDBClient.java:104) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.initializeAsFullIDBClient(AbstractIDBClient.java:860) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.(AbstractIDBClient.java:139) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.(AbfsIDBClient.java:74) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.getClient(AbfsIDBIntegration.java:287) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.serviceStart(AbfsIDBIntegration.java:240) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.fromDelegationTokenManager(AbfsIDBIntegration.java:205) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBDelegationTokenManager.bind(AbfsIDBDelegationTokenManager.java:66) > at > org.apache.hadoop.fs.azurebfs.extensions.ExtensionHelper.bind(ExtensionHelper.java:54) > at > org.apache.hadoop.fs.azurebfs.security.AbfsDelega
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713900#comment-17713900 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170825511 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java: ## @@ -196,6 +196,11 @@ public void initialize(URI uri, Configuration configuration) final AbfsConfiguration abfsConfiguration = abfsStore .getAbfsConfiguration(); + +// Ensures that configuration excludes incompatible credential providers Review Comment: Done. Note: there is 2x `excludeIncompatibleCredentialProviders` calls, because the AbfsConfiguration does it again. Another approach would be to move the `super.initialize` after the `AbfsConfiguration` is ready, there is a v1 patch uploaded to this PR, check that version too. The v2 is simpler and it's less error prone. > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68) > at > org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91) > at > org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450) > at > org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.getTruststorePassword(AbfsIDBClient.java:104) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.initializeAsFullIDBClient(AbstractIDBClient.java:860) > at > org.apache.knox.gateway.cloud.idbroker.AbstractIDBClient.(AbstractIDBClient.java:139) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.(AbfsIDBClient.java:74) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBIntegration.getClient(AbfsIDBIntegration.java:287) >
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713897#comment-17713897 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170822606 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; + +public class ITestAzureBlobFileSystemConfiguration extends AbstractAbfsIntegrationTest { Review Comment: done ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; Review Comment: done > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.File
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713898#comment-17713898 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170822736 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; + +public class ITestAzureBlobFileSystemConfiguration extends AbstractAbfsIntegrationTest { + + public ITestAzureBlobFileSystemConfiguration() throws Exception { + } + + @Test + public void testIncompatibleCredentialProviderIsExcluded() throws Exception { +Configuration rawConfig = getRawConfiguration(); +rawConfig.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH, +"jceks://abfs@a@b.c.d/tmp/a.jceks,jceks://file/tmp/secret.jceks"); +AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem.get(rawConfig); Review Comment: done > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.4.0, 3.3.5 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.a
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713626#comment-17713626 ] ASF GitHub Bot commented on HADOOP-18705: - steveloughran commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1169748503 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; Review Comment: import structure not what we prefer, which is ``` java javax not-org-apache org.apache.* statics ``` ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; + +public class ITestAzureBlobFileSystemConfiguration extends AbstractAbfsIntegrationTest { Review Comment: needs a name which explains what the test does, e.g "ITestABFSJceksFiltering" ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.junit.Test; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; + +public class ITestAzureBlobFileSystemConfiguration extends AbstractAbfsIntegrationTest { + + public ITestAzureBlobFileSystemConfiguration() throws Exception { + } + + @Test + public void testIncompatibleCredentialProviderIsExcluded() throws Exception { +Configuration rawConfig = getRawConfiguration(); +rawConfig.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH, +"jceks://abfs@a@b.c.d/tmp/a.jceks,jceks://file/tmp/secret.jceks"); +AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem.get(rawConfig); Review Comment: use try-with-resources to ensure that this is closed afterwards ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java: #
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713111#comment-17713111 ] ASF GitHub Bot commented on HADOOP-18705: - hadoop-yetus commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1511439932 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 47s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 44m 37s | | trunk passed | | +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | checkstyle | 0m 29s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | | trunk passed | | +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 14s | | trunk passed | | +1 :green_heart: | shadedclient | 23m 14s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | | the patch passed | | +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javac | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | javac | 0m 28s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 17s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | | the patch passed | | +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 | | +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | +1 :green_heart: | spotbugs | 1m 5s | | the patch passed | | +1 :green_heart: | shadedclient | 23m 37s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. | | | | 105m 11s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/5560 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux 785a46dea41b 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / c093fe1297cb91d261e100aa9c898ffe3de4d983 | | Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/1/testReport/ | | Max. process+thread count | 535 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5560/1/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > hadoop-azure:
[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713063#comment-17713063 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler opened a new pull request, #5560: URL: https://github.com/apache/hadoop/pull/5560 …atible credential providers when binding DelegationTokenManagers Change-Id: I1ad8b5856a0b8c0b75d4538019d43e7fdb1962d2 ### Description of PR ### How was this patch tested? I tested my change manually with a non-existing jceks file. Without my change I received the error described in the Jira: `Caused by: org.apache.hadoop.fs.PathIOException: `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if loading a JCEKS file, this means that the filesystem connector is trying to load the same file`. With my change the job run successfully, I also added some extra debug logs to see if the credential provider path is indeed correct. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-18705. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > hadoop-azure: AzureBlobFileSystem should exclude incompatible credential > providers when binding DelegationTokenManagers > --- > > Key: HADOOP-18705 > URL: https://issues.apache.org/jira/browse/HADOOP-18705 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 3.4.0 >Reporter: Tamas Domok >Priority: Major > > The DelegationTokenManager in AzureBlobFileSystem.initialize() gets the > untouched configuration which may contain a credentialProviderPath config > with incompatible credential providers (e.g.: jceks stored on abfs). This > results in an error: > {quote} > Caused by: org.apache.hadoop.fs.PathIOException: > `jceks://abfs@a@b.c.d/tmp/a.jceks': Recursive load of credential provider; if > loading a JCEKS file, this means that the filesystem connector is trying to > load the same file > {quote} > {code} > this.delegationTokenManager = > abfsConfiguration.getDelegationTokenManager(); > delegationTokenManager.bind(getUri(), configuration); > {code} > The abfsConfiguration excludes the incompatible credential providers already. > Reproduction steps: > {code} > export HADOOP_ROOT_LOGGER=DEBUG,console > hdfs dfs -rm -r -skipTrash /user/qa/sort_input; hadoop jar > hadoop-mapreduce-examples.jar randomwriter > "-Dmapreduce.randomwriter.totalbytes=100" > "-Dhadoop.security.credential.provider.path=jceks://abfs@a@b.c.d/tmp/a.jceks" > /user/qa/sort_input > {code} > Error: > {code} > ... > org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager.bind(AbfsDelegationTokenManager.java:96) > at > org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:224) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.security.alias.KeyStoreProvider.initFileSystem(KeyStoreProvider.java:84) > at > org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:85) > at > org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35) > at > org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68) > at > org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:91) > at > org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2450) > at > org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2388) > at > org.apache.knox.gateway.cloud.idbroker.abfs.AbfsIDBClient.getTruststo