Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2141433730 > @anujmodi2021 merged to trunk. Can you do the branch-3.4 cherrypick, rerun the tests and then push up as a PR to merge there too? thanks Thanks a lot Steve. Here is the Backport PR for branch_3.4 https://github.com/apache/hadoop/pull/6855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2140758208 @anujmodi2021 merged to trunk. Can you do the branch-3.4 cherrypick, rerun the tests and then push up as a PR to merge there too? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran merged PR #6552: URL: https://github.com/apache/hadoop/pull/6552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2138937445
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 40s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 1s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 128m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 0bf4c8d5d4fb 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 298640f65c486f62db490d2570b1db9402189cc9 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/17/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2138740413 Hi @steveloughran @mukund-thakur Requesting you to kindly review the latest changes and merge if they looks good. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2102708307
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 03s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 00s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 93m 50s | | trunk passed |
| +1 :green_heart: | compile | 5m 06s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 154m 16s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 156m 46s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | | the patch passed |
| +1 :green_heart: | compile | 2m 22s | | the patch passed |
| +1 :green_heart: | javac | 2m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 09s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 167m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 47s | | The patch does not
generate ASF License warnings. |
| | | 439m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | MINGW64_NT-10.0-17763 ca0f186c13b4 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 820e2e0d0c98b3441a71f0101915990911052959 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/3/testReport/
|
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/3/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2097509435
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 24s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 128m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 300cd269972c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 820e2e0d0c98b3441a71f0101915990911052959 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/16/testReport/ |
| Max. process+thread count | 705 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This i
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2097394199 > all of which are trivial Done. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2095321708
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 31s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 52s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/15/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 3 new + 9 unchanged - 0
fixed = 12 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 129m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 24be86ac3909 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecd151a7fac10629fb93f8f25b3a8567caf172ac |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/15/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-h
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2095234333 > I'm afraid you are very unlucky in that I have been staring at the ABFS auth code for diagnostics purposes today and so these lines of code are now very familiar. > > 1. getPasswordString/getTrimmedPasswordString() must be used for lookup. that way you can keep the secret in a JCEKS file. > 2. proposed a way to avoid double wrapping exceptions Thanks for the pointers here. Please let me know if changes made looks good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1590561673
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -980,33 +981,59 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken = this.getString(FS_AZURE_SAS_FIXED_TOKEN,
null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
+customSasTokenProviderImplementation, rawConfig);
+Preconditions.checkArgument(sasTokenProvider != null,
+"Failed to initialize %s", customSasTokenProviderImplementation);
+
+LOG.trace("Initializing {}",
customSasTokenProviderImplementation.getName());
+sasTokenProvider.initialize(rawConfig, accountName);
+LOG.trace("{} init complete",
customSasTokenProviderImplementation.getName());
+return sasTokenProvider;
+ } else {
+LOG.trace("Using FixedSASTokenProvider implementation");
+FixedSASTokenProvider fixedSASTokenProvider = new
FixedSASTokenProvider(configuredFixedToken);
+return fixedSASTokenProvider;
+ }
} catch (Exception e) {
Review Comment:
Taken but with SasTokenProviderException
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.java:
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by a
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1590561513
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -980,33 +981,59 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken = this.getString(FS_AZURE_SAS_FIXED_TOKEN,
null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
+customSasTokenProviderImplementation, rawConfig);
+Preconditions.checkArgument(sasTokenProvider != null,
Review Comment:
Yes.
I realized we do have a dedicated exception type for SAS Token related
issues.
Using SASTokenProviderException instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1590561164
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -980,33 +981,59 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken = this.getString(FS_AZURE_SAS_FIXED_TOKEN,
null);
Review Comment:
makes sense.
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2095179355 > +1 pending that change to abfs.md that @anmolanmol1234 spotted Which change are you referring to?? I can see a comment regarding the typo which anmol spotted and already fixed that. Am I missing any other comment? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1589326534
##
hadoop-tools/hadoop-azure/src/site/markdown/abfs.md:
##
@@ -609,21 +610,119 @@ In case delegation token is enabled, and the config
`fs.azure.delegation.token
### Shared Access Signature (SAS) Token Provider
-A Shared Access Signature (SAS) token provider supplies the ABFS connector
with SAS
-tokens by implementing the SASTokenProvider interface.
-
-```xml
-
- fs.azure.account.auth.type
- SAS
-
-
- fs.azure.sas.token.provider.type
-
{fully-qualified-class-name-for-implementation-of-SASTokenProvider-interface}
-
-```
-
-The declared class must implement
`org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider`.
+A shared access signature (SAS) provides secure delegated access to resources
in
+your storage account. With a SAS, you have granular control over how a client
can access your data.
+To know more about how SAS Authentication works refer to
+[Grant limited access to Azure Storage resources using shared access
signatures
(SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview)
+
+There are three types of SAS supported by Azure Storage:
+- [User Delegation
SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas):
Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It
is Identify based SAS that works at blob/directory level)
Review Comment:
this is the last change before we merge...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1589603358
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -980,33 +981,59 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken = this.getString(FS_AZURE_SAS_FIXED_TOKEN,
null);
Review Comment:
use getTrimmedPasswordString() so JECKS can be used as a store for this
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -980,33 +981,59 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1589326534
##
hadoop-tools/hadoop-azure/src/site/markdown/abfs.md:
##
@@ -609,21 +610,119 @@ In case delegation token is enabled, and the config
`fs.azure.delegation.token
### Shared Access Signature (SAS) Token Provider
-A Shared Access Signature (SAS) token provider supplies the ABFS connector
with SAS
-tokens by implementing the SASTokenProvider interface.
-
-```xml
-
- fs.azure.account.auth.type
- SAS
-
-
- fs.azure.sas.token.provider.type
-
{fully-qualified-class-name-for-implementation-of-SASTokenProvider-interface}
-
-```
-
-The declared class must implement
`org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider`.
+A shared access signature (SAS) provides secure delegated access to resources
in
+your storage account. With a SAS, you have granular control over how a client
can access your data.
+To know more about how SAS Authentication works refer to
+[Grant limited access to Azure Storage resources using shared access
signatures
(SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview)
+
+There are three types of SAS supported by Azure Storage:
+- [User Delegation
SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas):
Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It
is Identify based SAS that works at blob/directory level)
Review Comment:
this is the last change before we merge...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2085299632
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 31s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 52s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/14/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 9 unchanged - 0
fixed = 10 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 10s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 129m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 0058a3fedccc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5db5372240aeb83bd48ab3e774273e507820cdf2 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/14/testReport/ |
| Max. process+thread count | 562 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-h
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2085145286 Thanks @steveloughran for the review. This is good to be merged now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2085142098
--
AGGREGATED TEST RESULT
HNS-OAuth
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 620, Failures: 0, Errors: 0, Skipped: 76
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 54
HNS-SharedKey
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 620, Failures: 0, Errors: 0, Skipped: 28
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 604, Failures: 0, Errors: 0, Skipped: 268
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 620, Failures: 0, Errors: 0, Skipped: 78
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 78
Time taken: 56 mins 59 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2082335808
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 37m 18s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 6s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 136m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux f5aba3b9be2a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7c9d5b45ec896f5a6acdb418d6445fafde8e8cbe |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/13/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This i
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2082215735
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 45s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 132m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 86397bc683d5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b01a229fcbe160a9a89b91a243e3430c7c81fb98 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/12/testReport/ |
| Max. process+thread count | 709 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This i
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2082148111
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 33s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 55s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 131m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux a2f9fffab53c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f8e246d58162637a8b3a8a540be0faa3d0611f21 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/11/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This i
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1582642060
##
hadoop-tools/hadoop-azure/src/site/markdown/abfs.md:
##
@@ -609,21 +610,119 @@ In case delegation token is enabled, and the config
`fs.azure.delegation.token
### Shared Access Signature (SAS) Token Provider
-A Shared Access Signature (SAS) token provider supplies the ABFS connector
with SAS
-tokens by implementing the SASTokenProvider interface.
-
-```xml
-
- fs.azure.account.auth.type
- SAS
-
-
- fs.azure.sas.token.provider.type
-
{fully-qualified-class-name-for-implementation-of-SASTokenProvider-interface}
-
-```
-
-The declared class must implement
`org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider`.
+A shared access signature (SAS) provides secure delegated access to resources
in
+your storage account. With a SAS, you have granular control over how a client
can access your data.
+To know more about how SAS Authentication works refer to
+[Grant limited access to Azure Storage resources using shared access
signatures
(SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview)
+
+There are three types of SAS supported by Azure Storage:
+- [User Delegation
SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas):
Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It
is Identify based SAS that works at blob/directory level)
Review Comment:
typo:identity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1582640059
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -1419,16 +1421,17 @@ private String appendSASTokenToQuery(String path,
sasToken = cachedSasToken;
LOG.trace("Using cached SAS token.");
}
+
// if SAS Token contains a prefix of ?, it should be removed
if (sasToken.charAt(0) == '?') {
sasToken = sasToken.substring(1);
}
+
queryBuilder.setSASToken(sasToken);
LOG.trace("SAS token fetch complete for {} on {}", operation, path);
} catch (Exception ex) {
-throw new SASTokenProviderException(String.format("Failed to acquire a
SAS token for %s on %s due to %s",
-operation,
-path,
+throw new SASTokenProviderException(String.format(
Review Comment:
nit: changes in this class can be avoided as it adds to the diff and we are
adding only new lines (optional)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1582581353
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.java:
##
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
+
+public class FixedSASTokenProvider implements SASTokenProvider {
Review Comment:
It's there in the later commits...
```
/**
* In house implementation of {@link SASTokenProvider} to use a fixed SAS
token with ABFS.
* Use this to avoid implementing a Custom Token Provider just to return
fixed SAS.
* Fixed SAS Token to be provided using the config
"fs.azure.sas.fixed.token".
*/
public class FixedSASTokenProvider implements SASTokenProvider {
private String fixedSASToken;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1581305976
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -976,33 +977,60 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
Review Comment:
it's needed to support dynamic loading of the class provided in the
configuration file.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.java:
##
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
+
+public class FixedSASTokenProvider implements SASTokenProvider {
Review Comment:
...don't see them...
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/h
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1581300456
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
+
+// configuring a SASTokenProvider class: this provides a user delegation
SAS
+// user delegation SAS Provider is set
+// This easily distinguishes between results of filesystem level and blob
level operations to ensure correct SAS is chosen,
+// when both a provider class and fixed token is configured.
+testAbfsConfig.set(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
"org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider");
+
+// configuring the fixed SAS token
+testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS);
+
+// creating a new fs instance with the updated configs
+AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration());
+
+// testing a file system level operation
+TracingContext tracingContext = getTestTracingContext(newTestFs, true);
+// expected to fail in the ideal case, as delegation SAS will be chosen,
provider class is given preference when both are configured
+// this expectation is because filesystem level operations are beyond the
scope of Delegation SAS Token
+intercept(SASTokenProviderException.class,
+() -> {
+ newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
+});
+
+// testing blob level operation to ensure delegation SAS token is
otherwise valid and above operation fails only because it is fs level
+Path testPath = new Path("/testCorrectSASToken");
+newTestFs.create(test
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2076944140
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 03s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 123m 47s | | trunk passed |
| +1 :green_heart: | compile | 7m 35s | | trunk passed |
| +1 :green_heart: | checkstyle | 7m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 216m 15s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 220m 11s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 3m 54s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 3m 32s |
[/patch-compile-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | javac | 3m 32s |
[/patch-compile-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 12s | | the patch passed |
| -1 :x: | mvnsite | 3m 29s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed |
| -1 :x: | shadedclient | 221m 00s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 14s | | The patch does not
generate ASF License warnings. |
| | | 596m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | MINGW64_NT-10.0-17763 2dbfdbef420f 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / d06fe415497f1ca98787119b5f7e8680afed2547 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/testReport/
|
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6552/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2036297207
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 19s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 13s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 130m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux e708fd9cbad4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d06fe415497f1ca98787119b5f7e8680afed2547 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/10/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This i
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2034795234
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 33s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/9/artifact/out/blanks-eol.txt)
| The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 17s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 130m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 1a90c9db25d5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 554fc39efd805fcd975ee3a0983764223d887c15 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/9/testReport/ |
| Max. process+thread count | 553 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/9/console |
| v
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-2033798284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 55s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 16s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| -1 :x: | mvnsite | 0m 30s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/8/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 25s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/8/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 2 new + 15
unchanged - 0 fixed = 17 total (was 15) |
| -1 :x: | javadoc | 0m 26s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/8/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
|
hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 2 new +
15 unchanged - 0 fixed = 17 total (was 15) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 14s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 128m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 5a01db91f7a2 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1548938330
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.java:
##
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
+
+public class FixedSASTokenProvider implements SASTokenProvider {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1548934177
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##
@@ -1308,10 +1308,9 @@ public void access(final Path path, final FsAction mode)
throws IOException {
/**
* Incrementing exists() calls from superclass for statistic collection.
- *
* @param f source path.
* @return true if the path exists.
- * @throws IOException
+ * @throws IOException if some issue in checking path
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1548933832
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -976,33 +977,60 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
+customSasTokenProviderImplementation, rawConfig);
+Preconditions.checkArgument(sasTokenProvider != null,
+"Failed to initialize %s", customSasTokenProviderImplementation);
+
+LOG.trace("Initializing {}",
customSasTokenProviderImplementation.getName());
+sasTokenProvider.initialize(rawConfig, accountName);
+LOG.trace("{} init complete",
customSasTokenProviderImplementation.getName());
+return sasTokenProvider;
+ } else {
+LOG.trace("Using FixedSASTokenProvider implementation");
+FixedSASTokenProvider fixedSASTokenProvider = new
FixedSASTokenProvider(configuredFixedToken);
+return fixedSASTokenProvider;
+ }
} catch (Exception e) {
- throw new TokenAccessProviderException("Unable to load SAS token
provider class: " + e, e);
+ throw new TokenAccessProviderException(
+ "Unable to load SAS token provider class: " + e, e);
Review Comment:
Can you elaborate with an example?? Not familiar with the usage of {} in
concatenation..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1548931527
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -976,33 +977,60 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
Review Comment:
Why we want o avoid it??
Also it was there already. not added as part of this PR.
But if it is not recommended and we have a better way, we can change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1548929750
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##
@@ -51,13 +50,6 @@ public interface SASTokenProvider {
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
-
- // Filesystem Level Operations
Review Comment:
Yes, that's right.
Theya re not there anyway on trunk. They were added in this PR only and
removed later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1991044330
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 52s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| -1 :x: | mvnsite | 0m 29s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 14s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 138m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux f80ee1d270a6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 817f7cbc3bf21f013c61ff0fdabb9fadde02149f |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/7/testReport/ |
| Max. process+thread count | 695 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| P
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503721707
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -128,11 +130,18 @@ public void testOnlyFixedTokenConfigured() throws
Exception {
try (AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration())) {
- // Asserting that account SAS is used as both filesystem and blob level
operations succeed.
- newTestFs.getFileStatus(new Path("/"));
- Path testPath = new Path("/testCorrectSASToken");
- newTestFs.create(testPath).close();
- newTestFs.delete(new Path("/"), true);
+ // Asserting that FixedSASTokenProvider is used.
+ Assertions.assertThat(testAbfsConfig.getSASTokenProvider())
+ .describedAs("Custom SASTokenProvider Class must be used")
+ .isInstanceOf(FixedSASTokenProvider.class);
+
+ // Assert that Account SAS is used and only read operations are
permitted.
Review Comment:
Why was create passing in the last test case and would give Access Denied
exception now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503719618
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -97,16 +99,16 @@ public void testBothProviderFixedTokenConfigured() throws
Exception {
// Creating a new file system with updated configs.
try (AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration())) {
- TracingContext tracingContext = getTestTracingContext(newTestFs, true);
- // Asserting that filesystem level operations fails with User Delegation
SAS.
- intercept(SASTokenProviderException.class, () -> {
-newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
- });
+ // Asserting that MockDelegationSASTokenProvider is used.
+ Assertions.assertThat(testAbfsConfig.getSASTokenProvider())
+ .describedAs("Custom SASTokenProvider Class must be used")
+ .isInstanceOf(MockDelegationSASTokenProvider.class);
- // Asserting that User delegation SAS token is otherwise valid and blob
level operations succeed.
- Path testPath = new Path("/testCorrectSASToken");
+ // Assert that User Delegation SAS is used and both read and write
operations are permitted.
+ Path testPath = path(getMethodName());
newTestFs.create(testPath).close();
+ newTestFs.open(testPath).close();
Review Comment:
The testPath is already closed, the need for this additional statement ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1520976609
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/FixedSASTokenProvider.java:
##
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
+
+public class FixedSASTokenProvider implements SASTokenProvider {
Review Comment:
javadocs for the class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1520974197
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##
@@ -1308,10 +1308,9 @@ public void access(final Path path, final FsAction mode)
throws IOException {
/**
* Incrementing exists() calls from superclass for statistic collection.
- *
* @param f source path.
* @return true if the path exists.
- * @throws IOException
+ * @throws IOException if some issue in checking path
Review Comment:
. at the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1520972493
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -976,33 +977,60 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
+customSasTokenProviderImplementation, rawConfig);
+Preconditions.checkArgument(sasTokenProvider != null,
+"Failed to initialize %s", customSasTokenProviderImplementation);
+
+LOG.trace("Initializing {}",
customSasTokenProviderImplementation.getName());
+sasTokenProvider.initialize(rawConfig, accountName);
+LOG.trace("{} init complete",
customSasTokenProviderImplementation.getName());
+return sasTokenProvider;
+ } else {
+LOG.trace("Using FixedSASTokenProvider implementation");
+FixedSASTokenProvider fixedSASTokenProvider = new
FixedSASTokenProvider(configuredFixedToken);
+return fixedSASTokenProvider;
+ }
} catch (Exception e) {
- throw new TokenAccessProviderException("Unable to load SAS token
provider class: " + e, e);
+ throw new TokenAccessProviderException(
+ "Unable to load SAS token provider class: " + e, e);
Review Comment:
Use {} instead of concatenate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr..
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1520971108
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -976,33 +977,60 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null, SASTokenProvider.class);
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ customSasTokenProviderImplementation != null || configuredFixedToken
!= null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer Custom SASTokenProvider Implementation if configured.
+ if (customSasTokenProviderImplementation != null) {
+LOG.trace("Using Custom SASTokenProvider implementation because it is
given precedence when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
Review Comment:
Can we avoid using reflection API here in production code ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1990884707
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The
ownership o...
[INFO]
[ERROR] Tests run: 340, Failures: 0, Errors: 1, Skipped: 55
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 9
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181
Expecting org.apache.hadoop.security.AccessControlException with text "This
request is not authorized to perform this operation using this permission.",
403 but got : "void"
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemChooseSAS.testOnlyFixedTokenConfigured:145
» AccessDenied
[INFO]
[ERROR] Tests run: 607, Failures: 1, Errors: 1, Skipped: 268
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89
The actual value of 99 was not equal to the expected value
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The
ownership o...
[INFO]
[ERROR] Tests run: 340, Failures: 1, Errors: 1, Skipped: 79
Time taken: 25 mins 14 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1966046437
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 54s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 35m 14s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 7 unchanged - 0
fixed = 9 total (was 7) |
| -1 :x: | mvnsite | 0m 27s |
[/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 25s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/6/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 35s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/6/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 141m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux a0a84b9a3195 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503722016 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockSASTokenProvider.java: ## @@ -32,8 +32,7 @@ import org.apache.hadoop.fs.azurebfs.utils.ServiceSASGenerator; /** - * A mock SAS token provider implementation for testing purpose. - * Account SAS with full permission is created using storage account key. + * A mock SAS token provider implementation Review Comment: . at the end of the line -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503721707
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -128,11 +130,18 @@ public void testOnlyFixedTokenConfigured() throws
Exception {
try (AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration())) {
- // Asserting that account SAS is used as both filesystem and blob level
operations succeed.
- newTestFs.getFileStatus(new Path("/"));
- Path testPath = new Path("/testCorrectSASToken");
- newTestFs.create(testPath).close();
- newTestFs.delete(new Path("/"), true);
+ // Asserting that FixedSASTokenProvider is used.
+ Assertions.assertThat(testAbfsConfig.getSASTokenProvider())
+ .describedAs("Custom SASTokenProvider Class must be used")
+ .isInstanceOf(FixedSASTokenProvider.class);
+
+ // Assert that Account SAS is used and only read operations are
permitted.
Review Comment:
Why was create passing in the last test case and would give Access Denied
exception now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503719618
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -97,16 +99,16 @@ public void testBothProviderFixedTokenConfigured() throws
Exception {
// Creating a new file system with updated configs.
try (AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration())) {
- TracingContext tracingContext = getTestTracingContext(newTestFs, true);
- // Asserting that filesystem level operations fails with User Delegation
SAS.
- intercept(SASTokenProviderException.class, () -> {
-newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
- });
+ // Asserting that MockDelegationSASTokenProvider is used.
+ Assertions.assertThat(testAbfsConfig.getSASTokenProvider())
+ .describedAs("Custom SASTokenProvider Class must be used")
+ .isInstanceOf(MockDelegationSASTokenProvider.class);
- // Asserting that User delegation SAS token is otherwise valid and blob
level operations succeed.
- Path testPath = new Path("/testCorrectSASToken");
+ // Assert that User Delegation SAS is used and both read and write
operations are permitted.
+ Path testPath = path(getMethodName());
newTestFs.create(testPath).close();
+ newTestFs.open(testPath).close();
Review Comment:
The testPath is already closed, the need for this additional statement ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anmolanmol1234 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503713811
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##
@@ -51,13 +50,6 @@ public interface SASTokenProvider {
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
-
- // Filesystem Level Operations
Review Comment:
Is the purpose for removing these related to SAS issue for container API's ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1503698211
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final
String properties,
final AbfsUriQueryBuilder abfsUriQueryBuilder =
createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM);
+appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder);
Review Comment:
As discussed, we don't want to support any container API on SAS Token
Authentication. Removed these changes
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,33 +941,66 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The user can choose between a configured fixed sas token, and a user
+ * implementation of the SASTokenProvider interface. Preference will be given
+ * to SASTokenProvider class provided as the value of
"fs.azure.sas.token.provider.type".
+ * If above config is not set, it is expected that user wants to use a
+ * fixed SAS Token provided as value of "fs.azure.sas.fixed.token".
+ *
+ * If both the configs are not provided,
+ * initialization fails and {@link TokenAccessProviderException} is
thrown.
+ * If both are present, SASTokenProvider class will be used to
generate SAS Token.
+ * If only fixed SAS Token is configured, this will return null
+ * and Fixed SAS token will be used to sign requests.
+ *
+ * Avoid using a tokenProvider implementation just to read the configured
fixed token,
+ * as this could create confusion. Also,implementing the SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend
on the
+ * AbfsConfiguration with which a filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object.
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
+ sasTokenProviderImplementation != null || configuredFixedToken !=
null,
+ "At least one of the \"%s\" and \"%s\" must be set.",
+ FS_AZURE_SAS_TOKEN_PROVIDER_TYPE, FS_AZURE_SAS_FIXED_TOKEN);
+
+ // Prefer SASTokenProvider Implementation if configured.
+ if (sasTokenProviderImplementation != null) {
+LOG.trace("Using SASTokenProvider class because it is given precedence
when it is set.");
+SASTokenProvider sasTokenProvider = ReflectionUtils.newInstance(
+sasTokenProviderImplementation, rawConfig);
+Preconditions.checkArgument(sasTokenProvider != null,
+"Failed to initialize %s", sasTokenProviderImplementation);
+
+LOG.trace("Initializing {}", sasTokenProviderImplementation.getName());
+sasTokenProvider.initialize(rawConfig, accountName);
+LOG.trace("{} init complete",
sasTokenProviderImplementation.getName());
+return sasTokenProvider;
+ } else {
+// Configured Fixed SAS Token will be used to sign the requests.
+return null;
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497042989
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,33 +941,66 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The user can choose between a configured fixed sas token, and a user
+ * implementation of the SASTokenProvider interface. Preference will be given
+ * to SASTokenProvider class provided as the value of
"fs.azure.sas.token.provider.type".
+ * If above config is not set, it is expected that user wants to use a
+ * fixed SAS Token provided as value of "fs.azure.sas.fixed.token".
+ *
+ * If both the configs are not provided,
+ * initialization fails and {@link TokenAccessProviderException} is
thrown.
+ * If both are present, SASTokenProvider class will be used to
generate SAS Token.
+ * If only fixed SAS Token is configured, this will return null
+ * and Fixed SAS token will be used to sign requests.
+ *
+ * Avoid using a tokenProvider implementation just to read the configured
fixed token,
+ * as this could create confusion. Also,implementing the SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend
on the
+ * AbfsConfiguration with which a filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object.
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
Review Comment:
Setting configurations is upto users and it will be difficult to stop them
from setting both.
We have clear documentation in abfs,md file as well as javadocs that
SASTokenProvider will have preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
saxenapranav commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497115703 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java: ## @@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final String properties, final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder(); abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM); +appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder); Review Comment: `appendSASTokenToQuery` was not there before this patch. Is this added for fixed sas token only. If yes, then we would have to add if-condition to keep only fixedSasToken (if in config) applying to the container APIs, and in non-fixedSasToken case, `getSASToken` should not be called. Reason being, developers would already have their implementations and those might not be adaptable to the container APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552: URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1956073875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/6552 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6552 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/4/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497042989
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,33 +941,66 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The user can choose between a configured fixed sas token, and a user
+ * implementation of the SASTokenProvider interface. Preference will be given
+ * to SASTokenProvider class provided as the value of
"fs.azure.sas.token.provider.type".
+ * If above config is not set, it is expected that user wants to use a
+ * fixed SAS Token provided as value of "fs.azure.sas.fixed.token".
+ *
+ * If both the configs are not provided,
+ * initialization fails and {@link TokenAccessProviderException} is
thrown.
+ * If both are present, SASTokenProvider class will be used to
generate SAS Token.
+ * If only fixed SAS Token is configured, this will return null
+ * and Fixed SAS token will be used to sign requests.
+ *
+ * Avoid using a tokenProvider implementation just to read the configured
fixed token,
+ * as this could create confusion. Also,implementing the SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend
on the
+ * AbfsConfiguration with which a filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object.
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(
Review Comment:
Setting configurations is upto users and it will bee difficult to stop him
from setting both.
We have clear documentation in abfs,md file as well as javadocs that
SASTokenProvider will have preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497040276
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Tests to validate the choice between using a SASTokenProvider to generate
+ * a SAS or using a Fixed SAS Token configured by user.
+ */
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS = null;
+
+
+ /**
+ * To differentiate which config was used we will use different type of SAS
Tokens.
+ * For Fixed SAS Token we will use an Account SAS with permissions to do
File system level operations.
+ * For SASTokenProvider we will use a User Delegation SAS Token Provider
+ * such that File System level operations are not permitted.
+ */
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// SAS Token configured might not have permissions for creating file
system.
+// Shared Key must be configured to create one. Once created, a new
instance
+// of same file system will be used with SAS Authentication.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+generateAccountSAS(); }
+
+ /**
+ * Generates a Account SAS Token using the Account Shared Key to be used as
a fixed SAS Token.
+ * This will be used by individual tests to set in the configurations.
+ * @throws AzureBlobFileSystemException
+ */
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ /**
+ * Tests the scenario where both the SASTokenProvider and a fixed SAS token
are configured.
+ * SASTokenProvider class should be chosen and User Delegation SAS should be
used.
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1497039338 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java: ## @@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final String properties, final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder(); abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM); +appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder); Review Comment: Nice Catch. Operations name added for file system level operations as well. Just FYI The purpose of having operation name here is to determine what all permissions are needed to be set in SAS Token generated. It is up to the custom SAS Token Provider Implementation to use this information and set appropriate permissions. For Example, MockDelegationSASTokenProvider does not allow file system level operations hence it will error out for these operations with SASTokenProviderException. Where as AccountSASGenerator will have fixed permissions irrespective of operation type. Operation Name are added here so that if a user wants to define their own implementation, they can choose to consume this information as per their needs and logic. They can also choose to simply ignore them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1953656743
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| -1 :x: | javadoc | 0m 26s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 25s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08
with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 1s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 138m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux bfa437800130 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2912980eefe39cd43a3dcc0c7a7f8ac8608ecdaa |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1953633449
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| -1 :x: | javadoc | 0m 27s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 25s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08
with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 137m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux e1125968539d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 92cb6717eba8eee97373982cabc6db9f40f60c1f |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
saxenapranav commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495337702
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,156 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+
+import org.junit.Assume;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+/**
+ * Tests to validate the choice between using a SASTokenProvider to generate
+ * a SAS or using a Fixed SAS Token configured by user.
+ */
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS = null;
+
+
+ /**
+ * To differentiate which config was used we will use different type of SAS
Tokens.
+ * For Fixed SAS Token we will use an Account SAS with permissions to do
File system level operations.
+ * For SASTokenProvider we will use a User Delegation SAS Token Provider
+ * such that File System level operations are not permitted.
+ */
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// SAS Token configured might not have permissions for creating file
system.
+// Shared Key must be configured to create one. Once created, a new
instance
+// of same file system will be used with SAS Authentication.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+generateAccountSAS(); }
+
+ /**
+ * Generates a Account SAS Token using the Account Shared Key to be used as
a fixed SAS Token.
+ * This will be used by individual tests to set in the configurations.
+ * @throws AzureBlobFileSystemException
+ */
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ /**
+ * Tests the scenario where both the SASTokenProvider and a fixed SAS token
are configured.
+ * SASTokenProvider class should be chosen and User Delegation SAS should be
used.
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
Review Comment:
Since, these tests may run in parallel with test of other classes. Lets make
clone of the configuration object and use it in the test.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -332,6 +335,8 @@ public AbfsRestOperation setFilesystemProperties(final
String properties,
final AbfsUriQueryBuilder abfsUriQueryBuilder =
createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM);
+appendSASTokenToQuery(ROOT_PATH, "", abfsUriQueryBuilder);
Review Comment:
why operationName is not provided here. If the thing is that for container
APIs, existing SAS mechanism can not work, we should still prevent them. We
might have t
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495271173
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
+
+// configuring a SASTokenProvider class: this provides a user delegation
SAS
+// user delegation SAS Provider is set
+// This easily distinguishes between results of filesystem level and blob
level operations to ensure correct SAS is chosen,
+// when both a provider class and fixed token is configured.
+testAbfsConfig.set(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
"org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider");
+
+// configuring the fixed SAS token
+testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS);
+
+// creating a new fs instance with the updated configs
+AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration());
+
+// testing a file system level operation
+TracingContext tracingContext = getTestTracingContext(newTestFs, true);
+// expected to fail in the ideal case, as delegation SAS will be chosen,
provider class is given preference when both are configured
+// this expectation is because filesystem level operations are beyond the
scope of Delegation SAS Token
+intercept(SASTokenProviderException.class,
+() -> {
+ newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
+});
+
+// testing blob level operation to ensure delegation SAS token is
otherwise valid and above operation fails only because it is fs level
+Path testPath = new Path("/testCorrectSASToken");
+newTestFs.create(testP
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495270507
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
+
+// configuring a SASTokenProvider class: this provides a user delegation
SAS
+// user delegation SAS Provider is set
+// This easily distinguishes between results of filesystem level and blob
level operations to ensure correct SAS is chosen,
+// when both a provider class and fixed token is configured.
+testAbfsConfig.set(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
"org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider");
+
+// configuring the fixed SAS token
+testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS);
+
+// creating a new fs instance with the updated configs
+AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration());
+
+// testing a file system level operation
+TracingContext tracingContext = getTestTracingContext(newTestFs, true);
+// expected to fail in the ideal case, as delegation SAS will be chosen,
provider class is given preference when both are configured
+// this expectation is because filesystem level operations are beyond the
scope of Delegation SAS Token
+intercept(SASTokenProviderException.class,
+() -> {
+ newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
+});
+
+// testing blob level operation to ensure delegation SAS token is
otherwise valid and above operation fails only because it is fs level
+Path testPath = new Path("/testCorrectSASToken");
+newTestFs.create(testP
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495270368
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AccountSASGenerator.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
+
+import java.time.Instant;
+
+/**
+ * Account SAS Generator to be used by tests
+ */
+
+public class AccountSASGenerator extends SASGenerator {
+ /**
+ * Creates Account SAS
+ *
https://learn.microsoft.com/en-us/rest/api/storageservices/create-account-sas
+ * @param accountKey: the storage account key
+ */
+ public AccountSASGenerator(byte[] accountKey) {
+super(accountKey);
+ }
+
+ public String getAccountSAS(String accountName) throws
AzureBlobFileSystemException {
+// retaining only the account name
+accountName = getCanonicalAccountName(accountName);
+String sp = "racwdl";
+String sv = "2021-06-08";
Review Comment:
Doesn't matter. Anything later than 2012 will work
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -309,6 +310,8 @@ public AbfsRestOperation createFilesystem(TracingContext
tracingContext)
final AbfsUriQueryBuilder abfsUriQueryBuilder = new AbfsUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM);
+// appending SAS Token to query
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495269995
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
+
+// configuring a SASTokenProvider class: this provides a user delegation
SAS
+// user delegation SAS Provider is set
+// This easily distinguishes between results of filesystem level and blob
level operations to ensure correct SAS is chosen,
+// when both a provider class and fixed token is configured.
+testAbfsConfig.set(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
"org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider");
Review Comment:
Taken
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockSASTokenProvider.java:
##
@@ -35,10 +39,19 @@ public class MockSASTokenProvider implements
SASTokenProvider {
private byte[] accountKey;
private ServiceSASGenerator generator;
private boolean skipAuthorizationForTestSetup = false;
+ protected static final Logger LOG =
Review Comment:
Made private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495269757
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AccountSASGenerator.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
+
+import java.time.Instant;
+
+/**
+ * Account SAS Generator to be used by tests
+ */
+
+public class AccountSASGenerator extends SASGenerator {
+ /**
+ * Creates Account SAS
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495268940 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AccountSASGenerator.java: ## @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs.utils; + +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException; +import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder; + +import java.time.Instant; + +/** + * Account SAS Generator to be used by tests Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495268719
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockSASTokenProvider.java:
##
@@ -35,10 +39,19 @@ public class MockSASTokenProvider implements
SASTokenProvider {
private byte[] accountKey;
private ServiceSASGenerator generator;
private boolean skipAuthorizationForTestSetup = false;
+ protected static final Logger LOG =
+ LoggerFactory.getLogger(MockSASTokenProvider.class);
// For testing we use a container SAS for all operations.
private String generateSAS(byte[] accountKey, String accountName, String
fileSystemName) {
- return generator.getContainerSASWithFullControl(accountName,
fileSystemName);
+String containerSAS = "";
+try {
+ containerSAS = generator.getContainerSASWithFullControl(accountName,
fileSystemName);
+} catch (InvalidConfigurationValueException e) {
+ LOG.debug(e.getMessage());
Review Comment:
Yes, this exception might occur while getting the canonical account name.
If the account name configured is ".dfs.core.windows.net"
i.e. account name without endpoint is empty string.
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AccountSASGenerator.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
+
+import java.time.Instant;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267955
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
+ * @throws Exception
+ */
+ @Test
+ public void testBothProviderFixedTokenConfigured() throws Exception {
+AbfsConfiguration testAbfsConfig = getConfiguration();
+
+// configuring a SASTokenProvider class: this provides a user delegation
SAS
+// user delegation SAS Provider is set
+// This easily distinguishes between results of filesystem level and blob
level operations to ensure correct SAS is chosen,
+// when both a provider class and fixed token is configured.
+testAbfsConfig.set(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
"org.apache.hadoop.fs.azurebfs.extensions.MockDelegationSASTokenProvider");
+
+// configuring the fixed SAS token
+testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS);
+
+// creating a new fs instance with the updated configs
+AzureBlobFileSystem newTestFs = (AzureBlobFileSystem)
FileSystem.newInstance(testAbfsConfig.getRawConfiguration());
+
+// testing a file system level operation
+TracingContext tracingContext = getTestTracingContext(newTestFs, true);
+// expected to fail in the ideal case, as delegation SAS will be chosen,
provider class is given preference when both are configured
+// this expectation is because filesystem level operations are beyond the
scope of Delegation SAS Token
+intercept(SASTokenProviderException.class,
+() -> {
+ newTestFs.getAbfsStore().getFilesystemProperties(tracingContext);
+});
+
+// testing blob level operation to ensure delegation SAS token is
otherwise valid and above operation fails only because it is fs level
+Path testPath = new Path("/testCorrectSASToken");
+newTestFs.create(testP
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267742
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
+
+ private String accountSAS;
+
+ public ITestAzureBlobFileSystemChooseSAS() throws Exception {
+// The test uses shared key to create a random filesystem and then creates
another
+// instance of this filesystem using SAS authorization.
+Assume.assumeTrue(this.getAuthType() == AuthType.SharedKey);
+ }
+
+ private void generateAccountSAS() throws AzureBlobFileSystemException {
+final String accountKey = getConfiguration().getStorageAccountKey();
+AccountSASGenerator configAccountSASGenerator = new
AccountSASGenerator(Base64.decode(accountKey));
+accountSAS = configAccountSASGenerator.getAccountSAS(getAccountName());
+ }
+
+ @Override
+ public void setup() throws Exception {
+createFilesystemForSASTests();
+super.setup();
+// obtaining an account SAS token from in-built generator to set as
configuration for testing filesystem level operations
+generateAccountSAS();
+ }
+
+ /**
+ * Tests the scenario where both the token provider class and a fixed token
are configured:
+ * whether the correct choice is made (precedence given to token provider
class), and the chosen SAS Token works as expected
Review Comment:
Taken.
Checked Everywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267501
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java:
##
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator;
+import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.junit.Assume;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_FIXED_TOKEN;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestAzureBlobFileSystemChooseSAS extends
AbstractAbfsIntegrationTest{
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267360 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java: ## @@ -0,0 +1,145 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs.azurebfs; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException; +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException; +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException; +import org.apache.hadoop.fs.azurebfs.services.AuthType; +import org.apache.hadoop.fs.azurebfs.utils.AccountSASGenerator; +import org.apache.hadoop.fs.azurebfs.utils.Base64; +import org.apache.hadoop.fs.azurebfs.utils.TracingContext; +import org.junit.Assume; Review Comment: Modified as per the rules -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552: URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267242 ## hadoop-tools/hadoop-azure/src/site/markdown/abfs.md: ## @@ -311,10 +311,11 @@ driven by them. 1. With the storage account's authentication secret in the configuration: "Shared Key". -1. Using OAuth 2.0 tokens of one form or another. -1. Deployed in-Azure with the Azure VMs providing OAuth 2.0 tokens to the application, +2. Using OAuth 2.0 tokens of one form or another. Review Comment: Makes sense. Taken -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495267010
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -269,6 +269,9 @@ public static String accountProperty(String property,
String account) {
public static final String FS_AZURE_ENABLE_DELEGATION_TOKEN =
"fs.azure.enable.delegation.token";
public static final String FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE =
"fs.azure.delegation.token.provider.type";
+ /** Key for fixed SAS token **/
Review Comment:
Taken.
Added everywhere.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -1285,6 +1294,14 @@ public static String getDirectoryQueryParameter(final
String path) {
return directory;
}
+ private String chooseSASToken(String operation, String path) throws
IOException {
+// chooses the SAS token provider class if it is configured, otherwise
reads the configured fixed token
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495257805
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
+ * to not proceed further than thi stage itself when none of the options are
available.
+ * 2. avoid using similar tokenProvider implementation to just read the
configured fixed token,
+ * as this could create confusion. The configuration is introduced
+ * primarily to avoid using any tokenProvider class/interface.
Also,implementing the SASTokenProvider requires relying on the raw
configurations.
+ * It is more stable to depend on the AbfsConfiguration with which a
filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
- throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ throw new SASTokenProviderException(String.format("Invalid auth type: %s
is being used, expecting SAS", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(!(sasTokenProviderImplementation == null
Review Comment:
Preference will be given to SASTokenProvider implementation.
Modified javadocs to make it clear
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
+ * to not proceed further than thi stage itself when none of the options are
available.
+ * 2. avoid using similar tokenProvider implementation to just read the
configured fixed token,
+ * as this could create confusion. The configuration is introduced
+ * primarily to avoid using any tokenProvider class/interface.
Also,implementing the SASTokenProvider requires relying on the raw
configurations.
+ * It is more stable to depend on the AbfsConfiguration with which a
filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.Sh
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1495257474
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
Review Comment:
Taken
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
+ * to not proceed further than thi stage itself when none of the options are
available.
+ * 2. avoid using similar tokenProvider implementation to just read the
configured fixed token,
+ * as this could create confusion. The configuration is introduced
+ * primarily to avoid using any tokenProvider class/interface.
Also,implementing the SASTokenProvider requires relying on the raw
configurations.
+ * It is more stable to depend on the AbfsConfiguration with which a
filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
- throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ throw new SASTokenProviderException(String.format("Invalid auth type: %s
is being used, expecting SAS", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(!(sasTokenProviderImplementation == null
+ && configuredFixedToken == null),
+ String.format(
+ "The value for both \"%s\" and \"%s\" cannot be invalid.",
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
steveloughran commented on code in PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#discussion_r1489922525
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
+ * to not proceed further than thi stage itself when none of the options are
available.
+ * 2. avoid using similar tokenProvider implementation to just read the
configured fixed token,
+ * as this could create confusion. The configuration is introduced
+ * primarily to avoid using any tokenProvider class/interface.
Also,implementing the SASTokenProvider requires relying on the raw
configurations.
+ * It is more stable to depend on the AbfsConfiguration with which a
filesystem is initialized,
+ * and eliminate chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
- throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ throw new SASTokenProviderException(String.format("Invalid auth type: %s
is being used, expecting SAS", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
+ Class sasTokenProviderImplementation =
+ getTokenProviderClass(authType, FS_AZURE_SAS_TOKEN_PROVIDER_TYPE,
+ null,
SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ String configuredFixedToken =
this.rawConfig.get(FS_AZURE_SAS_FIXED_TOKEN,
+ null);
+
+ Preconditions.checkArgument(!(sasTokenProviderImplementation == null
+ && configuredFixedToken == null),
+ String.format(
+ "The value for both \"%s\" and \"%s\" cannot be invalid.",
Review Comment:
prefer a message "at least one of %s and %s must be set"
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The fixed token is read and used later. This is done to:
+ * 1. check for cases where both are not set, while initializing
AbfsConfiguration,
Review Comment:
nit: use html ol and li elements so it renders properly (in IDEs as well as
javadocs}
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##
@@ -941,31 +941,57 @@ public AccessTokenProvider getTokenProvider() throws
TokenAccessProviderExceptio
}
}
+ /**
+ * The following method chooses between a configured fixed sas token, and a
user implementation of the SASTokenProvider interface,
+ * depending on which one is available. In case a user SASTokenProvider
implementation is not present, and a fixed token is configured,
+ * it simply returns null, to set the sasTokenProvider object for current
configuration instance to null.
+ * The f
Re: [PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
hadoop-yetus commented on PR #6552:
URL: https://github.com/apache/hadoop/pull/6552#issuecomment-1943548839
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08
with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 generated 1 new + 15
unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 57s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 135m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6552/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 2530f81620ee 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / db03c5fd4049a916c180f2cb33467c1d886ec27c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~2
[PR] HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication [hadoop]
anujmodi2021 opened a new pull request, #6552: URL: https://github.com/apache/hadoop/pull/6552 ### Description of PR Jira: https://issues.apache.org/jira/browse/HADOOP-18516 Changes ported from PR: https://github.com/apache/hadoop/pull/5148 This PR introduces a new configuration for Fixed SAS Tokens. Using this new configuration fs.azure.sas.fixed.token, users can configure a fixed SAS Token in the account settings files itself. Ideally, this should be used with SAS Tokens that are scoped at a container or account level (Service or Account SAS), which can be considered to be a constant for one account or container, over multiple operations. The other method of using a SAS Token remains valid as well, where a user provides a custom implementation of the SASTokenProvider interface, using which a SAS Token are obtained. A change for appending SAS Tokens to the queries of 4 filesystem level APIs is also introduced, helped by the introduction of the fixed token. When an Account SAS Token is configured as the fixed SAS Token, and it is used, it is ensured that these operations are within the scope of the SAS Token. The configuration introduced also bypasses the need for having an implementation of the SASTokenProvider interface in cases where a token simply be directly set. The code checks for whether the fixed token and the token provider class implementation are configured. In the case of both being set, preference is given to the SASTokenProvider implementation. It must be noted that if such an implementation provides a SAS Token which has a lower scope than Account SAS, some filesystem and service level operations might be out of scope and may not succeed. ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
