[jira] [Resolved] (HADOOP-10992) Merge KMS to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur resolved HADOOP-10992.
-
Resolution: Fixed
Fix Version/s: 2.6.0
Completed.
Merge KMS to branch-2
-
Key: HADOOP-10992
URL: https://issues.apache.org/jira/browse/HADOOP-10992
Project: Hadoop Common
Issue Type: Task
Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
A pre-requisite for getting HDFS encryption in branch-2 is KMS, we need to
merge all related JIRAs:
{code}
052932e7299ff64d36287b368f94ccf8698d5c9d HADOOP-10141. Create KeyProvider API
to separate encryption key storage from the applications. (omalley)
b72026617b038f588581d43c323718fe8120b400 HADOOP-10201. Add listing to
KeyProvider API. (Larry McCay via omalley)
4a178b6736d54e1b1940babd7cbda34921957d01 HADOOP-10177. Create CLI tools for
managing keys. (Larry McCay via omalley)
0cf6ccf606fceb6c06f35d72b2c2b679d71ad96c HADOOP-10237. JavaKeyStoreProvider
needs to set keystore permissions correctly. (Larry McCay via omalley)
56d349b81d24ef1421ffcdfb822a8fe122f05c80 HADOOP-10432. Refactor SSLFactory to
expose static method to determine HostnameVerifier. (tucu)
0d3cb277937eb7ec6a281dc7f236efe387fd HADOOP-10429. KeyStores should have
methods to generate the materials themselves, KeyShell should use them. (tucu)
d9c1c42fdfddb810ebe2ec151f751d05e987f25e HADOOP-10427. KeyProvider
implementations should be thread safe. (tucu)
98be41ff908acd2fa55c0b302c8a3def55987e41 HADOOP-10428. JavaKeyStoreProvider
should accept keystore password via configuration falling back to ENV VAR.
(tucu)
b2b05181682c2a55f5ed1cfa2c44f3390eebd5c4 HADOOP-10244. TestKeyShell
improperly tests the results of delete (Larry McCay via omalley)
83f057e8e1d16949b94fe2e99f4232ced8156e6a HADOOP-10430. KeyProvider Metadata
should have an optional description, there should be a method to retrieve the
metadata from all keys. (tucu)
f6f52ca1c2df57d13fa596e074accc0f3549ff58 HADOOP-10431. Change visibility of
KeyStore.Options getter methods to public. (tucu)
05e59fd8058f21a52d4a268af3a189c89ebad2fe HADOOP-10534. KeyProvider
getKeysMetadata should take a list of names rather than returning all keys.
(omalley)
16be41a63e4b3bd79b1cee4edce6df374666ca58 HADOOP-10433. Key Management Server
based on KeyProvider API. (tucu)
4bcaa45a2ea36fb440069c7a458cdc225cb862ca HADOOP-10583. bin/hadoop key throws
NPE with no args and assorted other fixups. (clamb via tucu)
1727e235c3d3317b2ac6d7c25ea01505853653ca HADOOP-10586. KeyShell doesn't allow
setting Options via CLI. (clamb via tucu)
6b410f3b2e185fca963c7db664395e97d76cd6ee HADOOP-10645. TestKMS fails because
race condition writing acl files. (tucu)
7868054902590af6dbda941f2cc8324267c8bef8 HADOOP-10611. KMS, keyVersion name
should not be assumed to be keyName@versionNumber. (tucu)
725f087f3f2fc31190810344d0e508e34b4a126e HADOOP-10607. Create API to separate
credential/password storage from applications. (Larry McCay via omalley)
097254f094b004404ba4754f97f906f46a12b0e4 HADOOP-10696. Add optional
attributes to KeyProvider Options and Metadata. (tucu)
a283b91add9e9230b9597fd33355822517a1852e HADOOP-10695. KMSClientProvider
should respect a configurable timeout. (yoderme via tucu)
6cef126f29673704c345c52995890ff48395ec1a HADOOP-10757. KeyProvider KeyVersion
should provide the key name. (asuresh via tucu)
9b7a1cb122c6a6041e718986085ec7f6bab422c4 HADOOP-10719. Add
generateEncryptedKey and decryptEncryptedKey methods to KeyProvider. (asuresh
via tucu)
9c03a4b321db7950d5652ba03022f9ee3ebd2d6f HADOOP-10769. Create KeyProvider
extension to handle delegation tokens. Contributed by Arun Suresh.
db91ab3d02fddfd325fd308e46f65075c2c6cd93 HADOOP-10812. Delegate
KeyProviderExtension#toString to underlying KeyProvider. (wang)
7c7911bbd63d30932df71af536f45c20adba88ff HADOOP-10736. Add key attributes to
the key shell. Contributed by Mike Yoder.
cfb5943d356fef911f424ed8250a9c02b706ecc6 HADOOP-10824. Refactor KMSACLs to
avoid locking. (Benoy Antony via umamahesh)
6b9b985233c293d22f89a4deadf871230f09d7ed HADOOP-10816. KeyShell returns -1 on
error to the shell, should be 1. (Mike Yoder via wang)
ceea01cff5762115c58817ab696cd11641bc9a98 HADOOP-10841. EncryptedKeyVersion
should have a key name property. (asuresh via tucu)
468a4fc00921ea7bc61bb60666e9352b0ad3928b HADOOP-10842. CryptoExtension
generateEncryptedKey method should receive the key name. (asuresh via tucu)
c6d60c6db8b22d6dc45e63073bc5bb52dc041a8c HADOOP-10750. KMSKeyProviderCache
should be in hadoop-common. (asuresh via tucu)
c3eca9f2504ed619a3edcf3d3eafc286133911d0 HADOOP-10720. KMS: Implement
generateEncryptedKey and
[jira] [Moved] (HADOOP-10994) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
[ https://issues.apache.org/jira/browse/HADOOP-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur moved HDFS-6909 to HADOOP-10994: --- Component/s: (was: security) security Target Version/s: (was: 2.6.0) Affects Version/s: (was: 3.0.0) 3.0.0 Key: HADOOP-10994 (was: HDFS-6909) Project: Hadoop Common (was: Hadoop HDFS) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys --- Key: HADOOP-10994 URL: https://issues.apache.org/jira/browse/HADOOP-10994 Project: Hadoop Common Issue Type: Task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Currently is using JDK Cipher, with fs-encryption branch merged into trunk we can swap to CryptoCodec. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10994) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
[
https://issues.apache.org/jira/browse/HADOOP-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10994:
Attachment: HADOOP-10994.patch
besides converting to use {{CryptoCodec}} adding {{getConf()}} method to
{{KeyProvider}} so the {{CryptoCodec}} can be instantiated correctly.
KeyProviderCryptoExtension should use CryptoCodec for generation/decryption
of keys
---
Key: HADOOP-10994
URL: https://issues.apache.org/jira/browse/HADOOP-10994
Project: Hadoop Common
Issue Type: Task
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HADOOP-10994.patch
Currently is using JDK Cipher, with fs-encryption branch merged into trunk we
can swap to CryptoCodec.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10994) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
[ https://issues.apache.org/jira/browse/HADOOP-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10994: Status: Patch Available (was: Open) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys --- Key: HADOOP-10994 URL: https://issues.apache.org/jira/browse/HADOOP-10994 Project: Hadoop Common Issue Type: Task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10994.patch Currently is using JDK Cipher, with fs-encryption branch merged into trunk we can swap to CryptoCodec. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10994) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
[ https://issues.apache.org/jira/browse/HADOOP-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10994: Attachment: HADOOP-10994.patch new patch making conf final. KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys --- Key: HADOOP-10994 URL: https://issues.apache.org/jira/browse/HADOOP-10994 Project: Hadoop Common Issue Type: Task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10994.patch, HADOOP-10994.patch Currently is using JDK Cipher, with fs-encryption branch merged into trunk we can swap to CryptoCodec. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10956) Fix create-release script to include docs in the binary
[
https://issues.apache.org/jira/browse/HADOOP-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106139#comment-14106139
]
Alejandro Abdelnur commented on HADOOP-10956:
-
To get the CHANGES.txt, releasenotes.html, NOTICE.txt, README.txt, LICENCE.txt
files, we should do it via assembly descriptors, for example:
{code}
fileSet
useStrictFilteringtrue/useStrictFiltering
outputDirectory//outputDirectory
includes
includeNOTICE.txt/include
includeREADME.txt/include
includeLICENSE.txt/include
/includes
/fileSet
{code}
For the source tarball this should be in:
hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml
For the binary tarball this should be in the hadoop-dist/pom.xml, in the dist
profile where the run cp calls are done.
Fix create-release script to include docs in the binary
---
Key: HADOOP-10956
URL: https://issues.apache.org/jira/browse/HADOOP-10956
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.5.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Blocker
Attachments: hadoop-10956-1.patch, hadoop-10956-2.patch
The create-release script doesn't include docs in the binary tarball. We
should fix that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106455#comment-14106455 ] Alejandro Abdelnur commented on HADOOP-10880: - the extra javac warning is expected as a testcase is testing a deprecated method, I'll increment the javac warning count in test-patch by 1 on commit. Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10956) Fix create-release script to include docs in the binary
[ https://issues.apache.org/jira/browse/HADOOP-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106467#comment-14106467 ] Alejandro Abdelnur commented on HADOOP-10956: - the bin tarball has an api/ dir with the javadocs at top level (the hdfs javadocs seem missing from there, didn't check if other ones are missing), the share/docs/.. dir has only CHANGES/LICENSE/NOTICE/README files (there is no need for those there as they are at top level), all other docs are missing. Fix create-release script to include docs in the binary --- Key: HADOOP-10956 URL: https://issues.apache.org/jira/browse/HADOOP-10956 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 2.5.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Blocker Attachments: hadoop-10956-1.patch, hadoop-10956-2.patch, hadoop-10956-3.patch The create-release script doesn't include docs in the binary tarball. We should fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104146#comment-14104146 ] Alejandro Abdelnur commented on HADOOP-10880: - test failures are unrelated Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[ https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100737#comment-14100737 ] Alejandro Abdelnur commented on HADOOP-10307: - Looking at the patch, it duplicates part of the logic done in HADOOP-9054 (using the user-agent), if we do that, then we should built on HADOOP-9054, not duplicating the code. Also, personally, I don't like the idea of query string param to indicate the auth scheme to use. The other day, talking with [~daryn], he suggested that to support multiple authentication schemes we could multiple WWW-Authenticate headers. I was not aware this was possible, I've checked around and it is possible. You can either have multiple schemes in the same WWW-Authenticate header (parsing that is a bit tricky and HttpClient does not support it yet (HTTPCLIENT-1489). Or you can have multiple WWW-Authenticate headers. IMO, we should do as [~daryn] suggested me offline, add support for multiple authentication schemes. The AuthenticationFilter would have to support a list of AuthenticationHandlers, when a requests comes in and it is not authenticated (because of a cookie and because no handler found authentication info in it), then the response should include the challenges of all AuthenticationHandlers. Then the client should choose the strongest one it supports. Support multiple Authentication mechanisms for HTTP --- Key: HADOOP-10307 URL: https://issues.apache.org/jira/browse/HADOOP-10307 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HADOOP-10307.patch, HADOOP-10307.patch, HADOOP-10307.patch Currently it is possible to specify a custom Authentication Handler for HTTP authentication. We have a requirement to support multiple mechanisms to authenticate HTTP access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10970) Cleanup KMS configuration keys
[ https://issues.apache.org/jira/browse/HADOOP-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101334#comment-14101334 ] Alejandro Abdelnur commented on HADOOP-10970: - +1 pending jenkins. Cleanup KMS configuration keys -- Key: HADOOP-10970 URL: https://issues.apache.org/jira/browse/HADOOP-10970 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-10970.001.patch, hadoop-10970.002.patch It'd be nice to add descriptions to the config keys in kms-site.xml. Also, it'd be good to rename key.provider.path to key.provider.uri for clarity, or just drop .path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10970) Cleanup KMS configuration keys
[ https://issues.apache.org/jira/browse/HADOOP-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101536#comment-14101536 ] Alejandro Abdelnur commented on HADOOP-10970: - +1 again Cleanup KMS configuration keys -- Key: HADOOP-10970 URL: https://issues.apache.org/jira/browse/HADOOP-10970 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-10970.001.patch, hadoop-10970.002.patch, hadoop-10970.003.patch It'd be nice to add descriptions to the config keys in kms-site.xml. Also, it'd be good to rename key.provider.path to key.provider.uri for clarity, or just drop .path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101540#comment-14101540 ] Alejandro Abdelnur commented on HADOOP-10880: - I've talked with [~daryn] over the phone and he'd be OK on keeping the scope of this JIRA as initially intended, not adding digest stuff to it. For reasons along the lines the ones mentioned in my previous comment. Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10880: Status: Patch Available (was: Open) Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[ https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100142#comment-14100142 ] Alejandro Abdelnur commented on HADOOP-10307: - i want to review this patch, i'll look at it monday morning. Support multiple Authentication mechanisms for HTTP --- Key: HADOOP-10307 URL: https://issues.apache.org/jira/browse/HADOOP-10307 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0 Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HADOOP-10307.patch, HADOOP-10307.patch, HADOOP-10307.patch Currently it is possible to specify a custom Authentication Handler for HTTP authentication. We have a requirement to support multiple mechanisms to authenticate HTTP access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098690#comment-14098690 ] Alejandro Abdelnur commented on HADOOP-10698: - test failure is unrelated. KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk. KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 3.0.0 Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10970) Cleanup KMS configuration keys
[
https://issues.apache.org/jira/browse/HADOOP-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098698#comment-14098698
]
Alejandro Abdelnur commented on HADOOP-10970:
-
Larry, now that we are at it, KeyProvider and KeyCredentials implementation
lookup is completely different from how things are done in the rest of Hadoop.
Do you see any reason for not having a {{public static KeyProvider get(URI
uri)}} method (and one for {{KeyCredentials}})?
Cleanup KMS configuration keys
--
Key: HADOOP-10970
URL: https://issues.apache.org/jira/browse/HADOOP-10970
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hadoop-10970.001.patch
It'd be nice to add descriptions to the config keys in kms-site.xml.
Also, it'd be good to rename key.provider.path to key.provider.uri for
clarity, or just drop .path.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098734#comment-14098734 ] Alejandro Abdelnur commented on HADOOP-10880: - [~daryn], I’ve gone the digest approach to see how things would work but I’ve found a few issues with it which I think make the use of digest a moot thing: Digest typically assumes a challenge with a nonce from the server, the nonce is use to create the hash of the common secret. In our case, we would be doing a proactive digest (we don’t want 2 HTTP calls on every HTTP call using a DT), this means we don’t have a nonce from the server. We could get the server side to use proactively the AuthenticationInfo to provide a nonce in advance, that that would break pipelined requests. We could get the client to issue a nonce and the server honor it, but that would not work without using our client and server that understand that (curl would break). Even if we sort it out, we have another problem, the DT is an opaque string blob to the client. The DT includes the token identifier and the password, we would have to use the token identifier as the 'user' and the password as the 'password' in creating the Disgest 'user:digest(password)' client header. A curl client would not have a way to break down the DT opaque string into identifier and password. Another point, when getting, renewing or canceling a DT, we send the full DT opaque string. So using a digest for does not prevent for an attacker to get the full DT. Given all this, my take is, the current patch that moves the DTs from URL to headers and just use HTTPS. Thoughts? Am I missing something? Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10970) Cleanup KMS configuration keys
[
https://issues.apache.org/jira/browse/HADOOP-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098765#comment-14098765
]
Alejandro Abdelnur commented on HADOOP-10970:
-
The same we do for FileSystem, ie:
{code}
KeyProvider kp = KeyProvider.get(new
URI(jks://h...@nn1.example.com/my/keys.jks));
{code}
Cleanup KMS configuration keys
--
Key: HADOOP-10970
URL: https://issues.apache.org/jira/browse/HADOOP-10970
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hadoop-10970.001.patch
It'd be nice to add descriptions to the config keys in kms-site.xml.
Also, it'd be good to rename key.provider.path to key.provider.uri for
clarity, or just drop .path.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[ https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098778#comment-14098778 ] Alejandro Abdelnur commented on HADOOP-10904: - Why not do jks://hdfs/nn.example.com/foo/bar.jks and jks://har/hdfs/nn.example.com/foo/bar.jks, where nesting simply adds a path prefix, and the un-nesting bring it back as scheme? Provide Alt to Clear Text Passwords through Cred Provider API - Key: HADOOP-10904 URL: https://issues.apache.org/jira/browse/HADOOP-10904 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Larry McCay This is an umbrella jira to track various child tasks to uptake the credential provider API to enable deployments without storing passwords/credentials in clear text. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Attachment: HADOOP-10770.patch Thanks atm. Uploading patch with corrected indentation. committing momentarily. KMS add delegation token support Key: HADOOP-10770 URL: https://issues.apache.org/jira/browse/HADOOP-10770 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk. KMS add delegation token support Key: HADOOP-10770 URL: https://issues.apache.org/jira/browse/HADOOP-10770 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 3.0.0 Attachments: HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Attachment: HADOOP-10698.patch Thanks atm, attaching patch with updated docs. kicking test-patch, will commit after it comes back. KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Status: Patch Available (was: Open) KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10967) Improve DefaultCryptoExtension#generateEncryptedKey performance
[
https://issues.apache.org/jira/browse/HADOOP-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098198#comment-14098198
]
Alejandro Abdelnur commented on HADOOP-10967:
-
Nice, +1.
Improve DefaultCryptoExtension#generateEncryptedKey performance
Key: HADOOP-10967
URL: https://issues.apache.org/jira/browse/HADOOP-10967
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HADOOP-10967.001.patch
This JIRA is to improve generateEncryptedKey performance:
*1.* SecureRandom#generateSeed is very slow, we should use
SecureRandom#nextBytes to generate the {{IV}} which is much faster.
*2.* Define SecureRandom as threadlocal object which can improve the
performance a bit.
*3.* Use {{new SecureRandom()}} instead of SHA1PRNG, the former has better
entropy.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10970) Cleanup KMS configuration keys
[ https://issues.apache.org/jira/browse/HADOOP-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098202#comment-14098202 ] Alejandro Abdelnur commented on HADOOP-10970: - LGTM +1 pending jenkins. Cleanup KMS configuration keys -- Key: HADOOP-10970 URL: https://issues.apache.org/jira/browse/HADOOP-10970 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-10970.001.patch It'd be nice to add descriptions to the config keys in kms-site.xml. Also, it'd be good to rename key.provider.path to key.provider.uri for clarity, or just drop .path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10967) Improve DefaultCryptoExtension#generateEncryptedKey performance
[
https://issues.apache.org/jira/browse/HADOOP-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10967:
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Yi. Committed to trunk.
Improve DefaultCryptoExtension#generateEncryptedKey performance
Key: HADOOP-10967
URL: https://issues.apache.org/jira/browse/HADOOP-10967
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 3.0.0
Attachments: HADOOP-10967.001.patch
This JIRA is to improve generateEncryptedKey performance:
*1.* SecureRandom#generateSeed is very slow, we should use
SecureRandom#nextBytes to generate the {{IV}} which is much faster.
*2.* Define SecureRandom as threadlocal object which can improve the
performance a bit.
*3.* Use {{new SecureRandom()}} instead of SHA1PRNG, the former has better
entropy.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095943#comment-14095943 ] Alejandro Abdelnur commented on HADOOP-9902: Arpit, given the release notes, a bunch of incompatible changes. I've missed that before. So it cannot go in branch-2 as is, only trunk. My concern is that it will sit idle in trunk until a Hadoop 3 release. If others don't care about, well, I'm +0 on this. Ellen, you should have an explicit +1 before committing. Roman seems to have reviewed things in detail, I would ping him to stamp the +1. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Attachment: HADOOP-10836.patch [~atm], thxs for the review. yes, those 2 properties should be commented out, i've uploaded a new patch with the following comment at the beginning of the commented section: The following 2 properties within this comment are provided as an example to facilitate configuring HttpFS proxyusers. committing momentarily. Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10822) Refactor HTTP proxyuser support out of HttpFS into common
[ https://issues.apache.org/jira/browse/HADOOP-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved HADOOP-10822. - Resolution: Invalid converted HADOOP-10835 and HADOOP-10836 to issues as HADOOP-10836 should be an HDFS issue and subtasks cannot belong to different projects. Refactor HTTP proxyuser support out of HttpFS into common - Key: HADOOP-10822 URL: https://issues.apache.org/jira/browse/HADOOP-10822 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur HttpFS implements HTTP proxyuser support inline in httpfs code. For HADOOP-10698 we need similar functionality for KMS. Not to duplicate code, we should refactor existing code to common. We should also leverage HADOOP-10817. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Issue Type: Improvement (was: Sub-task)
Parent: (was: HADOOP-10822)
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch, HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Issue Type: Bug (was: Sub-task) Parent: (was: HADOOP-10822) Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Issue Type: Improvement (was: Bug) Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Attachment: HADOOP-10770.patch rebasing to trunk. KMS add delegation token support Key: HADOOP-10770 URL: https://issues.apache.org/jira/browse/HADOOP-10770 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Status: Patch Available (was: Open) KMS add delegation token support Key: HADOOP-10770 URL: https://issues.apache.org/jira/browse/HADOOP-10770 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10880: Attachment: HADOOP-10880.patch rebasing to trunk. Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Attachment: HADOOP-10698.patch rebasing to trunk KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096348#comment-14096348 ] Alejandro Abdelnur commented on HADOOP-10880: - daryn, i've thinking about this. if we do that, how non-java clients will do it? a curl client for example. Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094109#comment-14094109 ] Alejandro Abdelnur commented on HADOOP-10836: - existing httpfs testcases excercise the code refactoring, there was no need to modify them, thus the complain from test-patch, the test report shows all httpfs tests passing. Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093115#comment-14093115 ] Alejandro Abdelnur commented on HADOOP-9902: [~aw], I was under the impression we were targeting this for branch-2? is not the case? If we don't do that, given that we don't have imminent plans to create a branch-3 out of trunk, we are at risk of getting things stale in trunk as people add changes in branch-2 only. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093253#comment-14093253 ] Alejandro Abdelnur commented on HADOOP-9902: bq. If trunk is getting 'stale', then that sounds like an issue for the PMC to take up. I'm being proactive on this one. I'm trying to avoid getting into that situation. I'd love to get this in, just in a way it is exercised and refined ASAP. Else, a year from now or more we'll be battling with it. What are the key issues to be addressed for getting this in branch-2 and how can we take care of it? Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Attachment: HADOOP-10835.patch
thanks @atm, new patch addressing your comments.
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch, HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093482#comment-14093482
]
Alejandro Abdelnur commented on HADOOP-10835:
-
failure is unrelated.
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch, HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
committed to trunk and branch-2.
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch, HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Attachment: HADOOP-10836.patch rebasing patch on trunk now that HADOOP-10835 is committed. Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10947) Get rid of test krb5.conf files in test resources
[ https://issues.apache.org/jira/browse/HADOOP-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned HADOOP-10947: --- Assignee: (was: Alejandro Abdelnur) Get rid of test krb5.conf files in test resources - Key: HADOOP-10947 URL: https://issues.apache.org/jira/browse/HADOOP-10947 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur HADOOP-8078 added to trunk the possibility to run apacheds kdc for tests, downloading and configuring it by hand. Doing this, changed krb5.conf file in test/resources. krb5.conf is set as a system property, getDefaultRealm() resolves to that the first use. In trunk default realm is set to EXAMPLE.COM, in branch-2 to APACHE.ORG (because HADOOP-8078 was never merged into branch-2). This causes problems with tests using minikdc. I've noticed this while merging HADOOP-10771 into branch-2. We can get rid of krb5.conf files and revert HADOOP-8078 because we can use embedded minikdc in test cases now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10947) Get rid of test krb5.conf files in test resources
Alejandro Abdelnur created HADOOP-10947: --- Summary: Get rid of test krb5.conf files in test resources Key: HADOOP-10947 URL: https://issues.apache.org/jira/browse/HADOOP-10947 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur HADOOP-8078 added to trunk the possibility to run apacheds kdc for tests, downloading and configuring it by hand. Doing this, changed krb5.conf file in test/resources. krb5.conf is set as a system property, getDefaultRealm() resolves to that the first use. In trunk default realm is set to EXAMPLE.COM, in branch-2 to APACHE.ORG (because HADOOP-8078 was never merged into branch-2). This causes problems with tests using minikdc. I've noticed this while merging HADOOP-10771 into branch-2. We can get rid of krb5.conf files and revert HADOOP-8078 because we can use embedded minikdc in test cases now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10799) Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved HADOOP-10799. - Resolution: Invalid Refactor HTTP delegation token logic from httpfs into reusable code in hadoop-common. - Key: HADOOP-10799 URL: https://issues.apache.org/jira/browse/HADOOP-10799 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch, HADOOP-10799.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Attachment: HADOOP-10835.patch
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10835:
Status: Patch Available (was: Open)
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10835) Implement HTTP proxyuser support in HTTP authentication client/server libraries
[
https://issues.apache.org/jira/browse/HADOOP-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090355#comment-14090355
]
Alejandro Abdelnur commented on HADOOP-10835:
-
rebasing patch to trunk
Implement HTTP proxyuser support in HTTP authentication client/server
libraries
---
Key: HADOOP-10835
URL: https://issues.apache.org/jira/browse/HADOOP-10835
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: HADOOP-10835.patch, HADOOP-10835.patch,
HADOOP-10835.patch
This is to implement generic handling of proxyuser in the
{{DelegationTokenAuthenticatedURL}} and
{{DelegationTokenAuthenticationFilter}} classes and to wire properly UGI on
the server side.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10800) Refactor HttpFS to use hadoop-common HTTP delegation token support.
[ https://issues.apache.org/jira/browse/HADOOP-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10800: Resolution: Invalid Status: Resolved (was: Patch Available) Refactor HttpFS to use hadoop-common HTTP delegation token support. --- Key: HADOOP-10800 URL: https://issues.apache.org/jira/browse/HADOOP-10800 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10800.patch, HADOOP-10800.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10698) KMS, add proxyuser support
[ https://issues.apache.org/jira/browse/HADOOP-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10698: Attachment: HADOOP-10698.patch rebasing to trunk KMS, add proxyuser support -- Key: HADOOP-10698 URL: https://issues.apache.org/jira/browse/HADOOP-10698 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10698.patch, HADOOP-10698.patch, HADOOP-10698.patch Add proxyuser support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Status: Patch Available (was: Open) Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Attachment: COMBO.patch combo of HADOOP-10835 HADOOP-10836 for test-patch to run. Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10770) KMS add delegation token support
[ https://issues.apache.org/jira/browse/HADOOP-10770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10770: Attachment: HADOOP-10770.patch rebasing to trunk KMS add delegation token support Key: HADOOP-10770 URL: https://issues.apache.org/jira/browse/HADOOP-10770 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HADOOP-10770.patch, HADOOP-10770.patch, HADOOP-10770.patch This is a follow up on HADOOP-10769 for KMS itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10836) Replace HttpFS custom proxyuser handling with common implementation
[ https://issues.apache.org/jira/browse/HADOOP-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10836: Attachment: HADOOP-10836.patch rebasing patch to trunk Replace HttpFS custom proxyuser handling with common implementation --- Key: HADOOP-10836 URL: https://issues.apache.org/jira/browse/HADOOP-10836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: COMBO.patch, HADOOP-10836.patch, HADOOP-10836.patch, HADOOP-10836.patch Use HADOOP-10835 to implement proxyuser logic in HttpFS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10880: Attachment: HADOOP-10880.patch rebasing to trunk Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10880) Move HTTP delegation tokens out of URL querystring to a header
[ https://issues.apache.org/jira/browse/HADOOP-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090916#comment-14090916 ] Alejandro Abdelnur commented on HADOOP-10880: - only requests authenticated by the authenticatorhandler (ie kerberos...) issue hadoop-auth cookies, requests that present delegation tokens do not issue cookies (actually they issue a expired one to force a flushing). Move HTTP delegation tokens out of URL querystring to a header -- Key: HADOOP-10880 URL: https://issues.apache.org/jira/browse/HADOOP-10880 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Blocker Attachments: HADOOP-10880.patch, HADOOP-10880.patch Following up on a discussion in HADOOP-10799. Because URLs are often logged, delegation tokens may end up in LOG files while they are still valid. We should move the tokens to a header. We should still support tokens in the querystring for backwards compatibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10224) JavaKeyStoreProvider has to protect against corrupting underlying store
[ https://issues.apache.org/jira/browse/HADOOP-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091408#comment-14091408 ] Alejandro Abdelnur commented on HADOOP-10224: - +1 pending jenkins. JavaKeyStoreProvider has to protect against corrupting underlying store --- Key: HADOOP-10224 URL: https://issues.apache.org/jira/browse/HADOOP-10224 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Arun Suresh Attachments: HADOOP-10224.1.patch, HADOOP-10224.2.patch, HADOOP-10224.3.patch, HADOOP-10224.4.patch, HADOOP-10224.5.patch, HADOOP-10224.6.patch, HADOOP-10224.7.patch, HADOOP-10224.8.patch, HADOOP-10224.9.patch Java keystores get corrupted at times. A key management operation that writes the store to disk could cause a corruption and all protected data would then be unaccessible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10862) Miscellaneous trivial corrections to KMS classes
[
https://issues.apache.org/jira/browse/HADOOP-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091419#comment-14091419
]
Alejandro Abdelnur commented on HADOOP-10862:
-
+1
Miscellaneous trivial corrections to KMS classes
Key: HADOOP-10862
URL: https://issues.apache.org/jira/browse/HADOOP-10862
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Attachments: HADOOP-10862.1.patch, HADOOP-10862.2.patch
{{KMSRESTConstants.java}}, {{KEY_OP}} should be {{KEYS}} and value should be
{{keys}}.
{{KMS.java}} should be annotated with Jersey {{@Singleton}} to avoid creating
an instance on every request, it is thread safe already.
Make sure all KMS related classes are annotated with private audience.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10862) Miscellaneous trivial corrections to KMS classes
[
https://issues.apache.org/jira/browse/HADOOP-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10862:
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Arun. Committed to trunk.
Miscellaneous trivial corrections to KMS classes
Key: HADOOP-10862
URL: https://issues.apache.org/jira/browse/HADOOP-10862
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Fix For: 3.0.0
Attachments: HADOOP-10862.1.patch, HADOOP-10862.2.patch
{{KMSRESTConstants.java}}, {{KEY_OP}} should be {{KEYS}} and value should be
{{keys}}.
{{KMS.java}} should be annotated with Jersey {{@Singleton}} to avoid creating
an instance on every request, it is thread safe already.
Make sure all KMS related classes are annotated with private audience.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10224) JavaKeyStoreProvider has to protect against corrupting underlying store
[ https://issues.apache.org/jira/browse/HADOOP-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10224: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Arun, nice work. Thanks Larry for reviewing it in detail. Committed to trunk. JavaKeyStoreProvider has to protect against corrupting underlying store --- Key: HADOOP-10224 URL: https://issues.apache.org/jira/browse/HADOOP-10224 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Arun Suresh Fix For: 3.0.0 Attachments: HADOOP-10224.1.patch, HADOOP-10224.2.patch, HADOOP-10224.3.patch, HADOOP-10224.4.patch, HADOOP-10224.5.patch, HADOOP-10224.6.patch, HADOOP-10224.7.patch, HADOOP-10224.8.patch, HADOOP-10224.9.patch Java keystores get corrupted at times. A key management operation that writes the store to disk could cause a corruption and all protected data would then be unaccessible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10862) Miscellaneous trivial corrections to KMS classes
[
https://issues.apache.org/jira/browse/HADOOP-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088890#comment-14088890
]
Alejandro Abdelnur commented on HADOOP-10862:
-
*KMS.java*:
* On the message template constants, we were using 2 single quotes because the
single quote has a special meaning (take {} as literal) and you have to escape
it using 2 single quotes. Now that we are using {{String.format()}}, I believe
we should be just 1 single quote.
* The {{KMSOps}} enum seems identical to the {{KMSACLs.Type}} enum, would make
sense to bubble up KMSOps enum to top level class and use that in both places?
*KMSAudit.java*:
* I was not suggesting changing the slf4j message templates using {} to
String.format(), we should not use {{String.format()}} here, so the message is
resolved only if the log level is enabled (using {{String.format()}}, the
message is always resolved).
Miscellaneous trivial corrections to KMS classes
Key: HADOOP-10862
URL: https://issues.apache.org/jira/browse/HADOOP-10862
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Arun Suresh
Attachments: HADOOP-10862.1.patch
{{KMSRESTConstants.java}}, {{KEY_OP}} should be {{KEYS}} and value should be
{{keys}}.
{{KMS.java}} should be annotated with Jersey {{@Singleton}} to avoid creating
an instance on every request, it is thread safe already.
Make sure all KMS related classes are annotated with private audience.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10771) Refactor HTTP delegation support out of httpfs to common
[
https://issues.apache.org/jira/browse/HADOOP-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090048#comment-14090048
]
Alejandro Abdelnur commented on HADOOP-10771:
-
[~daryn], thanks for reviewing this, both by patch and by chatting with me over
the phone.
The patch is moving existing logic into common and doing minimal fixes, the
follow up JIRAs are taking care of most of your concerns. Specifically the UGI
handling and proxyuser handling.
The auth cookie is require to avoid doing SPNEGO handshake (requesting a
service ticket from the kdc) on every request. If network sniffing is an issue,
then HTTPS should be used to avoid that. If a process itself gives away the
cookie, then that is no different that a process via RPC asking for several
delegation tokens and giving them away.
I'll open a JIRA to clean up the exceptions and propagate info to the client to
regenerate and throw.
Thx
Refactor HTTP delegation support out of httpfs to common
Key: HADOOP-10771
URL: https://issues.apache.org/jira/browse/HADOOP-10771
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: COMBO.patch, COMBO.patch, HADOOP-10771.patch,
HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.sh
HttpFS implements delegation token support in {{AuthenticationFilter}}
{{AuthenticationHandler}} subclasses.
For HADOOP-10770 we need similar functionality for KMS.
Not to duplicate code, we should refactor existing code to common.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10771) Refactor HTTP delegation support out of httpfs to common
[
https://issues.apache.org/jira/browse/HADOOP-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10771:
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
[~daryn], [~atm], thanks for the discussions and reviews. Committed to trunk
and branch-2.
Refactor HTTP delegation support out of httpfs to common
Key: HADOOP-10771
URL: https://issues.apache.org/jira/browse/HADOOP-10771
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.6.0
Attachments: COMBO.patch, COMBO.patch, HADOOP-10771.patch,
HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.sh
HttpFS implements delegation token support in {{AuthenticationFilter}}
{{AuthenticationHandler}} subclasses.
For HADOOP-10770 we need similar functionality for KMS.
Not to duplicate code, we should refactor existing code to common.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10933) FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords
[ https://issues.apache.org/jira/browse/HADOOP-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086665#comment-14086665 ] Alejandro Abdelnur commented on HADOOP-10933: - LGTM, +1. FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords Key: HADOOP-10933 URL: https://issues.apache.org/jira/browse/HADOOP-10933 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Larry McCay Assignee: Larry McCay Attachments: HADOOP-10933.patch As part of HADOOP-10904, this jira represents the ability to leverage the credential provider API when clear text passwords on disk are unacceptable. By using the Configuration.getPassword method, the credential provider API may be used while maintaining backward compatibility for passwords stored in config/files. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
[
https://issues.apache.org/jira/browse/HADOOP-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10918:
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk.
JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 3.0.0
Attachments: HADOOP-10918.patch, HADOOP-10918.patch
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10791) AuthenticationFilter should support externalizing the secret for signing and provide rotation support
[ https://issues.apache.org/jira/browse/HADOOP-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086760#comment-14086760 ] Alejandro Abdelnur commented on HADOOP-10791: - +1 AuthenticationFilter should support externalizing the secret for signing and provide rotation support - Key: HADOOP-10791 URL: https://issues.apache.org/jira/browse/HADOOP-10791 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Robert Kanter Attachments: HADOOP-10791.patch, HADOOP-10791.patch, HADOOP-10791.patch, HADOOP-10791.patch It should be possible to externalize the secret used to sign the hadoop-auth cookies. In the case of WebHDFS the shared secret used by NN and DNs could be used. In the case of Oozie HA, the secret could be stored in Oozie HA control data in ZooKeeper. In addition, it is desirable for the secret to change periodically, this means that the AuthenticationService should remember a previous secret for the max duration of hadoop-auth cookie. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10791) AuthenticationFilter should support externalizing the secret for signing and provide rotation support
[ https://issues.apache.org/jira/browse/HADOOP-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10791: Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Robert. Committed to trunk and branch-2. AuthenticationFilter should support externalizing the secret for signing and provide rotation support - Key: HADOOP-10791 URL: https://issues.apache.org/jira/browse/HADOOP-10791 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Robert Kanter Fix For: 2.6.0 Attachments: HADOOP-10791.patch, HADOOP-10791.patch, HADOOP-10791.patch, HADOOP-10791.patch It should be possible to externalize the secret used to sign the hadoop-auth cookies. In the case of WebHDFS the shared secret used by NN and DNs could be used. In the case of Oozie HA, the secret could be stored in Oozie HA control data in ZooKeeper. In addition, it is desirable for the secret to change periodically, this means that the AuthenticationService should remember a previous secret for the max duration of hadoop-auth cookie. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10933) FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords
[ https://issues.apache.org/jira/browse/HADOOP-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086809#comment-14086809 ] Alejandro Abdelnur commented on HADOOP-10933: - I'm committing this right now to trunk and branch-2. FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords Key: HADOOP-10933 URL: https://issues.apache.org/jira/browse/HADOOP-10933 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Larry McCay Assignee: Larry McCay Attachments: HADOOP-10933.patch As part of HADOOP-10904, this jira represents the ability to leverage the credential provider API when clear text passwords on disk are unacceptable. By using the Configuration.getPassword method, the credential provider API may be used while maintaining backward compatibility for passwords stored in config/files. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10933) FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords
[ https://issues.apache.org/jira/browse/HADOOP-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10933: Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Larry. Committed to trunk and branch-2. FileBasedKeyStoresFactory Should use Configuration.getPassword for SSL Passwords Key: HADOOP-10933 URL: https://issues.apache.org/jira/browse/HADOOP-10933 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Larry McCay Assignee: Larry McCay Fix For: 2.6.0 Attachments: HADOOP-10933.patch As part of HADOOP-10904, this jira represents the ability to leverage the credential provider API when clear text passwords on disk are unacceptable. By using the Configuration.getPassword method, the credential provider API may be used while maintaining backward compatibility for passwords stored in config/files. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10802) Add metrics for KMS client and server encrypted key caches
[ https://issues.apache.org/jira/browse/HADOOP-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084654#comment-14084654 ] Alejandro Abdelnur commented on HADOOP-10802: - i've browsed the patch, the approach looks good. just not sure about the captured metrics. IMO, we shouldn't capture per key metrics. knowing a key is request more than others, I don't think brings much. i would keep track of going below watermarks and refill requests and I would log the key names. Add metrics for KMS client and server encrypted key caches -- Key: HADOOP-10802 URL: https://issues.apache.org/jira/browse/HADOOP-10802 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Arun Suresh Attachments: HADOOP-10802.WIP.patch HADOOP-10720 is adding KMS server and client caches for encrypted keys for performance reasons. It would be good to add metrics to make sure that the cache is working as expected, and to inform future dynamic cache sizing and refilling policies. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10791) AuthenticationFilter should support externalizing the secret for signing and provide rotation support
[ https://issues.apache.org/jira/browse/HADOOP-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084667#comment-14084667 ] Alejandro Abdelnur commented on HADOOP-10791: - All new classes (except tests) should have visibility/stability annotations. AuthenticationFilter should support externalizing the secret for signing and provide rotation support - Key: HADOOP-10791 URL: https://issues.apache.org/jira/browse/HADOOP-10791 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Robert Kanter Attachments: HADOOP-10791.patch, HADOOP-10791.patch, HADOOP-10791.patch It should be possible to externalize the secret used to sign the hadoop-auth cookies. In the case of WebHDFS the shared secret used by NN and DNs could be used. In the case of Oozie HA, the secret could be stored in Oozie HA control data in ZooKeeper. In addition, it is desirable for the secret to change periodically, this means that the AuthenticationService should remember a previous secret for the max duration of hadoop-auth cookie. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10224) JavaKeyStoreProvider has to protect against corrupting underlying store
[ https://issues.apache.org/jira/browse/HADOOP-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085202#comment-14085202 ] Alejandro Abdelnur commented on HADOOP-10224: - LGTM +1. [~lmccay], do you want to review the latest patch? I'll wait till WED morning to commit. JavaKeyStoreProvider has to protect against corrupting underlying store --- Key: HADOOP-10224 URL: https://issues.apache.org/jira/browse/HADOOP-10224 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Arun Suresh Attachments: HADOOP-10224.1.patch, HADOOP-10224.2.patch, HADOOP-10224.3.patch, HADOOP-10224.4.patch, HADOOP-10224.5.patch, HADOOP-10224.6.patch, HADOOP-10224.7.patch Java keystores get corrupted at times. A key management operation that writes the store to disk could cause a corruption and all protected data would then be unaccessible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10936) Change default KeyProvider bitlength to 128
[ https://issues.apache.org/jira/browse/HADOOP-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085330#comment-14085330 ] Alejandro Abdelnur commented on HADOOP-10936: - +1 Change default KeyProvider bitlength to 128 --- Key: HADOOP-10936 URL: https://issues.apache.org/jira/browse/HADOOP-10936 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-10936.001.patch You need to download unlimited strength JCE to work with 256-bit keys. It'd be good to change the default to 128 to avoid needing the unlimited strength JCE, and print out the bitlength being used in places. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
[
https://issues.apache.org/jira/browse/HADOOP-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10918:
Status: Patch Available (was: Open)
JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HADOOP-10918.patch
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
[
https://issues.apache.org/jira/browse/HADOOP-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10918:
Attachment: HADOOP-10918.patch
Patch that adds a method to the JMX servlet to allow subclassing and overriding
the authorization check. It also changes the UNAUTHORIZED response to
FORBIDDEN in HttpServer2 as UNAUTHORIZED on its own should indicate also the
expected authentication mechanism.
It also wires a subclassed JMX servlet for KMS.
I've tested on local deployment that JMX works fine from KMS with the change
(cannot do testcase for that).
JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HADOOP-10918.patch
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10224) JavaKeyStoreProvider has to protect against corrupting underlying store
[
https://issues.apache.org/jira/browse/HADOOP-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082553#comment-14082553
]
Alejandro Abdelnur commented on HADOOP-10224:
-
In the constructor, the {{throw new IOException(Keystore cannot be loaded
!!);}} should be more descriptive of the problem, why is the exception is
being thrown, that CURRENT _NEW exists, and it should not be the case, only
one should be there.
In the constructor, when loading the CURRENT and having an error other than bad
password we should log that CURRENT was corrupted and we loaded OLD. We should
rename CURRENT to _BAD_$DATE, and report that as well for an admin to delete it.
When loading _NEW and corrupt, we should report that in the exception message
clearly. Also, shouldn’t we be renaming _NEW to CURRENT here?
// Check if _NEW exists (incase flush had finished writing but not, typo in
case
KeyStore intialized anew sucessfully 2 typos, KeyStore initialized anew
successfully
{{isBadorWrongPassword()}} method, always use {}s for IF blocks.
{{loadFromPath()}}, you don’t want to rename until you know you can load the
keystore, i would do the rename outside of here.
{{flush()}}, when renaming files verify rename is successful (boolean return
value) and fail if not.
JavaKeyStoreProvider has to protect against corrupting underlying store
---
Key: HADOOP-10224
URL: https://issues.apache.org/jira/browse/HADOOP-10224
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Larry McCay
Assignee: Arun Suresh
Attachments: HADOOP-10224.1.patch, HADOOP-10224.2.patch,
HADOOP-10224.3.patch
Java keystores get corrupted at times. A key management operation that writes
the store to disk could cause a corruption and all protected data would then
be unaccessible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
[
https://issues.apache.org/jira/browse/HADOOP-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10918:
Attachment: HADOOP-10918.patch
new patch fixing testcases because of the change from unauthorized to forbidden.
JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HADOOP-10918.patch, HADOOP-10918.patch
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[
https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080552#comment-14080552
]
Alejandro Abdelnur commented on HADOOP-10904:
-
got it, sounds good. BTW, the
{{Configuration.getPasswordFromCredenitalProviders()}} method has a typo.
Provide Alt to Clear Text Passwords through Cred Provider API
-
Key: HADOOP-10904
URL: https://issues.apache.org/jira/browse/HADOOP-10904
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Larry McCay
Assignee: Larry McCay
This is an umbrella jira to track various child tasks to uptake the
credential provider API to enable deployments without storing
passwords/credentials in clear text.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10224) JavaKeyStoreProvider has to protect against corrupting underlying store
[
https://issues.apache.org/jira/browse/HADOOP-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081147#comment-14081147
]
Alejandro Abdelnur commented on HADOOP-10224:
-
*JavaKeyStoreProvider.java*:
* {{if ((pwFile != null)(pwdFile == null))}}, no need to check for pwFile not
NULL here, we can be here only if it is not NULL already.
I think we should go over all corner cases (even if not happening under normal
circumstances).
*On startup should be something like:*
{code}
boolean loaded = false;
Path keyStorePath =
Path newPath = constructNewPath(path);
Path oldPath = constructOldPath(path);
FSPermission permission = ...
if (fs.exists(keyStorePath)) {
if (fs.exists(newPath)) {
//THROW EXCEPTION, something weird happened, admin should take care of
}
keyStore = loadKeyStore(path, password);
if (fs.exists(oldPath)) {
fs.delete(oldPath);
}
loaded = true;
//LOG
} else {
if (fs.exists(newPath) || fs.exists(oldPath)) {
if (fs.exists(newPath)) {
try {
keyStore = loadKeyStore(newPath, password);
fs.rename(newPath, path);
fs.delete(oldPath);
loaded = true;
//LOG
} catch (Exception ex) {
//THROW EXCEPTION if password issue, we don’t want to trash the new
file because of wrong password, admin should take care
}
}
if (!loaded) {
if (fs.exists(oldPath)) {
try {
keyStore = loadKeyStore(oldPath, password);
fs.rename(oldPath, path);
loaded = true;
//LOG
} catch (Exception ex) {
//THROW EXCEPTION if password issue, we don’t want to trash the new
file because of wrong password, admin should take care
}
} else {
//LOG
}
}
} else {
//LOG
}
}
if (!loaded) {
// creating an empty store
keyStore = KeyStore.getInstance(SCHEME_NAME);
OutputStream out = FileSystem.create(fs, path, permissions);
keyStore.store(out, password);
out.close();
//LOG
}
{code}
*On flush code should be something like:*
{code}
Path keyStorePath =
Path newPath = constructNewPath(path);
Path oldPath = constructOldPath(path);
FSPermission permission = ...
if (fs.exists(newPath) || fs.exists(oldPath)) {
//THROW EXCEPTION, something weird happened, admin should take care of
}
fs.rename(path, oldPath);
try {
OutputStream out = FileSystem.create(fs, newPath, permissions);
keyStore.store(out, password);
out.close();
} catch (Exception ex) {
fs.rename(oldPath, path);
//THROW EXCEPTION
}
fs.rename(newPath, path); //assert it happens else we need to revert and
throw exception
fs.delete(oldPath); //LOG WARN if does not happen.
{code}
JavaKeyStoreProvider has to protect against corrupting underlying store
---
Key: HADOOP-10224
URL: https://issues.apache.org/jira/browse/HADOOP-10224
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Larry McCay
Assignee: Arun Suresh
Attachments: HADOOP-10224.1.patch, HADOOP-10224.2.patch
Java keystores get corrupted at times. A key management operation that writes
the store to disk could cause a corruption and all protected data would then
be unaccessible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10771) Refactor HTTP delegation support out of httpfs to common
[
https://issues.apache.org/jira/browse/HADOOP-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081344#comment-14081344
]
Alejandro Abdelnur commented on HADOOP-10771:
-
[~daryn], ping. BTW, HADOOP-10880 already has a patch.
Refactor HTTP delegation support out of httpfs to common
Key: HADOOP-10771
URL: https://issues.apache.org/jira/browse/HADOOP-10771
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: COMBO.patch, COMBO.patch, HADOOP-10771.patch,
HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.patch, HADOOP-10771.sh
HttpFS implements delegation token support in {{AuthenticationFilter}}
{{AuthenticationHandler}} subclasses.
For HADOOP-10770 we need similar functionality for KMS.
Not to duplicate code, we should refactor existing code to common.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
Alejandro Abdelnur created HADOOP-10918:
---
Summary: JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10918) JMXJsonServlet fails when used within Tomcat
[
https://issues.apache.org/jira/browse/HADOOP-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081560#comment-14081560
]
Alejandro Abdelnur commented on HADOOP-10918:
-
Adding a protected method {{isInstrumentationAccessAllowed}} that can be
overridden by services not running with Jetty
JMXJsonServlet fails when used within Tomcat
Key: HADOOP-10918
URL: https://issues.apache.org/jira/browse/HADOOP-10918
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
{{JMXJsonServlet.doGet()}} has the following check:
{code}
if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(),
request, response)) {
{code}
Loading the class {{HttpServer2}} triggers loading Jetty specific classes:
{code}
SEVERE: Servlet.service() for servlet jmx-servlet threw exception
java.lang.ClassNotFoundException: org.mortbay.jetty.servlet.Context
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.crypto.key.kms.server.KMSMDCFilter.doFilter(KMSMDCFilter.java:84)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:438)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:255)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:408)
at
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:128)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:695)
Jul 31, 2014 2:46:24 PM org.apache.catalina.core.StandardWrapperValve invoke
{code}
Because of this the JMX servlet fails to work in KMS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10850) KerberosAuthenticator should not do the SPNEGO handshake
[
https://issues.apache.org/jira/browse/HADOOP-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-10850:
Attachment: testorder.patch
[~daryn], I believe the tests are passing for you because the order they are
running. If the {{testNotAuthenticated()}} runs before the
{{testAuthenticated()}}, then localhost will be blacklisted as not supported,
if the run in the opposite order, then things work because localhost is not
blacklisted.
I'm attaching a patch with a new test {{TestX}} (based on the
{{TestKerberosAuthenticator}} test) which demonstrates this, it has 2 methods:
{{testOrderOk()}} and {{testOrderFailing()}}. these method simply invoke in
diff order the original 2 testcases.
*Using hadoop-auth client SPNEGO*:
Apply the {{testorder.patch}} patch on trunk and run 2 diff mvn test
invocations, one running {{TestX#testOrderOk}} and the other running
{{TestX#testOrderFailing}}:
{code}
$ mvn test -Dtest=TestX#testOrderOk
$ mvn test -Dtest=TestX#testOrderFailing
{code}
Both testcases run OK.
*Using JDK client SPNEGO*:
Apply both, the {{testorder.patch}} and the HADOOP-10850.patch, patches on
trunk and run 2 diff mvn test invocations, one running {{TestX#testOrderOk}}
and the other running {{TestX#testOrderFailing}}:
{code}
$ mvn test -Dtest=TestX#testOrderOk
$ mvn test -Dtest=TestX#testOrderFailing
{code}
Now one test fails.
This is using both Sun JDK6 and JDK7:
java version 1.7.0_51
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
java version 1.6.0_65
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
KerberosAuthenticator should not do the SPNEGO handshake
Key: HADOOP-10850
URL: https://issues.apache.org/jira/browse/HADOOP-10850
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HADOOP-10850.patch, testFailures.png, testorder.patch
As mentioned in HADOOP-10453, the JDK automatically does a SPNEGO handshake
when opening a connection with a URL within a Kerberos login context, there
is no need to do the SPNEGO handshake in the {{KerberosAuthenticator}},
simply extract the auth token (hadoop-auth cookie) and do the fallback if
necessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[
https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079477#comment-14079477
]
Alejandro Abdelnur commented on HADOOP-10607:
-
Steve, Larry, Owen,
I'm not disputing the usefulness of the {{CredentialsProvider}} at all. I also
have concrete usecases outside of Hadoop.
Still, my concern is that Hadoop is being use a dumping ground of random stuff
for the convenience of other projects. Hadoop should only ship code that Hadoop
itself uses.
I understand the convenience, for all downstream projects, to have this API in
Hadoop itself: It is available for free.
IMO, the {{CredentialProvider}} should not be shipped in a Hadoop release until
Hadoop makes use of it. It can leave in trunk until then.
I don't have anything to add, I've explained clearly my position, now it is
your call to decide if the {{CredentialProvider}} should be shipped with Hadoop
at the moment or not.
Create an API to Separate Credentials/Password Storage from Applications
Key: HADOOP-10607
URL: https://issues.apache.org/jira/browse/HADOOP-10607
Project: Hadoop Common
Issue Type: New Feature
Components: security
Reporter: Larry McCay
Assignee: Larry McCay
Fix For: 3.0.0, 2.6.0
Attachments: 10607-10.patch, 10607-11.patch, 10607-12.patch,
10607-2.patch, 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch,
10607-7.patch, 10607-8.patch, 10607-9.patch, 10607-branch-2.patch, 10607.patch
As with the filesystem API, we need to provide a generic mechanism to support
multiple credential storage mechanisms that are potentially from third
parties.
We need the ability to eliminate the storage of passwords and secrets in
clear text within configuration files or within code.
Toward that end, I propose an API that is configured using a list of URLs of
CredentialProviders. The implementation will look for implementations using
the ServiceLoader interface and thus support third party libraries.
Two providers will be included in this patch. One using the credentials cache
in MapReduce jobs and the other using Java KeyStores from either HDFS or
local file system.
A CredShell CLI will also be included in this patch which provides the
ability to manage the credentials within the stores.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079499#comment-14079499 ] Alejandro Abdelnur commented on HADOOP-10756: - [~asuresh], much simpler approach. One minor NIT in the docs, instead saying 'end-points' use 'operations'. +1 after that. KMS audit log should consolidate successful similar requests Key: HADOOP-10756 URL: https://issues.apache.org/jira/browse/HADOOP-10756 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh Attachments: HADOOP-10756.1.patch, HADOOP-10756.10.patch, HADOOP-10756.2.patch, HADOOP-10756.3.patch, HADOOP-10756.4.patch, HADOOP-10756.5.patch, HADOOP-10756.6.patch, HADOOP-10756.7.patch, HADOOP-10756.8.patch, HADOOP-10756.9.patch Every rejected access should be audited, but successful accesses should be consolidated within a given amount of time if the request is from the same user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10607) Create an API to Separate Credentials/Password Storage from Applications
[ https://issues.apache.org/jira/browse/HADOOP-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079546#comment-14079546 ] Alejandro Abdelnur commented on HADOOP-10607: - [~lmccay], passwords in the SSL config make sense, thanks for bringing that up again (I've missed before). Create an API to Separate Credentials/Password Storage from Applications Key: HADOOP-10607 URL: https://issues.apache.org/jira/browse/HADOOP-10607 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Larry McCay Assignee: Larry McCay Fix For: 3.0.0, 2.6.0 Attachments: 10607-10.patch, 10607-11.patch, 10607-12.patch, 10607-2.patch, 10607-3.patch, 10607-4.patch, 10607-5.patch, 10607-6.patch, 10607-7.patch, 10607-8.patch, 10607-9.patch, 10607-branch-2.patch, 10607.patch As with the filesystem API, we need to provide a generic mechanism to support multiple credential storage mechanisms that are potentially from third parties. We need the ability to eliminate the storage of passwords and secrets in clear text within configuration files or within code. Toward that end, I propose an API that is configured using a list of URLs of CredentialProviders. The implementation will look for implementations using the ServiceLoader interface and thus support third party libraries. Two providers will be included in this patch. One using the credentials cache in MapReduce jobs and the other using Java KeyStores from either HDFS or local file system. A CredShell CLI will also be included in this patch which provides the ability to manage the credentials within the stores. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10756: Issue Type: Improvement (was: Bug) KMS audit log should consolidate successful similar requests Key: HADOOP-10756 URL: https://issues.apache.org/jira/browse/HADOOP-10756 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh Attachments: HADOOP-10756.1.patch, HADOOP-10756.10.patch, HADOOP-10756.11.patch, HADOOP-10756.12.patch, HADOOP-10756.2.patch, HADOOP-10756.3.patch, HADOOP-10756.4.patch, HADOOP-10756.5.patch, HADOOP-10756.6.patch, HADOOP-10756.7.patch, HADOOP-10756.8.patch, HADOOP-10756.9.patch Every rejected access should be audited, but successful accesses should be consolidated within a given amount of time if the request is from the same user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079635#comment-14079635 ] Alejandro Abdelnur commented on HADOOP-10756: - +1 KMS audit log should consolidate successful similar requests Key: HADOOP-10756 URL: https://issues.apache.org/jira/browse/HADOOP-10756 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh Attachments: HADOOP-10756.1.patch, HADOOP-10756.10.patch, HADOOP-10756.11.patch, HADOOP-10756.12.patch, HADOOP-10756.2.patch, HADOOP-10756.3.patch, HADOOP-10756.4.patch, HADOOP-10756.5.patch, HADOOP-10756.6.patch, HADOOP-10756.7.patch, HADOOP-10756.8.patch, HADOOP-10756.9.patch Every rejected access should be audited, but successful accesses should be consolidated within a given amount of time if the request is from the same user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10756) KMS audit log should consolidate successful similar requests
[ https://issues.apache.org/jira/browse/HADOOP-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HADOOP-10756: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Arun. Committed to trunk. KMS audit log should consolidate successful similar requests Key: HADOOP-10756 URL: https://issues.apache.org/jira/browse/HADOOP-10756 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Arun Suresh Fix For: 3.0.0 Attachments: HADOOP-10756.1.patch, HADOOP-10756.10.patch, HADOOP-10756.11.patch, HADOOP-10756.12.patch, HADOOP-10756.2.patch, HADOOP-10756.3.patch, HADOOP-10756.4.patch, HADOOP-10756.5.patch, HADOOP-10756.6.patch, HADOOP-10756.7.patch, HADOOP-10756.8.patch, HADOOP-10756.9.patch Every rejected access should be audited, but successful accesses should be consolidated within a given amount of time if the request is from the same user for he same key. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10791) AuthenticationFilter should support externalizing the secret for signing and provide rotation support
[
https://issues.apache.org/jira/browse/HADOOP-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079746#comment-14079746
]
Alejandro Abdelnur commented on HADOOP-10791:
-
*AuthenticationFilter.java*:
The boostrap of the signer secret provider logic is a bit complex, how about?
{code}
validity = Long.parseLong(config.getProperty(AUTH_TOKEN_VALIDITY, 36000))
* 1000; //10 hours
secretProvider = (SignerSecretProvider) filterConfig.getServletContext().
getAttribute(SIGNATURE_PROVIDER_ATTRIBUTE);
if (secretProvider == null) {
String signerSecretProviderClassName =
config.getProperty(configPrefix + SIGNER_SECRET_PROVIDER_CLASS, null);
if (signerSecretProviderClassName == null) {
String signatureSecret =
config.getProperty(configPrefix + SIGNATURE_SECRET, null);
if (signatureSecret != null) {
secretProvider = new StringSignerSecretProvider(signatureSecret);
} else {
secretProvider = new RandomSignerSecretProvider();
}
} else {
try {
Class? klass = Thread.currentThread().getContextClassLoader().
loadClass(signerSecretProviderClassName);
secretProvider = (SignerSecretProvider) klass.newInstance();
} catch (ClassNotFoundException ex) {
throw new ServletException(ex);
} catch (InstantiationException ex) {
throw new ServletException(ex);
} catch (IllegalAccessException ex) {
throw new ServletException(ex);
}
}
try {
secretProvider.init(config, validity);
} catch (Exception ex) {
throw new ServletException(ex);
}
}
{code}
Note the {{StringSignerSecretProvider}} would have a constructor that takes a
secret besides the default one.
*RollingSignerSecretProvider.java*:
* the scheduler should be created in the init() method, to avoid a run away
thread if an exception happens before init.
* if {{rollSecret()}} is synched so it cannot run simultaneously with
{{destroy()}} then we need to have a boolean that indicates if the provider is
destroyed, and check that in {{rollSecret()}}, if destroy do a NOP.
*Signer.java*:
* we can get rid of the Signer(byte[]) constructor.
*SignerSecretProvider.java*:
* {{init()}} should not have a {{secretStr}} param, that is impl specific.
*Logic change:*
Now we are creating a new array on every {{getAllSecrets()}} call on every
{{getCurrentSecret()}} call. this is because we don’t want a caller to be able
to modify the secret.
How about moving the signing/verification logic into the
{{SignerSecretProvider}}, then you don’t give away the secrets, then you don’t
have to clone them either.
AuthenticationFilter should support externalizing the secret for signing and
provide rotation support
-
Key: HADOOP-10791
URL: https://issues.apache.org/jira/browse/HADOOP-10791
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Robert Kanter
Attachments: HADOOP-10791.patch, HADOOP-10791.patch
It should be possible to externalize the secret used to sign the hadoop-auth
cookies.
In the case of WebHDFS the shared secret used by NN and DNs could be used. In
the case of Oozie HA, the secret could be stored in Oozie HA control data in
ZooKeeper.
In addition, it is desirable for the secret to change periodically, this
means that the AuthenticationService should remember a previous secret for
the max duration of hadoop-auth cookie.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10900) CredentialShell args should use single-dash style
[ https://issues.apache.org/jira/browse/HADOOP-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079753#comment-14079753 ] Alejandro Abdelnur commented on HADOOP-10900: - LGTM, +1 CredentialShell args should use single-dash style - Key: HADOOP-10900 URL: https://issues.apache.org/jira/browse/HADOOP-10900 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Attachments: hadoop-10900.001.patch As was discussed in HADOOP-10793 related to KeyShell, we should standardize on single-dash flags for things in branch-2. CredentialShell also needs to be updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10793) KeyShell args should use single-dash style
[ https://issues.apache.org/jira/browse/HADOOP-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079751#comment-14079751 ] Alejandro Abdelnur commented on HADOOP-10793: - LGTM, +1 KeyShell args should use single-dash style -- Key: HADOOP-10793 URL: https://issues.apache.org/jira/browse/HADOOP-10793 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 3.0.0 Reporter: Mike Yoder Assignee: Andrew Wang Attachments: hadoop-10793.001.patch Follow-on from HADOOP-10736 as per [~andrew.wang] - the key shell uses the gnu double dash style for command line args, while other command line programs use a single dash. Consider changing this, and consider another argument parsing scheme, like the CommandLine class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10904) Provide Alt to Clear Text Passwords through Cred Provider API
[
https://issues.apache.org/jira/browse/HADOOP-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079859#comment-14079859
]
Alejandro Abdelnur commented on HADOOP-10904:
-
[~lmccay], how about following a common pattern such as...
All components that load a password (i.e. {{FileBaseKeyStoresFactory}} for SSL
stuff) would add a new property (the old one + .ref, ie:
{{ssl.server.keystore.password.ref}}) that points to the credential ID. If the
*REF* the property is defined, it takes precedence over the old property, if
the *REF* property is not present, we print a deprecation warning and use the
old property.
Provide Alt to Clear Text Passwords through Cred Provider API
-
Key: HADOOP-10904
URL: https://issues.apache.org/jira/browse/HADOOP-10904
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Larry McCay
Assignee: Larry McCay
This is an umbrella jira to track various child tasks to uptake the
credential provider API to enable deployments without storing
passwords/credentials in clear text.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10910) Increase findbugs maxHeap size
[ https://issues.apache.org/jira/browse/HADOOP-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080047#comment-14080047 ] Alejandro Abdelnur commented on HADOOP-10910: - +1 Increase findbugs maxHeap size -- Key: HADOOP-10910 URL: https://issues.apache.org/jira/browse/HADOOP-10910 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.5.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Blocker Attachments: hadoop-10910.001.patch The release build fails because of an obscure findbugs error. Testing reveals that this is related to the findbugs heap size. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10911) hadoop.auth cookie after HADOOP-10710 still not proper according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080171#comment-14080171
]
Alejandro Abdelnur commented on HADOOP-10911:
-
Please don't remove the quotes.
By looking at RFC2109:
{code}
set-cookie = Set-Cookie: cookies
cookies = 1#cookie
cookie = NAME = VALUE *(; cookie-av)
NAME= attr
VALUE = value
cookie-av = Comment = value
| Domain = value
| Max-Age = value
| Path = value
| Secure
| Version = 1*DIGIT
{code}
It seems we are only off with {{Max-Age}} (ignoring HttpOnly which is not
breaking things).
Good idea on adding a test for HttpClient.
hadoop.auth cookie after HADOOP-10710 still not proper according to RFC2109
---
Key: HADOOP-10911
URL: https://issues.apache.org/jira/browse/HADOOP-10911
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.5.0
Reporter: Gregory Chanan
Attachments: HADOOP-10911.patch
I'm seeing the same problem reported in HADOOP-10710 (that is, httpclient is
unable to authenticate with servers running the authentication filter), even
with HADOOP-10710 applied.
From my reading of the spec, the problem is as follows:
Expires is not a valid directive according to the RFC, though it is mentioned
for backwards compatibility with netscape draft spec. When httpclient sees
Expires, it parses according to the netscape draft spec, but note from
RFC2109:
{code}
Note that the Expires date format contains embedded spaces, and that old
cookies did not have quotes around values.
{code}
and note that AuthenticationFilter puts quotes around the value:
https://github.com/apache/hadoop-common/blob/6b11bff94ebf7d99b3a9e513edd813cb82538400/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java#L437-L439
So httpclient's parsing appears to be kosher.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10911) hadoop.auth cookie after HADOOP-10710 still not proper according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080263#comment-14080263
]
Alejandro Abdelnur commented on HADOOP-10911:
-
on the quotes, we had them, they got removed, that broke things, we added
again. They don't do any harm if they are there.
On Max-Age Expired, i don't think we want to break old browsers. It seems to
me an HttpClient bug that uses the presence of Expire to go back to old cookie
format, the precense of Version=1 should trump. Can you dig on HttpClient side?
hadoop.auth cookie after HADOOP-10710 still not proper according to RFC2109
---
Key: HADOOP-10911
URL: https://issues.apache.org/jira/browse/HADOOP-10911
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.5.0
Reporter: Gregory Chanan
Attachments: HADOOP-10911.patch
I'm seeing the same problem reported in HADOOP-10710 (that is, httpclient is
unable to authenticate with servers running the authentication filter), even
with HADOOP-10710 applied.
From my reading of the spec, the problem is as follows:
Expires is not a valid directive according to the RFC, though it is mentioned
for backwards compatibility with netscape draft spec. When httpclient sees
Expires, it parses according to the netscape draft spec, but note from
RFC2109:
{code}
Note that the Expires date format contains embedded spaces, and that old
cookies did not have quotes around values.
{code}
and note that AuthenticationFilter puts quotes around the value:
https://github.com/apache/hadoop-common/blob/6b11bff94ebf7d99b3a9e513edd813cb82538400/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java#L437-L439
So httpclient's parsing appears to be kosher.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
