[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17080 thanks. One thing I realised last night is that logging the session token, even at debug level, would have been a security risk. So it's very good that the log statement got cut, even at the cost of making it slightly harder to track down where credentials came from. One thing which could be added to all the setters would be the use of `Configuration.set(key, value, origin)` and set the origin to be something like "Env var $varname on $hostname", so the provenance does get tracked. There's not much which is offered in terms of examining that though; hard to justify --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17080 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17080 cc @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73797/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73797/testReport)** for PR 17080 at commit [`976dc57`](https://github.com/apache/spark/commit/976dc57d8afa48a7c76f74b07f272b4c1bc55b97). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73797/testReport)** for PR 17080 at commit [`976dc57`](https://github.com/apache/spark/commit/976dc57d8afa48a7c76f74b07f272b4c1bc55b97). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73765/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73765/testReport)** for PR 17080 at commit [`d8fd8dc`](https://github.com/apache/spark/commit/d8fd8dcb22c21a749049180f9802e5c9c7b300bc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73765/testReport)** for PR 17080 at commit [`d8fd8dc`](https://github.com/apache/spark/commit/d8fd8dcb22c21a749049180f9802e5c9c7b300bc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17080 @srowen dont worry, been tracking this: I filed the JIRA. Core code is good (i.e. property/env var names). One thing to bear in mind, the existing code propagates the env vars even if you are submitting work to a cluster running in EC2, which will be using EC2 instance metadata as a source for its credentials. Nobody has publicly complained about that in JIRA/stack overflow, and changing the behaviour may have adverse consequences. This patch does not change that situation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17080 \cc @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17080 @steveloughran OK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17080 I agree. I was just checking the files to make sure the strings were consistent/correct, rather than trusting the documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/17080 @steveloughran IMHO, there is no need to use `org.apache.hadoop.fs.s3a.Constants` and `com.amazonaws.SDKGlobalConfiguration`, otherwise we will import `hadoop-aws` and `aws-java-sdk-core` into Spark core. also cc @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17080 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73501/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73501/testReport)** for PR 17080 at commit [`0ae5aa7`](https://github.com/apache/spark/commit/0ae5aa73c70ae2f46a2d16087b5c55652d1e0282). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17080 **[Test build #73501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73501/testReport)** for PR 17080 at commit [`0ae5aa7`](https://github.com/apache/spark/commit/0ae5aa73c70ae2f46a2d16087b5c55652d1e0282). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17080: [SPARK-19739][CORE] propagate S3 session token to cluser
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17080 LGTM. Verified option name in `org.apache.hadoop.fs.s3a.Constants` file; env var name in `com.amazonaws.SDKGlobalConfiguration' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
