[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14439: Resolution: Won't Fix Status: Resolved (was: Patch Available) I'm going to close as a WONTFIX. The fact that we've had so few complaints shows that people have been moving off using this mechanism at all; I think we should drop it completely from Hadoop 3.3 > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Assignee: Vinayakumar B >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14439: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14831 > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Assignee: Vinayakumar B >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-14439: Target Version/s: 2.8.3 (was: 2.8.2) > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Assignee: Vinayakumar B >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14439: Release Note: We have restored the retention of a username:secret in an s3a: URL if you authenticate with S3 using embedded URI secrets. This is because some applications using strings to marshall the URLs were breaking. However, using secrets in this way is dangerous as it will end up in logs. It will be unsupported in Hadoop 3. To use different credentials in different buckets, move to per-bucket configuration > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Assignee: Vinayakumar B >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14439: Target Version/s: 2.8.2 > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Assignee: Vinayakumar B >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-14439: --- Attachment: HADOOP-14439-02.patch Attached the patch with findbugs,checkstyle and javadoc fix. > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14439-01.patch, HADOOP-14439-02.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-14439: --- Status: Patch Available (was: Open) > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14439-01.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-14439: --- Attachment: HADOOP-14439-01.patch Re-adding the secret to S3X urI in the same secret encoded format. Following are the highlights of changes: 1. Builds the FSUri containing only schema and authority part. 2. Strips down secret part if {{user}} was not provided in the {{userinfo}} section. 3. Encodes the secret part, even if original passed uri was not containing encoded secret. So, direct comparison of fs uri and provided uri still might fail in above cases. Please review. > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14439-01.patch > > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14439) regression: secret stripping from S3x URIs breaks some downstream code
[ https://issues.apache.org/jira/browse/HADOOP-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14439: Summary: regression: secret stripping from S3x URIs breaks some downstream code (was: regression: secret stripping from S3x URIs breaks some mapping code) > regression: secret stripping from S3x URIs breaks some downstream code > -- > > Key: HADOOP-14439 > URL: https://issues.apache.org/jira/browse/HADOOP-14439 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Spark 2.1 >Reporter: Steve Loughran >Priority: Minor > > Surfaced in SPARK-20799 > Spark is listing the contents of a path with getFileStatus(path), then > looking up the path value doing a lookup of the contents. > Apparently the lookup is failing to find files if you have a secret in the > key, {{s3a://key:secret@bucket/path}}. > Presumably this is because the stripped values aren't matching. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org