[jira] [Resolved] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation
[ https://issues.apache.org/jira/browse/HADOOP-16826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16826. -- Resolution: Fixed > ABFS: update abfs.md to include config keys for identity transformation > --- > > Key: HADOOP-16826 > URL: https://issues.apache.org/jira/browse/HADOOP-16826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2 >Reporter: Da Zhou >Assignee: Karthick Narendran >Priority: Major > > Update the abfs.md to include the config keys for identity transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16826) ABFS: update abfs.md to include config keys for identity transformation
Da Zhou created HADOOP-16826: Summary: ABFS: update abfs.md to include config keys for identity transformation Key: HADOOP-16826 URL: https://issues.apache.org/jira/browse/HADOOP-16826 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2 Reporter: Da Zhou Update the abfs.md to include the config keys for identity transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16750) Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16750. -- Resolution: Fixed > Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue > to branch-3.2 > -- > > Key: HADOOP-16750 > URL: https://issues.apache.org/jira/browse/HADOOP-16750 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Mandar Inamdar >Assignee: Sneha Vijayarajan >Priority: Minor > Fix For: 3.2.2, 3.2 > > > Make flush operation enabled/disabled through configuration. This is part of > performance improvements for ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16752) ABFS: test failure testLastModifiedTime()
Da Zhou created HADOOP-16752: Summary: ABFS: test failure testLastModifiedTime() Key: HADOOP-16752 URL: https://issues.apache.org/jira/browse/HADOOP-16752 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2 Reporter: Da Zhou -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16640) WASB: Override getCanonicalServiceName() to return full url of WASB filesystem
[ https://issues.apache.org/jira/browse/HADOOP-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16640. -- Resolution: Fixed > WASB: Override getCanonicalServiceName() to return full url of WASB filesystem > -- > > Key: HADOOP-16640 > URL: https://issues.apache.org/jira/browse/HADOOP-16640 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > > HBase calls getCanonicalServiceName() to check if two FS are the same: > [https://github.com/apache/hbase/blob/10180e232ebf886c9577d77eb91ce64b51564dfc/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L117] > This is creating some issues for customer because the current WASB relied on > the default implementation of getCanonicalServiceName() and will return > "ip:port". > Will override getCanonicalServiceName() in WASB to return the full URI of > the fs, and this would be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16652) Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16652. -- Resolution: Fixed > Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to > branch-2 > -- > > Key: HADOOP-16652 > URL: https://issues.apache.org/jira/browse/HADOOP-16652 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 2.0 > > > Make AAD endpoint configurable on all Auth flows -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16659) ABFS: add missing docs for configuration
Da Zhou created HADOOP-16659: Summary: ABFS: add missing docs for configuration Key: HADOOP-16659 URL: https://issues.apache.org/jira/browse/HADOOP-16659 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.2 Reporter: Da Zhou double-check the docs for ABFS and WASB configurations and add the missing ones. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16640) WASB: Override getCanonicalServiceName() to return full url of WASB filesystem
Da Zhou created HADOOP-16640: Summary: WASB: Override getCanonicalServiceName() to return full url of WASB filesystem Key: HADOOP-16640 URL: https://issues.apache.org/jira/browse/HADOOP-16640 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.2 Reporter: Da Zhou HBase calls getCanonicalServiceName() to check if two FS are the same: [https://github.com/apache/hbase/blob/10180e232ebf886c9577d77eb91ce64b51564dfc/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L117] This is creating some issues for customer because the current WASB relied on the default implementation of getCanonicalServiceName() and will return "ip:port". Will override getCanonicalServiceName() in WASB to return the full URI of the fs, and this would be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16578) ABFS: fileSystemExists() should not call container level apis
[ https://issues.apache.org/jira/browse/HADOOP-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-16578. -- Resolution: Fixed > ABFS: fileSystemExists() should not call container level apis > - > > Key: HADOOP-16578 > URL: https://issues.apache.org/jira/browse/HADOOP-16578 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Da Zhou >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > ABFS driver should not use container level api "Get Container Properties" as > there is no concept of container in HDFS, and this caused some RBAC check > issue. > Fix: use getFileStatus() to check if the container exists. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16578) ABFS: fileSystemExists() should not call container level apis
Da Zhou created HADOOP-16578: Summary: ABFS: fileSystemExists() should not call container level apis Key: HADOOP-16578 URL: https://issues.apache.org/jira/browse/HADOOP-16578 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Da Zhou Fix For: 3.3.0 ABFS driver should not use container level api "Get Container Properties" as there is no concept of container in HDFS, and this caused some RBAC check issue. Fix: use getFileStatus() to check if the container exists. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16539) ABFS: Add missing query parameter for getFileStatus
Da Zhou created HADOOP-16539: Summary: ABFS: Add missing query parameter for getFileStatus Key: HADOOP-16539 URL: https://issues.apache.org/jira/browse/HADOOP-16539 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou When calling [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16401) ABFS: port Azure doc to 3.2 branch
Da Zhou created HADOOP-16401: Summary: ABFS: port Azure doc to 3.2 branch Key: HADOOP-16401 URL: https://issues.apache.org/jira/browse/HADOOP-16401 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Need to port the latest Azure markdown docs from trunk to 3.2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16376) ABFS: Override access() to return true
Da Zhou created HADOOP-16376: Summary: ABFS: Override access() to return true Key: HADOOP-16376 URL: https://issues.apache.org/jira/browse/HADOOP-16376 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Gen1 driver override FileSystem.access() and forward it to storage service, but ABFS doesn't have this and is having some hive permission issue. As a short term fix, ABFS could override this to return true. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16317) ABFS: improve random read performance
Da Zhou created HADOOP-16317: Summary: ABFS: improve random read performance Key: HADOOP-16317 URL: https://issues.apache.org/jira/browse/HADOOP-16317 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Improving random read performance is an interesting topic. ABFS doesn't perform well when reading column format files as the process involves with many seek operations which make the readAhead no use, and if readAhead is used unwisely it would lead to unnecessary data request. Hence creating this Jira as a reminder to track the investigation and progress of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16315) ABFS: transform full UPN for named user in AclStatus
Da Zhou created HADOOP-16315: Summary: ABFS: transform full UPN for named user in AclStatus Key: HADOOP-16315 URL: https://issues.apache.org/jira/browse/HADOOP-16315 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou When converting the identity in AclStatus, only "owner" and "owning group" are transformed. We need to add the conversion for the AclEntry in AclStatus too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16281) ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side
Da Zhou created HADOOP-16281: Summary: ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side Key: HADOOP-16281 URL: https://issues.apache.org/jira/browse/HADOOP-16281 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16269) ABFS: add listFileStatus with StartFrom
Da Zhou created HADOOP-16269: Summary: ABFS: add listFileStatus with StartFrom Key: HADOOP-16269 URL: https://issues.apache.org/jira/browse/HADOOP-16269 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Adding a ListFileStatus in a path from a entry name in lexical order. This is added to AzureBlobFileSystemStore and won't be exposed to FS level api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest
Da Zhou created HADOOP-16251: Summary: ABFS: add FSMainOperationsBaseTest Key: HADOOP-16251 URL: https://issues.apache.org/jira/browse/HADOOP-16251 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Just happened to see "hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java", ABFS could inherit this test to increase its test coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16242) ABFS: add bufferpool to AbfsOutputStream
Da Zhou created HADOOP-16242: Summary: ABFS: add bufferpool to AbfsOutputStream Key: HADOOP-16242 URL: https://issues.apache.org/jira/browse/HADOOP-16242 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Environment: In current AbfsOutPutStream, new temporary buffers always get allocated when uploading the data to server. These big chunk creates huge pressure on the GC, and in some extreme cases that GC doesn't happen in time(observed in some customer's env), it also lead to OutOfMemory issue easily. Hence we need to replace the current implementation with a bufferpool. Reporter: Da Zhou Assignee: Da Zhou -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16169) ABFS: Bug fix for getPathProperties
Da Zhou created HADOOP-16169: Summary: ABFS: Bug fix for getPathProperties Key: HADOOP-16169 URL: https://issues.apache.org/jira/browse/HADOOP-16169 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Da Zhou Assignee: Da Zhou There is a bug in AbfsClient, getPathProperties(). For both xns accnout and non-xns account, it should use AbfsRestOperationType.GetPathStatus -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16136) ABFS: Transform username only when shortName is enabled
Da Zhou created HADOOP-16136: Summary: ABFS: Transform username only when shortName is enabled Key: HADOOP-16136 URL: https://issues.apache.org/jira/browse/HADOOP-16136 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.0 Reporter: Da Zhou Assignee: Da Zhou When short name is enabled, IdentityTransformer should only transform user name to a short name, and the group name should remains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16074) WASB: Update container not found error code
Da Zhou created HADOOP-16074: Summary: WASB: Update container not found error code Key: HADOOP-16074 URL: https://issues.apache.org/jira/browse/HADOOP-16074 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Due to service error code change, container is not automatically created for WRITE operation in WASB. Fix: need to update the corresponding error code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16048) ABFS: Fix Date format parser
Da Zhou created HADOOP-16048: Summary: ABFS: Fix Date format parser Key: HADOOP-16048 URL: https://issues.apache.org/jira/browse/HADOOP-16048 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Environment: Parsing of the US date does not work on a machine with European time format, need to set "Locale.*_US_*" when creating SimpleDateFormat. Reporter: Da Zhou Assignee: Da Zhou Fix For: 3.2.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
Da Zhou created HADOOP-16044: Summary: ABFS: Better exception handling of DNS errors followup Key: HADOOP-16044 URL: https://issues.apache.org/jira/browse/HADOOP-16044 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou reopened HADOOP-15662: -- 001 patch doesn't handle transient DNS resolution error, it could require manual retry when such things happen. The more obvious issue in current implementation is that the HostNotFoundException is logged at DEBUG level, so error msg in the long time retry could not be captured by user until DEBUG level is set for the logger. Will provide a fix so that the error msg is logged at WARN level and make it easier for user to read. > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15662-001.patch > > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15934) ABFS: make retry policy configurable
[ https://issues.apache.org/jira/browse/HADOOP-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou resolved HADOOP-15934. -- Resolution: Won't Fix > ABFS: make retry policy configurable > > > Key: HADOOP-15934 > URL: https://issues.apache.org/jira/browse/HADOOP-15934 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Labels: won't-fix > > Currently the retry policy parameter is hard coded, should make it > configurable for user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16040) ABFS: Bug fix for tolerateOobAppends configuration
Da Zhou created HADOOP-16040: Summary: ABFS: Bug fix for tolerateOobAppends configuration Key: HADOOP-16040 URL: https://issues.apache.org/jira/browse/HADOOP-16040 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Cause: configuration for tolerateOobAppends is never used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16036) WASB: Disable jetty logging configuration announcement
Da Zhou created HADOOP-16036: Summary: WASB: Disable jetty logging configuration announcement Key: HADOOP-16036 URL: https://issues.apache.org/jira/browse/HADOOP-16036 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Da Zhou Assignee: Da Zhou Remove the unnecessary logging configuration announcement from WASB console output. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16004) ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException
Da Zhou created HADOOP-16004: Summary: ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException Key: HADOOP-16004 URL: https://issues.apache.org/jira/browse/HADOOP-16004 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou In AbfsInputStream and AbfsOutPutStream, client error response is used to create an IOException. We should convert 404 error response to FileNotFoundException -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15975) ABFS: remove timeout check for DELETE and RENAME
Da Zhou created HADOOP-15975: Summary: ABFS: remove timeout check for DELETE and RENAME Key: HADOOP-15975 URL: https://issues.apache.org/jira/browse/HADOOP-15975 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Currently, ABFS rename and delete is doing a timeout check, which will fail the request for rename/delete when the target contains tons of file/dirs. Because timeout check is already there for each HTTP call, we should remove the timeout check in RENAME and DELETE. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15972) ABFS: update LIST_MAX_RESULTS
Da Zhou created HADOOP-15972: Summary: ABFS: update LIST_MAX_RESULTS Key: HADOOP-15972 URL: https://issues.apache.org/jira/browse/HADOOP-15972 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou This will be the temporary fix, as the service fix take much longer time to roll out. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15969) ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
Da Zhou created HADOOP-15969: Summary: ABFS: getNamespaceEnabled can fail blocking user access thru ACLs Key: HADOOP-15969 URL: https://issues.apache.org/jira/browse/HADOOP-15969 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15968) ABFS: add try catch for UGI failure when initializing ABFS
Da Zhou created HADOOP-15968: Summary: ABFS: add try catch for UGI failure when initializing ABFS Key: HADOOP-15968 URL: https://issues.apache.org/jira/browse/HADOOP-15968 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou There are cases that primary group doesn't exist for the user, need to add try catch when fetching usergroup info. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15957) WASB: Add asterisk wildcard support for Page Blob Dir Set
Da Zhou created HADOOP-15957:
Summary: WASB: Add asterisk wildcard support for Page Blob Dir Set
Key: HADOOP-15957
URL: https://issues.apache.org/jira/browse/HADOOP-15957
Project: Hadoop Common
Issue Type: Improvement
Components: fs/azure
Affects Versions: 3.2.0
Reporter: Da Zhou
Assignee: Da Zhou
In WASB, property "*fs.azure.page.blob.dir*" only support literal directory
name.
We need to add support for wildcard '*' to represent for any directory name.
For example, the following pattern should be supported:
{code:java}
/dir1/dir2
/dir1/*/dir3
/dir1/*/*/dir4
/dir1/*/*/file
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15940) ABFS: For HNS account, avoid unnecessary get call when doing Rename
Da Zhou created HADOOP-15940: Summary: ABFS: For HNS account, avoid unnecessary get call when doing Rename Key: HADOOP-15940 URL: https://issues.apache.org/jira/browse/HADOOP-15940 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou When rename, there is always a GET dst file status call, this is not necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15934) ABFS: make retry policy configurable
Da Zhou created HADOOP-15934: Summary: ABFS: make retry policy configurable Key: HADOOP-15934 URL: https://issues.apache.org/jira/browse/HADOOP-15934 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou Currently the retry policy parameter is hard coded, should make it configurable for user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15812) ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console
Da Zhou created HADOOP-15812: Summary: ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console Key: HADOOP-15812 URL: https://issues.apache.org/jira/browse/HADOOP-15812 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou AbfsRestOperationException msg contains multiple "\n" for displaying nicely, however, hadoop cmd class split the error msg by "\n" and display the first string in console, which lead to the incomplete error msg shown in console. Hence we need to improve this format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15809) ABFS: better exception handling when making getAccessToken call
Da Zhou created HADOOP-15809: Summary: ABFS: better exception handling when making getAccessToken call Key: HADOOP-15809 URL: https://issues.apache.org/jira/browse/HADOOP-15809 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Da Zhou Assignee: Da Zhou Currently getAccessToken throws only IOException and it is never checked for cases like 401, 403, which lead to unnecessary retry, this should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15795) Making HTTPS as default for ABFS
Da Zhou created HADOOP-15795: Summary: Making HTTPS as default for ABFS Key: HADOOP-15795 URL: https://issues.apache.org/jira/browse/HADOOP-15795 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou HTTPS should be used as default in ABFS, but also we provide a configuration key for user to disable it in non-secure mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15757) ABFS: remove dependency on common-codec Base64
Da Zhou created HADOOP-15757: Summary: ABFS: remove dependency on common-codec Base64 Key: HADOOP-15757 URL: https://issues.apache.org/jira/browse/HADOOP-15757 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou Currently ABFS relies on common-codec Base64, because different versions of common-codec are widely used and some are missing the methods needed by ABFS, it cause lots of "no such method" exception, hence we decide to add util for Base64 to avoid such issues in future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"
Da Zhou created HADOOP-15753: Summary: ABFS: support path "abfs://mycluster/file/path" Key: HADOOP-15753 URL: https://issues.apache.org/jira/browse/HADOOP-15753 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, which caused some issues for customer. I will add support for this path format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15728) ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature
Da Zhou created HADOOP-15728: Summary: ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature Key: HADOOP-15728 URL: https://issues.apache.org/jira/browse/HADOOP-15728 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou For non-namespaceenbaled account, UnsupportedOperationException is thrown when setOwner() or setPermission() is called, which cause problems for others. They should behave the same as before, so should make them no-op if the storage account is not namespace enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15661) ABFS: Add support for ACL
Da Zhou created HADOOP-15661: Summary: ABFS: Add support for ACL Key: HADOOP-15661 URL: https://issues.apache.org/jira/browse/HADOOP-15661 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou - Add support for ACL -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
Da Zhou created HADOOP-15659: Summary: ABFS: Code changes for bug fix and new tests Key: HADOOP-15659 URL: https://issues.apache.org/jira/browse/HADOOP-15659 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou - add bug fixes. - remove unnecessary dependencies. - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15560) ABFS: removed dependency injection and unnecessary dependencies
Da Zhou created HADOOP-15560: Summary: ABFS: removed dependency injection and unnecessary dependencies Key: HADOOP-15560 URL: https://issues.apache.org/jira/browse/HADOOP-15560 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou # Removed dependency injection and unnecessary dependencies. # Added tool to clean up test containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
