from:"Vishwajeet Dusane \(JIRA\)"

[jira] [Commented] (HADOOP-16405) Upgrade Wildfly Openssl version to 1.0.7.Final

2019-07-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879132#comment-16879132
 ] 

Vishwajeet Dusane commented on HADOOP-16405:


Thanks [~ste...@apache.org]. Will publish patch with results once i am 
satisfied.

> Upgrade Wildfly Openssl version to 1.0.7.Final
> --
>
> Key: HADOOP-16405
> URL: https://issues.apache.org/jira/browse/HADOOP-16405
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> Upgrade Wildfly Openssl version to 1.0.7.Final. This version has SNI support 
> which is essential for firewall enabled clusters along with many stability 
> related fixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16405) Upgrade Wildfly Openssl version to 1.0.7.Final

2019-07-03 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane reassigned HADOOP-16405:
--

Assignee: Vishwajeet Dusane

> Upgrade Wildfly Openssl version to 1.0.7.Final
> --
>
> Key: HADOOP-16405
> URL: https://issues.apache.org/jira/browse/HADOOP-16405
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> Upgrade Wildfly Openssl version to 1.0.7.Final. This version has SNI support 
> which is essential for firewall enabled clusters along with many stability 
> related fixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16405) Upgrade Wildfly Openssl version to 1.0.7.Final

2019-07-03 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877647#comment-16877647
 ] 

Vishwajeet Dusane commented on HADOOP-16405:


[~ste...@apache.org], 

If the issue is reproable from s3a then can you try if 1.0.7.Final fixes those? 
If not, i can take a shot at the issue as a seperate issue? We need SNI support 
to access storage account from firewall enabled cluster. I believe same would 
be needed for s3a as well.

 

 
{panel:title=Wildfly 1.0.7.Final Changelist}


1. Add server name indication (SNI) during SSL handshake
2. Fix JVM crash during client SSL session invalidation
3. Fix a potential race condition in SSL Instance
4. Enable TLS_DHE_DSS_WITH_AES_128_CBC_SHA for TLSv1 
5. Fix NPE with wildfly-openssl using OpenSSL 1.1.1 
6. Handle intermediate certificates correctly
{panel}
 

 

> Upgrade Wildfly Openssl version to 1.0.7.Final
> --
>
> Key: HADOOP-16405
> URL: https://issues.apache.org/jira/browse/HADOOP-16405
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build, fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Priority: Major
>
> Upgrade Wildfly Openssl version to 1.0.7.Final. This version has SNI support 
> which is essential for firewall enabled clusters along with many stability 
> related fixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16405) Upgrade Wildfly Openssl version to 1.0.7.Final

2019-07-02 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-16405:
--

 Summary: Upgrade Wildfly Openssl version to 1.0.7.Final
 Key: HADOOP-16405
 URL: https://issues.apache.org/jira/browse/HADOOP-16405
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/azure
Reporter: Vishwajeet Dusane


Upgrade Wildfly Openssl version to 1.0.7.Final. This version has SNI support 
which is essential for firewall enabled clusters along with many stability 
related fixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16402) AAD MSI flow is broken

2019-06-30 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-16402:
--

 Summary: AAD MSI flow is broken
 Key: HADOOP-16402
 URL: https://issues.apache.org/jira/browse/HADOOP-16402
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/adl
Affects Versions: 2.9.0
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane


For AAD with MSI flow to work, ADL driver needs to initialize MsiTokenProvider 
class with AAD Client id and tenant id. With the current implementation, AAD 
MSI flow is broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16182) Update abfs storage back-end with "close" flag when application is done writing to a file

2019-03-18 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795059#comment-16795059
 ] 

Vishwajeet Dusane commented on HADOOP-16182:


Thank you [~ste...@apache.org] . I will update this thread once I run test 
against 3.2 branch as well.

> Update abfs storage back-end with "close" flag when application is done 
> writing to a file 
> --
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16182.001.patch
>
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16182) Update abfs storage back-end with "close" flag when application is done writing to a file

2019-03-14 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793307#comment-16793307
 ] 

Vishwajeet Dusane commented on HADOOP-16182:


Thank you [~DanielZhou] . [~ste...@apache.org] Could you please also review 
this patch for commit?

> Update abfs storage back-end with "close" flag when application is done 
> writing to a file 
> --
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-16182.001.patch
>
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-13 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16791743#comment-16791743
 ] 

Vishwajeet Dusane commented on HADOOP-16182:


Below is the test result
{code:java}
[INFO] Results:
[INFO]
[WARNING] Tests run: 1159, Failures: 0, Errors: 0, Skipped: 741
[INFO]

{code}
No new test are added as part of this patch since this is change trigger 
backend functionality. However, verified that the close=true query param is 
passed when flush is invoked from AbfsOutputStream::close() API. Otherwise 
close=false.

 

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-16182.001.patch
>
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-16182:
---
Target Version/s: 3.3.0  (was: 3.2.0)
  Status: Patch Available  (was: Open)

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-16182.001.patch
>
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-16182:
---
Attachment: HADOOP-16182.001.patch

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-16182.001.patch
>
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-12 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane reassigned HADOOP-16182:
--

Assignee: (was: Vishwajeet Dusane)

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Priority: Major
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-12 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane reassigned HADOOP-16182:
--

Assignee: Vishwajeet Dusane

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-12 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-16182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane reassigned HADOOP-16182:
--

Assignee: Vishwajeet Dusane

> Update storage back-end with "close" flag when application is done writing to 
> a file 
> -
>
> Key: HADOOP-16182
> URL: https://issues.apache.org/jira/browse/HADOOP-16182
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> As part of Azure Data Lake Storage Gen2 notifications design, customers are 
> interested in knowing when a client is done writing to a file so they can 
> take certain actions like initiate a pipeline, or replicate a file, or start 
> certain processing. To satisfy that, ABFS client should send "close" flag 
> during Flush rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file

2019-03-12 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-16182:
--

 Summary: Update storage back-end with "close" flag when 
application is done writing to a file 
 Key: HADOOP-16182
 URL: https://issues.apache.org/jira/browse/HADOOP-16182
 Project: Hadoop Common
  Issue Type: Task
  Components: fs/azure
Affects Versions: 3.2.0
Reporter: Vishwajeet Dusane


As part of Azure Data Lake Storage Gen2 notifications design, customers are 
interested in knowing when a client is done writing to a file so they can take 
certain actions like initiate a pipeline, or replicate a file, or start certain 
processing. To satisfy that, ABFS client should send "close" flag during Flush 
rest API when invoked by OutputStream::close() API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-18 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724715#comment-16724715
 ] 

Vishwajeet Dusane commented on HADOOP-15965:


[~ste...@apache.org] could you please take a look at my review?

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch, HADOOP-15965-002.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-18 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15965:
---
Release Note: 


  Status: Patch Available  (was: Open)

Remove the additional log line printed by wildfly.

Test result.
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s 
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s 
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s - 
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s 
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4

{code}

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch, HADOOP-15965-002.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-18 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15965:
---
Status: Open  (was: Patch Available)

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch, HADOOP-15965-002.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-18 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15965:
---
Attachment: HADOOP-15965-002.patch

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch, HADOOP-15965-002.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-06 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711159#comment-16711159
 ] 

Vishwajeet Dusane commented on HADOOP-15965:


[~ste...@apache.org] - Will raise another patch with minor version change in 
SDK release. Will remove the additional log line printed by wildfly. Reference 
to similar issue addressed in abfs - -HADOOP-15851.-

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710058#comment-16710058
 ] 

Vishwajeet Dusane commented on HADOOP-15965:


Ahh sorry to hear that. Such issues are hard to trace. I have raised a patch. 
Could you please take a look?

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-05 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15965:
---
Attachment: HADOOP-15965-001.patch

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-05 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15965:
---
Assignee: Vishwajeet Dusane
  Status: Patch Available  (was: Open)

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15965-001.patch
>
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710014#comment-16710014
 ] 

Vishwajeet Dusane edited comment on HADOOP-15965 at 12/5/18 12:37 PM:
--

Thanks [~ste...@apache.org].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother 
warning people at info/warn{quote} * Wildfly dependency is shaded with Adls 
Java SDK hence no dependency package is needed. Verified that even if native 
libraries are failed to load, Java SDK uses JSE instead. However, not 
recommended since would hit hard on performance.

 
 * For info/warn - You might be referring to HADOOP-15851? In that case, Java 
SDK 2.3.2 does not incorporate this change. Java SDK 2.3.3 would have the 
change, timeline yet to be decided. Would it be fine to bypass this check for 
this Jira since not a functional issue? We have to unblock priority customer 
hence request to relax on this logline.

 
{quote} * can you make sure the SDK checks content tyoe of Auth & raises 
meaningful error if != json{quote} * This issue is addressed in the [Java SDK 
2.3.2 
version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]

 
{quote} * review all transitive dependencies: is it going to force in new 
versions of anything?{quote} * No other dependency version changes.

 
{quote} * and of course, usual "test everything process" applies{quote}
 
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s 
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s 
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s - 
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s 
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4
{code}


was (Author: vishwajeet.dusane):
Thanks [~ste...@apache.org].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother 
warning people at info/warn{quote} * Wildfly dependency is shaded with Adls 
Java SDK hence no dependency package is needed. Verified that even if native 
libraries are failed to load, Java SDK uses JSE instead. However, not 
recommended since would hit hard on performance.
 * For info/warn - You might be referring to HADOOP-15851? In that case, Java 
SDK 2.3.2 does not incorporate this change. Java SDK 2.3.3 would have the 
change, timeline yet to be decided. Would it be fine to bypass this check for 
this Jira since not a functional issue? We have to unblock priority customer 
hence request to relax on this logline.

 
{quote} * can you make sure the SDK checks content tyoe of Auth & raises 
meaningful error if != json{quote} * This issue is addressed in the [Java SDK 
2.3.2 
version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]

{quote} * review all transitive dependencies: is it going to force in new 
versions of anything?{quote} * No other dependency version changes.

{quote} * and of course, usual "test everything process" applies{quote}
 
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s 
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s 
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s - 
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s 
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO]

[jira] [Commented] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710014#comment-16710014
 ] 

Vishwajeet Dusane commented on HADOOP-15965:


Thanks [~ste...@apache.org].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother 
warning people at info/warn{quote} * Wildfly dependency is shaded with Adls 
Java SDK hence no dependency package is needed. Verified that even if native 
libraries are failed to load, Java SDK uses JSE instead. However, not 
recommended since would hit hard on performance.
 * For info/warn - You might be referring to HADOOP-15851? In that case, Java 
SDK 2.3.2 does not incorporate this change. Java SDK 2.3.3 would have the 
change, timeline yet to be decided. Would it be fine to bypass this check for 
this Jira since not a functional issue? We have to unblock priority customer 
hence request to relax on this logline.

 
{quote} * can you make sure the SDK checks content tyoe of Auth & raises 
meaningful error if != json{quote} * This issue is addressed in the [Java SDK 
2.3.2 
version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]

{quote} * review all transitive dependencies: is it going to force in new 
versions of anything?{quote} * No other dependency version changes.

{quote} * and of course, usual "test everything process" applies{quote}
 
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s 
- in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s 
- in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s - 
in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s 
- in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4
{code}

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> --
>
> Key: HADOOP-15965
> URL: https://issues.apache.org/jira/browse/HADOOP-15965
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Vishwajeet Dusane
>Priority: Major
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in 
> one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to 
> [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
> over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress

2018-12-03 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-15965:
--

 Summary: Upgrade to ADLS SDK which has major performance 
improvement for ingress/egress
 Key: HADOOP-15965
 URL: https://issues.apache.org/jira/browse/HADOOP-15965
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/adl
Reporter: Vishwajeet Dusane


Upgrade ADLS SDK to version 2.3.2 which has major improvements
 # Add special handling for 404 errors when requesting tokens from MSI
 # Fix liststatus response parsing when filestatus object contains array in one 
field.
 # Use wildfly openssl native binding with Java. This is a workaround to 
[https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost 
over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651104#comment-16651104
 ] 

Vishwajeet Dusane commented on HADOOP-15851:


Thank you [~ste...@apache.org]

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650391#comment-16650391
 ] 

Vishwajeet Dusane commented on HADOOP-15851:


Thank you [~ste...@apache.org] - I ran the test hadoop-azure test. All test are 
running except a couple of test failing with/without this patch.

 
{code:java}
[ERROR] Errors:
[ERROR] 
ITestAzureBlobFileSystemRandomRead.testRandomRead:112->verifyConsistentReads:580
 » IO
[ERROR] 
ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance:429->randomRead:498
 » IO
[INFO]
[ERROR] Tests run: 307, Failures: 0, Errors: 2, Skipped: 197

{code}
Investigating on the failure, based on the initial analysis, will raise 
separate Jira.

Failure is with/without this patch. So this patch is clean and has not 
introduced any issues.

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15851:
---
Status: Patch Available  (was: Open)

[~ste...@apache.org] - Could you please take a look at this patch? This patch 
is verified manually.

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15851:
---
Attachment: HADOOP-15851-001.patch

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649914#comment-16649914
 ] 

Vishwajeet Dusane edited comment on HADOOP-15851 at 10/15/18 8:57 AM:
--

Wildfly ssl and it is logging its [version information - source code 
link|https://github.com/wildfly/wildfly-openssl/blob/ace72ba07d0c746b6eb46635f4a8b122846c47c8/java/src/main/java/org/wildfly/openssl/SSL.java#L196].
 Disabling these log message when successfully loaded openssl permanently would 
be useful.


was (Author: vishwajeet.dusane):
Wildfly ssl and it is logging its [version information - source code 
link|[https://github.com/wildfly/wildfly-openssl/blob/ace72ba07d0c746b6eb46635f4a8b122846c47c8/java/src/main/java/org/wildfly/openssl/SSL.java#L196].]
 Disabling these log message when successfully loaded openssl permanently would 
be useful.

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649914#comment-16649914
 ] 

Vishwajeet Dusane commented on HADOOP-15851:


Wildfly ssl and it is logging its [version information - source code 
link|[https://github.com/wildfly/wildfly-openssl/blob/ace72ba07d0c746b6eb46635f4a8b122846c47c8/java/src/main/java/org/wildfly/openssl/SSL.java#L196].]
 Disabling these log message when successfully loaded openssl permanently would 
be useful.

> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
>  Issue Type: Task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a 
> predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15851) Disable wildfly logs to the console

2018-10-15 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-15851:
--

 Summary: Disable wildfly logs to the console
 Key: HADOOP-15851
 URL: https://issues.apache.org/jira/browse/HADOOP-15851
 Project: Hadoop Common
  Issue Type: Task
  Components: fs/azure
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane
 Fix For: 3.2.0


On loading OpenSSL library successfully, Wildfly logging messages like below
{code:java}
Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
{code}
These messages may fiddle with existing scripts which parses logs with a 
predefined schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-09-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604812#comment-16604812
 ] 

Vishwajeet Dusane commented on HADOOP-15700:


[~tmarquardt] and [~DanielZhou] - I disagree on below comments.
{quote}No need to port HADOOP-15669 to branch-2, since it cannot work on Java 7.
{quote}
{quote}But for branch-2, which is released on JDK 1.7, HADOOP-15669 introduced 
unnecessary dependency which will never be used, and we already know it will 
cause UnsupportedClassVersionError during run time.
{quote}
Please note that branch-2 does not restrict the runtime environment to only 
Java 7. Branch-2 minimum requirement is to support >=Java 7. branch-2 can also 
be built and run on Java 8. Example Current Azure HDInsight 3.6 version which 
is based on Hadoop 2.x and Java 8. ISV could also land similar stack with 
branch-2 and Java 8.
{quote}And in future development for branch-2, wildfly.openssl might cause some 
other issues we can not foresee at this moment.
{quote}
branch-3 is superset of branch-2. So those issues should also surface to 
branch-3.

 

If backporting of HADOOP-15407 (Since HADOOP-15669 subtask of this effort) to 
2.x version is not supported then this patch would not be required. Please 
suggest.

 

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-09-05 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604663#comment-16604663
 ] 

Vishwajeet Dusane commented on HADOOP-15700:


[~tmarquardt] - ABFS sources compilation should not fail using Java 7 due to 
wildfly.openssl dependency. However, would cause an error at runtime during app 
initialization with below exception.
{code:java}
java.lang.UnsupportedClassVersionError: org/wildfly/openssl/OpenSSLProvider : 
Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
{code}

This patch would swallow this exception and fall back to JSSE. Please correct 
me if you have already tried building ABFS code with java 7 and found 
compilation error due to wildfly.openssl dependency.

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-29 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596501#comment-16596501
 ] 

Vishwajeet Dusane commented on HADOOP-15700:


The scenario i am trying to address is ABFS client builds using JDK 1.7 and 
runs on Java 7. Wildfly OpenSSL, however, is a targeted for Java 8+.

On runtime, ABFS client initialization would fail 
with.{{UnsupportedClassVersionError exception}} This patch would allow ABFS 
client to fall back to JSSE SSL provider instead of blocking ABFS client usage.

Please suggest if HADOOP-15669 and HADOOP-15407 is targetted strictly Java 8+ 
only and backporting to 2.x version is not supported. In that case, this patch 
would not be required.

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-29 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596443#comment-16596443
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


Thank you for clarification [~ste...@apache.org]. In Azure Data Lake Gen 1, we 
had the same strategy to collect cluster details. However, the config is a 
custom and set by us during cluster creation. Please correct me, is there a 
standard config available which stores the app details?

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-29 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596439#comment-16596439
 ] 

Vishwajeet Dusane commented on HADOOP-15700:


Precisely the scenario i was trying to address with this patch which introduced 
by HADOOP-15669. i.e. ABFS client builds and runs on Java 7 but the SSL Lib 
used is Java8. This patch would address that issue.

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595006#comment-16595006
 ] 

Vishwajeet Dusane commented on HADOOP-15700:


[~ste...@apache.org] and [~tmarquardt] -  Change is specific to target version 
hence does not have test cases. 

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-28 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594994#comment-16594994
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


You are right [~ste...@apache.org],  i am looking at currently for collecting 
such information. {{org.apache.hadoop.util.VersionInfo.getBuildVersion() return 
user as well which may lead to privacy issue?}}

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15700:
---
Status: Patch Available  (was: Open)

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15700:
---
Attachment: (was: HADOOP-15700-001.patch)

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15700:
---
Attachment: HADOOP-15700-HADOOP-15407-01.patch

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-HADOOP-15407-01.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15700:
---
Attachment: HADOOP-15700-001.patch

> ABFS: Failure in OpenSSLProvider should fall back to JSSE
> -
>
> Key: HADOOP-15700
> URL: https://issues.apache.org/jira/browse/HADOOP-15700
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15700-001.patch
>
>
> Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
> initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
> back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE

2018-08-28 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-15700:
--

 Summary: ABFS: Failure in OpenSSLProvider should fall back to JSSE
 Key: HADOOP-15700
 URL: https://issues.apache.org/jira/browse/HADOOP-15700
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/azure
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane


Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE 
initialization. This is needed to support Java 7 in case the HADOOP-15669 is 
back-ported to support Java7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-16 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582793#comment-16582793
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


Thank you [~tmarquardt] and [~ste...@apache.org]

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-15 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581997#comment-16581997
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


Thank you [~tmarquardt] patch 04. Looks good to me +1.

 

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Status: Patch Available  (was: Open)

Fixed findbug reported issue.

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: HADOOP-15669-HADOOP-15407-03.patch

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Status: Open  (was: Patch Available)

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580198#comment-16580198
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


All tests pass with fs.azure.scale.test.enabled=true set for WASB 
(azure-auth-keys.xml) and ABFS (azure-bfs-auth-keys.xml):

Tests run: 263, Failures: 0, Errors: 0, Skipped: 11
Tests run: 966, Failures: 0, Errors: 0, Skipped: 107

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Status: Patch Available  (was: Open)

[~tmarquardt] and [~ste...@apache.org] - Could you please have a look at the 
02.patch? I have removed jar shading from this patch.

[~tmarquardt] - I have reverted change from AbfsRestOperation, however, 
AbfsHttpOperation class need to set Custom SSLSocketFactory instance. Please 
suggest if you had a different approach in mind.

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: HADOOP-15669-HADOOP-15407-02.patch

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Status: Open  (was: Patch Available)

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579477#comment-16579477
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


Thank you for the comments [~tmarquardt] and [~ste...@apache.org]. I agree with 
all the comments and I will incorporate those comments.

[~tmarquardt] - Performance numbers are documented in the [^ABFS - Improve 
HTTPS Performance Over Java Based Client.pdf].

[~ste...@apache.org] - Thank you for HADOOP-15387.

 

 

 

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-14 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579477#comment-16579477
 ] 

Vishwajeet Dusane edited comment on HADOOP-15669 at 8/14/18 9:01 AM:
-

Thank you for the comments [~tmarquardt] and [~ste...@apache.org]. I agree with 
all the comments and I will incorporate those comments.

[~tmarquardt] - Performance numbers are documented in the [^ABFS - Improve 
HTTPS Performance Over Java Based Client.pdf].

[~ste...@apache.org] - Thank you for HADOOP-15387.


was (Author: vishwajeet.dusane):
Thank you for the comments [~tmarquardt] and [~ste...@apache.org]. I agree with 
all the comments and I will incorporate those comments.

[~tmarquardt] - Performance numbers are documented in the [^ABFS - Improve 
HTTPS Performance Over Java Based Client.pdf].

[~ste...@apache.org] - Thank you for HADOOP-15387.

 

 

 

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Status: Patch Available  (was: Open)

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: (was: HADOOP-15669.patch)

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: HADOOP-15669-HADOOP-15407-01.patch

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: HADOOP-15669.patch

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578022#comment-16578022
 ] 

Vishwajeet Dusane commented on HADOOP-15669:


Attached detail root cause and proposed solution. 

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-13 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15669:
---
Attachment: ABFS - Improve HTTPS Performance Over Java Based Client.pdf

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-15669) ABFS: Improve HTTPS Performance

2018-08-12 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane reassigned HADOOP-15669:
--

Assignee: Vishwajeet Dusane

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-25 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Attachment: (was: HDFS-13344-002.patch)

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-25 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Attachment: (was: HDFS-13344-001.patch)

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-25 Thread Vishwajeet Dusane (JIRA)



[ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521961#comment-16521961
 ] 

Vishwajeet Dusane commented on HADOOP-15515:


Thanks [~chris.douglas] for comment.

 
 * _@if close is holding the lock on openFileStreams while cleaning up, if 
another thread is waiting on its monitor then it may add elements to the 
collection after the filesystem is closed._

Could you please suggest if the _openFileStreams_ modification is disallowed 
and create/append/creteNonRecursive should fail with IOException? Looked at 
DfsClient code and operation's after FileSystem.Close is allowed.

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch, HDFS-13344-001.patch, 
> HDFS-13344-002.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-25 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Attachment: HADOOP-15515-001.patch

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch, HDFS-13344-001.patch, 
> HDFS-13344-002.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-25 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Attachment: (was: HADOOP-15515-001.patch)

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch, HDFS-13344-001.patch, 
> HDFS-13344-002.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-24 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Attachment: HADOOP-15515-001.patch

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch, HDFS-13344-001.patch, 
> HDFS-13344-002.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15515) adl.AdlFilesystem.close() doesn't release locks on open files

2018-06-24 Thread Vishwajeet Dusane (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15515:
---
Status: Open  (was: Patch Available)

> adl.AdlFilesystem.close() doesn't release locks on open files
> -
>
> Key: HADOOP-15515
> URL: https://issues.apache.org/jira/browse/HADOOP-15515
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl
>Affects Versions: 2.7.3
> Environment: HDInsight on MS Azure:
>  
> Hadoop 2.7.3.2.6.2.25-1
> Subversion g...@github.com:hortonworks/hadoop.git -r 
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using 
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
>Reporter: Jay Hankinson
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15515-001.patch, HDFS-13344-001.patch, 
> HDFS-13344-002.patch
>
>
> If you write to a file on and Azure ADL filesystem and close the file system 
> but not the file before the process exits, the next time you try open the 
> file for append it fails with:
> Exception in thread "main" java.io.IOException: APPEND failed with error 
> 0x83090a16 (Failed to perform the requested operation because the file is 
> currently open in write mode by another user or process.). 
> [a67c6b32-e78b-4852-9fac-142a3e2ba963][2018-03-22T20:54:08.3520940-07:00]
>  The following moves local file to HDFS if it doesn't exist or appends it's 
> contents if it does:
>  
> {code:java}
> public void addFile(String source, String dest, Configuration conf) throws 
> IOException {
> FileSystem fileSystem = FileSystem.get(conf);
> // Get the filename out of the file path
> String filename = source.substring(source.lastIndexOf('/') + 
> 1,source.length());
> // Create the destination path including the filename.
> if (dest.charAt(dest.length() - 1) != '/')
> { dest = dest + "/" + filename; }
> else {
> dest = dest + filename;
> }
> // Check if the file already exists
> Path path = new Path(dest);
> FSDataOutputStream out;
> if (fileSystem.exists(path)) {
> System.out.println("File " + dest + " already exists appending");
> out = fileSystem.append(path);
> } else {
> out = fileSystem.create(path);
> }
> // Create a new file and write data to it.
> InputStream in = new BufferedInputStream(new FileInputStream(new File(
> source)));
> byte[] b = new byte[1024];
> int numBytes = 0;
> while ((numBytes = in.read(b)) > 0) {
> out.write(b, 0, numBytes);
> }
> // Close the file system not the file
> in.close();
> //out.close();
> fileSystem.close();
> }
> {code}
>  If "dest" is an adl:// location, invoking the function a second time (after 
> the process has exited) it raises the error. If it's a regular hdfs:// file 
> system, it doesn't as all the locks are released. The same exception is also 
> raised if a subsequent append is done using: hdfs dfs  -appendToFile.
> As I can't see a way to force lease recovery in this situation, this seems 
> like a bug. org.apache.hadoop.fs.adl.AdlFileSystem inherits close() from 
> org.apache.hadoop.fs.FileSystem
> [https://hadoop.apache.org/docs/r3.0.0/api/org/apache/hadoop/fs/adl/AdlFileSystem.html]
> Which states:
> Close this FileSystem instance. Will release any held locks. This does not 
> seem to be the case



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15187) Remove ADL mock test dependency on REST call invoked from Java SDK

2018-02-13 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362210#comment-16362210
 ] 

Vishwajeet Dusane commented on HADOOP-15187:


Thanks [~ste...@apache.org]

> Remove ADL mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-02-11 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360312#comment-16360312
 ] 

Vishwajeet Dusane commented on HADOOP-15187:


[~ste...@apache.org] - Please let me know if you need more detail on the 
executed test ?

> Remove mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-02-07 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356442#comment-16356442
 ] 

Vishwajeet Dusane commented on HADOOP-15187:


I reran test against production environment hosted in east us region. All test 
passed.

> Remove mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-02-07 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355164#comment-16355164
 ] 

Vishwajeet Dusane commented on HADOOP-15187:


[~ste...@apache.org] - Could you please review this patch ?

> Remove mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-02-06 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15187:
---
Status: Patch Available  (was: Open)

The test that stub network call initiated from Java SDK has been removed.

> Remove mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-02-06 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15187:
---
Attachment: HADOOP-15187-001.patch

> Remove mock test dependency on REST call invoked from Java SDK 
> ---
>
> Key: HADOOP-15187
> URL: https://issues.apache.org/jira/browse/HADOOP-15187
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15187-001.patch
>
>
> Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to be set on the command line

2018-01-29 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344463#comment-16344463
 ] 

Vishwajeet Dusane edited comment on HADOOP-15186 at 1/30/18 3:54 AM:
-

Thanks [~ste...@apache.org]. Is it possible to back port this CR to 3.0.0, 2.9 
and 2.8 branch ?


was (Author: vishwajeet.dusane):
Thanks [~ste...@apache.org]

> Allow Azure Data Lake SDK dependency version to be set on the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to be set on the command line

2018-01-29 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344463#comment-16344463
 ] 

Vishwajeet Dusane commented on HADOOP-15186:


Thanks [~ste...@apache.org]

> Allow Azure Data Lake SDK dependency version to be set on the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342916#comment-16342916
 ] 

Vishwajeet Dusane edited comment on HADOOP-15186 at 1/29/18 5:33 AM:
-

[~ste...@apache.org] - Could you please review this patch ?


was (Author: vishwajeet.dusane):
[~ste...@apache.org] - Could you please review this path ?

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342916#comment-16342916
 ] 

Vishwajeet Dusane commented on HADOOP-15186:


[~ste...@apache.org] - Could you please review this path ?

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Attachment: HADOOP-15186-003.patch

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Open  (was: Patch Available)

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Patch Available  (was: Open)

Formatted patch after sync with latest changes in master.

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch, 
> HADOOP-15186-003.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Patch Available  (was: Open)

Resubmit patch.

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Open  (was: Patch Available)

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-28 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Attachment: HADOOP-15186-002.patch

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch, HADOOP-15186-002.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Patch Available  (was: Open)

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Open  (was: Patch Available)

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK

2018-01-23 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-15187:
--

 Summary: Remove mock test dependency on REST call invoked from 
Java SDK 
 Key: HADOOP-15187
 URL: https://issues.apache.org/jira/browse/HADOOP-15187
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/adl
Affects Versions: 3.0.0
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane


Cleanup unit test which mocks REST calls invoked within dependency SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Component/s: build

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Status: Patch Available  (was: Open)

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-15186:
---
Attachment: HADOOP-15186-001.patch

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Attachments: HADOOP-15186-001.patch
>
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335557#comment-16335557
 ] 

Vishwajeet Dusane commented on HADOOP-15186:


Proposal is to expose property in pom.xml as 
{{azure.data.lake.store.sdk.version. This property can be override during maven 
build phase like. }}
{code:java}
mvn clean install -Dazure.data.lake.store.sdk.version=3.0.0{code}

> Allow Azure Data Lake SDK dependency version to override from the command line
> --
>
> Key: HADOOP-15186
> URL: https://issues.apache.org/jira/browse/HADOOP-15186
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
>
> For backward/forward release of Java SDK compatibility test against Hadoop 
> driver. Allow Azure Data Lake Java SDK dependency version to override from 
> command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line

2018-01-23 Thread Vishwajeet Dusane (JIRA)

Vishwajeet Dusane created HADOOP-15186:
--

 Summary: Allow Azure Data Lake SDK dependency version to override 
from the command line
 Key: HADOOP-15186
 URL: https://issues.apache.org/jira/browse/HADOOP-15186
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/adl
Affects Versions: 3.0.0
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane


For backward/forward release of Java SDK compatibility test against Hadoop 
driver. Allow Azure Data Lake Java SDK dependency version to override from 
command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15090) add adl troubleshooting doc

2017-12-04 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277094#comment-16277094
 ] 

Vishwajeet Dusane commented on HADOOP-15090:


Thanks [~ste...@apache.org] for initiating trouble shooting doc.
+1 for the patch.

> add adl troubleshooting doc
> ---
>
> Key: HADOOP-15090
> URL: https://issues.apache.org/jira/browse/HADOOP-15090
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: documentation, fs/adl
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-15090-001.patch
>
>
> Add a troubleshooting section/doc to the ADL docs based on our experiences.
> this should not be a substitute for improving the diagnostics/fixing the 
> error messages. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14730) Support protobuf FileStatus in AdlFileSystem

2017-08-08 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118212#comment-16118212
 ] 

Vishwajeet Dusane commented on HADOOP-14730:


Thanks a lot [~chris.douglas] and [~jzhuge].

> Support protobuf FileStatus in AdlFileSystem
> 
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14730.001.patch, HADOOP-14730.002.patch, 
> HADOOP-14730.003.patch, HADOOP-14730.004.patch, HADOOP-14730.005.patch, 
> HADOOP-14730.006.patch
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order

2017-08-05 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115391#comment-16115391
 ] 

Vishwajeet Dusane commented on HADOOP-14730:


+1 for v004, mainly to not break 2.x linkage. Having a {{FileStatus}} shim 
layer to continue support linkage with 2.x is better approach.

> hasAcl property always set to false, regardless of FsPermission higher bit 
> order 
> -
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14730.001.patch, HADOOP-14730.002.patch, 
> HADOOP-14730.003.patch, HADOOP-14730.004.patch
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order

2017-08-03 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113441#comment-16113441
 ] 

Vishwajeet Dusane commented on HADOOP-14730:


Thanks [~chris.douglas] for Patch 003 and +1 on the change.

Had a offline sync with [~chris.douglas]. Agree that HDFS-6984 is not backward 
compatible change. So as {{hadoop-azure-datalake.jar}} with Patch 003 would not 
link to Hadoop 2.x common jar, mainly for the HDFS-6984 introduced new 
{{FileStatus}} constructor. Patching Hadoop 2.X cluster with Hadoop Adl jar 
will not work.

CC: [~liuml07] and [~jzhuge]



> hasAcl property always set to false, regardless of FsPermission higher bit 
> order 
> -
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14730.001.patch, HADOOP-14730.002.patch, 
> HADOOP-14730.003.patch
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order

2017-08-03 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-14730:
---
Status: Patch Available  (was: Open)

> hasAcl property always set to false, regardless of FsPermission higher bit 
> order 
> -
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14730.001.patch
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order

2017-08-03 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-14730:
---
Attachment: HADOOP-14730.001.patch

Proposed change while initializing FileStatus. Set hasAcl value based on the 
FsPermission AclBit property if available.

> hasAcl property always set to false, regardless of FsPermission higher bit 
> order 
> -
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14730.001.patch
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order

2017-08-03 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-14730:
---
Fix Version/s: 3.0.0-beta1

> hasAcl property always set to false, regardless of FsPermission higher bit 
> order 
> -
>
> Key: HADOOP-14730
> URL: https://issues.apache.org/jira/browse/HADOOP-14730
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Vishwajeet Dusane
>Assignee: Chris Douglas
> Fix For: 3.0.0-beta1
>
>
> 2 Unit Test cases are failing  [Azure-data-lake Module 
> |https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
>  caused after HDFS-6984 commit.
> Issue seems to be {{hasAcl}} is hard coded to {{false}}. 
> {code:java}
> public FileStatus(long length, boolean isdir,
> int block_replication,
> long blocksize, long modification_time, long access_time,
> FsPermission permission, String owner, String group, 
> Path symlink,
> Path path) {
> this(length, isdir, block_replication, blocksize, modification_time,
> access_time, permission, owner, group, symlink, path,
> false, false, false);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

1 2 3 4 >

1 - 100 of 318 matches

Mail list logo