[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015549#comment-17015549
]
Hudson commented on HADOOP-16005:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17862 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17862/])
HADOOP-16005. NativeAzureFileSystem does not support setXAttr. (dazhou: rev
c36f09deb91454c086926c01f872d8ca4419aee0)
* (edit)
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
* (add)
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAttributes.java
* (edit)
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
* (edit)
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
* (edit)
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeFileSystemStore.java
* (edit)
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
* (edit)
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012066#comment-17012066
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
50s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 72a3738d9792 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12
10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a40dc9e |
| Default Java | 1.8.0_232 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/11/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/11/console |
| versions | git=2.7.4 maven=3.3.9
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992839#comment-16992839
]
Hadoop QA commented on HADOOP-16005:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:104ccca9169.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/10/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992827#comment-16992827
]
Hadoop QA commented on HADOOP-16005:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
10s{color} | {color:red} Docker failed to build yetus/hadoop:104ccca9169.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/9/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985463#comment-16985463
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 29m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
48s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cc953942734b 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12
10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/8/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985264#comment-16985264
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
56s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e4cb54c0831e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/6/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985262#comment-16985262
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
53s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 24241755d1fe 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/5/testReport/ |
| Max. process+thread count | 295 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985255#comment-16985255
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
51s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 79ceaaf65ff9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/7/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985254#comment-16985254
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
53s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e1f92a9e389d 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12
10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/4/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985252#comment-16985252
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
54s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 299679ebd1f0 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12
10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6b2d6d4 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/3/testReport/ |
| Max. process+thread count | 295 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983034#comment-16983034
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9e270c2aaaf6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ef950b0 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982783#comment-16982783
]
Clemens Wolff commented on HADOOP-16005:
[~virajith] [~ste...@apache.org] Apologies for the delayed response. I had to
deprioritize this pull request for a while but am now working again on this.
I'm updating my pull request and fixing the merge conflicts. Expect an update
on Github soon.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943793#comment-16943793
]
Virajith Jalaparti commented on HADOOP-16005:
-
Hi [~c-w] - I wanted to follow up on this. Do you have any updates on this
JIRA? It's been a while since the last comment. We are also interested in ABFS
supporting XAttrs
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775697#comment-16775697
]
Da Zhou commented on HADOOP-16005:
--
[~c-w] After sync, I downloaded the patch
"[https://github.com/apache/hadoop/pull/452.patch]; and applied locally:
error: patch failed:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:106
error:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
patch does not apply
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775524#comment-16775524
]
Steve Loughran commented on HADOOP-16005:
-
[~c-w] looks like your test running JVM crashed. There might be something in
the -output file; sadly the XML DOM gets buffered until the process exits
cleanly.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775522#comment-16775522
]
Steve Loughran commented on HADOOP-16005:
-
bq. why don't we send PR through github? Seems it is better than what we do
today: upload/download patch, make comments in text box.
Github integration with yetus is only just coming together. HADOOP-16035 was
the latest change. We don't just want review in there, we need yetus to kick
off a test run; the move of the asf repo to g
now one little bit of fun: someone has somehow done a successful merge through
git of a patch, but it ends up signed by the github key, whereas those of us
who have been doing commit -S get it signed with our local keys, so giving an
audit log of who really committed a patch.
Compare: git show --show-signature d33f0666f66
With: git show --show-signature a868f59d523
Mixed feelings there. I like the one-click merge, and I wasn't trying to be
strict about code signing, happy. But I am used to GPG signing all commits
these days. I think what I'd really like is some way client side to do a PR
merge from github. There's probably a way, but
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774405#comment-16774405
]
Clemens Wolff commented on HADOOP-16005:
[~DanielZhou] Thanks for the comments. Some notes:
1) I've pushed some changes to the branch to address your comments.
2) Could you point me to the breaking commit that you mentioned? I rebased my
local branch onto trunk (at 6c4ab03) and don't see any conflicts.
3) When running {{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8
clean verify}} on trunk, I get a [bunch of
errors|https://user-images.githubusercontent.com/1086421/53193268-9a584a00-35de-11e9-90a0-95781ed3bda1.png].
For reference, here's my
[azure-auth-keys.xml|https://user-images.githubusercontent.com/1086421/53193465-08047600-35df-11e9-974d-8b2d3fb202c8.png].
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761190#comment-16761190
]
Da Zhou commented on HADOOP-16005:
--
BTW, [~ste...@apache.org] why don't we send PR through github? Seems it is
better than what we do today: upload/download patch, make comments in text box.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761181#comment-16761181
]
Da Zhou commented on HADOOP-16005:
--
Thank you [~c-w] for your work! I like your GitHub PR and the test screenshots.
I've added some comments for it , and because of a recent commit, you need to
resolve some conflicts.
Regarding the test, I would suggest you to test ABFS using following command:
{code:java}
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
{code}
This will run ABFS tests only, similarly you can run wasb tests using:
{code:java}
mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
{code}
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761026#comment-16761026
]
Steve Loughran commented on HADOOP-16005:
-
[~tmarquardt], [~DanielZhou] - have you two looked at this?
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727540#comment-16727540
]
ASF GitHub Bot commented on HADOOP-16005:
-
GitHub user c-w opened a pull request:
https://github.com/apache/hadoop/pull/452
HADOOP-16005: Add XAttr support to WASB and ABFS
As discussed in
[HADOOP-16005](https://issues.apache.org/jira/browse/HADOOP-16005), this pull
request implements `getXAttr` and `setXAttr` on hadoop-azure's WASB and ABFS
file-systems.
The changes were tested against the following Azure storage account
configurations:
- WASB: StorageV2, RA-GRS replication in East US (primary) West US
(secondary). [WASB test session
screenshot](https://user-images.githubusercontent.com/1086421/50362109-699f5a00-0534-11e9-97c9-e8a7cee6e6c6.png).
All tests pass and the ABFS tests are skipped as expected.
- ABFS: StorageV2 with Data Lake Storage Gen2 preview enabled, RA-GRS
replication in East US (primary) West US (secondary). [ABFS test session
screenshot](https://user-images.githubusercontent.com/1086421/50361278-fea05400-0530-11e9-9cb4-cc23dec87cfc.png).
All ABFS tests pass but the WASB tests fail since the storage account hasn't
implemented the blob endpoints yet.
The test-patch script passed: [test-patch
output](https://user-images.githubusercontent.com/1086421/50377952-50aaad80-05f5-11e9-8ea2-b7bf99fc7509.png).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CatalystCode/hadoop hadoop-16005
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/452.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #452
commit 1c8303a5af1016455d23ce78508f911a10af4e77
Author: Clemens Wolff
Date: 2018-12-20T21:30:56Z
Add setXAttr and getXAttr to WASB and ABFS
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726243#comment-16726243
]
Clemens Wolff commented on HADOOP-16005:
It looks like {{getXAttr}} and {{setXAttr}} now work for the wasb connector
([screenshot of the attributes set by the unit
test|https://user-images.githubusercontent.com/1086421/50312485-7b222c80-0476-11e9-8c8f-4595aedd115c.png]).
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726239#comment-16726239
]
Clemens Wolff commented on HADOOP-16005:
I started working on a branch here:
[CatalystCode/hadoop-16005|https://github.com/apache/hadoop/compare/trunk...CatalystCode:hadoop-16005].
Note that for now I'm implementing {{getXAttr}} and {{setXAttr}} on the old
wasb connector. This is because I'm familiar with the Azure Blob Storage SDK
but not with the new Azure Datalake Gen 2 API -- implementing {{getXAttr}} and
{{setXAttr}} in terms of the SDKs that's more familiar for me will let me
better focus on understanding the hadoop-azure codebase itself instead of also
having to learn about the new Azure Datalake Gen 2 API at the same time. I'll
port the changes required for {{getXAttr}} and {{setXAttr}} to abfs once I have
a better understanding of what's going on in the hadoop-azure codebase overall.
Hope that's okay, [~ste...@apache.org].
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725285#comment-16725285
]
Clemens Wolff commented on HADOOP-16005:
Thanks for the pointers, [~ste...@apache.org]. I'll take a stab at implementing
this over the next couple of weeks.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722876#comment-16722876
]
Steve Loughran commented on HADOOP-16005:
-
I should add: serving up the etag as the file checksum would be nice —lets you
do backups which use a change in the etag as the sign of a file being out of
date
Look at
* class to describe the etag
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store/EtagChecksum.java
*HADOOP-13282 is the change to S3A to add this; HADOOP-15287 the discovery
we'd better make it optional to stop distcp backups from HDFS failing, as too
many jobs weren't using {{-skipCrc}} on the command line, it
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722609#comment-16722609
]
Steve Loughran commented on HADOOP-16005:
-
Now that Azure Datalake Gen 2 is public, abfs is the connector getting active
development of new features & performance. That'd be the one to work on ... I
don't see it there either.
Note the strict test policy for object store patches...it's in the
documentation for that tools/hadoop-azure module. Thanks
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
