[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038700#comment-14038700 ] Hudson commented on HDFS-6492: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 2.5.0 Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038814#comment-14038814 ] Hudson commented on HDFS-6492: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 2.5.0 Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038903#comment-14038903 ] Hudson commented on HDFS-6492: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1807 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1807/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 2.5.0 Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14037560#comment-14037560 ] Andrew Wang commented on HDFS-6492: --- This is a known issue with test-patch, I'll update editsStored at commit time. Will commit shortly, thanks all. Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14037576#comment-14037576 ] Hudson commented on HDFS-6492: -- FAILURE: Integrated in Hadoop-trunk-Commit #5735 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5735/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036504#comment-14036504
]
Colin Patrick McCabe commented on HDFS-6492:
{code}
import java.util.*;
{code}
Shouldn't use wildcard includes (turn off this setting in your editor)
{code}
@VisibleForTesting
ListXAttr filterINodeXAttrs(ListXAttr existingXAttrs,
ListXAttr filterXAttrs, ListXAttr filteredXAttrs) {
{code}
{{filterXAttrs}} sounds confusingly similar to {{filteredXAttrs}}. Perhaps
rename these to {{toFilter}} and {{removed}}?
{{FSNamesystem#removeXAttr}}: we don't log the name of the xattr that was
removed? Seems like we should do this. Maybe in a separate JIRA, though.
+1 once these are addressed
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036825#comment-14036825
]
Hadoop QA commented on HDFS-6492:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651295/HDFS-6492.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7170//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7170//console
This message is automatically generated.
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch,
HDFS-6492.003.patch
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034229#comment-14034229 ] Charles Lamb commented on HDFS-6492: LGTM. I only have a few picky little nits. The javadoc you added needs an @returns. final decls. I see a few in args, but generally none in method bodies. I'm whiny about this as you know. I'd be happy if you did a query-replace s/ListXAttr/final ListXAttr. s/verifing/verifying/ I prefer 'for (int i = 0; i numToAdd; i++)' to 'for (int i=0; inumToAdd; i++)'. I think the Java coding standards support that. We discussed (offline) the lack of mkdir support in fseditlog. I just wanted to add it to the Jira for the record. I thought import foo.* was not in the Java coding standard. Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032236#comment-14032236
]
Hadoop QA commented on HDFS-6492:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650512/HDFS-6492.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestEditLogFileInputStream
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.TestPersistBlocks
org.apache.hadoop.hdfs.TestFileAppendRestart
org.apache.hadoop.hdfs.server.namenode.TestEditLog
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7128//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7128//console
This message is automatically generated.
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-6492.001.patch
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032261#comment-14032261
]
Yi Liu commented on HDFS-6492:
--
Thanks [~andrew.wang], nice work. I just have two comments for small
improvements.
*1.*
{code}
ListXAttr setINodeXAttrs(ListXAttr existingXAttrs, ListXAttr toSet,
EnumSetXAttrSetFlag flag) throws IOException {
// Check for duplicate XAttrs in toSet
// We need to use a custom comparator, so using a HashSet is not suitable
for (int i=0; itoSet.size(); i++) {
for (int j=0; jtoSet.size(); j++) {
if (i==j) {
continue;
}
if (toSet.get(i).equalsIgnoreValue(toSet.get(j))) {
throw new IOException(Cannot specify the same XAttr to be set +
more than once);
}
}
}
..
{code}
The two {{for}} can be improved as following. Since we don’t need to compare
the elements i two times.
{code}
for(int i = 0; i toSet.size(); i++) {
for(int j = i+1; j toSet.size(); j++) {
if (toSet.get(i).equalsIgnoreValue(toSet.get(j))) {
throw new IOException(Cannot specify the same XAttr to be set +
more than once);
}
}
}
{code}
*2.*
In {{FSDirectory#setINodeXAttrs}}, if change like following could be a bit
more efficient? (Can save one iteration)
*1).*
{code}
if (existingXAttrs != null) {
for (XAttr a: existingXAttrs) {
if (isUserVisible(a)) {
userVisibleXAttrsNum++;
}
}
}
{code}
Remove this snippet code.
*2).*
{code}
XAttrSetFlag.validate(xAttr.getName(), exist, flag);
// add the new XAttr since it passed validation
xAttrs.add(xAttr);
if (isUserVisible(xAttr) !exist) {
userVisibleXAttrsNum++;
}
{code}
Change it to :
{code}
XAttrSetFlag.validate(xAttr.getName(), exist, flag);
// add the new XAttr since it passed validation
xAttrs.add(xAttr);
if (isUserVisible(xAttr)) {
userVisibleXAttrsNum++;
}
{code}
*3).*
{code}
if (!alreadySet) {
xAttrs.add(existing);
}
{code}
Change it to:
{code}
if (!alreadySet) {
xAttrs.add(existing);
if (isUserVisible(existing)) {
userVisibleXAttrsNum++;
}
}
{code}
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-6492.001.patch
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033193#comment-14033193
]
Hadoop QA commented on HDFS-6492:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650648/HDFS-6492.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7138//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7138//console
This message is automatically generated.
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026736#comment-14026736 ] Colin Patrick McCabe commented on HDFS-6492: Technically it could still be atomic if we create all the separate edit log ops under the FSN lock. The only cases I can see that might be problematic are where the edit log got truncated (how?) or there was corruption. But it's simple enough to add this to the ops, and perhaps it will be helpful when making the FSN lock finer grained... I agree that we don't have to do any client-side API work here, since this is intended for encryption at the moment. Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019331#comment-14019331 ] Andrew Wang commented on HDFS-6492: --- For HDFS-6134, we need to set both a key and IV for newly-created encrypted files. It'd be better to not do this as separate edit log ops, since it means we need to deal with a file with no xattrs, or just a key but no IV. Thus, create/mkdir time xattrs. Multi-set seems potentially useful for similar reasons, so let's future-proof the metadata format before it gets released. I'll also note that clients can pack multiple k-v pairs into a single xattr via something like protobuf, and then do a read/modify/write to do atomic multi-updates. So, even if we don't immediately expose multi-set to clients, they still have somewhat worse workarounds available. Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019498#comment-14019498
]
Andrew Wang commented on HDFS-6492:
---
Hi [~hitliuyi], thanks for taking a look,
Yea, I'd like to add a {{ListXAttr}} field to OP_MKDIR and OP_ADD. The idea
here is that we can be assured that a file will have the xattrs even in the
presence of failures. With separate ops, the NN might fail between creating a
file and then setting xattrs, leading to the situations I mentioned.
I'll note that other RPCs do log multiple ops, like create with overwrite and
mkdirs, and thus I believe are not atomic. That feels more okay, since it's a
user operation that needs to be cleaned up by the user, but here we'd have a
messy system operation that *we'd* have to clean up. I'd prefer to avoid making
such a mess in the first place ;)
Support create-time xattrs and atomically setting multiple xattrs
-
Key: HDFS-6492
URL: https://issues.apache.org/jira/browse/HDFS-6492
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Ongoing work in HDFS-6134 requires being able to set system namespace
extended attributes at create and mkdir time, as well as being able to
atomically set multiple xattrs at once. There's currently no need to expose
this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019508#comment-14019508 ] Yi Liu commented on HDFS-6492: -- Thanks for the clarification Andrew. Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
