Re: Group mismatches?
Hi Edward, I'm not sure I understand. We all share the primary group domain_users (since these come from Active Directory over Samba). Still, we are able to be in multiple projects by using secondary groups, and it seems dfsgroups/mrgroups tries to report secondary groups. Regardless, if I use the newgrp(1) command to switch my primary group, Hadoop still seems blind to the foobarcommander group, however: [clayb@hamster ~]$ newgrp foobarcommander [clayb@hamster ~]$ id uid=16777217(clayb) gid=16777316(foobarcommander) groups=5(xxx_rec_eng),16777216(domain users),16777218(all),16777221(all_north america),16777223(batchlogon),16777226(xxx-s),16777229(xxx03-s),16777235(xxx1-admins),16777236(xxx-emr-users),16777237(xxx-emr-admins),16777265(xxx1-users),16777285(BUILTIN\users),16777316(foobarcommander) [clayb@hamster ~]$ hadoop dfsgroups log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users -Clay On Mon, 16 Jul 2012, Edward Capriolo wrote: In all places I have found it only to be the primary group, not all the users supplemental groups. On Mon, Jul 16, 2012 at 3:05 PM, Clay B. wrote: Hi all, I have a Hadoop cluster which uses Samba to map an Active Directory domain to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with groups. Does anyone have any debugging advice, or how to refresh the DFS groups mapping? If not, should I file a bug at https://issues.apache.org/jira/browse/HADOOP? # I see the following error: [clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". ls: could not get get listing for 'hdfs://hamster:9000/projects/foobarcommander' : org.apache.hadoop.security.AccessControlException: Permission denied: user=clayb, access=READ_EXECUTE, inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx--- # I verify group membership -- look a mismatch! [clayb@hamster ~]$ which groups /usr/bin/groups [clayb@hamster ~]$ groups foobarcommander xxx_rec_eng domain users all all_north america batchlogon xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users BUILTIN\users [clayb@hamster ~]$ hadoop dfsgroups log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users Notice, in particular the foobarcommander group is only shown for my /usr/bin/groups output. It looks like the following from the HDFS Permissions Guide[1] is not correct, in my case: "The group list is the equivalent of `bash -c groups`." # I have tried the following to no useful effect: [admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". # I do, however, see other users with the foobarcommander group, so the group should be "visible" to Hadoop: [clayb@hamster ~]$ hadoop dfsgroups pat log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". pat : domain users all_north america all_san diego all foobarcommander BUILTIN\users # And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me: [clayb@hamster ~]$ hadoop mrgroups log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users # And I see that the system is getting the right data via getent(1): [clayb@hamster ~]$ getent group foobarcommander foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15 # I am using Cloudera's CDH3u4 Hadoop: [clayb@hamster ~]$ hadoop version Hadoop 0.20.2-cdh3u4 Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r 214dd731e3bdb687cb55988d3f47dd9e248c5690 Compiled by root on Mon May 7 14:03:02 PDT 2012 From source with checksum a60c9795e41a3248b212344fb131c12c I also do not see any obviously useful errors in my namenode logs. -Clay [1]: http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity
Re: Group mismatches?
In all places I have found it only to be the primary group, not all the users supplemental groups. On Mon, Jul 16, 2012 at 3:05 PM, Clay B. wrote: > Hi all, > > I have a Hadoop cluster which uses Samba to map an Active Directory domain > to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with > groups. Does anyone have any debugging advice, or how to refresh the DFS > groups mapping? If not, should I file a bug at > https://issues.apache.org/jira/browse/HADOOP? > > # I see the following error: > [clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander > log4j:ERROR Could not find value for key log4j.appender.NullAppender > log4j:ERROR Could not instantiate appender named "NullAppender". > ls: could not get get listing for > 'hdfs://hamster:9000/projects/foobarcommander' : > org.apache.hadoop.security.AccessControlException: Permission denied: > user=clayb, access=READ_EXECUTE, > inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx--- > > # I verify group membership -- look a mismatch! > [clayb@hamster ~]$ which groups > /usr/bin/groups > [clayb@hamster ~]$ groups > foobarcommander xxx_rec_eng domain users all all_north america batchlogon > xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users > BUILTIN\users > [clayb@hamster ~]$ hadoop dfsgroups > log4j:ERROR Could not find value for key log4j.appender.NullAppender > log4j:ERROR Could not instantiate appender named "NullAppender". > clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon > all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users > > Notice, in particular the foobarcommander group is only shown for my > /usr/bin/groups output. It looks like the following from the HDFS > Permissions Guide[1] is not correct, in my case: > "The group list is the equivalent of `bash -c groups`." > > # I have tried the following to no useful effect: > [admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings > log4j:ERROR Could not find value for key log4j.appender.NullAppender > log4j:ERROR Could not instantiate appender named "NullAppender". > > # I do, however, see other users with the foobarcommander group, so the > group should be "visible" to Hadoop: > [clayb@hamster ~]$ hadoop dfsgroups pat > log4j:ERROR Could not find value for key log4j.appender.NullAppender > log4j:ERROR Could not instantiate appender named "NullAppender". > pat : domain users all_north america all_san diego all foobarcommander > BUILTIN\users > # And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me: > [clayb@hamster ~]$ hadoop mrgroups > log4j:ERROR Could not find value for key log4j.appender.NullAppender > log4j:ERROR Could not instantiate appender named "NullAppender". > clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon > all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users > > # And I see that the system is getting the right data via getent(1): > [clayb@hamster ~]$ getent group foobarcommander > foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15 > > # I am using Cloudera's CDH3u4 Hadoop: > [clayb@hamster ~]$ hadoop version > Hadoop 0.20.2-cdh3u4 > Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r > 214dd731e3bdb687cb55988d3f47dd9e248c5690 > Compiled by root on Mon May 7 14:03:02 PDT 2012 > From source with checksum a60c9795e41a3248b212344fb131c12c > > I also do not see any obviously useful errors in my namenode logs. > > -Clay > > [1]: > http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity >
Group mismatches?
Hi all, I have a Hadoop cluster which uses Samba to map an Active Directory domain to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with groups. Does anyone have any debugging advice, or how to refresh the DFS groups mapping? If not, should I file a bug at https://issues.apache.org/jira/browse/HADOOP? # I see the following error: [clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". ls: could not get get listing for 'hdfs://hamster:9000/projects/foobarcommander' : org.apache.hadoop.security.AccessControlException: Permission denied: user=clayb, access=READ_EXECUTE, inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx--- # I verify group membership -- look a mismatch! [clayb@hamster ~]$ which groups /usr/bin/groups [clayb@hamster ~]$ groups foobarcommander xxx_rec_eng domain users all all_north america batchlogon xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users BUILTIN\users [clayb@hamster ~]$ hadoop dfsgroups log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users Notice, in particular the foobarcommander group is only shown for my /usr/bin/groups output. It looks like the following from the HDFS Permissions Guide[1] is not correct, in my case: "The group list is the equivalent of `bash -c groups`." # I have tried the following to no useful effect: [admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". # I do, however, see other users with the foobarcommander group, so the group should be "visible" to Hadoop: [clayb@hamster ~]$ hadoop dfsgroups pat log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". pat : domain users all_north america all_san diego all foobarcommander BUILTIN\users # And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me: [clayb@hamster ~]$ hadoop mrgroups log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users # And I see that the system is getting the right data via getent(1): [clayb@hamster ~]$ getent group foobarcommander foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15 # I am using Cloudera's CDH3u4 Hadoop: [clayb@hamster ~]$ hadoop version Hadoop 0.20.2-cdh3u4 Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r 214dd731e3bdb687cb55988d3f47dd9e248c5690 Compiled by root on Mon May 7 14:03:02 PDT 2012 From source with checksum a60c9795e41a3248b212344fb131c12c I also do not see any obviously useful errors in my namenode logs. -Clay [1]: http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity
