Re: I am a new comer and have a question about HDFS code check out
Hi, Jay, Thanks for reply. I can see "TestBlockRecovery" and "TestDataDirs" for datanode; also "TestGetImageServlet","TestINodeFile" and "TestNNLeaseRecovery" for namenode. I will start from these first then. Thanks. Best Regards, Min (Catherine) Long IBM China Systems and Technology Lab Jay Booth 09/14/2010 09:59 AM Please respond to hdfs-dev@hadoop.apache.org To hdfs-dev@hadoop.apache.org cc Subject Re: I am a new comer and have a question about HDFS code check out Hi Min, look at the unit tests which make use of MiniDFSCluster -- you can run those in debug mode and step through the code, and they include a bunch of use cases. That's generally much easier than running all of the different services and debugging each separately.
Re: I am a new comer and have a question about HDFS code check out
Hi Min, look at the unit tests which make use of MiniDFSCluster -- you can run those in debug mode and step through the code, and they include a bunch of use cases. That's generally much easier than running all of the different services and debugging each separately. On Mon, Sep 13, 2010 at 9:30 PM, Min Long wrote: > Thanks for help. I could check out codes. Are there any documentation for > setting up debug environment for HDFS? Should Single Node HDFS be set up > for debug purpose? Should Hadoop Common codes be checked out for > running/debugging HDFS? Or can HDFS alone be enough for run/debug? Sorry > for the simple questions. Hope to get reply or any advice to new comer for > HDFS. > > Best Regards, > Min (Catherine) Long > IBM China Systems and Technology Lab > > > > > > 09/13/2010 11:25 AM > Please respond to > hdfs-dev@hadoop.apache.org > > > To > > cc > > Subject > Re: I am a new comer and have a question about HDFS code check out > > > > > > > You need to use Git (http://git-scm.com). It is a source control tool like > CVS, but better suited to large distributed projects. > > > - Original Message - > From: Min Long > To: hdfs-dev@hadoop.apache.org > Sent: Sun Sep 12 23:13:34 2010 > Subject: Re: I am a new comer and have a question about HDFS code check > out > > Thanks for reply. But I find the codes are not zipped, so not to be > downloaded at one shot. Can I check out the code via CVS? If so, what's > the server configuration and directory info for downloading the codes via > CVS? Sorry for the simple question. Hope to get your reply. Thanks. > > Best Regards, > Min (Catherine) Long > IBM China Systems and Technology Lab >
Re: I am a new comer and have a question about HDFS code check out
Thanks for help. I could check out codes. Are there any documentation for setting up debug environment for HDFS? Should Single Node HDFS be set up for debug purpose? Should Hadoop Common codes be checked out for running/debugging HDFS? Or can HDFS alone be enough for run/debug? Sorry for the simple questions. Hope to get reply or any advice to new comer for HDFS. Best Regards, Min (Catherine) Long IBM China Systems and Technology Lab 09/13/2010 11:25 AM Please respond to hdfs-dev@hadoop.apache.org To cc Subject Re: I am a new comer and have a question about HDFS code check out You need to use Git (http://git-scm.com). It is a source control tool like CVS, but better suited to large distributed projects. - Original Message - From: Min Long To: hdfs-dev@hadoop.apache.org Sent: Sun Sep 12 23:13:34 2010 Subject: Re: I am a new comer and have a question about HDFS code check out Thanks for reply. But I find the codes are not zipped, so not to be downloaded at one shot. Can I check out the code via CVS? If so, what's the server configuration and directory info for downloading the codes via CVS? Sorry for the simple question. Hope to get your reply. Thanks. Best Regards, Min (Catherine) Long IBM China Systems and Technology Lab
[jira] Created: (HDFS-1398) HDFS federation: Upgrade and rolling back of Federation
HDFS federation: Upgrade and rolling back of Federation --- Key: HDFS-1398 URL: https://issues.apache.org/jira/browse/HDFS-1398 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tanping Wang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1397) HDFS federation: Storage directory of VERSION(/ID) file
HDFS federation: Storage directory of VERSION(/ID) file Key: HDFS-1397 URL: https://issues.apache.org/jira/browse/HDFS-1397 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tanping Wang -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1396) reloginFromKeytab in Hftp clients
[ https://issues.apache.org/jira/browse/HDFS-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey resolved HDFS-1396. Resolution: Duplicate marking this is as duplicate of HDFS-1364 > reloginFromKeytab in Hftp clients > - > > Key: HDFS-1396 > URL: https://issues.apache.org/jira/browse/HDFS-1396 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1396) reloginFromKeytab in Hftp clients
reloginFromKeytab in Hftp clients - Key: HDFS-1396 URL: https://issues.apache.org/jira/browse/HDFS-1396 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: hadoop.job.ugi backwards compatibility
On Mon, Sep 13, 2010 at 1:04 PM, Owen O'Malley wrote:
> On Mon, Sep 13, 2010 at 11:10 AM, Todd Lipcon wrote:
> > Yep, but there are plenty of 10 node clusters out there that do important
> > work at small startups or single-use-case installations, too. We need to
> > provide scalability and security features that work for the 100+ node
> > clusters but also not leave the beginners in the dust.
>
> 10 node clusters are an important use case, but creating the user
> accounts on those clusters is very easy because of the few users.
> Futhermore, if the accounts aren't there it just means the users have
> no groups. Which for a single use system with security turned off
> isn't the end of the world.
>
> > But I think there are plenty of people out there who have built small
> > webapps, shell scripts, cron jobs, etc that use hadoop.job.ugi on some
> > shared account to impersonate other users.
>
> I'd be surprised. At Yahoo, the primary problem came with people
> screen scraping the jobtracker http. With security turned off that
> isn't an issue. Again, it isn't hard, just the evolving interface of
> UserGroupInformation changed. With security, we tried really hard to
> maintain backwards compatibility and succeeded for the vast (99%+)
> majority of the users.
>
> > Perhaps I am estimating
> > incorrectly - that's why I wanted this discussion on a user-facing list
> > rather than a dev-facing list.
>
> Obviously the pointer is there for them to follow into the rabbit hole
> of the dev lists. *grin*
>
> > Another example use case that I do a lot on non-secure clusters is:
> hadoop
> > fs -Dhadoop.job.ugi=hadoop,hadoop superuser>.
> > The permissions model we have in 0.20 obviously isn't secure, but it's
> nice
> > to avoid accidental mistakes, and making it easy to "sudo" like that is
> > handy.
>
> It might make sense to add a new switch ( -user ?) to hadoop fs that
> does a doAs before doing the
> shell command. You could even make it fancy and try to be a proxy user
> if security is turned on.
>
Yep, I agree - I think either (ab)using proxyuser functionality or adding
some new "sudoers" like configuration would be very handy and we should do
it.
>
> > Regardless of our particular opinions, isn't our policy that we cannot
> break
> > API compatibility between versions without a one-version deprecation
> period?
>
> There wasn't a way to keep UGI stable. It was a broken design before
> the security work. It is marked evolving so we try to minimize
> breakage, but it isn't prohibited.
>
>
I agree that keeping API compatibility for UGI was probably impossible, and
respect that. But it would certainly be very easy to do a patch like the
following:
JobClient(Configuration conf) {
if (conf.get("hadoop.job.ugi") != null &&
UserGroupInformation.isSecurityEnabled()) {
LOG.warn("Stop being evil. Don't use hadoop.job.ugi! RAAWR");
UserGroupInformation.createRemoteUser(...).doAs() { create proxy }
} else {
create normal RPC proxy;
}
}
... and the same on the HDFS side.
Would you -1 such a compatibility layer?
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: namenode crash - recovery model?
use linux heartbeat project and DRBD to build a backup namenode. Jimmy -- From: "John Hui" Sent: Monday, September 13, 2010 2:33 PM To: Subject: namenode crash - recovery model? Hi All, I am new to Hadoop. I have been reading and playing around with Hadoop, it seems like the namenode is a single point of failure. I read about the backup node which is basically a copy of the live namenode. So if the namenode were to crash, what is some of the recovery model that people have in place. Is there a way to configure hadoop to flip the the backup node and use that as the primary name node if the primary namenode goes down.. Thanks in advance, John
namenode crash - recovery model?
Hi All, I am new to Hadoop. I have been reading and playing around with Hadoop, it seems like the namenode is a single point of failure. I read about the backup node which is basically a copy of the live namenode. So if the namenode were to crash, what is some of the recovery model that people have in place. Is there a way to configure hadoop to flip the the backup node and use that as the primary name node if the primary namenode goes down.. Thanks in advance, John
Re: hadoop.job.ugi backwards compatibility
On Mon, Sep 13, 2010 at 11:10 AM, Todd Lipcon wrote: > Yep, but there are plenty of 10 node clusters out there that do important > work at small startups or single-use-case installations, too. We need to > provide scalability and security features that work for the 100+ node > clusters but also not leave the beginners in the dust. 10 node clusters are an important use case, but creating the user accounts on those clusters is very easy because of the few users. Futhermore, if the accounts aren't there it just means the users have no groups. Which for a single use system with security turned off isn't the end of the world. > But I think there are plenty of people out there who have built small > webapps, shell scripts, cron jobs, etc that use hadoop.job.ugi on some > shared account to impersonate other users. I'd be surprised. At Yahoo, the primary problem came with people screen scraping the jobtracker http. With security turned off that isn't an issue. Again, it isn't hard, just the evolving interface of UserGroupInformation changed. With security, we tried really hard to maintain backwards compatibility and succeeded for the vast (99%+) majority of the users. > Perhaps I am estimating > incorrectly - that's why I wanted this discussion on a user-facing list > rather than a dev-facing list. Obviously the pointer is there for them to follow into the rabbit hole of the dev lists. *grin* > Another example use case that I do a lot on non-secure clusters is: hadoop > fs -Dhadoop.job.ugi=hadoop,hadoop . > The permissions model we have in 0.20 obviously isn't secure, but it's nice > to avoid accidental mistakes, and making it easy to "sudo" like that is > handy. It might make sense to add a new switch ( -user ?) to hadoop fs that does a doAs before doing the shell command. You could even make it fancy and try to be a proxy user if security is turned on. > Regardless of our particular opinions, isn't our policy that we cannot break > API compatibility between versions without a one-version deprecation period? There wasn't a way to keep UGI stable. It was a broken design before the security work. It is marked evolving so we try to minimize breakage, but it isn't prohibited. -- Owen
[jira] Created: (HDFS-1395) Add @Override annotation to FSDataset methods that implement FSDatasetInterface
Add @Override annotation to FSDataset methods that implement FSDatasetInterface --- Key: HDFS-1395 URL: https://issues.apache.org/jira/browse/HDFS-1395 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.22.0 @Override annotations are inconsistently added to methods implementing the interface. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: hadoop.job.ugi backwards compatibility
On Mon, Sep 13, 2010 at 10:59 AM, Owen O'Malley wrote: > On Mon, Sep 13, 2010 at 10:05 AM, Todd Lipcon wrote: > > > This is not MR-specific, since the strangely named hadoop.job.ugi > determines > > HDFS permissions as well. > > Yeah, after I hit send, I realized that I should have used common-dev. > This is really a dev issue. > > > "or the user must write a custom group mapper" above refers to this > plugin > > capability. But I think most users do not want to spend the time to write > > (or even setup) such a plugin beyond the default shell-based mapping > > service. > > Sure, which is why it is easiest to just have the (hopefully disabled) > user accounts on the jt/nn. Any installs > 100 nodes should be using > HADOOP-6864 to avoid the fork in the JT/NN. > Yep, but there are plenty of 10 node clusters out there that do important work at small startups or single-use-case installations, too. We need to provide scalability and security features that work for the 100+ node clusters but also not leave the beginners in the dust. > > > As someone who spends an awful lot of time doing downstream support of > lots > > of different clusters, I actually disagree. > > Normal applications never need to do doAs. They run as the default > user. This only comes up in servers that deal with multiple users. In > *that* context, it sucks having servers that only work in non-secure > mode. If some server X only works without security that sucks. Doing > doAs isn't harder, it is just different. Having two different > semantics models *will* cause lots of grief. > I agree that all real (ie community) projects should support both security and non-security and shouldn't be using hadoop.job.ugi to impersonate users. But I think there are plenty of people out there who have built small webapps, shell scripts, cron jobs, etc that use hadoop.job.ugi on some shared account to impersonate other users. Perhaps I am estimating incorrectly - that's why I wanted this discussion on a user-facing list rather than a dev-facing list. Another example use case that I do a lot on non-secure clusters is: hadoop fs -Dhadoop.job.ugi=hadoop,hadoop . The permissions model we have in 0.20 obviously isn't secure, but it's nice to avoid accidental mistakes, and making it easy to "sudo" like that is handy. Regardless of our particular opinions, isn't our policy that we cannot break API compatibility between versions without a one-version deprecation period? I see this as an important API (even if it isn't one we like) and breaking it without such a transition period is against our own rules. Like you said, doAs() isn't any harder, but we need to give people a grace period to switch over, and we probably need to write some command line tools to allow fs operations as superuser, etc. -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: hadoop.job.ugi backwards compatibility
On Mon, Sep 13, 2010 at 10:05 AM, Todd Lipcon wrote: > This is not MR-specific, since the strangely named hadoop.job.ugi determines > HDFS permissions as well. Yeah, after I hit send, I realized that I should have used common-dev. This is really a dev issue. > "or the user must write a custom group mapper" above refers to this plugin > capability. But I think most users do not want to spend the time to write > (or even setup) such a plugin beyond the default shell-based mapping > service. Sure, which is why it is easiest to just have the (hopefully disabled) user accounts on the jt/nn. Any installs > 100 nodes should be using HADOOP-6864 to avoid the fork in the JT/NN. > As someone who spends an awful lot of time doing downstream support of lots > of different clusters, I actually disagree. Normal applications never need to do doAs. They run as the default user. This only comes up in servers that deal with multiple users. In *that* context, it sucks having servers that only work in non-secure mode. If some server X only works without security that sucks. Doing doAs isn't harder, it is just different. Having two different semantics models *will* cause lots of grief. -- Owen
[jira] Created: (HDFS-1394) modify -format option for namenode to generated new blockpool id and accept newcluster
modify -format option for namenode to generated new blockpool id and accept newcluster -- Key: HDFS-1394 URL: https://issues.apache.org/jira/browse/HDFS-1394 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: hadoop.job.ugi backwards compatibility
On Sep 13, 2010, at 10:05 AM, Todd Lipcon wrote: > On Mon, Sep 13, 2010 at 9:31 AM, Owen O'Malley wrote: > >> Moving the discussion over to the more appropriate mapreduce-dev. >> > > This is not MR-specific, since the strangely named hadoop.job.ugi determines > HDFS permissions as well. +CC hdfs-dev... though I actually think this is an > issue that users will have interest in, which is why I posted to general > initially rather than a dev list. +1
Re: hadoop.job.ugi backwards compatibility
On Mon, Sep 13, 2010 at 9:31 AM, Owen O'Malley wrote:
> Moving the discussion over to the more appropriate mapreduce-dev.
>
This is not MR-specific, since the strangely named hadoop.job.ugi determines
HDFS permissions as well. +CC hdfs-dev... though I actually think this is an
issue that users will have interest in, which is why I posted to general
initially rather than a dev list.
> On Mon, Sep 13, 2010 at 9:08 AM, Todd Lipcon wrote:
>
> > 1) Groups resolution happens on the server side, where it used to happen
> on
> > the client. Thus, all Hadoop users must exist on the NN/JT machines in
> order
> > for group mapping to succeed (or the user must write a custom group
> mapper).
>
> There is a plugin that performs the group lookup. See HADOOP-4656.
> There is no requirement for having the user accounts on the NN/JT
> although that is the easiest approach. It is not recommended that the
> users be allowed to login.
>
"or the user must write a custom group mapper" above refers to this plugin
capability. But I think most users do not want to spend the time to write
(or even setup) such a plugin beyond the default shell-based mapping
service.
> I think it is important that turning security on and off doesn't
> drastically change the semantics or protocols. That will become much
> much harder to support downstream.
>
>
As someone who spends an awful lot of time doing downstream support of lots
of different clusters, I actually disagree. I believe the majority of users
do *not* plan on turning on security, so keeping things simpler for them is
worth a lot. In many of these clusters the users and the ops team and the
developers are all one and the same - it's not the multitenant "internal
service" model that we see at the larger installations like Yahoo or
Facebook.
> > 2) The hadoop.job.ugi parameter is ignored - instead the user has to use
> the
> > new UGI.createRemoteUser("foo").doAs() API, even in simple security.
>
> User code that counts on hadoop.job.ugi working will be horribly
> broken once you turn on security. Turning on and off security should
> not involve testing all of your applications. It is unfortunate that
> we ever used the configuration value as the user, but continuing to
> support it will make our user's code much much more brittle.
>
The assumption above is "once you turn on security" - but many users will
not and probably never will turn on security. Providing a transition plan
for one version is our usual policy here - I agree that long term we would
like to do away with this hack of a configuration parameter. Since it's not
hard to provide a backwards compatibility path with a deprecation warning
for one version, are you against it? Or just saying that on your particular
clusters you will choose not to take advantage of it?
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
