Hoping to merge HDFS-1073 branch soon
Hey all, Work on the HDFS-1073 branch has been progressing steadily, and I believe we're coming close to the point where it can be merged. To briefly summarize the status: - NameNode and SecondaryNameNode are both fully working and have undergone some stress/fault testing in addition to a over 3000 lines worth of new unit tests. - Most of the existing unit tests have been updated, though a few more need some small tweaks (HDFS-2101) - The BackupNode and CheckpointNode are not currently working, though I am working on it locally and making good progress (HDFS-1979) - There are a few various and sundry small improvements that should probably be done before release, but I think could be done either before or after merge (eg HDFS-2104) Given this, I am expecting that we can merge this into trunk by the end of July if not earlier, as soon as the BN/CN work is complete. If you are hoping to review the code or tests before merge time, this is your early warning! Please do so now! Thanks! -Todd P.S. I will also be giving a short talk about the motivations and current status of this project at Friday's contributor meeting, for those who are able to attend. If we're lucky, maybe even a demo! -- Todd Lipcon Software Engineer, Cloudera
Re: Hoping to merge HDFS-1073 branch soon
Hi all, Just an update on this project: - The current list of uncommitted patches up for review is: 1bea9d3 HDFS-1979. Fix BackupNode and CheckpointNode 32db384 Amend HDFS-2011. Fix TestCheckpoint test for double close/abort of ELFOS b6a55a4 HDFS-2101. Update remaining unit tests for new layout ca0ace6 HDFS-2133. Address TODOs left in code b46825d HDFS-1780. reduce need to rewrite fsimage on statrtup 30c858d HDFS-2104. Add flag to SecondaryNameNode to format it during startup 942eaef HDFS-2135. Fix regression of HDFS-1955 in branch I believe Eli is going to work on reviewing these this week. - I've set up a Hudson job for the branch here: https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/ It's currently failing because it's missing some of the patches above. After the above patches go in, I expect a pretty clean build, modulo maybe one or two things that are environment issues, which I'll tackle later this week. - BackupNode and CheckpointNode are working. I've done some basic functional testing by pounding edits into the NN while both a 2NN and a BN are checkpointing every 2 seconds. - I merged with trunk as of this morning, so I think we should be up-to-date with trunk patches. Aaron was very helpful and went through all NN-related patches in trunk from the last 3 months to make sure we didn't inadvertently regress anything - he discovered one bug but everything else looks good. Once the above patches are in the branch, I would like to merge. So, if you plan on reviewing pre-merge, please do so *this week*. Of course, if you don't have time and you find issues post-merge, I absolutely plan on fixing them ASAP ;-) Thanks -Todd On Thu, Jun 30, 2011 at 12:11 AM, Todd Lipcon wrote: > Hey all, > > Work on the HDFS-1073 branch has been progressing steadily, and I believe > we're coming close to the point where it can be merged. To briefly summarize > the status: > - NameNode and SecondaryNameNode are both fully working and have undergone > some stress/fault testing in addition to a over 3000 lines worth of new unit > tests. > - Most of the existing unit tests have been updated, though a few more need > some small tweaks (HDFS-2101) > - The BackupNode and CheckpointNode are not currently working, though I am > working on it locally and making good progress (HDFS-1979) > - There are a few various and sundry small improvements that should > probably be done before release, but I think could be done either before or > after merge (eg HDFS-2104) > > Given this, I am expecting that we can merge this into trunk by the end of > July if not earlier, as soon as the BN/CN work is complete. If you are > hoping to review the code or tests before merge time, this is your early > warning! Please do so now! > > Thanks! > > -Todd > P.S. I will also be giving a short talk about the motivations and current > status of this project at Friday's contributor meeting, for those who are > able to attend. If we're lucky, maybe even a demo! > -- > Todd Lipcon > Software Engineer, Cloudera > -- Todd Lipcon Software Engineer, Cloudera
Re: Hoping to merge HDFS-1073 branch soon
We can merge 1580 after 1073 is merged in. Looks like the biggest thing in your 1073 list is the Backup NN related changes. Are you shooting for end of this month? sanjay On Jul 6, 2011, at 8:03 PM, Todd Lipcon wrote: > Hi all, > > Just an update on this project: > - The current list of uncommitted patches up for review is: > > 1bea9d3 HDFS-1979. Fix BackupNode and CheckpointNode > 32db384 Amend HDFS-2011. Fix TestCheckpoint test for double close/abort of > ELFOS > b6a55a4 HDFS-2101. Update remaining unit tests for new layout > ca0ace6 HDFS-2133. Address TODOs left in code > b46825d HDFS-1780. reduce need to rewrite fsimage on statrtup > 30c858d HDFS-2104. Add flag to SecondaryNameNode to format it during startup > 942eaef HDFS-2135. Fix regression of HDFS-1955 in branch > > I believe Eli is going to work on reviewing these this week. > > - I've set up a Hudson job for the branch here: > https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/ > It's currently failing because it's missing some of the patches above. After > the above patches go in, I expect a pretty clean build, modulo maybe one or > two things that are environment issues, which I'll tackle later this week. > > - BackupNode and CheckpointNode are working. I've done some basic functional > testing by pounding edits into the NN while both a 2NN and a BN are > checkpointing every 2 seconds. > - I merged with trunk as of this morning, so I think we should be up-to-date > with trunk patches. Aaron was very helpful and went through all NN-related > patches in trunk from the last 3 months to make sure we didn't inadvertently > regress anything - he discovered one bug but everything else looks good. > > Once the above patches are in the branch, I would like to merge. So, if you > plan on reviewing pre-merge, please do so *this week*. Of course, if you > don't have time and you find issues post-merge, I absolutely plan on fixing > them ASAP ;-) > > Thanks > -Todd > > On Thu, Jun 30, 2011 at 12:11 AM, Todd Lipcon wrote: > >> Hey all, >> >> Work on the HDFS-1073 branch has been progressing steadily, and I believe >> we're coming close to the point where it can be merged. To briefly summarize >> the status: >> - NameNode and SecondaryNameNode are both fully working and have undergone >> some stress/fault testing in addition to a over 3000 lines worth of new unit >> tests. >> - Most of the existing unit tests have been updated, though a few more need >> some small tweaks (HDFS-2101) >> - The BackupNode and CheckpointNode are not currently working, though I am >> working on it locally and making good progress (HDFS-1979) >> - There are a few various and sundry small improvements that should >> probably be done before release, but I think could be done either before or >> after merge (eg HDFS-2104) >> >> Given this, I am expecting that we can merge this into trunk by the end of >> July if not earlier, as soon as the BN/CN work is complete. If you are >> hoping to review the code or tests before merge time, this is your early >> warning! Please do so now! >> >> Thanks! >> >> -Todd >> P.S. I will also be giving a short talk about the motivations and current >> status of this project at Friday's contributor meeting, for those who are >> able to attend. If we're lucky, maybe even a demo! >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera
Re: Hoping to merge HDFS-1073 branch soon
On Tue, Jul 12, 2011 at 10:38 AM, sanjay Radia wrote: > We can merge 1580 after 1073 is merged in. > > Looks like the biggest thing in your 1073 list is the Backup NN related > changes. > The BN-related changes are done and just awaiting code review. See HDFS-1979. The current list of patches awaiting review are: HDFS-1979, HDFS-2101, HDFS-2133, HDFS-1780, HDFS-2104, HDFS-2135. > Are you shooting for end of this month? > I'm hoping as early as next week, assuming folks feel the branch is in good shape. If all goes well, I'll have code reviews back for the above in the next day or two, can respond to review comments and commit over the weekend, and call a vote to merge early next week. Thanks -Todd > On Jul 6, 2011, at 8:03 PM, Todd Lipcon wrote: > > > Hi all, > > > > Just an update on this project: > > - The current list of uncommitted patches up for review is: > > > > 1bea9d3 HDFS-1979. Fix BackupNode and CheckpointNode > > 32db384 Amend HDFS-2011. Fix TestCheckpoint test for double close/abort > of > > ELFOS > > b6a55a4 HDFS-2101. Update remaining unit tests for new layout > > ca0ace6 HDFS-2133. Address TODOs left in code > > b46825d HDFS-1780. reduce need to rewrite fsimage on statrtup > > 30c858d HDFS-2104. Add flag to SecondaryNameNode to format it during > startup > > 942eaef HDFS-2135. Fix regression of HDFS-1955 in branch > > > > I believe Eli is going to work on reviewing these this week. > > > > - I've set up a Hudson job for the branch here: > > https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/ > > It's currently failing because it's missing some of the patches above. > After > > the above patches go in, I expect a pretty clean build, modulo maybe one > or > > two things that are environment issues, which I'll tackle later this > week. > > > > - BackupNode and CheckpointNode are working. I've done some basic > functional > > testing by pounding edits into the NN while both a 2NN and a BN are > > checkpointing every 2 seconds. > > - I merged with trunk as of this morning, so I think we should be > up-to-date > > with trunk patches. Aaron was very helpful and went through all > NN-related > > patches in trunk from the last 3 months to make sure we didn't > inadvertently > > regress anything - he discovered one bug but everything else looks good. > > > > Once the above patches are in the branch, I would like to merge. So, if > you > > plan on reviewing pre-merge, please do so *this week*. Of course, if you > > don't have time and you find issues post-merge, I absolutely plan on > fixing > > them ASAP ;-) > > > > Thanks > > -Todd > > > > On Thu, Jun 30, 2011 at 12:11 AM, Todd Lipcon wrote: > > > >> Hey all, > >> > >> Work on the HDFS-1073 branch has been progressing steadily, and I > believe > >> we're coming close to the point where it can be merged. To briefly > summarize > >> the status: > >> - NameNode and SecondaryNameNode are both fully working and have > undergone > >> some stress/fault testing in addition to a over 3000 lines worth of new > unit > >> tests. > >> - Most of the existing unit tests have been updated, though a few more > need > >> some small tweaks (HDFS-2101) > >> - The BackupNode and CheckpointNode are not currently working, though I > am > >> working on it locally and making good progress (HDFS-1979) > >> - There are a few various and sundry small improvements that should > >> probably be done before release, but I think could be done either before > or > >> after merge (eg HDFS-2104) > >> > >> Given this, I am expecting that we can merge this into trunk by the end > of > >> July if not earlier, as soon as the BN/CN work is complete. If you are > >> hoping to review the code or tests before merge time, this is your early > >> warning! Please do so now! > >> > >> Thanks! > >> > >> -Todd > >> P.S. I will also be giving a short talk about the motivations and > current > >> status of this project at Friday's contributor meeting, for those who > are > >> able to attend. If we're lucky, maybe even a demo! > >> -- > >> Todd Lipcon > >> Software Engineer, Cloudera > >> > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > -- Todd Lipcon Software Engineer, Cloudera
Re: Hoping to merge HDFS-1073 branch soon
On Tue, Jul 12, 2011 at 3:44 PM, Todd Lipcon wrote: > On Tue, Jul 12, 2011 at 10:38 AM, sanjay Radia wrote: > >> We can merge 1580 after 1073 is merged in. >> >> Looks like the biggest thing in your 1073 list is the Backup NN related >> changes. >> > > The BN-related changes are done and just awaiting code review. See > HDFS-1979. The current list of patches awaiting review are: HDFS-1979, > HDFS-2101, HDFS-2133, HDFS-1780, HDFS-2104, HDFS-2135. I'm reviewing these now. I started HDFS-1979 last week, it's a big change so taking me a little while. Thanks, Eli
