Re: Please join me in welcoming the following people as committers to the Hadoop project
congrats all..I hope i'll be committer some day as well:) On Thu, Jan 6, 2011 at 10:23 AM, li ping wrote: > congratulations > > On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman wrote: > > > On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome > > to the following people, > > who have all chosen to accept the role of committers on Hadoop. > > > > In no alphabetical order: > > > > - Aaron Kimball > > - Allen Wittenauer > > - Amar Kamat > > - Dmytro Molkov > > - Jitendra Pandey > > - Kan Zhang > > - Ravi Gummadi > > - Sreekanth Ramakrishna > > - Todd Lipcon > > > > I appreciate all the hard work these people have put into the project so > > far, and look forward to future contributions they will make to Hadoop in > > the future > > > > Well done guys! > > > > > > --Ian > > > > > > > -- > -李平 > -- Deepak Sharma http://www.linkedin.com/in/rikindia
Re: Moving to a new namenode
could you move this traffic to HDFS? thanks! On Jan 5, 2011, at 6:29 PM, Mag Gam wrote: > What is the correct procedure to move to a new namenode? My current > namenode is running on a 2 core with 4gb of memory. I am moving to a > new host, 8 core x 32 GB of memory. My question are: what is the > proper way to move to the new namenode, and what can I do to enhance > the performance of it since the new hardware is much superior than the > old one.
Re: Patch testing
On Wed, Jan 05, 2011 at 08:46PM, Nigel Daley wrote: > Lol. At least now we know how much it's valued. I could get a foot long from > subway. But hurry Nige - pretty soon it's gonna be worthless ;( > Cheers, > Nige > > On Jan 5, 2011, at 8:11 PM, Stack wrote: > > > I'll give you $5.00 Nigel if you add HBase to the list below. > > St.Ack > > > > On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley wrote: > >> I think ZK, PIG, etc will also be included in the update I'm working on. > >> > >> Cheers, > >> Nige > >> > >> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote: > >> > >>> Great. > >>> > >>> Nigel, > >>> Can you please document in somewhere on how you fixed it? We¹d like to fix > >>> it for ZooKeeper as well. > >>> > >>> Thanks > >>> mahadev > >>> > >>> > >>> On 10/20/10 1:23 PM, "Owen O'Malley" wrote: > >>> > > > On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote: > > > I'm working to get the pre-commit patch testing running again for > > HDFS, HADOOP, and MAPREDUCE patches. > > That would be great, Nigel. > > Thanks, > Owen > > >>> > >> > >> signature.asc Description: Digital signature
Re: Please join me in welcoming the following people as committers to the Hadoop project
congratulations On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman wrote: > On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome > to the following people, > who have all chosen to accept the role of committers on Hadoop. > > In no alphabetical order: > > - Aaron Kimball > - Allen Wittenauer > - Amar Kamat > - Dmytro Molkov > - Jitendra Pandey > - Kan Zhang > - Ravi Gummadi > - Sreekanth Ramakrishna > - Todd Lipcon > > I appreciate all the hard work these people have put into the project so > far, and look forward to future contributions they will make to Hadoop in > the future > > Well done guys! > > > --Ian > > -- -李平
Re: Patch testing
Lol. At least now we know how much it's valued. I could get a foot long from subway. Cheers, Nige On Jan 5, 2011, at 8:11 PM, Stack wrote: > I'll give you $5.00 Nigel if you add HBase to the list below. > St.Ack > > On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley wrote: >> I think ZK, PIG, etc will also be included in the update I'm working on. >> >> Cheers, >> Nige >> >> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote: >> >>> Great. >>> >>> Nigel, >>> Can you please document in somewhere on how you fixed it? We¹d like to fix >>> it for ZooKeeper as well. >>> >>> Thanks >>> mahadev >>> >>> >>> On 10/20/10 1:23 PM, "Owen O'Malley" wrote: >>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote: > I'm working to get the pre-commit patch testing running again for > HDFS, HADOOP, and MAPREDUCE patches. That would be great, Nigel. Thanks, Owen >>> >> >>
Re: Please join me in welcoming the following people as committers to the Hadoop project
Congratulations! Great to see the community growing once again. On Wed, Jan 5, 2011 at 8:34 PM, Jay Booth wrote: > Congrats, all! > > On Wed, Jan 5, 2011 at 11:09 PM, Stack wrote: > > > Congrats lads. > > St.Ack > > > > On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman wrote: > > > On behalf of the Apache Hadoop PMC, I would like to extend a warm > welcome > > to the following people, > > > who have all chosen to accept the role of committers on Hadoop. > > > > > > In no alphabetical order: > > > > > > - Aaron Kimball > > > - Allen Wittenauer > > > - Amar Kamat > > > - Dmytro Molkov > > > - Jitendra Pandey > > > - Kan Zhang > > > - Ravi Gummadi > > > - Sreekanth Ramakrishna > > > - Todd Lipcon > > > > > > I appreciate all the hard work these people have put into the project > so > > far, and look forward to future contributions they will make to Hadoop in > > the future > > > > > > Well done guys! > > > > > > > > > --Ian > > > > > > > > >
Re: Please join me in welcoming the following people as committers to the Hadoop project
Congrats, all! On Wed, Jan 5, 2011 at 11:09 PM, Stack wrote: > Congrats lads. > St.Ack > > On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman wrote: > > On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome > to the following people, > > who have all chosen to accept the role of committers on Hadoop. > > > > In no alphabetical order: > > > > - Aaron Kimball > > - Allen Wittenauer > > - Amar Kamat > > - Dmytro Molkov > > - Jitendra Pandey > > - Kan Zhang > > - Ravi Gummadi > > - Sreekanth Ramakrishna > > - Todd Lipcon > > > > I appreciate all the hard work these people have put into the project so > far, and look forward to future contributions they will make to Hadoop in > the future > > > > Well done guys! > > > > > > --Ian > > > > >
Re: Patch testing
I'll give you $5.00 Nigel if you add HBase to the list below. St.Ack On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley wrote: > I think ZK, PIG, etc will also be included in the update I'm working on. > > Cheers, > Nige > > On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote: > >> Great. >> >> Nigel, >> Can you please document in somewhere on how you fixed it? We¹d like to fix >> it for ZooKeeper as well. >> >> Thanks >> mahadev >> >> >> On 10/20/10 1:23 PM, "Owen O'Malley" wrote: >> >>> >>> >>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote: >>> I'm working to get the pre-commit patch testing running again for HDFS, HADOOP, and MAPREDUCE patches. >>> >>> That would be great, Nigel. >>> >>> Thanks, >>> Owen >>> >> > >
Re: Please join me in welcoming the following people as committers to the Hadoop project
Congrats lads. St.Ack On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman wrote: > On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome to > the following people, > who have all chosen to accept the role of committers on Hadoop. > > In no alphabetical order: > > - Aaron Kimball > - Allen Wittenauer > - Amar Kamat > - Dmytro Molkov > - Jitendra Pandey > - Kan Zhang > - Ravi Gummadi > - Sreekanth Ramakrishna > - Todd Lipcon > > I appreciate all the hard work these people have put into the project so far, > and look forward to future contributions they will make to Hadoop in the > future > > Well done guys! > > > --Ian > >
Please join me in welcoming the following people as committers to the Hadoop project
On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome to the following people, who have all chosen to accept the role of committers on Hadoop. In no alphabetical order: - Aaron Kimball - Allen Wittenauer - Amar Kamat - Dmytro Molkov - Jitendra Pandey - Kan Zhang - Ravi Gummadi - Sreekanth Ramakrishna - Todd Lipcon I appreciate all the hard work these people have put into the project so far, and look forward to future contributions they will make to Hadoop in the future Well done guys! --Ian
Moving to a new namenode
What is the correct procedure to move to a new namenode? My current namenode is running on a 2 core with 4gb of memory. I am moving to a new host, 8 core x 32 GB of memory. My question are: what is the proper way to move to the new namenode, and what can I do to enhance the performance of it since the new hardware is much superior than the old one.
Re: decommission a rack
Thanks for the response. I did decommission the server and it took couple of minutes. However, I see the server in my "Dead Datanotes" page. How can I completely forget about this box? On Wed, Jan 5, 2011 at 10:01 AM, Segel, Mike wrote: > Change your topology script? > > One thing about rack awareness. You never want to have an unbalanced > configuration. The caveat is if your uplinks between the ToR Switch are > 'fat' enough. (4 10GBe should do it.) > > If you watched your system with the one machine on a separate rack (assuming > it was also on a separate switch) you'd see a major network bottle neck. (We > use Ganglia) > > If you're seeing the rack count wrong, you could just bounce the cloud and it > should show only two. > > Note: Deprecating a machine can take a long time. You may want to consider > just killing the node, waiting till the cluster recognizes that its down and > I believe just kicking off an FSCK or maybe running the balancer program will > clean that up faster than deprecating. (YMMV) > > HTH > > -Mike > > > -Original Message- > From: Mag Gam [mailto:magaw...@gmail.com] > Sent: Wednesday, January 05, 2011 5:56 AM > To: general@hadoop.apache.org > Subject: decommission a rack > > Using 0.21 > > I have a server which was in rack3. It was the only server in this > rack so I decommissioned the server. fsck shows only 2 racks however > topology shows 3 racks. How can I fix this? > > > The information contained in this communication may be CONFIDENTIAL and is > intended only for the use of the recipient(s) named above. If you are not > the intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its contents, is > strictly prohibited. If you have received this communication in error, > please notify the sender and delete/destroy the original message and any copy > of it from your computer or paper files. >
Re: Patch testing
On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley wrote: > Thanks for looking into it Todd. Let's first see if you think it can be > fixed quickly. Let me know. > > No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which fixes this test timeout for me. -Todd > On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote: > > > On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley wrote: > > > >> Todd, would love to get > >> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since > >> this is failing every night on trunk. > >> > > > > What if we disable that test, move that issue to 0.22 blocker, and then > > enable the test-patch? I'll also look into that one today, but if it's > > something that will take a while to fix, I don't think we should hold off > > the useful testing for all the other patches. > > > > -Todd > > > > On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote: > >> > >>> Hi Nigel, > >>> > >>> MAPREDUCE-2172 has been fixed for a while. Are there any other > particular > >>> JIRAs you think need to be fixed before the MR test-patch queue gets > >>> enabled? I have a lot of outstanding patches and doing all the > test-patch > >>> turnaround manually on 3 different boxes is a real headache. > >>> > >>> Thanks > >>> -Todd > >>> > >>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley wrote: > >>> > Ok, HDFS is now enabled. You'll see a stream of updates shortly on > the > >> ~30 > Patch Available HDFS issues. > > Nige > > On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote: > > > I committed HDFS-1511 this morning. We should be good to go. I can > > haz snooty robot butler? > > > > On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik > wrote: > >> Thanks Jacob. I am wasted already but I can do it on Sun, I think, > >> unless it is done earlier. > >> -- > >> Take care, > >> Konstantin (Cos) Boudnik > >> > >> > >> > >> On Fri, Dec 17, 2010 at 19:41, Jakob Homan > wrote: > >>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants > to > >>> whip one up tonight. > >>> > >>> > >>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley > wrote: > I agree with Cos on fixing HDFS-1511 first. Once that is done I'll > enable hdfs patch testing. > > Cheers, > Nige > > Sent from my iPhone4 > > On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik > wrote: > > > One more issue needs to be addressed before test-patch is turned > on > HDFS is > > https://issues.apache.org/jira/browse/HDFS-1511 > > -- > > Take care, > > Konstantin (Cos) Boudnik > > > > > > > > On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik < > c...@apache.org> > wrote: > >> Considering that because of these 4 faulty cases every patch > will > >> be > >> -1'ed a patch author will still have to look at it and make a > comment > >> why this particular -1 isn't valid. Lesser work, perhaps, but > messier > >> IMO. I'm not blocking it - I just feel like there's a better > way. > >> > >> -- > >> Take care, > >> Konstantin (Cos) Boudnik > >> > >> > >> > >> On Fri, Dec 17, 2010 at 15:55, Jakob Homan > wrote: > If HDFS is added to the test-patch queue right now we get > nothing but dozens of -1'ed patches. > >>> There aren't dozens of patches being submitted currently. The > -1 > >>> isn't the important thing, it's the grunt work of actually > >> running > >>> (and waiting) for the tests, test-patch, etc. that Hudson does > so > that > >>> the developer doesn't have to. > >>> > >>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur < > dhr...@gmail.com> wrote: > +1, thanks for doing this. > > On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan < > jgho...@gmail.com > >>> > wrote: > > > So, with test-patch updated to show the failing tests, saving > >> the > > developers the need to go and verify that the failed tests > are > all > > known, how do people feel about turning on test-patch again > for > HDFS > > and mapred? I think it'll help prevent any more tests from > entering > > the "yeah, we know" category. > > > > Thanks, > > jg > > > > > > On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan < > jho...@yahoo-inc.com> wrote: > >> True, each patch would get a -1 and the failing tests would > >> need > to be > >> verified as those known bad (BTW, it would be great if > Hudson > could list > >> which tests failed in the message it posts to JIRA).
Re: Patch testing
Thanks for looking into it Todd. Let's first see if you think it can be fixed quickly. Let me know. Thanks, Nige On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote: > On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley wrote: > >> Todd, would love to get >> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since >> this is failing every night on trunk. >> > > What if we disable that test, move that issue to 0.22 blocker, and then > enable the test-patch? I'll also look into that one today, but if it's > something that will take a while to fix, I don't think we should hold off > the useful testing for all the other patches. > > -Todd > > On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote: >> >>> Hi Nigel, >>> >>> MAPREDUCE-2172 has been fixed for a while. Are there any other particular >>> JIRAs you think need to be fixed before the MR test-patch queue gets >>> enabled? I have a lot of outstanding patches and doing all the test-patch >>> turnaround manually on 3 different boxes is a real headache. >>> >>> Thanks >>> -Todd >>> >>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley wrote: >>> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the >> ~30 Patch Available HDFS issues. Nige On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote: > I committed HDFS-1511 this morning. We should be good to go. I can > haz snooty robot butler? > > On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik wrote: >> Thanks Jacob. I am wasted already but I can do it on Sun, I think, >> unless it is done earlier. >> -- >> Take care, >> Konstantin (Cos) Boudnik >> >> >> >> On Fri, Dec 17, 2010 at 19:41, Jakob Homan wrote: >>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to >>> whip one up tonight. >>> >>> >>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley wrote: I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing. Cheers, Nige Sent from my iPhone4 On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik wrote: > One more issue needs to be addressed before test-patch is turned on HDFS is > https://issues.apache.org/jira/browse/HDFS-1511 > -- > Take care, > Konstantin (Cos) Boudnik > > > > On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik wrote: >> Considering that because of these 4 faulty cases every patch will >> be >> -1'ed a patch author will still have to look at it and make a comment >> why this particular -1 isn't valid. Lesser work, perhaps, but messier >> IMO. I'm not blocking it - I just feel like there's a better way. >> >> -- >> Take care, >> Konstantin (Cos) Boudnik >> >> >> >> On Fri, Dec 17, 2010 at 15:55, Jakob Homan wrote: If HDFS is added to the test-patch queue right now we get nothing but dozens of -1'ed patches. >>> There aren't dozens of patches being submitted currently. The -1 >>> isn't the important thing, it's the grunt work of actually >> running >>> (and waiting) for the tests, test-patch, etc. that Hudson does so that >>> the developer doesn't have to. >>> >>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur < dhr...@gmail.com> wrote: +1, thanks for doing this. On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan >> wrote: > So, with test-patch updated to show the failing tests, saving >> the > developers the need to go and verify that the failed tests are all > known, how do people feel about turning on test-patch again for HDFS > and mapred? I think it'll help prevent any more tests from entering > the "yeah, we know" category. > > Thanks, > jg > > > On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan < jho...@yahoo-inc.com> wrote: >> True, each patch would get a -1 and the failing tests would >> need to be >> verified as those known bad (BTW, it would be great if Hudson could list >> which tests failed in the message it posts to JIRA). But >> that's still > quite >> a bit less error-prone work than if the developer runs the >> tests and >> test-patch themselves. Also, with 22 being cut, there are a >> lot of > patches >> up in the air and several developers are juggling multiple patches. The >> more automation we can have, even if it's not perfect, will decrease > errors
Re: Patch testing
On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley wrote: > Todd, would love to get > https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since > this is failing every night on trunk. > What if we disable that test, move that issue to 0.22 blocker, and then enable the test-patch? I'll also look into that one today, but if it's something that will take a while to fix, I don't think we should hold off the useful testing for all the other patches. -Todd On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote: > > > Hi Nigel, > > > > MAPREDUCE-2172 has been fixed for a while. Are there any other particular > > JIRAs you think need to be fixed before the MR test-patch queue gets > > enabled? I have a lot of outstanding patches and doing all the test-patch > > turnaround manually on 3 different boxes is a real headache. > > > > Thanks > > -Todd > > > > On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley wrote: > > > >> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the > ~30 > >> Patch Available HDFS issues. > >> > >> Nige > >> > >> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote: > >> > >>> I committed HDFS-1511 this morning. We should be good to go. I can > >>> haz snooty robot butler? > >>> > >>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik > >> wrote: > Thanks Jacob. I am wasted already but I can do it on Sun, I think, > unless it is done earlier. > -- > Take care, > Konstantin (Cos) Boudnik > > > > On Fri, Dec 17, 2010 at 19:41, Jakob Homan wrote: > > Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to > > whip one up tonight. > > > > > > On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley wrote: > >> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll > >> enable hdfs patch testing. > >> > >> Cheers, > >> Nige > >> > >> Sent from my iPhone4 > >> > >> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik > >> wrote: > >> > >>> One more issue needs to be addressed before test-patch is turned on > >> HDFS is > >>> https://issues.apache.org/jira/browse/HDFS-1511 > >>> -- > >>> Take care, > >>> Konstantin (Cos) Boudnik > >>> > >>> > >>> > >>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik > >> wrote: > Considering that because of these 4 faulty cases every patch will > be > -1'ed a patch author will still have to look at it and make a > >> comment > why this particular -1 isn't valid. Lesser work, perhaps, but > >> messier > IMO. I'm not blocking it - I just feel like there's a better way. > > -- > Take care, > Konstantin (Cos) Boudnik > > > > On Fri, Dec 17, 2010 at 15:55, Jakob Homan > >> wrote: > >> If HDFS is added to the test-patch queue right now we get > >> nothing but dozens of -1'ed patches. > > There aren't dozens of patches being submitted currently. The -1 > > isn't the important thing, it's the grunt work of actually > running > > (and waiting) for the tests, test-patch, etc. that Hudson does so > >> that > > the developer doesn't have to. > > > > On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur < > >> dhr...@gmail.com> wrote: > >> +1, thanks for doing this. > >> > >> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan > > >> wrote: > >> > >>> So, with test-patch updated to show the failing tests, saving > the > >>> developers the need to go and verify that the failed tests are > >> all > >>> known, how do people feel about turning on test-patch again for > >> HDFS > >>> and mapred? I think it'll help prevent any more tests from > >> entering > >>> the "yeah, we know" category. > >>> > >>> Thanks, > >>> jg > >>> > >>> > >>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan < > >> jho...@yahoo-inc.com> wrote: > True, each patch would get a -1 and the failing tests would > need > >> to be > verified as those known bad (BTW, it would be great if Hudson > >> could list > which tests failed in the message it posts to JIRA). But > that's > >> still > >>> quite > a bit less error-prone work than if the developer runs the > tests > >> and > test-patch themselves. Also, with 22 being cut, there are a > lot > >> of > >>> patches > up in the air and several developers are juggling multiple > >> patches. The > more automation we can have, even if it's not perfect, will > >> decrease > >>> errors > we may make. > -jg > > Nigel Daley wrote: > > > > On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote: > > > >>> It's also ready to run o
Re: Patch testing
Todd, would love to get https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since this is failing every night on trunk. Nige On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote: > Hi Nigel, > > MAPREDUCE-2172 has been fixed for a while. Are there any other particular > JIRAs you think need to be fixed before the MR test-patch queue gets > enabled? I have a lot of outstanding patches and doing all the test-patch > turnaround manually on 3 different boxes is a real headache. > > Thanks > -Todd > > On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley wrote: > >> Ok, HDFS is now enabled. You'll see a stream of updates shortly on the ~30 >> Patch Available HDFS issues. >> >> Nige >> >> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote: >> >>> I committed HDFS-1511 this morning. We should be good to go. I can >>> haz snooty robot butler? >>> >>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik >> wrote: Thanks Jacob. I am wasted already but I can do it on Sun, I think, unless it is done earlier. -- Take care, Konstantin (Cos) Boudnik On Fri, Dec 17, 2010 at 19:41, Jakob Homan wrote: > Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to > whip one up tonight. > > > On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley wrote: >> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll >> enable hdfs patch testing. >> >> Cheers, >> Nige >> >> Sent from my iPhone4 >> >> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik >> wrote: >> >>> One more issue needs to be addressed before test-patch is turned on >> HDFS is >>> https://issues.apache.org/jira/browse/HDFS-1511 >>> -- >>> Take care, >>> Konstantin (Cos) Boudnik >>> >>> >>> >>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik >> wrote: Considering that because of these 4 faulty cases every patch will be -1'ed a patch author will still have to look at it and make a >> comment why this particular -1 isn't valid. Lesser work, perhaps, but >> messier IMO. I'm not blocking it - I just feel like there's a better way. -- Take care, Konstantin (Cos) Boudnik On Fri, Dec 17, 2010 at 15:55, Jakob Homan >> wrote: >> If HDFS is added to the test-patch queue right now we get >> nothing but dozens of -1'ed patches. > There aren't dozens of patches being submitted currently. The -1 > isn't the important thing, it's the grunt work of actually running > (and waiting) for the tests, test-patch, etc. that Hudson does so >> that > the developer doesn't have to. > > On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur < >> dhr...@gmail.com> wrote: >> +1, thanks for doing this. >> >> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan >> wrote: >> >>> So, with test-patch updated to show the failing tests, saving the >>> developers the need to go and verify that the failed tests are >> all >>> known, how do people feel about turning on test-patch again for >> HDFS >>> and mapred? I think it'll help prevent any more tests from >> entering >>> the "yeah, we know" category. >>> >>> Thanks, >>> jg >>> >>> >>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan < >> jho...@yahoo-inc.com> wrote: True, each patch would get a -1 and the failing tests would need >> to be verified as those known bad (BTW, it would be great if Hudson >> could list which tests failed in the message it posts to JIRA). But that's >> still >>> quite a bit less error-prone work than if the developer runs the tests >> and test-patch themselves. Also, with 22 being cut, there are a lot >> of >>> patches up in the air and several developers are juggling multiple >> patches. The more automation we can have, even if it's not perfect, will >> decrease >>> errors we may make. -jg Nigel Daley wrote: > > On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote: > >>> It's also ready to run on MapReduce and HDFS but we won't >> turn it on >>> until these projects build and test cleanly. Looks like both >> these >>> projects >>> currently have test failures. >> >> Assuming the projects are compiling and building, is there a >> reason to >> not turn it on despite the test failures? Hudson is invaluable >> to >>> developers >> who then don't have to run the tests and test-patch >> themselves. We >>> didn't >> turn Hudson off when it was working p
Hadoop 0.22 Release Preparation
Folks, Happy New Year! As Hadoop 0.22 release manager I'd like to start moving towards a release. There are 2 main lists of issues that I'd like to review with folks. I propose doing this on the Hadoop IRC channel, reviewing issues one-on-one with issue assignee (or reporter if no assignee) at a designated time. Of course anyone else is free to chime in during the IRC discussion. Decisions and rationale will be recorded on the respective Jiras. The 2 lists are: 1) All COMMON, HDFS, and MAPREDUCE issues that are currently marked BLOCKER against any release: http://tinyurl.com/2a4sh6s The goal is to correctly set FIX VERSION on these issues and try to assign most of these issues. 2) All COMMON, HDFS, and MAPREDUCE issues that are BUGS and CRITICAL against any release: http://tinyurl.com/29w9bxw The goal is to assess whether the issues should be moved to BLOCKER state for Hadoop 0.22 If you have other BUG issues that you believe are blockers but are not in the BLOCKER or CRITICAL Jira state, then please mark them as such before this review occurs so that they can be evaluated. ***I will be removing the 0.22 FIX VERSION for all non-blocker issues before these reviews start.*** As always, test and doc IMPROVEMENTS are always eligible for commit to 0.22. They should be marked as BLOCKER with FIX VERSION set to 0.22. I may downgrade them from BLOCKER once we have the above reviews complete. I know this is a volunteer effort, but it doesn't hurt to have some goals. I'd like to start the review of these issues early next week and complete the review by Jan 21. I'd love to see blockers fixed by Feb 21 and have our first release candidate shortly thereafter. I'll try to coordinate test resources closer to that time. Thoughts on the above? Thanks, Nige
Re: svn commit: r1055684 - /hadoop/common/branches/branch-0.20/CHANGES.txt
Is 20.3 a 'dead' release ? I haven't seen any discussion of this on the apache lists about creating a 20.3 release, and kind of goes against all the discussion that we recently had with StAck about creating a 'append' release on 0.20. I'm not against 20.3, but I would like to see some discussion and not have things reverted out of it without discussion. On Jan 6, 2011, at 10:12 AM, omal...@apache.org wrote: > Author: omalley > Date: Wed Jan 5 23:12:49 2011 > New Revision: 1055684 > > URL: http://svn.apache.org/viewvc?rev=1055684&view=rev > Log: > HADOOP-1734. Move to release 0.20.4 since I already made the tag 0.20.3. > > Modified: >hadoop/common/branches/branch-0.20/CHANGES.txt > > Modified: hadoop/common/branches/branch-0.20/CHANGES.txt > URL: > http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=1055684&r1=1055683&r2=1055684&view=diff > == > --- hadoop/common/branches/branch-0.20/CHANGES.txt (original) > +++ hadoop/common/branches/branch-0.20/CHANGES.txt Wed Jan 5 23:12:49 2011 > @@ -8,6 +8,9 @@ Release 0.20.4 - Unreleased > > IMPROVEMENTS > > +MAPREDUCE-1734. Un-deprecate the old MapReduce API in the 0.20 branch. > +(todd) > + > Release 0.20.3 - 2011-1-5 > > NEW FEATURES > @@ -89,9 +92,6 @@ Release 0.20.3 - 2011-1-5 > > MAPREDUCE-1832. Allow file sizes less than 1MB in DFSIO benchmark. (shv) > > -MAPREDUCE-1734. Un-deprecate the old MapReduce API in the 0.20 branch. > -(todd) > - > Release 0.20.2 - 2010-2-19 > > NEW FEATURES > >
Re: Patch testing
Hi Nigel, MAPREDUCE-2172 has been fixed for a while. Are there any other particular JIRAs you think need to be fixed before the MR test-patch queue gets enabled? I have a lot of outstanding patches and doing all the test-patch turnaround manually on 3 different boxes is a real headache. Thanks -Todd On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley wrote: > Ok, HDFS is now enabled. You'll see a stream of updates shortly on the ~30 > Patch Available HDFS issues. > > Nige > > On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote: > > > I committed HDFS-1511 this morning. We should be good to go. I can > > haz snooty robot butler? > > > > On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik > wrote: > >> Thanks Jacob. I am wasted already but I can do it on Sun, I think, > >> unless it is done earlier. > >> -- > >> Take care, > >> Konstantin (Cos) Boudnik > >> > >> > >> > >> On Fri, Dec 17, 2010 at 19:41, Jakob Homan wrote: > >>> Ok. I'll get a patch out for 1511 tomorrow, unless someone wants to > >>> whip one up tonight. > >>> > >>> > >>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley wrote: > I agree with Cos on fixing HDFS-1511 first. Once that is done I'll > enable hdfs patch testing. > > Cheers, > Nige > > Sent from my iPhone4 > > On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik > wrote: > > > One more issue needs to be addressed before test-patch is turned on > HDFS is > > https://issues.apache.org/jira/browse/HDFS-1511 > > -- > > Take care, > > Konstantin (Cos) Boudnik > > > > > > > > On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik > wrote: > >> Considering that because of these 4 faulty cases every patch will be > >> -1'ed a patch author will still have to look at it and make a > comment > >> why this particular -1 isn't valid. Lesser work, perhaps, but > messier > >> IMO. I'm not blocking it - I just feel like there's a better way. > >> > >> -- > >> Take care, > >> Konstantin (Cos) Boudnik > >> > >> > >> > >> On Fri, Dec 17, 2010 at 15:55, Jakob Homan > wrote: > If HDFS is added to the test-patch queue right now we get > nothing but dozens of -1'ed patches. > >>> There aren't dozens of patches being submitted currently. The -1 > >>> isn't the important thing, it's the grunt work of actually running > >>> (and waiting) for the tests, test-patch, etc. that Hudson does so > that > >>> the developer doesn't have to. > >>> > >>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur < > dhr...@gmail.com> wrote: > +1, thanks for doing this. > > On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan > wrote: > > > So, with test-patch updated to show the failing tests, saving the > > developers the need to go and verify that the failed tests are > all > > known, how do people feel about turning on test-patch again for > HDFS > > and mapred? I think it'll help prevent any more tests from > entering > > the "yeah, we know" category. > > > > Thanks, > > jg > > > > > > On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan < > jho...@yahoo-inc.com> wrote: > >> True, each patch would get a -1 and the failing tests would need > to be > >> verified as those known bad (BTW, it would be great if Hudson > could list > >> which tests failed in the message it posts to JIRA). But that's > still > > quite > >> a bit less error-prone work than if the developer runs the tests > and > >> test-patch themselves. Also, with 22 being cut, there are a lot > of > > patches > >> up in the air and several developers are juggling multiple > patches. The > >> more automation we can have, even if it's not perfect, will > decrease > > errors > >> we may make. > >> -jg > >> > >> Nigel Daley wrote: > >>> > >>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote: > >>> > > It's also ready to run on MapReduce and HDFS but we won't > turn it on > > until these projects build and test cleanly. Looks like both > these > > projects > > currently have test failures. > > Assuming the projects are compiling and building, is there a > reason to > not turn it on despite the test failures? Hudson is invaluable > to > > developers > who then don't have to run the tests and test-patch > themselves. We > > didn't > turn Hudson off when it was working previously and there were > known > failures. I think one of the reasons we have more failing > tests now is > > the > higher cost of doing Hudson's work (not a great excuse I > know). This > > is > particularly true n
RE: decommission a rack
Change your topology script? One thing about rack awareness. You never want to have an unbalanced configuration. The caveat is if your uplinks between the ToR Switch are 'fat' enough. (4 10GBe should do it.) If you watched your system with the one machine on a separate rack (assuming it was also on a separate switch) you'd see a major network bottle neck. (We use Ganglia) If you're seeing the rack count wrong, you could just bounce the cloud and it should show only two. Note: Deprecating a machine can take a long time. You may want to consider just killing the node, waiting till the cluster recognizes that its down and I believe just kicking off an FSCK or maybe running the balancer program will clean that up faster than deprecating. (YMMV) HTH -Mike -Original Message- From: Mag Gam [mailto:magaw...@gmail.com] Sent: Wednesday, January 05, 2011 5:56 AM To: general@hadoop.apache.org Subject: decommission a rack Using 0.21 I have a server which was in rack3. It was the only server in this rack so I decommissioned the server. fsck shows only 2 racks however topology shows 3 racks. How can I fix this? The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
decommission a rack
Using 0.21 I have a server which was in rack3. It was the only server in this rack so I decommissioned the server. fsck shows only 2 racks however topology shows 3 racks. How can I fix this?
