Re: Update on hadoop-0.23

2011-10-19 Thread Steve Loughran
On 19/10/11 00:40, Todd Lipcon wrote: On Tue, Oct 18, 2011 at 4:36 AM, Steve Loughranste...@apache.org wrote: One more thing: are the ProtocolBuffers needed for all installations, or is that a compile-time requirement? If the binaries are going to be required, there's going to have to be one

Re: Update on hadoop-0.23

2011-10-18 Thread Todd Lipcon
On Tue, Oct 18, 2011 at 4:36 AM, Steve Loughran ste...@apache.org wrote: One more thing: are the ProtocolBuffers needed for all installations, or is that a compile-time requirement? If the binaries are going to be required, there's going to have to be one built for the various platforms, and

Re: Update on hadoop-0.23

2011-10-18 Thread Harsh J
HBase trunk compiles with 0.23 now, after Todd's work on it -- I've updated https://issues.apache.org/jira/browse/HBASE-4510 with further details. On Tue, Oct 18, 2011 at 1:57 AM, Ted Yu yuzhih...@gmail.com wrote: On behalf of Harsh w.r.t. HBASE-4510 HDFS-1620 related changes downstream (For

Update on hadoop-0.23

2011-10-17 Thread Arun C Murthy
Folks, Quick note - the dev community continues to scramble to get things wrapped up on hadoop-0.23. We are down to ~30 blockers and I hope to see them resolved over the next two weeks! Also, I feel Alejandro and Tom can finish up the remaining mavenization bits by then too - as I see

Re: Update on hadoop-0.23

2011-10-03 Thread Eric Baldeschwieler
) From: Konstantin Shvachko shv.had...@gmail.com To: general@hadoop.apache.org Sent: Friday, September 30, 2011 2:23 AM Subject: Re: Update on hadoop-0.23 On Thu, Sep 29, 2011 at 10:27 PM, Eric Baldeschwieler eri...@hortonworks.com wrote: Hi Doug, Jeff, Roman I'd like to request that folks

Re: Update on hadoop-0.23

2011-10-01 Thread Konstantin Shvachko
in the East Bay is that my 500 users will. I also hear that if 0.22 was available people would use it now. Thanks, Konstajntin From: Konstantin Shvachko shv.had...@gmail.com To: general@hadoop.apache.org Sent: Friday, September 30, 2011 2:23 AM Subject: Re: Update on hadoop

Re: Update on hadoop-0.23

2011-10-01 Thread Konstantin Shvachko
I am very glad that the development and testing of 0.23 is going so well. I see a lot of commits and hundreds of changes going in literally every day. It is great to see the new technology building! On the criticism of the 0.22 release. Arun has a top-down view and I agree a lot of progress have

Re: Update on hadoop-0.23

2011-09-30 Thread Konstantin Shvachko
On Thu, Sep 29, 2011 at 10:27 PM, Eric Baldeschwieler eri...@hortonworks.com wrote: Hi Doug, Jeff, Roman I'd like to request that folks take bigtop project discussions onto the bigtop lists and don't greet status reports on general@hadoop I am personally very interested in the results of

Re: Update on hadoop-0.23

2011-09-30 Thread Steve Loughran
On 29/09/2011 22:38, Doug Cutting wrote: The ASF discourages the use of Java's @author tag in large part because it tends to mark code as the territory of particular contributors. We want ASF codebases to be the responsibility of an entire community. There's another reason which is if you

Re: Update on hadoop-0.23

2011-09-30 Thread Steve Loughran
On 30/09/2011 06:27, Eric Baldeschwieler wrote: Hi Doug, Jeff, Roman Let me rephrase my point. I'd like to request that folks take bigtop project discussions onto the bigtop lists and don't greet status reports on general@hadoop with insinuations that folks who are working really hard on

Re: Update on hadoop-0.23

2011-09-30 Thread Milind.Bhandarkar
3. There's 0.22 which is going to combine the API of 0.21 with the fixes of 0.20.20x *and* will be the last release of the MR1.0 engine. For that last reason, I think there's value in pushing it out, though it's going to take time, and there's a risk of it adding another branch to be maintained

Re: Update on hadoop-0.23

2011-09-30 Thread Andrew Purtell
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) From: Konstantin Shvachko shv.had...@gmail.com To: general@hadoop.apache.org Sent: Friday, September 30, 2011 2:23 AM Subject: Re: Update on hadoop-0.23 On Thu, Sep 29

Re: Update on hadoop-0.23

2011-09-30 Thread Doug Cutting
On 09/30/2011 03:17 AM, Steve Loughran wrote: 4. There are the third party compatible projects, CDH, MapR, EMC HD, Amazon Elastic MR, which are all declaring compatibility with 0.20.x; no stated plans when/how to move to 0.23+ CDH4 will include 0.23 (hopefully without any patches). I expect

Re: Update on hadoop-0.23

2011-09-30 Thread Arun C Murthy
On Sep 30, 2011, at 3:17 AM, Steve Loughran wrote: 3. There's 0.22 which is going to combine the API of 0.21 with the fixes of 0.20.20x *and* will be the last release of the MR1.0 engine. For that last reason, I think there's value in pushing it out, though it's going to take time, and

Re: Update on hadoop-0.23

2011-09-30 Thread Arun C Murthy
On Sep 30, 2011, at 3:17 AM, Steve Loughran wrote: 3. There's 0.22 which is going to combine the API of 0.21 with the fixes of 0.20.20x *and* will be the last release of the MR1.0 engine. For that last reason, I think there's value in pushing it out, though it's going to take time, and

Re: Update on hadoop-0.23

2011-09-30 Thread Todd Lipcon
On Fri, Sep 30, 2011 at 11:44 AM, Roman Shaposhnik r...@apache.org wrote: I apologize if my level of institutional knowledge of these things is lacking, but do you have any benchmarking results between 0.22 and 0.20.2xx? The reason I'm asking is twofold -- I really would like to see an

Re: Update on hadoop-0.23

2011-09-30 Thread Arun C Murthy
On Sep 30, 2011, at 1:13 PM, Todd Lipcon wrote: On Fri, Sep 30, 2011 at 11:44 AM, Roman Shaposhnik r...@apache.org wrote: I apologize if my level of institutional knowledge of these things is lacking, but do you have any benchmarking results between 0.22 and 0.20.2xx? The reason I'm asking

Re: Update on hadoop-0.23

2011-09-30 Thread Konstantin Boudnik
BTW, Roman I have a recollection of Hadoop performance suite for iTest (aka BigTop now) which I have put together during the initial development phase of the framework. I don't see in BigTop's source tree - has this work ever been committed to open source along with the iTest? This way if any

Re: HBase trunk on 0.23 Was: Update on hadoop-0.23

2011-09-29 Thread Uma Maheswara Rao G 72686
: Thursday, September 29, 2011 7:27 am Subject: HBase trunk on 0.23 Was: Update on hadoop-0.23 To: general@hadoop.apache.org Cc: bigtop-...@incubator.apache.org Minor correction to Todd's report, currently HBase TRUNK doesn't compileagainst 0.23 ( https://builds.apache.org/view/G-L/view/HBase/job

Re: Update on hadoop-0.23

2011-09-29 Thread Eric Baldeschwieler
Hi Jeff, This sees like a great opportunity for you to add some value. I'd welcome that. It seems rude to me to beat up the folks who have been driving the majority of the work on a project to do more. In general I don't think its good open source etiquete to ask others to contribute their

Re: Update on hadoop-0.23

2011-09-29 Thread Doug Cutting
On 09/29/2011 12:35 PM, Eric Baldeschwieler wrote: If you're wondering why I am asserting that arun and company have done the majority of the work on 23, check out the last graph on this post, or look at the commit logs. The ASF discourages the use of Java's @author tag in large part because

Re: Update on hadoop-0.23

2011-09-29 Thread Eric Baldeschwieler
Hi Doug, Jeff, Roman Let me rephrase my point. I'd like to request that folks take bigtop project discussions onto the bigtop lists and don't greet status reports on general@hadoop with insinuations that folks who are working really hard on this project should be contributing different things

Re: Update on hadoop-0.23

2011-09-28 Thread Arun C Murthy
Roman, On Sep 27, 2011, at 12:24 PM, Roman Shaposhnik wrote: I'm sorry if I had too many questions in my original email, but I can't find an answer to my integration tests question. Could you, please, share a URL with us where I can find out more about them? As you know, me my team at HW

HBase trunk on 0.23 Was: Update on hadoop-0.23

2011-09-28 Thread Ted Yu
Minor correction to Todd's report, currently HBase TRUNK doesn't compile against 0.23 ( https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/42/console ): [ERROR]

Re: HBase trunk on 0.23 Was: Update on hadoop-0.23

2011-09-28 Thread Roman Shaposhnik
On Wed, Sep 28, 2011 at 6:56 PM, Ted Yu yuzhih...@gmail.com wrote: Minor correction to Todd's report, currently HBase TRUNK doesn't compile against 0.23 ( https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/42/console ): [ERROR]

Re: HBase trunk on 0.23 Was: Update on hadoop-0.23

2011-09-28 Thread Konstantin Boudnik
I don't see hadoop-hdfs artifact being downloaded during the build. I see hadoop-hdfs-test, but not hdfs itself. This is likely to be a problem because location of the interface in question is the same as it was back in 0.22 Cos On Wed, Sep 28, 2011 at 07:16PM, Roman Shaposhnik wrote: On Wed,

Re: Update on hadoop-0.23

2011-09-27 Thread Arun C Murthy
Roman, In general, we'll need to make changes upstream: # I believe someone got HBase working. # We made changes to Pig - rather we got help from the Pig team, particularly Daniel. So, we plan to work through the rest of the stack - Hive, Oozie etc. very soon and we'll depend on updated

Re: Update on hadoop-0.23

2011-09-27 Thread Arun C Murthy
On Sep 26, 2011, at 11:20 PM, Arun C Murthy wrote: Roman, In general, we'll need to make changes upstream: # I believe someone got HBase working. # We made changes to Pig - rather we got help from the Pig team, particularly Daniel. So, we plan to work through the rest of the stack

Re: Update on hadoop-0.23

2011-09-27 Thread Doug Cutting
On 09/26/2011 02:07 PM, Arun C Murthy wrote: We are about to finish performance certification for both HDFS MapReduce in the next couple of weeks too, after which we start integration tests with HBase, Hive, Oozie etc. Who's 'we' here? I haven't seen this happening on the list, so I assume

Re: Update on hadoop-0.23

2011-09-27 Thread Roman Shaposhnik
Hi Arun! Thanks for the quick reply! I'm sorry if I had too many questions in my original email, but I can't find an answer to my integration tests question. Could you, please, share a URL with us where I can find out more about them? On Mon, Sep 26, 2011 at 11:20 PM, Arun C Murthy

Update on hadoop-0.23

2011-09-26 Thread Arun C Murthy
Greetings, I thought I'd drop a note to update folks on progress of hadoop-0.23. Things are have been very busy in hadoop-0.23 land. We continue to crank through the issues and get ready to ship. We are mostly pass the initial teething pains of moving our entire build infrastructure to Maven

Re: Update on hadoop-0.23

2011-09-26 Thread Roman Shaposhnik
Hi Arun! Great news! Hopefuly you wouldn't mind answering some of the questions below... On Mon, Sep 26, 2011 at 2:07 PM, Arun C Murthy a...@hortonworks.com wrote: NextGen MapReduce (aka MRv2, aka YARN) is coming along great: # We are happy to report we've done extensive scale testing to