Re: hadoop incremental build

[Haopeng Liu]

I have to use old version to look inside some bugs.

I'm wondering whether maven support incremental build? Can I configure 
maven command to achieve it?


Regards

On 10/15/15 5:24 PM, Ted Yu wrote:

hadoop-0.23.1 is an old release.

I am not even sure whether there would be new release for that branch.

Can you upgrade to more recent hadoop release ?

Cheers

On Thu, Oct 15, 2015 at 3:03 PM, Haopeng Liu > wrote:


Thank you. The patch of Hadoop-12194 seems will be used in
Hadoop-2.8.0 version.

I work on hadoop-0.23.1. How can I build old version hadoop
incrementally?

Regards

On 10/15/15 4:44 PM, Ted Yu wrote:

After a brief search, I found HADOOP-12194

Can you clarify what your goal is ?

Cheers

On Thu, Oct 15, 2015 at 2:42 PM, Haopeng Liu mailto:hyli...@gmail.com>> wrote:

Hi All,

I'm building hadoop system from the source code via maven.
Does it support incremental build? Thank you!

Best,
Haopeng

Re: hadoop 2.6 jar command doesn't load text files in classpath

[Bourgon, Armel]

Hello Chris,

Thanks for your interest in this issue. I set up a local installation of 2.5.2 
and I am actually running into the same issue.
I will work on a jira ticket tomorrow.

Best
—
Armel Bourgon

> On Oct 15, 2015, at 3:41 PM, Chris Nauroth  wrote:
> 
> Hello Armel,
> 
> That's an interesting find.  Thank you for reporting it.
> 
> I know you mentioned an upgrade from 0.20 to 2.6.  Do you also have the
> ability to test against an earlier 2.x release, like 2.5.2?  If the
> problem repros in 2.6.0, but not in 2.5.2, then I wonder if it's a
> regression introduced by the client-side classloader isolation that we
> shipped in 2.6.0.
> 
> https://issues.apache.org/jira/browse/HADOOP-10893
> 
> 
> If you're interested in looking at the code, the most relevant piece is
> the RunJar class, which is the main entry point of the "hadoop jar"
> command.
> 
> If you have a simplified consistent repro that demonstrates the problem,
> then I suggest filing a JIRA for further investigation.  Thanks again.
> 
> --Chris Nauroth
> 
> 
> 
> 
> On 10/15/15, 2:40 PM, "Bourgon, Armel" 
> wrote:
> 
>> Hello,
>> 
>> To give you a bit of context, I wrote a java library that aims to provide
>> an easy way to coordinate multiple MR jobs and execute them with a single
>> jar submission. The final result is a "fat jar² (build using the maven
>> assembly plugin) that contains the different Mapper and Reducer classes
>> and a Main class that has the logic to submit the different jobs to the
>> cluster.
>> 
>> To accomplish this, the Main relies on some text files (packaged in the
>> jar) to be present. Those files are not needed by the MR jobs themselves,
>> it¹s some kind of configuration for the Main to know how it should
>> schedule the different MR jobs.
>> 
>> The jar is executed like that:
>> hadoop jar the_jar_file.jar 
>> 
>> It has been used in production for a long time now but recently we
>> decided to upgrade to hadoop 2.6 (we were using 0.20). All our jobs
>> packaged like that are failing because the Main cannot locate the text
>> files in the classpath.
>> 
>> I did a bit of debugging by replacing the Main with a piece of code that
>> print the content of the classpath. When running the jar with:
>> java -jar the_jar_file.jar 
>> 
>> I can see the text files in the list. But when I run the same jar with:
>> hadoop jar the_jar_file.jar 
>> 
>> The text files are missing. I assume that something changed in the way
>> the hadoop jar command read the jar and build the classpath. I found
>> someone complaining about the same issue on stakoverflow
>> (http://stackoverflow.com/questions/31670390/accessing-jar-resource-when-r
>> un-in-hadoop) but nobody replied.
>> 
>> I would like to be able to keep the same mechanism (keep those conf files
>> in the jar and access them at runtime from the classpath), maybe their is
>> an options to alter the way the jar command behave? Can someone point me
>> to the source code of the jar command?
>> 
>> Thanks!
>> 
> 

Re: hadoop 2.6 jar command doesn't load text files in classpath

[Chris Nauroth]

Hello Armel,

That's an interesting find.  Thank you for reporting it.

I know you mentioned an upgrade from 0.20 to 2.6.  Do you also have the
ability to test against an earlier 2.x release, like 2.5.2?  If the
problem repros in 2.6.0, but not in 2.5.2, then I wonder if it's a
regression introduced by the client-side classloader isolation that we
shipped in 2.6.0.

https://issues.apache.org/jira/browse/HADOOP-10893


If you're interested in looking at the code, the most relevant piece is
the RunJar class, which is the main entry point of the "hadoop jar"
command.

If you have a simplified consistent repro that demonstrates the problem,
then I suggest filing a JIRA for further investigation.  Thanks again.

--Chris Nauroth




On 10/15/15, 2:40 PM, "Bourgon, Armel" 
wrote:

>Hello,
>
>To give you a bit of context, I wrote a java library that aims to provide
>an easy way to coordinate multiple MR jobs and execute them with a single
>jar submission. The final result is a "fat jar² (build using the maven
>assembly plugin) that contains the different Mapper and Reducer classes
>and a Main class that has the logic to submit the different jobs to the
>cluster.
>
>To accomplish this, the Main relies on some text files (packaged in the
>jar) to be present. Those files are not needed by the MR jobs themselves,
>it¹s some kind of configuration for the Main to know how it should
>schedule the different MR jobs.
>
>The jar is executed like that:
>hadoop jar the_jar_file.jar 
>
>It has been used in production for a long time now but recently we
>decided to upgrade to hadoop 2.6 (we were using 0.20). All our jobs
>packaged like that are failing because the Main cannot locate the text
>files in the classpath.
>
>I did a bit of debugging by replacing the Main with a piece of code that
>print the content of the classpath. When running the jar with:
>java -jar the_jar_file.jar 
>
>I can see the text files in the list. But when I run the same jar with:
>hadoop jar the_jar_file.jar 
>
>The text files are missing. I assume that something changed in the way
>the hadoop jar command read the jar and build the classpath. I found
>someone complaining about the same issue on stakoverflow
>(http://stackoverflow.com/questions/31670390/accessing-jar-resource-when-r
>un-in-hadoop) but nobody replied.
>
>I would like to be able to keep the same mechanism (keep those conf files
>in the jar and access them at runtime from the classpath), maybe their is
>an options to alter the way the jar command behave? Can someone point me
>to the source code of the jar command?
>
>Thanks!
>

Re: hadoop incremental build

[Ted Yu]

hadoop-0.23.1 is an old release.

I am not even sure whether there would be new release for that branch.

Can you upgrade to more recent hadoop release ?

Cheers

On Thu, Oct 15, 2015 at 3:03 PM, Haopeng Liu  wrote:

> Thank you. The patch of Hadoop-12194 seems will be used in Hadoop-2.8.0
> version.
>
> I work on hadoop-0.23.1. How can I build old version hadoop incrementally?
>
> Regards
>
> On 10/15/15 4:44 PM, Ted Yu wrote:
>
> After a brief search, I found HADOOP-12194
>
> Can you clarify what your goal is ?
>
> Cheers
>
> On Thu, Oct 15, 2015 at 2:42 PM, Haopeng Liu  wrote:
>
>> Hi All,
>>
>> I'm building hadoop system from the source code via maven. Does it
>> support incremental build? Thank you!
>>
>> Best,
>> Haopeng
>>
>
>
>

Re: hadoop incremental build

[Haopeng Liu]

Thank you. The patch of Hadoop-12194 seems will be used in Hadoop-2.8.0 
version.


I work on hadoop-0.23.1. How can I build old version hadoop incrementally?

Regards

On 10/15/15 4:44 PM, Ted Yu wrote:

After a brief search, I found HADOOP-12194

Can you clarify what your goal is ?

Cheers

On Thu, Oct 15, 2015 at 2:42 PM, Haopeng Liu > wrote:


Hi All,

I'm building hadoop system from the source code via maven. Does it
support incremental build? Thank you!

Best,
Haopeng

Re: hadoop incremental build

[Ted Yu]

After a brief search, I found HADOOP-12194

Can you clarify what your goal is ?

Cheers

On Thu, Oct 15, 2015 at 2:42 PM, Haopeng Liu  wrote:

> Hi All,
>
> I'm building hadoop system from the source code via maven. Does it support
> incremental build? Thank you!
>
> Best,
> Haopeng
>

hadoop incremental build

[Haopeng Liu]

Hi All,

I'm building hadoop system from the source code via maven. Does it 
support incremental build? Thank you!


Best,
Haopeng

hadoop 2.6 jar command doesn't load text files in classpath

[Bourgon, Armel]

Hello,

To give you a bit of context, I wrote a java library that aims to provide an 
easy way to coordinate multiple MR jobs and execute them with a single jar 
submission. The final result is a "fat jar” (build using the maven assembly 
plugin) that contains the different Mapper and Reducer classes and a Main class 
that has the logic to submit the different jobs to the cluster.

To accomplish this, the Main relies on some text files (packaged in the jar) to 
be present. Those files are not needed by the MR jobs themselves, it’s some 
kind of configuration for the Main to know how it should schedule the different 
MR jobs. 

The jar is executed like that:
hadoop jar the_jar_file.jar 

It has been used in production for a long time now but recently we decided to 
upgrade to hadoop 2.6 (we were using 0.20). All our jobs packaged like that are 
failing because the Main cannot locate the text files in the classpath.

I did a bit of debugging by replacing the Main with a piece of code that print 
the content of the classpath. When running the jar with:
java -jar the_jar_file.jar 

I can see the text files in the list. But when I run the same jar with:
hadoop jar the_jar_file.jar 

The text files are missing. I assume that something changed in the way the 
hadoop jar command read the jar and build the classpath. I found someone 
complaining about the same issue on stakoverflow 
(http://stackoverflow.com/questions/31670390/accessing-jar-resource-when-run-in-hadoop)
 but nobody replied.

I would like to be able to keep the same mechanism (keep those conf files in 
the jar and access them at runtime from the classpath), maybe their is an 
options to alter the way the jar command behave? Can someone point me to the 
source code of the jar command?

Thanks!