Re: hadoop version profiles

[Eugene Koontz]

Thanks Hyunsik and Avery for your comments!

I've added a JIRA here: 
https://issues.apache.org/jira/browse/GIRAPH-168  to present a patch 
that adds two new profiles:


hadoop_0.20.203
hadoop_trunk

hadoop_0.20.203 is the default (i.e. same as if you do not give a -P 
argument).


I tried "mvn clean verify && mvn -Phadoop_0.20.203 clean verify && mvn 
-Phadoop_trunk clean verify" and this worked with the GIRAPH-168 patch 
applied.


As I said on GIRAPH-168, I could not get hadoop_non_secure or 
hadoop_facebook to compile.


I agree that munge is hacky - in my patch I tried to modify our use of 
munge a bit to make it simpler: I removed the usages of HADOOP (e.g. 
"/*if_not[HADOOP]"). I felt like this usage is misleading, because it 
might make you think that Giraph can be compiled without Hadoop support 
at all, which is not what is meant by "/*if_not[HADOOP]": in fact, what 
it actually means is that the Hadoop to be used is the default version 
(Hadoop 0.205.0).


-Eugene




On 3/21/12 3:46 AM, Hyunsik Choi wrote:

This approach looks good.
+1

On Wed, Mar 21, 2012 at 3:53 PM, Avery Ching  wrote:


I agree with this approach, although munge is kinda hacky.  It is easy
though. =)

Avery


On 3/20/12 5:52 PM, Eugene Koontz wrote:


Hi Giraphers,
I think it might be good to look at how we can add support for new
hadoop versions. Currently we have hadoop_facebook (
https://issues.apache.org/**jira/browse/GIRAPH-14).
I am considering adding new ones such as hadoop_0.24. Looking at the code,
it seems that the main hadoop variation between the stock hadoop used
(0.203.0) versus facebook has to do with the new security-related APIs in
the latter that is, fortunately, also available in hadoop 0.23 and 0.24.
So, hopefully we can make use of the existing work that Avery has done for
hadoop_facebook and apply it to other hadoop versions. Therefore I would
propose that:

1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java and
a few other places, where we are currently checking for HADOOP_FACEBOOK and
HADOOP.
2. we make a new profile called hadoop_secure, which, as with
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default
hadoop and make it the default profile (activeByDefault=true). This will
makes it easier to handle the differences in the hadoop library dependency
set that have happened between 0.20.203 and hadoop trunk.

Please see 
https://github.com/ekoontz/**giraph/tree/security-profilefor
 my branch that implements the above.

Thanks,

-Eugene

Re: hadoop version profiles

[Hyunsik Choi]

This approach looks good.
+1

On Wed, Mar 21, 2012 at 3:53 PM, Avery Ching  wrote:

> I agree with this approach, although munge is kinda hacky.  It is easy
> though. =)
>
> Avery
>
>
> On 3/20/12 5:52 PM, Eugene Koontz wrote:
>
>> Hi Giraphers,
>>I think it might be good to look at how we can add support for new
>> hadoop versions. Currently we have hadoop_facebook (
>> https://issues.apache.org/**jira/browse/GIRAPH-14).
>> I am considering adding new ones such as hadoop_0.24. Looking at the code,
>> it seems that the main hadoop variation between the stock hadoop used
>> (0.203.0) versus facebook has to do with the new security-related APIs in
>> the latter that is, fortunately, also available in hadoop 0.23 and 0.24.
>> So, hopefully we can make use of the existing work that Avery has done for
>> hadoop_facebook and apply it to other hadoop versions. Therefore I would
>> propose that:
>>
>> 1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java and
>> a few other places, where we are currently checking for HADOOP_FACEBOOK and
>> HADOOP.
>> 2. we make a new profile called hadoop_secure, which, as with
>> hadoop_facebook, will use the above munge flag.
>> 3. we make a new profile "hadoop_0.20.203" for the existing default
>> hadoop and make it the default profile (activeByDefault=true). This will
>> makes it easier to handle the differences in the hadoop library dependency
>> set that have happened between 0.20.203 and hadoop trunk.
>>
>> Please see 
>> https://github.com/ekoontz/**giraph/tree/security-profilefor
>>  my branch that implements the above.
>>
>> Thanks,
>>
>> -Eugene
>>
>
>

Re: hadoop version profiles

[Avery Ching]

I agree with this approach, although munge is kinda hacky.  It is easy 
though. =)


Avery

On 3/20/12 5:52 PM, Eugene Koontz wrote:

Hi Giraphers,
I think it might be good to look at how we can add support for new 
hadoop versions. Currently we have hadoop_facebook 
(https://issues.apache.org/jira/browse/GIRAPH-14). I am considering 
adding new ones such as hadoop_0.24. Looking at the code, it seems 
that the main hadoop variation between the stock hadoop used (0.203.0) 
versus facebook has to do with the new security-related APIs in the 
latter that is, fortunately, also available in hadoop 0.23 and 0.24. 
So, hopefully we can make use of the existing work that Avery has done 
for hadoop_facebook and apply it to other hadoop versions. Therefore I 
would propose that:


1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java 
and a few other places, where we are currently checking for 
HADOOP_FACEBOOK and HADOOP.
2. we make a new profile called hadoop_secure, which, as with 
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default 
hadoop and make it the default profile (activeByDefault=true). This 
will makes it easier to handle the differences in the hadoop library 
dependency set that have happened between 0.20.203 and hadoop trunk.


Please see https://github.com/ekoontz/giraph/tree/security-profile for 
my branch that implements the above.


Thanks,

-Eugene

hadoop version profiles

[Eugene Koontz]

Hi Giraphers,
I think it might be good to look at how we can add support for new 
hadoop versions. Currently we have hadoop_facebook 
(https://issues.apache.org/jira/browse/GIRAPH-14). I am considering 
adding new ones such as hadoop_0.24. Looking at the code, it seems that 
the main hadoop variation between the stock hadoop used (0.203.0) versus 
facebook has to do with the new security-related APIs in the latter that 
is, fortunately, also available in hadoop 0.23 and 0.24. So, hopefully 
we can make use of the existing work that Avery has done for 
hadoop_facebook and apply it to other hadoop versions. Therefore I would 
propose that:


1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java 
and a few other places, where we are currently checking for 
HADOOP_FACEBOOK and HADOOP.
2. we make a new profile called hadoop_secure, which, as with 
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default 
hadoop and make it the default profile (activeByDefault=true). This will 
makes it easier to handle the differences in the hadoop library 
dependency set that have happened between 0.20.203 and hadoop trunk.


Please see https://github.com/ekoontz/giraph/tree/security-profile for 
my branch that implements the above.


Thanks,

-Eugene