Hi Colin,
I have a requirement to be able to run all the Hadoop functionality
that I build from a variety of platforms. This was the original
motivation for wanting to use libhdfs. I followed your
suggestion and looked into using webhdfs and it's looking
promising. Thanks for that. However I also need to be able
to launch Map/Reduce jobs from any platform.
In particular from Windows. I looked into this by hacking
the bin/hadoop script to extract the required class path
and various arguments so that I could launch a Map/Reduce job
just by invoking java with the correct arguments.
However I ran into HADOOP-7682.
I can see that there is a workaround here
https://github.com/congainc/patch-hadoop_7682-1.0.x-win
but it suggests that this is not really appropriate for
deployment. I suspect that I can get it to work reliably
by using cygwin and making loads of modifications
but that all seems rather a large effort, error-prone and
difficult to maintain.
Given that I plan to have a relatively small repertoire of Map/Reduce
jobs that I need to launch, I'm tempted to have all the jars pre-packed
on the Name-Node and have the ability to run them there. Then
have a daemon running so that I can use any appropriate ad-hoc RPC
mechanism from Windows to launch them.
Am I missing something? Is there a way to launch Map/Reduce
jobs in a platform neutral way, which runs out of the box, on Windows?
Again, any suggestions welcome.
Peter Marron
-Original Message-
From: Peter Marron
[mailto:Peter.Marron@trilliumsoftware.
com]
Sent: 26 October 2012 00:53
To: common-dev@hadoop.apache.org
Subject: RE: libhdfs on windows
Hi Colin,
OK, I didn't know there was a hdfs-dev.
I'm happy to ask there.
(However there's a lot of mail on
dev@hadoop and user@hadoop as well
as user@hive and it's a bit of a
commitment to track them all.) As for
webhdfs, I did think about that, and in
some ways it's a beautiful solution as it
gives me a platform- and language-
neutral access mechanism. I was just a
little worried about the HTTP overhead if
I am reading a single record at a time.
Also I will need some way to launch my
Map/Reduce jobs as well. So I'll probably
end up using the C++/JNI/Java route to
do that anyway. Unless there's a better
way?
Is there a web Map/Reduce interface?
Many thanks,
Z
-Original Message-
From: rarecac...@gmail.com
[mailto:rarecac...@gmail.com] On
Behalf Of Colin McCabe
Sent: 25 October 2012 18:24
To: common-dev@hadoop.apache.org
Subject: Re: libhdfs on windows
Hi Peter,
This might be a good question for hdfs-
dev?
As Harsh pointed out below, HDFS-573
was never committed. I don't even see
a patch attached, although
there is some discussion.
In the mean time, might I suggest using
the webhdfs interface on
Windows?
webhdfs was intended as a stable REST
interface that can be accessed from
any platform.
cheers,
Colin
On Thu, Oct 25, 2012 at 7:19 AM, Peter
Marron
peter.mar...@trilliumsoftware.com
wrote:
Hi,
I've been looking at using libhdfs and
I
would like to use it on windows.
I have found HDFS-573 and the
information on this page:
http://issues.apache.org/jira/browse/HD
FS-573?page=com.atlassian.jira.
plugin.system.issuetabpanels:all-
tabpanel
which suggests that quite a lot of
work
was done on this way back in 2009.
So is there some source from this
effort retained somewhere? If so,
where?
Or do I have to start from scratch?
Apologies if this has already been asked
recently.
Any help appreciated.
Peter Marron
