In hadoop 0.19.1, (and 19.0) libhdfs (which is used by the fuse package for
hdfs access) explicitly denies open requests that pass O_RDWR
If you have binary applications that pass the flag, but would work correctly
given the limitations of HDFS, you may alter the code in
src/c++/libhdfs/hdfs.c to allow it, or build a shared library that you
preload that changes the flags passed to the real open. Hacking hdfs.c is
much simpler.
Line 407 of hdfs.c
jobject jFS = (jobject)fs;
if (flags & O_RDWR) {
fprintf(stderr, "ERROR: cannot open an hdfs file in O_RDWR mode\n");
errno = ENOTSUP;
return NULL;
}
On Fri, May 1, 2009 at 6:34 PM, Philip Zeyliger <phi...@cloudera.com> wrote:
> HDFS does not allow you to overwrite bytes of a file that have already been
> written. The only operations it supports are read (an existing file),
> write
> (a new file), and (in newer versions, not always enabled) append (to an
> existing file).
>
> -- Philip
>
> On Fri, May 1, 2009 at 5:56 PM, Robert Engel <enge...@ligo.caltech.edu
> >wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hello,
> >
> > I am using Hadoop on a small storage cluster (x86_64, CentOS 5.3,
> > Hadoop-0.19.1). The hdfs is mounted using fuse and everything seemed
> > to work just fine so far. However, I noticed that I cannot:
> >
> > 1) use svn to check out files on the mounted hdfs partition
> > 2) request that stdout and stderr of Globus jobs is written to the
> > hdfs partition
> >
> > In both cases I see following error message in /var/log/messages:
> >
> > fuse_dfs: ERROR: could not connect open file fuse_dfs.c:1364
> >
> > When I run fuse_dfs in debugging mode I get:
> >
> > ERROR: cannot open an hdfs file in O_RDWR mode
> > unique: 169, error: -5 (Input/output error), outsize: 16
> >
> > My question is if this is a general limitation of Hadoop or if this
> > operation is just not currently supported? I searched Google and JIRA
> > but could not find an answer.
> >
> > Thanks,
> > Robert
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.9 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> >
> > iEYEARECAAYFAkn7mksACgkQrxCAtr5BXdMx5wCeICTHQbOwjZoGpVTO6ayd7l7t
> > LXoAn0WBwfo6ZYdJX1sh2eO2owAR0HLm
> > =PUCc
> > -----END PGP SIGNATURE-----
> >
> >
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
