libhdfs append question

2009-01-13 Thread Tamás Szokol 
Hi!


I'm using the latest stable 0.19.0 version of hadoop. I'd like to try the new 
append functionality. Is it available from libhdfs? I didn't find it in hdfs.h 
interface nor in the hdfs.c implementation.

I saw the hdfsOpenFile's new flag O_APPEND in: HADOOP-4494
(http://mail-archives.apache.org/mod_mbox/hadoop-core-dev/200810.mbox/%3c270645717.1224722204314.javamail.j...@brutus%3e),
but I still don't find it in the latest release.

Is it available as a patch, or maybe only available in the svn repository?

Could you please give me some pointers how to use the append functionality from 
libhdfs?
Thank you in advance!

Cheers,
Tamas

getting null from CompressionCodecFactory.getCodec(Path file)

2009-01-13 Thread Gert Pfeifer 
Hi,
I want to use an lzo file as input for a mapper. The record reader
determines the codec using a CompressionCodecFactory, like this:

(Hadoop version 0.19.0)

compressionCodecs = new CompressionCodecFactory(job);
System.out.println(Using codecFactory: +compressionCodecs.toString());
final CompressionCodec codec = compressionCodecs.getCodec(file);
System.out.println(Using codec: +codec+ for file +file.getName());


The output that I get is:

Using codecFactory: { etalfed_ozl.: org.apache.hadoop.io.compress.LzoCodec }
Using codec: null for file test.lzo

Of course, the mapper does not work without codec. What could be the
problem?

Thanks,
Gert

Re: getting null from CompressionCodecFactory.getCodec(Path file)

2009-01-13 Thread Arun C Murthy 


On Jan 13, 2009, at 7:29 AM, Gert Pfeifer wrote:


Hi,
I want to use an lzo file as input for a mapper. The record reader
determines the codec using a CompressionCodecFactory, like this:

(Hadoop version 0.19.0)



http://hadoop.apache.org/core/docs/r0.19.0/native_libraries.html

hth,
Arun


compressionCodecs = new CompressionCodecFactory(job);
System.out.println(Using codecFactory:  
+compressionCodecs.toString());

final CompressionCodec codec = compressionCodecs.getCodec(file);
System.out.println(Using codec: +codec+ for file +file.getName());


The output that I get is:

Using codecFactory: { etalfed_ozl.:  
org.apache.hadoop.io.compress.LzoCodec }

Using codec: null for file test.lzo

Of course, the mapper does not work without codec. What could be the
problem?

Thanks,
Gert

Hadoop 0.17.1 = EOFException reading FSEdits file, what causes this? how to prevent?

2009-01-13 Thread Joe Montanez 
Hi:

 

I'm using Hadoop 0.17.1 and I'm encountering EOFException reading the
FSEdits file.  I don't have a clear understanding what is causing this
and how to prevent this.  Has anyone seen this and can advise?

 

Thanks in advance,

Joe

 

2009-01-12 22:51:45,573 ERROR org.apache.hadoop.dfs.NameNode:
java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:180)

at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)

at
org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)

at
org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:599)

at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:766)

at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:640)

at
org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:223)

at
org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)

at
org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:274)

at
org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:255)

at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:133)

at org.apache.hadoop.dfs.NameNode.init(NameNode.java:178)

at org.apache.hadoop.dfs.NameNode.init(NameNode.java:164)

at
org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:848)

at org.apache.hadoop.dfs.NameNode.main(NameNode.java:857)

 

2009-01-12 22:51:45,574 INFO org.apache.hadoop.dfs.NameNode:
SHUTDOWN_MSG:

HDFS read/write question

2009-01-13 Thread Manish Katyal 
I'm trying out the new append feature  (
https://issues.apache.org/jira/browse/HADOOP-1700).
[Hadoop 0.19, distributed mode with a single data node]

The following scenario as per the JIRA documentation (Appends.doc) *i
assume* it should have worked but does not:

...//initialize FileSystem fs
//*(1) create a new file*
FSDataOutputStream os = fs.create(name, true, fs.getConf().getInt(
io.file.buffer.size, 4096), fs.getDefaultReplication(), fs
.getDefaultBlockSize(), null);
os.writeUTF(hello);
os.flush();
os.close(); *//closed*

//*(2) open* the file for append
os = fs.append(name);
os.writeUTF(world);
os.flush(); //file is *not* closed

//*(3) read existing data from the file*
DataInputStream dis = fs.open(name);
String data = dis.readUTF();
dis.close();
System.out.println(Read:  + data); //*expected hello*

//finally close
os.close();

I get an exception: hdfs.DFSClient: Could not obtain block
blk_3192362259459791054_10766 from any node... (on *Step 3:*).
What am I missing?

Thanks.
- Manish

Re: HDFS read/write question

2009-01-13 Thread Jonathan Cao 
I encountered the same issue before (not only the append operation failed,
the appended file became corrupted after the append operation), my test
indicated this issue only showed up when the file size is small (as in your
case. i.e. less than a block). The append seems to work fine with large
files (~100M).
Jonathan

On Tue, Jan 13, 2009 at 9:59 AM, Manish Katyal manish.kat...@gmail.comwrote:

 I'm trying out the new append feature  (
 https://issues.apache.org/jira/browse/HADOOP-1700).
 [Hadoop 0.19, distributed mode with a single data node]

 The following scenario as per the JIRA documentation (Appends.doc) *i
 assume* it should have worked but does not:

...//initialize FileSystem fs
//*(1) create a new file*
FSDataOutputStream os = fs.create(name, true, fs.getConf().getInt(
io.file.buffer.size, 4096), fs.getDefaultReplication(), fs
.getDefaultBlockSize(), null);
os.writeUTF(hello);
os.flush();
os.close(); *//closed*

//*(2) open* the file for append
os = fs.append(name);
os.writeUTF(world);
os.flush(); //file is *not* closed

//*(3) read existing data from the file*
DataInputStream dis = fs.open(name);
String data = dis.readUTF();
dis.close();
System.out.println(Read:  + data); //*expected hello*

//finally close
os.close();

 I get an exception: hdfs.DFSClient: Could not obtain block
 blk_3192362259459791054_10766 from any node... (on *Step 3:*).
 What am I missing?

 Thanks.
 - Manish

@hadoop on twitter

2009-01-13 Thread Philip (flip) Kromer 
Hey all,
There is no @hadoop on twitter, but there should be.
http://twitter.com/datamapper and http://twitter.com/rails both set good
examples.

I'd be glad to either help get that going or to nod approvingly if someone
on core does so.

flip