I think at present only SequenceFiles can be compressed.
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/io/SequenceFile.html
If you have plain text files, they are stored as is into blocks. You can store
them as .gz and hadoop recognizes it and process the gz files. But its not
splittable, meaning each map will consume whole of .gz
Thanks,
Lohit
- Original Message
From: Michael K. Tung <[EMAIL PROTECTED]>
To: core-user@hadoop.apache.org
Sent: Friday, August 8, 2008 1:09:01 PM
Subject: How to enable compression of blockfiles?
Hello, I have a simple question. How do I configure DFS to store compressed
block files? I've noticed by looking at the "blk_" files that the text
documents I am storing are uncompressed. Currently our hadoop deployment is
taking up 10x the diskspace as compared to our system before moving to
hadoop. I've tried modifying the io.seqfile.compress.blocksize option
without success and haven't been able to find anything online regarding
this. Is there any way to do this or do I need to manually compress my data
before storing to HDFS?
Thanks,
Michael Tung
