Re: dfs.write.packet.size set to 2G

2011-11-08 Thread Ted Dunning
By snapshots, I mean that you can freeze a copy of a portion of the the file system for later use as a backup or reference. By mirror, I mean that a snapshot can be transported to another location in the same cluster or to another cluster and the mirrored image will be updated atomically to the ne

Re: dfs.write.packet.size set to 2G

2011-11-08 Thread Uma Maheswara Rao G 72686
- Original Message - From: donal0412 Date: Tuesday, November 8, 2011 1:04 pm Subject: dfs.write.packet.size set to 2G To: hdfs-user@hadoop.apache.org > Hi, > I want to store lots of files in HDFS, the file size is <= 2G. > I don't want the file to split into blocks,b

Re: dfs.write.packet.size set to 2G

2011-11-08 Thread donal0412
Thanks! That's exactly what I want. And Ted, what do you mean by "snapshots and mirrors" ? On 2011/11/8 16:21, Harsh J wrote: Block sizes are per-file, not permanently set on the HDFS. So create your files with a sufficiently large block size (2G is OK if it fits your usecase well). This way you

Re: dfs.write.packet.size set to 2G

2011-11-08 Thread Harsh J
Block sizes are per-file, not permanently set on the HDFS. So create your files with a sufficiently large block size (2G is OK if it fits your usecase well). This way you won't have block splits, as you desire. For example, to upload a file via the shell with a tweaked blocksize, I'd do: hadoop d

dfs.write.packet.size set to 2G

2011-11-07 Thread donal0412
Hi, I want to store lots of files in HDFS, the file size is <= 2G. I don't want the file to split into blocks,because I need the whole file while processing it, and I don't want to transfer blocks to one node when processing it. A easy way to do this would be set dfs.write.packet.size to 2G, I w