On Wed, Jul 21, 2010 at 12:42 PM, Xavier Stevens <[email protected]> wrote:
>  Hi Urckle,
>
> A lot of the more "advanced" setups just record data directly to HDFS to
> start with.  You have to write some custom code using the HDFS API but
> that way you don't need to import large masses of data.  People also use
> "distcp" to do large scale imports, but if you're hitting something like
> an NFS server it will probably fall over if you have a descent sized
> cluster hitting it.
>
> In cases where writing data directly to HDFS can't be done for some
> reason.  You can install Hadoop as a "client" (no processes actually
> running) on the machines that will be putting the data.  Then you can
> invoke hadoop fs -put in parallel from all of those machines.  It should
> go significantly faster than using a single mount point.
>
> Hope this helps,
>
> -Xavier
>
> On 7/21/10 9:30 AM, Urckle wrote:
>> Hi, I have a newbie question.
>>
>> Scenario:
>> Hadoop version: 0.20.2
>> MR coding will be done in java.
>>
>>
>> Just starting out with my first Hadoop setup. I would like to know are
>> there any best practice ways to load data into the dfs? I have
>> (obviously) manually put data files into hdfs using the shell commands
>> while playing with it at setup but going forward I will want to be
>> retrieving large numbers of data feeds from remote, 3rd party
>> locations and throwing them into hadoop for analysis later. What is
>> the best way to automate this? Is it to gather the retrieved files
>> into known locations to be mounted and then automate via script etc.
>> to put the files into hdfs? Or is there some other practice? I've not
>> been able to find specific use case yet... all docs cover the basic fs
>> command without giving much details about more advanced setups.
>>
>> thanks for any info
>>
>> regards
>

Hadoop Mailing list presents...
An ASF film...
from the producer of 'dfs -copyFromLocal'...

How to move logs

Starting (in order of appearance)
Chukwa
Chukwa is an open source data collection system for monitoring large
distributed systems. Chukwa is built on top of the Hadoop Distributed
File System (HDFS) and Map/Reduce framework and inherits Hadoop’s
scalability and robustness. Chukwa also includes a flexible and
powerful toolkit for displaying, monitoring and analyzing results to
make the best use of the collected data.

Scribe

Scribe is a server for aggregating log data that's streamed in real
time from clients. It is designed to be scalable and reliable.

Flume

Flume is a distributed service that makes it very easy to collect and
aggregate your data into a persistent store such as HDFS. Flume can
read data from almost any source – log files, Syslog packets, the
standard output of any Unix process – and can deliver it to a batch
processing system like Hadoop or a real-time data store like HBase

Co Starting:

FSDataOutputStream

Reply via email to