Do I need to create the /flume/messages/ directories?
On Mon, Jun 16, 2014 at 10:14 AM, kishore alajangi < [email protected]> wrote: > checked, nothing is written in hdfs. > > > On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <[email protected]> wrote: > >> That just means source has done its work and is waiting for more data to >> read. Did you check hdfs to see if all data has been written? >> >> >> >> On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi < >> [email protected]> wrote: >> >>> Hi Mohit and sharminder, >>> >>> Thanks for reply, after I called with -n tier, source is not directory >>> error came, I changed the source to /tmp/ and hdfs.path to /flume/messages/ >>> in config file, and run the command, the INFO i am getting now is "spooling >>> directory source runner has shutdown" >>> what could be the problem, please help me. >>> >>> >>> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal < >>> [email protected]> wrote: >>> >>>> Replace -n agent with -n tier1 >>>> >>>> >>>> On Sunday, June 15, 2014, kishore alajangi <[email protected]> >>>> wrote: >>>> >>>>> Dear Sharminder, >>>>> >>>>> Thanks for your reply, yes I am playing with flume, as you suggested i >>>>> am using spool directory source, My configuration file looks like >>>>> >>>>> *tier1.sources = source1 >>>>> tier1.channels = channel1 >>>>> tier1.sinks = sink1 >>>>> >>>>> tier1.sources.source1.type = spooldir >>>>> tier1.sources.source1.spoolDir = /var/log/messages >>>>> tier1.sources.source1.channels = channel1 >>>>> tier1.channels.channel1.type = memory >>>>> >>>>> tier1.sinks.sink1.type = hdfs >>>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages >>>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile >>>>> tier1.sinks.sink1.hdfs.filePrefix = data >>>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq >>>>> >>>>> # Roll based on the block size only >>>>> tier1.sinks.sink1.hdfs.rollCount=0 >>>>> tier1.sinks.sink1.hdfs.rollInterval=0 >>>>> tier1.sinks.sink1.hdfs.rollSize = 120000000 >>>>> # seconds to wait before closing the file. >>>>> tier1.sinks.sink1.hdfs.idleTimeout = 60 >>>>> tier1.sinks.sink1.channel = channel1 >>>>> >>>>> tier1.channels.channel1.capacity = 100000* >>>>> tier1.sources.source1.deserializer.maxLineLength = 32768 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> the command I used is >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf >>>>> -Dflume.root.logger=DEBUG,console -n agent >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> it gives warn after created sources, channels, sinks for tier1 agent is >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> no configuration found for this host:agent >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> any help? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> I want to copy my local data to hdfs using flume in a single machine >>>>>>> which isrunning hadoop, How can I do that, please help me. >>>>>>> >>>>>>> What is this "local data" ? >>>>>> >>>>>> If it's just files, why not use the hadoop fs copy command instead? >>>>>> If you want to play around with flume, take a look at the spool directory >>>>>> source or the exec source and you should be able to put something >>>>>> together >>>>>> that'll push data through flume to hadoop. >>>>>> >>>>>> -- >>>>>> Sharninder >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Kishore. >>>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Kishore. >>> >> >> > > > -- > Thanks, > Kishore. > -- Thanks, Kishore.
