Instead just mentioning hdsfs.path = /flume/messages/, do i need to mention something else?
On Mon, Jun 16, 2014 at 10:25 AM, kishore alajangi < [email protected]> wrote: > I created the /flume/messages directories, but still nothing is written > with flume in those directories. please help me. > > > On Mon, Jun 16, 2014 at 10:15 AM, kishore alajangi < > [email protected]> wrote: > >> Do I need to create the /flume/messages/ directories? >> >> >> >> On Mon, Jun 16, 2014 at 10:14 AM, kishore alajangi < >> [email protected]> wrote: >> >>> checked, nothing is written in hdfs. >>> >>> >>> On Mon, Jun 16, 2014 at 10:10 AM, Sharninder <[email protected]> >>> wrote: >>> >>>> That just means source has done its work and is waiting for more data >>>> to read. Did you check hdfs to see if all data has been written? >>>> >>>> >>>> >>>> On Mon, Jun 16, 2014 at 11:34 AM, kishore alajangi < >>>> [email protected]> wrote: >>>> >>>>> Hi Mohit and sharminder, >>>>> >>>>> Thanks for reply, after I called with -n tier, source is not directory >>>>> error came, I changed the source to /tmp/ and hdfs.path to >>>>> /flume/messages/ >>>>> in config file, and run the command, the INFO i am getting now is >>>>> "spooling >>>>> directory source runner has shutdown" >>>>> what could be the problem, please help me. >>>>> >>>>> >>>>> On Sun, Jun 15, 2014 at 10:21 PM, Mohit Durgapal < >>>>> [email protected]> wrote: >>>>> >>>>>> Replace -n agent with -n tier1 >>>>>> >>>>>> >>>>>> On Sunday, June 15, 2014, kishore alajangi <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Dear Sharminder, >>>>>>> >>>>>>> Thanks for your reply, yes I am playing with flume, as you suggested >>>>>>> i am using spool directory source, My configuration file looks like >>>>>>> >>>>>>> *tier1.sources = source1 >>>>>>> tier1.channels = channel1 >>>>>>> tier1.sinks = sink1 >>>>>>> >>>>>>> tier1.sources.source1.type = spooldir >>>>>>> tier1.sources.source1.spoolDir = /var/log/messages >>>>>>> tier1.sources.source1.channels = channel1 >>>>>>> tier1.channels.channel1.type = memory >>>>>>> >>>>>>> tier1.sinks.sink1.type = hdfs >>>>>>> tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/flume/messages >>>>>>> tier1.sinks.sink1.hdfs.fileType = SequenceFile >>>>>>> tier1.sinks.sink1.hdfs.filePrefix = data >>>>>>> tier1.sinks.sink1.hdfs.fileSuffix = .seq >>>>>>> >>>>>>> # Roll based on the block size only >>>>>>> tier1.sinks.sink1.hdfs.rollCount=0 >>>>>>> tier1.sinks.sink1.hdfs.rollInterval=0 >>>>>>> tier1.sinks.sink1.hdfs.rollSize = 120000000 >>>>>>> # seconds to wait before closing the file. >>>>>>> tier1.sinks.sink1.hdfs.idleTimeout = 60 >>>>>>> tier1.sinks.sink1.channel = channel1 >>>>>>> >>>>>>> tier1.channels.channel1.capacity = 100000* >>>>>>> tier1.sources.source1.deserializer.maxLineLength = 32768 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> the command I used is >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ./flume-ng agent --conf ./conf/ -f bin/example.conf >>>>>>> -Dflume.root.logger=DEBUG,console -n agent >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> it gives warn after created sources, channels, sinks for tier1 agent is >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> no configuration found for this host:agent >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> any help? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Jun 15, 2014 at 11:18 AM, Sharninder <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> I want to copy my local data to hdfs using flume in a single >>>>>>>>> machine which isrunning hadoop, How can I do that, please help me. >>>>>>>>> >>>>>>>>> What is this "local data" ? >>>>>>>> >>>>>>>> If it's just files, why not use the hadoop fs copy command instead? >>>>>>>> If you want to play around with flume, take a look at the spool >>>>>>>> directory >>>>>>>> source or the exec source and you should be able to put something >>>>>>>> together >>>>>>>> that'll push data through flume to hadoop. >>>>>>>> >>>>>>>> -- >>>>>>>> Sharninder >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Kishore. >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Kishore. >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Kishore. >>> >> >> >> >> -- >> Thanks, >> Kishore. >> > > > > -- > Thanks, > Kishore. > -- Thanks, Kishore.
