Hmm yeah, we should definitely fail fast in this case. The correct way to enable support for HDFS sink needs to be documented somewhere... but the right way to include the jars is to set HADOOP_HOME or add the hadoop bin directory to your PATH in order to allow the flume-ng script to find the right libs for you. The problem is that basically every version of HDFS is not protocol compatible with any other version of Hadoop, but the libs are (mostly) binary compatible. Hence, the use of the technique that can be seen in flume-ng, of appending the output from "hadoop classpath" to the Flume classpath at startup time, which seems to be the lesser of many evils. Of course if you don't use HDFS sink then you never have to worry about this.
Best, Mike On Apr 17, 2012, at 9:28 AM, Inder Pall wrote: > Mike & Brock, > > Yes FLUME-1093 is the issue. I checked out the latest source and > hadoop-core*jar is missing from flume-ng-dist/target/flume-ng-1.2*/lib/ > after the build. > > After putting hadoop-core.jar it works. > > Thanks, > - Inder > > On Tue, Apr 17, 2012 at 9:18 PM, Brock Noland <[email protected]> wrote: > >> I wonder if the hadoop jars are not being added to your classpath: >> >> https://issues.apache.org/jira/browse/FLUME-1093 >> >> Brock >> >> On Tue, Apr 17, 2012 at 3:23 PM, Inder Pall <[email protected]> wrote: >>> thanks, didn't knew that. Here is the log - http://pastebin.com/TQQc9TWU >>> >>> Thanks, >>> - Inder >>> >>> On Tue, Apr 17, 2012 at 8:37 PM, Brock Noland <[email protected]> >> wrote: >>> >>>> The mailing list doesn't like attachements, yours was not included. I >>>> would use pastebin or something similar. >>>> >>>> On Tue, Apr 17, 2012 at 3:04 PM, Inder Pall <[email protected]> >> wrote: >>>>> Brock, >>>>> >>>>> looking at the code and from past usage that wasn't an issue, however >> i >>>>> tried the suggested change >>>>> i.e. agent1.sinks.log-sink1.hdfs.path = /tmp/flume-data/ with no luck. >>>>> >>>>> attaching flume.log with this email for reference. Server does startup >>>>> however telnet/connection fails. >>>>> Weirdly if i change the sink type to logger it does work. >>>>> >>>>> Not sure if i am missing something obvious here. >>>>> >>>>> Thanks, >>>>> - inder >>>>> >>>>> >>>>> >>>>> On Tue, Apr 17, 2012 at 8:17 PM, Brock Noland <[email protected]> >>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> inline >>>>>> >>>>>> On Tue, Apr 17, 2012 at 2:41 PM, Inder Pall <[email protected]> >>>> wrote: >>>>>>> folks, >>>>>>> >>>>>>> i am not sure whether anyone else is facing this or i am missing >>>>>>> something >>>>>>> here. Here are some observations - >>>>>>> >>>>>>> 1. checkout flume trunk >>>>>>> 2. provide the following config >>>>>>> >>>>>>> # Define a memory channel called ch1 on agent1 >>>>>>> agent1.channels.ch1.type = memory >>>>>>> >>>>>>> # Define an Avro source called avro-source1 on agent1 and tell it >>>>>>> # to bind to 0.0.0.0:41414. Connect it to channel ch1. >>>>>>> agent1.sources.avro-source1.type = avro >>>>>>> agent1.sources.avro-source1.bind = 0.0.0.0 >>>>>>> agent1.sources.avro-source1.port = 41414 >>>>>> >>>>>> 1) Does localhost work? >>>>>> 2) Can you paste the output of the following? >>>>>> >>>>>> sudo /usr/sbin/lsof -i tcp | grep 41414 >>>>>> >>>>>>> agent1.sources.avro-source1.channels = ch1 >>>>>>> >>>>>>> # Define a logger sink that simply logs all events it receives >>>>>>> # and connect it to the other end of the same channel. >>>>>>> agent1.sinks.log-sink1.type = hdfs >>>>>>> agent1.sinks.log-sink1.channel = ch1 >>>>>>> agent1.sinks.log-sink1.hdfs.path = hdfs://localhost >>>>>> >>>>>> I think this should be something like /user/noland/ >>>>>> >>>>>>> >>>>>>> >>>>>>> # Finally, now that we've defined all of our components, tell >>>>>>> # agent1 which ones we want to activate. >>>>>>> agent1.sources = avro-source1 >>>>>>> agent1.sinks = log-sink1 >>>>>>> agent1.channels = ch1 >>>>>>> flume.cfg (END) >>>>>>> >>>>>>> 3. start agent - ./flume-ng agent --conf ../conf/ -f flume.cfg -n >>>> agent1 >>>>>> >>>>>> Does flume.log tell you its getting started up? >>>>>> >>>>>> -- >>>>>> Apache MRUnit - Unit testing MapReduce - >>>>>> http://incubator.apache.org/mrunit/ >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> - Inder >>>>> Tech Platforms @Inmobi >>>>> Linkedin - http://goo.gl/eR4Ub >>>> >>>> >>>> >>>> -- >>>> Apache MRUnit - Unit testing MapReduce - >>>> http://incubator.apache.org/mrunit/ >>>> >>> >>> >>> >>> -- >>> Thanks, >>> - Inder >>> Tech Platforms @Inmobi >>> Linkedin - http://goo.gl/eR4Ub >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - >> http://incubator.apache.org/mrunit/ >> > > > > -- > Thanks, > - Inder > Tech Platforms @Inmobi > Linkedin - http://goo.gl/eR4Ub
