Unfortunately, there is no simple mechanism. There are one thing that makes things complicated - file to transfer into the HDFS consists of arbitrary number of events and as soon SpoolDir starts to read file it fills the channel with them.
This solution sounds like a hack, but it can be implemented with least amount of software development and it's probably the best one: You can use fileHeaderKey param for SpoolDir Source that adds file's full path to the event header. Also, you'll need to store event header into HDFS. Then, you can do line count for a file and compare to number of events in HDFS that has the given header. On Fri, Dec 5, 2014 at 1:47 PM, Manohar CS <[email protected]> wrote: > Thanks for the response. With these monitoring JSON result we can basic > metrics such as no.of events processed. But I have custom requirement to > signify completion of transfer of file. > > > > *From:* Ahmed Vila [mailto:[email protected]] > *Sent:* Friday, December 5, 2014 5:59 PM > *To:* [email protected] > *Subject:* Re: Notification support from flume? > > > > Hi Manohar, > > > > You can turn Flume monitoring web server that reports the numbers as JSON. > > > http://archive.cloudera.com/cdh/3/flume-ng/FlumeUserGuide.html#json-reporting > > > > It explains that flume-ng agentshould be started with parameters > "-Dflume.monitoring.type=HTTP -Dflume.monitoring.port=34545", but there is > no info on how to configure those params in packed Flume for f.e. Ubuntu. > > > > For ubunti package that you install via apt-get, you can > change /etc/flume-ng/conf/flume-env.sh and add those params into JAVA_OPTS > variable export... something like this: > > export JAVA_OPTS="-Xms256m -Xmx512m -Dflume.monitoring.type=HTTP > -Dflume.monitoring.port=34545" > > > > > > > > On Fri, Dec 5, 2014 at 1:19 PM, Manohar CS <[email protected]> > wrote: > > Hi All, > > > > I wanted to know if there is a way of notification mechanism or some way > of finding out if flume has finished transfer of certain file from spoolDir > to HDFS sink? We know by looking at .COMPLETED files in spoolDir we can > assume its completed but wanted to know if there is more reliable way of > call back mechanism ? > > > > > > Thanks, > > Manohar. > > > > > Please consider the environment before printing this e-mail > > > Disclaimer: This communication is for the exclusive use of the intended > recipient(s) and shall not attach any liability on the originator or ITC > Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group > Companies. If you are the addressee, the contents of this e-mail are intended > for your use only and it shall not be forwarded to any third party, without > first obtaining written authorization from the originator or ITC Infotech > India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It > may contain information which is confidential and legally privileged and the > same shall not be used or dealt with by any third party in any manner > whatsoever without the specific consent of ITC Infotech India Ltd./ its > Holding company/ its Subsidiaries/ its Group Companies. > > > > > > -- > > Best regards, > > Ahmed Vila | Senior software developer > > DevLogic | Sarajevo | Bosnia and Herzegovina > > > > Office : +387 33 942 123 > > Mobile: +387 62 139 348 > > > > Website: www.devlogic.eu > > E-mail : [email protected] > > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies. > > > --------------------------------------------------------------------- > This e-mail and any attachment is for authorised use by the intended > recipient(s) only. This email contains confidential information. It should > not be copied, disclosed to, retained or used by, any party other than the > intended recipient. Any unauthorised distribution, dissemination or copying > of this E-mail or its attachments, and/or any use of any information > contained in them, is strictly prohibited and may be illegal. If you are > not an intended recipient then please promptly delete this e-mail and any > attachment and all copies and inform the sender directly via email. Any > emails that you send to us may be monitored by systems or persons other > than the named communicant for the purposes of ascertaining whether the > communication complies with the law and company policies. > > > > Please consider the environment before printing this e-mail > > > Disclaimer: This communication is for the exclusive use of the intended > recipient(s) and shall not attach any liability on the originator or ITC > Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group > Companies. If you are the addressee, the contents of this e-mail are intended > for your use only and it shall not be forwarded to any third party, without > first obtaining written authorization from the originator or ITC Infotech > India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It > may contain information which is confidential and legally privileged and the > same shall not be used or dealt with by any third party in any manner > whatsoever without the specific consent of ITC Infotech India Ltd./ its > Holding company/ its Subsidiaries/ its Group Companies. > -- --------------------------------------------------------------------- This e-mail and any attachment is for authorised use by the intended recipient(s) only. This email contains confidential information. It should not be copied, disclosed to, retained or used by, any party other than the intended recipient. Any unauthorised distribution, dissemination or copying of this E-mail or its attachments, and/or any use of any information contained in them, is strictly prohibited and may be illegal. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender directly via email. Any emails that you send to us may be monitored by systems or persons other than the named communicant for the purposes of ascertaining whether the communication complies with the law and company policies.
