This is what i wanted .waiting for flume 1.6.0 Release to use this features 😊 Thanks.
Date: Mon, 16 Feb 2015 16:26:25 +0530 Subject: Re: No threads used in Spooling Directory From: [email protected] To: [email protected] This could be of help https://issues.apache.org/jira/browse/FLUME-2413 (you need this at Sink level but this should work as well) HTH! On Mon, Feb 16, 2015 at 4:01 PM, mahendran m <[email protected]> wrote: I am moving the IIS logs into remote hdfs using spooling directory and I need to update how much data (size not no of events) has been moved successfully in real time in UI Thanks, Date: Mon, 16 Feb 2015 12:59:06 +0530 Subject: Re: No threads used in Spooling Directory From: [email protected] To: [email protected] What Flume Dev's meant was, Flume is not intended to do File transfers as a whole. A File shall be read and an Event created out of the content based on how an Event needs to be created (each line as an Event by default). To answer you question (to the best of my knowledge) Q1. I would like to know how much file has been processed in parallel by SpoolDir ? Nope Q2. Is it possible to how much size of file has been moved by SpoolDir to sink at any point using built-in API? Nope If you can share what you want to achieve, we may be able to provide some pointers. thanks ashish On Fri, Feb 13, 2015 at 5:43 PM, mahendran m <[email protected]> wrote: I need to move IIS logs to hdfs , then which is suitable for this purpose ? Thanks , From: [email protected] To: [email protected] Subject: RE: No threads used in Spooling Directory Date: Fri, 13 Feb 2015 11:00:48 +0000 I had posted for similar requirement few days back , but from core team members got to know that spoolDir is not suitable for fileBased handling, its more of event based content transfer that is being done. From: mahendran m [mailto:[email protected]] Sent: Friday, February 13, 2015 11:02 AM To: FlumeMaling List Subject: No threads used in Spooling Directory HI , Q1. I would like to know how much file has been processed in parallel by SpoolDir ? Q2. Is it possible to how much size of file has been moved by SpoolDir to sink at any point using built-in API? Thanks , Please consider the environment before printing this e-mail Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
