Yes Ashish is right. Which flume version your are using? Version 1.4.0 does not support recursive directory checks, so, if you store your logs in a directory with a timestamp (i.e. /logs/project_name/2013/12/26), you will have to move all files to a single directory (i.e. /logs/project_name).
-----Original Message----- From: Ashish [mailto:[email protected]] Sent: Tuesday, November 18, 2014 11:32 AM To: [email protected] Subject: Re: Need help on spooling directory source AFAIK, it doesn't work recursively. It would pick files only from specified directory. Here is the patch you can try https://issues.apache.org/jira/browse/FLUME-1899 HTH ! On Tue, Nov 18, 2014 at 11:21 AM, mahendran m <[email protected]> wrote: > Hi All, > > I was working on moving the IIS logs into HDFS using flume spooling > directory source i was running the 3 website in IIS server , so my > logging directory will looks like below > > LogFiles > > | > |-- W3SVC1 > > | | > > | `--website1.log > > | > > |-- W3SVC1 > > | | > > | `--website2.log > > | > > `-- W3SCV3 > | > `-- website3.log > > > > Now my use case is that i want to move the log of website 3 that is > website3.log file to HDFS. So i have configured the CONF file as > below > > > > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3 > > > and it is works perfectly . now My use case is changed that i want > move all the log files under the folder LogFiles , now i changed the > CONF as below > > > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles > > > but it does not working . not moving any files from LogFiles > directory. How can i achieve my use case ? > > > Thanks, > > Mahendran > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal Please consider the environment before printing this e-mail Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
