Hi Ashish Thanks for the help. Can you pls help with procedure to update or pull the patch into my system . Thanks.
Regards, PavaN -----Original Message----- From: Ashish [mailto:paliwalash...@gmail.com] Sent: Saturday, July 18, 2015 12:44 AM To: user@flume.apache.org Subject: Re: How to read content from files which are in nested directory One more way to achieve the same is using this patch https://issues.apache.org/jira/browse/FLUME-1899 On Fri, Jul 17, 2015 at 7:14 AM, Johny Rufus <jru...@cloudera.com> wrote: > The spooling directory source as of now supports only reading from a > flat directory and wont read files from subdirectories. > You could write an external script that transfers all the files in all > the date directories to a common directory which spooling source > points to. (If this fits your use case) > > Thanks, > Rufus > > On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <pavan.jak...@cgi.com> wrote: >> >> Hi Folks, >> >> >> >> I am consuming Apache flume to read logs which are stored in >> following format , logs under date directory . How do I read the logs >> using spooldir source. >> >> >> >> >> >> Directory Structure : >> >> >> >> >> >> flume]# ll /ADAPTORS/MAIL_CONNECT/logs/ >> >> total 0 >> >> drwxr-xr-x 2 root root 29 Jun 3 18:04 06-03-15 >> >> drwxr-xr-x 2 root root 56 Jun 4 18:16 06-04-15 >> >> drwxr-xr-x 2 root root 29 Jun 5 14:03 06-05-15 >> >> drwxr-xr-x 2 root root 29 Jun 8 12:43 06-08-15 >> >> drwxr-xr-x 2 root root 29 Jun 9 14:47 06-09-15 >> >> drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15 >> >> drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15 >> >> drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15 >> >> drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15 >> >> drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15 >> >> drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15 >> >> drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15 >> >> >> >> >> >> My current Configuration : >> >> agent1.sources.source1.type = spooldir >> >> agent1.sources.source1.channels = channel1 >> >> agent1.sources.source1.spoolDir = >> /ADAPTORS/MAIL_CONNECT/logs/07-17-15 >> >> agent1.sources.source1.fileHeader = true >> >> >> >> >> >> Pls suggest if i need to use some other source type . >> >> >> >> Can I use any script inside conf file to change the directory >> dynamically . Thanks >> >> >> >> >> >> Regards, >> >> PavaN >> >> > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
