Hi Phil, Nice approach. How is the spooling directory source working for you? Any thoughts on how it could be improved?
Mike On Tue, May 21, 2013 at 8:17 AM, Phil Scala <[email protected]>wrote: > Hi,**** > > ** ** > > Based on my use and understanding that setting “fileSuffix” is simpy the > extension to the file to be added once the file was consumed and placed > onto the channel. I don’t think it was intended to be used to indicate a > completely uploaded file. In the dev newsgroup there was a discussion > about having the spooler ”wait” for a little while before ingesting the > file. An ignore pattern may also be a good idea.**** > > ** ** > > Currently I am using an upload directory that I monitor and when lsof > reports a file is not in use/open I then move it to my spool directory. * > *** > > ** ** > > HTH**** > > Phil**** > > ** ** > > ** ** > > ** ** > > Phil Scala**** > > Software Developer / Architect > Global Relay > > [email protected] > > *866.484.6630* | > [email protected]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net> > | > globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f> > **** > > ** ** > > *From:* ZORAIDA HIDALGO SANCHEZ [mailto:[email protected]] > *Sent:* Tuesday, May 21, 2013 9:21 AM > *To:* Flume User List > *Subject:* Spooling fileSuffix attribute ignored**** > > ** ** > > Dear all, **** > > I am using the Spooling attribute "fileSuffix" to ensure that only when a > file has been completely upload into the Spooling dir, is consumed by the > source. However, files with no suffix are also appended to the channel and > then processed by the sink. **** > > My configuration:**** > > ** ** > > tier1.sources = s1**** > > tier1.channels = c1**** > > tier1.sinks = s1**** > > ** ** > > # For each source, channel, and sink, set**** > > # standard properties.**** > > tier1.sources.s1.type = spooldir**** > > tier1.sources.s1.spoolDir = /home/user/flume/data**** > > tier1.sources.s1.deletePolicy = immediate**** > > tier1.sources.s1.batchSize = 1000**** > > tier1.sources.s1.bufferMaxLines = 3000**** > > tier1.sources.s1.fileHeader = true**** > > tier1.sources.s1.fileSuffix=.COMPLETED**** > > ** ** > > Is that ok? **** > > ** ** > > Thanks.**** > > ** ** > ------------------------------ > > > Este mensaje se dirige exclusivamente a su destinatario. Puede consultar > nuestra política de envío y recepción de correo electrónico en el enlace > situado más abajo. > This message is intended exclusively for its addressee. We only send and > receive email on the basis of the terms set out at: > http://www.tid.es/ES/PAGINAS/disclaimer.aspx**** >
