Jason, thank you for the report! There are basically two issues.
I will try to reproduce and fix the problem with the rsync file replacement. GitHub issue for that: https://github.com/Graylog2/collector/issues/46 This is probably related to the second issue below. (not reading old data yet) The second one with the new files showing up is due to the constraint that we currently only start reading files at the end. So when a new file appears, the collector opens the file, seeks to the end and waits for new data. We are currently working on support for keeping file reader state and reading old log files. Regards, Bernd Jason Haar [Fri, Jul 24, 2015 at 01:27:41AM -0700] wrote: >Hi there > >I'm wanting to feed our (multiple) squid server logs into graylog and want >to simply rsync the logs into a staging directory on the server, and have >the collector pipe them in via the GELF connector. (ie I don't want them >put into syslog, nor do I want to install java on the proxies just so I can >run the collector) > >For testing I'm running it in a shell with the output going to stdout. > >inputs { > squid { >type = "file" >path-glob-root = "/var/spool/squid-logs" >path-glob-pattern = "*access.log" > } >} > >I have a "server1-access.log" file in there, and if I "echo squidline >> >server1-access.log" it triggers graylog-collector nicely and I see the >GELF. However, rsync doesn't work like that: it creates a new file with a >temporary filename, copies the original file to that, appends the new data >and then renames it over the original file. End result is the file is >updated, but has a new inode. It appears graylog-collector doesn't notice >that change, nor the fact the file is now a different size? Also, I'm using >globbing, but if I rsync a totally new filename into that directory (eg 2nd >squid server), then nothing happens - it doesn't pick it up either. In >fact, if I create a new file with one line of data, nothing happens; but if >I then append a new line to that file, that is picked up??? > >Am I doing it wrong? > >Jason > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
