[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527956#comment-17527956 ] Apoorva Maheshwari commented on IO-278: --- Issue is still present in version 2.7. As it is in reopen state, Kindly confirm if it is being planned to fix in upcoming release. > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa >Priority: Major > Fix For: 2.4 > > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251757#comment-13251757 ] Sergio Bossa commented on IO-278: - Hi James, I think this is not the right place to discuss issues related to my fork, please use Tayler github tracker for that. Thanks, Sergio B. > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251745#comment-13251745 ] James Herrmann commented on IO-278: --- Looks like theses patches have been rolled in. So, congrats to all on that. I went ahead and maven repo'ed your fork anyway because of the IO-269 issue. Tailer would read from start of file on frequent occasions. Not good when you're sending alerts! Sergio, does your fork handle the IO-269 bug? Right now, that's the last known issue I'm concerned about. Thanks! Jim > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225376#comment-13225376 ] Gary D. Gregory commented on IO-278: Hello, Can you create your patches on top of SVN trunk? I'll take a look this week. Thank you, Gary > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225372#comment-13225372 ] Sergio Bossa commented on IO-278: - Hi Jim, Commons-IO team doesn't seem to be much responsive, so I had to create a fork from that at: https://github.com/sbtourist/tayler Feel free to give it a look. > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223498#comment-13223498 ] James Herrmann commented on IO-278: --- Hi Sergio, Have you heard if these last two fixes (IO-278/IO-279) under evaluation for inclusion into the official build? Thanks, Jim > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (IO-278) Improve Tailer performance with buffered reads
[ https://issues.apache.org/jira/browse/IO-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066415#comment-13066415 ] Sergio Bossa commented on IO-278: - Also, while I was at it I fixed a concurrency bug in TestTailerListener causing intermittent test failures (the lines list needed to be thread safe). > Improve Tailer performance with buffered reads > -- > > Key: IO-278 > URL: https://issues.apache.org/jira/browse/IO-278 > Project: Commons IO > Issue Type: Improvement >Affects Versions: 2.0.1 >Reporter: Sergio Bossa > Attachments: Tailer.diff, TailerTest.diff > > > I noticed Tailer read performances are pretty poor when dealing with large, > frequently written, log files, and this is due to the use of RandomAccessFile > which does unbuffered reads, hence causing lots of disk I/O. > So I improved the Tailer implementation by introducing buffered reads: it > works by loading large (configurable) file chunks in memory, and reading > lines from there; this enhances performances in my tests from 10x to 30x > depending on the file size. > I also added two test cases: one to simulate reading of a large file (you can > use it to compare performances), the other to verify correct handling on > buffer breaks; obviously, all tests pass. > I'm attaching the diff files, let me know if it's okay for you guys! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
