[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578669#comment-14578669 ] Hudson commented on RAT-138: SUCCESS: Integrated in tobago-trunk #1386 (See [https://builds.apache.org/job/tobago-trunk/1386/]) updated rat-plugin to fix slowness (see RAT-138) (deki: http://svn.apache.org/viewvc/?view=revrev=1684365) * /myfaces/tobago/trunk/pom.xml RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.8, 0.9 Reporter: Sebb Labels: perfomance, regression Fix For: 0.10 Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Curly.match0(Pattern.java:4166) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661378#comment-13661378 ] Sebb commented on RAT-138: -- URL: http://svn.apache.org/r1484128 Log: RAT-138 RAT runs very slowly on some input Use String matching by default rather than building patterns Modified: creadur/rat/trunk/apache-rat-core/src/main/java/org/apache/rat/analysis/generation/GeneratedLicenseNotRequired.java RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.9 Reporter: Sebb Labels: perfomance, regression Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Curly.match0(Pattern.java:4166) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661069#comment-13661069 ] Sebb commented on RAT-138: -- Seems it is mainly the extra matchers that are being run; 0.9 additionally runs: GPL1License GPL2License GPL3License MITLicense These are all FullTextMatchers which currently use regexes unnecessarily. RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.9 Reporter: Sebb Labels: perfomance, regression Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Curly.match0(Pattern.java:4166) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13661134#comment-13661134 ] Sebb commented on RAT-138: -- Actually, the issue seems to be multi-line matching. The OASIS license matcher looks for the initial copyright line, and only then tries using the full text matcher. If the copyright line is not present in a file, it's very quick to scan. However, if the OASIS copyright line is present, but the rest of the license is not, then the matching is very slow, even in 0.8. It helps to use String.contains rather than regexes, but that does not entirely solve the slowness. The technique used in FullTextMatchingLicense is clearly not very efficient. I think the problem is that the text buffer keeps getting bigger even if there is no chance of a match for the first part of the license, so matching gets more and more expensive. The OASIS code avoids this for valid licences, but allows the initial Copyright to be far away from the rest of the license (which is why it is slow if that is missing) If the initial line is not immediately followed by the rest of the license, then it should be possible to restart looking for the first line. However the entire buffer cannot be discarded without first checking whether the first line has been seen again already. This is unlikely, but possible. RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.9 Reporter: Sebb Labels: perfomance, regression Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Curly.match0(Pattern.java:4166) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657801#comment-13657801 ] Sebb commented on RAT-138: -- Seems to be just the Javadocs that causes the problem. The following file (amongst others) causes the spinning to occur: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io/apidocs/index-all.html RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.9 Reporter: Sebb Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xecc runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Slice.match(Pattern.java:3867) at java.util.regex.Pattern$Curly.match0(Pattern.java:4170) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.generation.GeneratedLicenseNotRequired.match(GeneratedLicenseNotRequired.java:71) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (RAT-138) RAT runs very slowly on some input
[ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657822#comment-13657822 ] Sebb commented on RAT-138: -- Looks like GeneratedLicenseNotRequired does not need to use regexes either - it does not even use case-insensitive matching. RAT runs very slowly on some input -- Key: RAT-138 URL: https://issues.apache.org/jira/browse/RAT-138 Project: Apache Rat Issue Type: Bug Components: engine Affects Versions: 0.9 Reporter: Sebb Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting to 0.8 fixes the problem. Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here is a sample stack trace from a thread dump: main prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$Curly.match0(Pattern.java:4166) at java.util.regex.Pattern$Curly.match(Pattern.java:4132) at java.util.regex.Matcher.match(Matcher.java:1221) at java.util.regex.Matcher.matches(Matcher.java:559) at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79) at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42) at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113) at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68) at org.apache.rat.Report.report(Report.java:393) at org.apache.rat.Report.report(Report.java:373) at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462) at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148) at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310) at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210) I assume there must be a problem with one of the REs which is triggering lots of backtracking when applied to files under site-content/, which is a working copy of: https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io Last Changed Rev: 861378 This directory should not have been included in the RAT scan, but the files don't cause problems for RAT 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
