I shitlisted CommonGramsQueryFilter in r1409199 On Wed, Nov 14, 2012 at 5:40 AM, Robert Muir <[email protected]> wrote: > CommonGramsQueryFilter is not safe for a graph, such as the one here > coming from ShingleFilter. > > I am not sure how easy this would be to fix: the problem is > essentially just like having two ShingleFilters in your analysis > chain. I suspect this gives strange results as well. > > On Tue, Nov 13, 2012 at 10:54 PM, Policeman Jenkins Server > <[email protected]> wrote: >> Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Windows/1614/ >> Java: 32bit/jdk1.6.0_37 -server -XX:+UseSerialGC >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.analysis.core.TestRandomChains.testRandomChains >> >> Error Message: >> stage 3: inconsistent endOffset at pos=5: 15 vs 10; token=ޞ i ivqk_i >> >> Stack Trace: >> java.lang.IllegalStateException: stage 3: inconsistent endOffset at pos=5: >> 15 vs 10; token=ޞ i ivqk_i >> at >> __randomizedtesting.SeedInfo.seed([5D0182766DE523A4:60E0AB172AF73E64]:0) >> at >> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:135) >> at >> org.apache.lucene.analysis.hi.HindiNormalizationFilter.incrementToken(HindiNormalizationFilter.java:51) >> at >> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:78) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:632) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:542) >> at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:443) >> at >> org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:859) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) >> at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> at >> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >> at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >> at java.lang.Thread.run(Thread.java:662) >> >> >> >> >> Build Log: >> [...truncated 3859 lines...] >> [junit4:junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains >> [junit4:junit4] 2> TEST FAIL: useCharFilter=false text='\ue726 >> \u06715\u01b8\ue6ac\u8e51\u079e i ivqk |\u0528 ' >> [junit4:junit4] 2> Exception from random analyzer: >> [junit4:junit4] 2> charfilters= >> [junit4:junit4] 2> tokenizer= >> [junit4:junit4] 2> >> org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer(LUCENE_50, >> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory@1cc6a6a, >> >> org.apache.lucene.analysis.core.TestRandomChains$CheckThatYouDidntReadAnythingReaderWrapper@1a2ac94) >> [junit4:junit4] 2> filters= >> [junit4:junit4] 2> >> org.apache.lucene.analysis.shingle.ShingleFilter(org.apache.lucene.analysis.ValidatingTokenFilter@6c11df8, >> 58) >> [junit4:junit4] 2> >> org.apache.lucene.analysis.ar.ArabicNormalizationFilter(org.apache.lucene.analysis.ValidatingTokenFilter@6c11df8) >> [junit4:junit4] 2> >> org.apache.lucene.analysis.commongrams.CommonGramsQueryFilter(org.apache.lucene.analysis.commongrams.CommonGramsFilter@6c11df8) >> [junit4:junit4] 2> >> org.apache.lucene.analysis.hi.HindiNormalizationFilter(org.apache.lucene.analysis.ValidatingTokenFilter@d162a12d) >> [junit4:junit4] 2> offsetsAreCorrect=true >> [junit4:junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestRandomChains -Dtests.method=testRandomChains >> -Dtests.seed=5D0182766DE523A4 -Dtests.slow=true -Dtests.locale=be_BY >> -Dtests.timezone=Australia/Lord_Howe -Dtests.file.encoding=UTF-8 >> [junit4:junit4] ERROR 3.15s | TestRandomChains.testRandomChains <<< >> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: stage 3: >> inconsistent endOffset at pos=5: 15 vs 10; token=? i ivqk_i >> [junit4:junit4] > at >> __randomizedtesting.SeedInfo.seed([5D0182766DE523A4:60E0AB172AF73E64]:0) >> [junit4:junit4] > at >> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:135) >> [junit4:junit4] > at >> org.apache.lucene.analysis.hi.HindiNormalizationFilter.incrementToken(HindiNormalizationFilter.java:51) >> [junit4:junit4] > at >> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:78) >> [junit4:junit4] > at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:632) >> [junit4:junit4] > at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:542) >> [junit4:junit4] > at >> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:443) >> [junit4:junit4] > at >> org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:859) >> [junit4:junit4] > at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> [junit4:junit4] > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> [junit4:junit4] > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) >> [junit4:junit4] > at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) >> [junit4:junit4] > at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) >> [junit4:junit4] > at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> [junit4:junit4] > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) >> [junit4:junit4] > at java.lang.Thread.run(Thread.java:662) >> [junit4:junit4] 2> NOTE: test params are: codec=Lucene41: >> {dummy=MockVariableIntBlock(baseBlockSize=31)}, sim=DefaultSimilarity, >> locale=be_BY, timezone=Australia/Lord_Howe >> [junit4:junit4] 2> NOTE: Windows 7 6.1 x86/Sun Microsystems Inc. 1.6.0_37 >> (32-bit)/cpus=2,threads=1,free=19381432,total=47677440 >> [junit4:junit4] 2> NOTE: All tests run in this JVM: >> [TestShingleFilterFactory, TestPortugueseStemFilter, HunspellStemFilterTest, >> TestStopAnalyzer, TestAllAnalyzersHaveFactories, >> TestItalianLightStemFilterFactory, QueryAutoStopWordAnalyzerTest, >> TestUAX29URLEmailAnalyzer, TestDelimitedPayloadTokenFilterFactory, >> TestBugInSomething, TestFrenchLightStemFilterFactory, TestDuelingAnalyzers, >> CommonGramsFilterTest, TestItalianAnalyzer, >> TestDictionaryCompoundWordTokenFilterFactory, TestBasqueAnalyzer, >> TestGreekStemmer, TestCzechStemmer, TestCharArrayMap, TestKeepWordFilter, >> TestIrishAnalyzer, TestIndicNormalizer, NGramTokenizerTest, >> TestRemoveDuplicatesTokenFilter, TestLatvianAnalyzer, >> TestLengthFilterFactory, TestHindiNormalizer, TestPortugueseAnalyzer, >> TestSolrSynonymParser, TestGreekStemFilterFactory, TestElisionFilterFactory, >> ShingleFilterTest, TestNorwegianMinimalStemFilterFactory, GreekAnalyzerTest, >> TestNorwegianMinimalStemFilter, TokenTypeSinkTokenizerTest, >> TestEnglishMinimalStemFilterFactory, TestUAX29URLEmailTokenizerFactory, >> TestWikipediaTokenizerFactory, TestKeepFilterFactory, >> TestIrishLowerCaseFilter, TestGermanLightStemFilter, >> TestPortugueseMinimalStemFilter, TestTrimFilterFactory, TestLengthFilter, >> TestCJKBigramFilterFactory, TestGalicianMinimalStemFilter, TestCharArraySet, >> TestUAX29URLEmailTokenizer, TestCJKWidthFilterFactory, >> TestThaiWordFilterFactory, TestBulgarianStemFilterFactory, >> TestGermanNormalizationFilter, TestSpanishAnalyzer, >> TestFrenchLightStemFilter, TestPatternReplaceCharFilterFactory, >> TestPatternReplaceFilter, TestTeeSinkTokenFilter, >> TestLatvianStemFilterFactory, TestCommonGramsFilterFactory, >> TestCharTokenizers, TestPorterStemFilter, TestIndonesianStemmer, >> TestSnowballPorterFilterFactory, TestMappingCharFilter, >> HTMLStripCharFilterTest, TestPatternReplaceCharFilter, >> TestPorterStemFilterFactory, TestFactories, TestArabicAnalyzer, >> PositionFilterTest, TestPortugueseStemFilterFactory, >> TestCommonGramsQueryFilterFactory, TestFrenchAnalyzer, TestElision, >> TestGalicianStemFilterFactory, TestSwedishLightStemFilterFactory, >> TestFinnishLightStemFilter, TestDutchStemmer, TestSnowball, >> TestNorwegianAnalyzer, TestSwedishLightStemFilter, TestFinnishAnalyzer, >> TestStandardFactories, TestPatternTokenizerFactory, HunspellStemmerTest, >> TestClassicAnalyzer, TestReversePathHierarchyTokenizer, TestCJKBigramFilter, >> NumericPayloadTokenFilterTest, TestWordnetSynonymParser, TestCJKAnalyzer, >> TestAnalysisSPILoader, TestBulgarianAnalyzer, TestKStemFilterFactory, >> TestFrenchMinimalStemFilter, TestIndonesianAnalyzer, >> TestCzechStemFilterFactory, EdgeNGramTokenizerTest, >> ShingleAnalyzerWrapperTest, TestTurkishLowerCaseFilter, >> TestGermanStemFilterFactory, TestRussianLightStemFilter, >> TestCollationKeyAnalyzer, TestHindiAnalyzer, TestPortugueseLightStemFilter, >> TestHTMLStripCharFilterFactory, TestSynonymMapFilter, TestAnalyzers, >> TestStandardAnalyzer, TestRandomChains] >> [junit4:junit4] Completed in 3.83s, 1 test, 1 error <<< FAILURES! >> >> [...truncated 315 lines...] >> BUILD FAILED >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:337: The >> following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The >> following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:520: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1692: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\analysis\build.xml:105: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\analysis\build.xml:38: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:61: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1163: >> The following error occurred while executing this line: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:827: >> There were test failures: 213 suites, 1078 tests, 1 error, 1 ignored >> >> Total time: 10 minutes 37 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Recording test results >> Description set: Java: 32bit/jdk1.6.0_37 -server -XX:+UseSerialGC >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected]
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
