Here is a test case:
string text = @"Califórnia";
Lucene.Net.Analysis.KeywordTokenizer tokenizer = new KeywordTokenizer(new
StringReader(text));
Lucene.Net.Analysis.Snowball.SnowballFilter stemmer=
new Lucene.Net.Analysis.Snowball.SnowballFilter(tokenizer,
"Portuguese");
Lucene.Net.Analysis.Token token;
while ((token = stemmer.Next()) != null)
{
System.Console.WriteLine(tokenText);
}
Seems to go into infinite loop. Call to stemmer.Next() never returns. Not
sure if this is the only stemmer I am having trouble with. And it does happen
to us on a near daily basis.
Thanks,
Bob
On Sep 13, 2011, at 9:37 AM, Robert Stewart wrote:
> Are there any known issues with snowball stemmers (portuguese in particular)
> going into some infinite loop? I have a problem that happens on a recurring
> basis where IndexWriter locks up on AddDocument and never returns (it has
> taken up to 3 days before we realize it), requiring manual killing of the
> process. It seems to happen only on portuguese documents from what I can
> tell so far, and the stack trace when thread is aborted is always as follows:
>
> System.Threading.ThreadAbortException: Thread was being aborted.
> at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method,
> Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes
> methodAttributes, RuntimeType typeOwner)
> at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method,
> Object target, Object[] arguments, Signature sig, MethodAttributes
> methodAttributes, RuntimeType typeOwner)
> at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
> invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean
> skipVisibilityChecks)
> at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
> invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
> at Lucene.Net.Analysis.Snowball.SnowballFilter.Next()
> System.SystemException: System.Threading.ThreadAbortException: Thread was
> being aborted.
> at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method,
> Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes
> methodAttributes, RuntimeType typeOwner)
> at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method,
> Object target, Object[] arguments, Signature sig, MethodAttributes
> methodAttributes, RuntimeType typeOwner)
> at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
> invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean
> skipVisibilityChecks)
> at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags
> invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
> at Lucene.Net.Analysis.Snowball.SnowballFilter.Next()
> at Lucene.Net.Analysis.Snowball.SnowballFilter.Next()
> at Lucene.Net.Analysis.TokenStream.IncrementToken()
> at Lucene.Net.Index.DocInverterPerField.ProcessFields(Fieldable[] fields,
> Int32 count)
> at Lucene.Net.Index.DocFieldProcessorPerThread.ProcessDocument()
> at Lucene.Net.Index.DocumentsWriter.UpdateDocument(Document doc, Analyzer
> analyzer, Term delTerm)
> at Lucene.Net.Index.IndexWriter.AddDocument(Document doc, Analyzer analyzer)
>
>
> Is there another list of contrib/snowball issues? I have not been able to
> reproduce a small test case yet however. Have there been any such issues
> with stemmers in the past?
>
> Thanks,
> Bob
