Re: solr.StandardTokenizerFactory: more info needed
Thanks, Erick. On Wed, Jul 6, 2011 at 6:27 PM, Erick Erickson wrote: > See ..src/test/org/apache/solr/analysis. > > But... you'll be changing the grammar, so > I don't know how tests would actually help you. Actually > I'd expect them to break. And you'd have to write some > new ones of your own to exercise your changes to insure > that they do what you want > > Best > Erick > > On Wed, Jul 6, 2011 at 9:31 AM, Dmitry Kan wrote: > > OK, thanks. Do you know if there are tokenizer specific tests to run > after > > compilation? > > > > On Wed, Jul 6, 2011 at 4:25 PM, Steven A Rowe wrote: > > > >> Yes, you can change the rules and recompile. > >> > >> Before you recompile, you have to run 'ant jflex' to generate the java > >> source. > >> > >> Steve > >> > >> -----Original Message- > >> From: Dmitry Kan [mailto:dmitry@gmail.com] > >> Sent: Wednesday, July 06, 2011 9:21 AM > >> To: solr-user@lucene.apache.org > >> Subject: Re: solr.StandardTokenizerFactory: more info needed > >> > >> Hi Steven, > >> > >> This looks very good. Thanks. Do I understand correctly, that I were to > >> change the tokenizer rules, I could go and change e.g. the token class > >> definitions (like ) in this file and recompile the code? > >> > >> On Wed, Jul 6, 2011 at 3:45 PM, Steven A Rowe wrote: > >> > >> > Hi Dmitry, > >> > > >> > The underlying Lucene implementation is here: > >> > > http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java > >> > /org/apache/lucene/analysis/standard/ > >> > > >> > StandardTokenizerImpl.jflex is probably where you should start. > >> > > >> > Steve > >> > > >> > -Original Message- > >> > From: Dmitry Kan [mailto:dmitry@gmail.com] > >> > Sent: Wednesday, July 06, 2011 3:23 AM > >> > To: solr-user@lucene.apache.org > >> > Subject: solr.StandardTokenizerFactory: more info needed > >> > > >> > Hi all! > >> > > >> > solr.StandardTokenizerFactory -- is it possible to see the full > >> > description of its behaviour for solr.1.4 somewhere? Wiki > >> > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.Stand > >> > ardTokenizerFactory > >> > is > >> > very short.. > >> > > >> > -- > >> > Regards, > >> > > >> > Dmitry Kan > >> > > >> > >> > >> > >> -- > >> Regards, > >> > >> Dmitry Kan > >> > > > > > > > > -- > > Regards, > > > > Dmitry Kan > > > -- Regards, Dmitry Kan
Re: solr.StandardTokenizerFactory: more info needed
See ..src/test/org/apache/solr/analysis. But... you'll be changing the grammar, so I don't know how tests would actually help you. Actually I'd expect them to break. And you'd have to write some new ones of your own to exercise your changes to insure that they do what you want Best Erick On Wed, Jul 6, 2011 at 9:31 AM, Dmitry Kan wrote: > OK, thanks. Do you know if there are tokenizer specific tests to run after > compilation? > > On Wed, Jul 6, 2011 at 4:25 PM, Steven A Rowe wrote: > >> Yes, you can change the rules and recompile. >> >> Before you recompile, you have to run 'ant jflex' to generate the java >> source. >> >> Steve >> >> -Original Message- >> From: Dmitry Kan [mailto:dmitry@gmail.com] >> Sent: Wednesday, July 06, 2011 9:21 AM >> To: solr-user@lucene.apache.org >> Subject: Re: solr.StandardTokenizerFactory: more info needed >> >> Hi Steven, >> >> This looks very good. Thanks. Do I understand correctly, that I were to >> change the tokenizer rules, I could go and change e.g. the token class >> definitions (like ) in this file and recompile the code? >> >> On Wed, Jul 6, 2011 at 3:45 PM, Steven A Rowe wrote: >> >> > Hi Dmitry, >> > >> > The underlying Lucene implementation is here: >> > http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java >> > /org/apache/lucene/analysis/standard/ >> > >> > StandardTokenizerImpl.jflex is probably where you should start. >> > >> > Steve >> > >> > -Original Message- >> > From: Dmitry Kan [mailto:dmitry@gmail.com] >> > Sent: Wednesday, July 06, 2011 3:23 AM >> > To: solr-user@lucene.apache.org >> > Subject: solr.StandardTokenizerFactory: more info needed >> > >> > Hi all! >> > >> > solr.StandardTokenizerFactory -- is it possible to see the full >> > description of its behaviour for solr.1.4 somewhere? Wiki >> > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.Stand >> > ardTokenizerFactory >> > is >> > very short.. >> > >> > -- >> > Regards, >> > >> > Dmitry Kan >> > >> >> >> >> -- >> Regards, >> >> Dmitry Kan >> > > > > -- > Regards, > > Dmitry Kan >
Re: solr.StandardTokenizerFactory: more info needed
OK, thanks. Do you know if there are tokenizer specific tests to run after compilation? On Wed, Jul 6, 2011 at 4:25 PM, Steven A Rowe wrote: > Yes, you can change the rules and recompile. > > Before you recompile, you have to run 'ant jflex' to generate the java > source. > > Steve > > -Original Message- > From: Dmitry Kan [mailto:dmitry@gmail.com] > Sent: Wednesday, July 06, 2011 9:21 AM > To: solr-user@lucene.apache.org > Subject: Re: solr.StandardTokenizerFactory: more info needed > > Hi Steven, > > This looks very good. Thanks. Do I understand correctly, that I were to > change the tokenizer rules, I could go and change e.g. the token class > definitions (like ) in this file and recompile the code? > > On Wed, Jul 6, 2011 at 3:45 PM, Steven A Rowe wrote: > > > Hi Dmitry, > > > > The underlying Lucene implementation is here: > > http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java > > /org/apache/lucene/analysis/standard/ > > > > StandardTokenizerImpl.jflex is probably where you should start. > > > > Steve > > > > -Original Message- > > From: Dmitry Kan [mailto:dmitry@gmail.com] > > Sent: Wednesday, July 06, 2011 3:23 AM > > To: solr-user@lucene.apache.org > > Subject: solr.StandardTokenizerFactory: more info needed > > > > Hi all! > > > > solr.StandardTokenizerFactory -- is it possible to see the full > > description of its behaviour for solr.1.4 somewhere? Wiki > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.Stand > > ardTokenizerFactory > > is > > very short.. > > > > -- > > Regards, > > > > Dmitry Kan > > > > > > -- > Regards, > > Dmitry Kan > -- Regards, Dmitry Kan
RE: solr.StandardTokenizerFactory: more info needed
Yes, you can change the rules and recompile. Before you recompile, you have to run 'ant jflex' to generate the java source. Steve -Original Message- From: Dmitry Kan [mailto:dmitry@gmail.com] Sent: Wednesday, July 06, 2011 9:21 AM To: solr-user@lucene.apache.org Subject: Re: solr.StandardTokenizerFactory: more info needed Hi Steven, This looks very good. Thanks. Do I understand correctly, that I were to change the tokenizer rules, I could go and change e.g. the token class definitions (like ) in this file and recompile the code? On Wed, Jul 6, 2011 at 3:45 PM, Steven A Rowe wrote: > Hi Dmitry, > > The underlying Lucene implementation is here: > http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java > /org/apache/lucene/analysis/standard/ > > StandardTokenizerImpl.jflex is probably where you should start. > > Steve > > -Original Message- > From: Dmitry Kan [mailto:dmitry@gmail.com] > Sent: Wednesday, July 06, 2011 3:23 AM > To: solr-user@lucene.apache.org > Subject: solr.StandardTokenizerFactory: more info needed > > Hi all! > > solr.StandardTokenizerFactory -- is it possible to see the full > description of its behaviour for solr.1.4 somewhere? Wiki > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.Stand > ardTokenizerFactory > is > very short.. > > -- > Regards, > > Dmitry Kan > -- Regards, Dmitry Kan
Re: solr.StandardTokenizerFactory: more info needed
Hi Steven, This looks very good. Thanks. Do I understand correctly, that I were to change the tokenizer rules, I could go and change e.g. the token class definitions (like ) in this file and recompile the code? On Wed, Jul 6, 2011 at 3:45 PM, Steven A Rowe wrote: > Hi Dmitry, > > The underlying Lucene implementation is here: > http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java/org/apache/lucene/analysis/standard/ > > StandardTokenizerImpl.jflex is probably where you should start. > > Steve > > -Original Message- > From: Dmitry Kan [mailto:dmitry@gmail.com] > Sent: Wednesday, July 06, 2011 3:23 AM > To: solr-user@lucene.apache.org > Subject: solr.StandardTokenizerFactory: more info needed > > Hi all! > > solr.StandardTokenizerFactory -- is it possible to see the full description > of its behaviour for solr.1.4 somewhere? Wiki > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.StandardTokenizerFactory > is > very short.. > > -- > Regards, > > Dmitry Kan > -- Regards, Dmitry Kan
RE: solr.StandardTokenizerFactory: more info needed
Hi Dmitry, The underlying Lucene implementation is here: http://svn.apache.org/repos/asf/lucene/java/tags/lucene_2_9_1/src/java/org/apache/lucene/analysis/standard/ StandardTokenizerImpl.jflex is probably where you should start. Steve -Original Message- From: Dmitry Kan [mailto:dmitry@gmail.com] Sent: Wednesday, July 06, 2011 3:23 AM To: solr-user@lucene.apache.org Subject: solr.StandardTokenizerFactory: more info needed Hi all! solr.StandardTokenizerFactory -- is it possible to see the full description of its behaviour for solr.1.4 somewhere? Wiki http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.StandardTokenizerFactory is very short.. -- Regards, Dmitry Kan
