Wildcard Terms and total word or phrase count
Hi - This is my first Lucene project, my other search projects have used Solr. I would like to find the total number of WildCard terms in a set of documents with 0-N matches per document. I would prefer not have to open each document where a match is found. I need to be able to support wildcards but my requirements are somewhat flexible in about phrase search support. Whatever is easier. This is what I have so far. public static void main(String args[]) throws IOException, ParseException { Directory idx = FSDirectory.open(path); index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); Term term = new Term("Doc", "quar*"); WildcardQuery wc = new WildcardQuery(term); SpanQuery spanTerm = new SpanMultiTermQueryWrapper(wc); IndexReader indexReader = DirectoryReader.open(idx); System.out.println("Term freq=" + indexReader.totalTermFreq(term)); System.out.println("Term freq=" + indexReader.getSumTotalTermFreq("Doc")); IndexSearcher isearcher = new IndexSearcher(indexReader); IndexReaderContext indexReaderContext = isearcher.getTopReaderContext(); TermContext context = TermContext.build(indexReaderContext, term); TermStatistics termStatistics = isearcher.termStatistics(term, context); System.out.println("termStatics=" + termStatistics.totalTermFreq()); } Does anyone have any suggestions? totalTermFreq is zero, but when search using quartz we find matches. I'm searching the Quartz user's guide as an example. Thanks, Doug
Re: Wildcard Terms and total word or phrase count
It is because your index does not contain term quar* and this statistics function is not a query (you have to pass exact form of the term). To count terms that meet search criteria you may run search query with custom collector and count results. Or use normal search query returning TopDocs and just check totalHitCount (however, first option is faster because no results are gathered and sorted). MW Sent from Mi phone On 27 Nov 2015 22:06, "Kunzman, Douglas *" wrote: > Hi - > > This is my first Lucene project, my other search projects have used Solr. > I would like to find the total number of WildCard terms in a set of > documents with 0-N matches per document. > I would prefer not have to open each document where a match is found. I > need to be able to support wildcards but my requirements are somewhat > flexible in about phrase search support. > Whatever is easier. > > This is what I have so far. > >public static void main(String args[]) throws IOException, > ParseException { > Directory idx = FSDirectory.open(path); > index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); > > Term term = new Term("Doc", "quar*"); > > WildcardQuery wc = new WildcardQuery(term); > > SpanQuery spanTerm = new > SpanMultiTermQueryWrapper(wc); > IndexReader indexReader = DirectoryReader.open(idx); > > System.out.println("Term freq=" + indexReader.totalTermFreq(term)); > System.out.println("Term freq=" + > indexReader.getSumTotalTermFreq("Doc")); > > IndexSearcher isearcher = new IndexSearcher(indexReader); > > IndexReaderContext indexReaderContext = > isearcher.getTopReaderContext(); > TermContext context = TermContext.build(indexReaderContext, term); > TermStatistics termStatistics = isearcher.termStatistics(term, > context); > System.out.println("termStatics=" + > termStatistics.totalTermFreq()); > } > > Does anyone have any suggestions? totalTermFreq is zero, but when search > using quartz we find matches. > I'm searching the Quartz user's guide as an example. > > Thanks, > Doug > > > > > >
RE: Wildcard Terms and total word or phrase count
Everyone - Thanks for getting back to me. Unfortunately, in the sample code even when I pass a term with no wild cards and it is a string with multiple instances in my document the totalHitCount is never more than one. Does anyone have any ideas what I could be doing wrong? Thanks, Doug -Original Message- From: Michael Wilkowski [mailto:m...@silenteight.com] Sent: Sunday, November 29, 2015 3:38 AM To: java-user@lucene.apache.org Subject: Re: Wildcard Terms and total word or phrase count It is because your index does not contain term quar* and this statistics function is not a query (you have to pass exact form of the term). To count terms that meet search criteria you may run search query with custom collector and count results. Or use normal search query returning TopDocs and just check totalHitCount (however, first option is faster because no results are gathered and sorted). MW Sent from Mi phone On 27 Nov 2015 22:06, "Kunzman, Douglas *" wrote: > Hi - > > This is my first Lucene project, my other search projects have used Solr. > I would like to find the total number of WildCard terms in a set of > documents with 0-N matches per document. > I would prefer not have to open each document where a match is found. I > need to be able to support wildcards but my requirements are somewhat > flexible in about phrase search support. > Whatever is easier. > > This is what I have so far. > >public static void main(String args[]) throws IOException, > ParseException { > Directory idx = FSDirectory.open(path); > index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); > > Term term = new Term("Doc", "quar*"); > > WildcardQuery wc = new WildcardQuery(term); > > SpanQuery spanTerm = new > SpanMultiTermQueryWrapper(wc); > IndexReader indexReader = DirectoryReader.open(idx); > > System.out.println("Term freq=" + indexReader.totalTermFreq(term)); > System.out.println("Term freq=" + > indexReader.getSumTotalTermFreq("Doc")); > > IndexSearcher isearcher = new IndexSearcher(indexReader); > > IndexReaderContext indexReaderContext = > isearcher.getTopReaderContext(); > TermContext context = TermContext.build(indexReaderContext, term); > TermStatistics termStatistics = isearcher.termStatistics(term, > context); > System.out.println("termStatics=" + > termStatistics.totalTermFreq()); > } > > Does anyone have any suggestions? totalTermFreq is zero, but when search > using quartz we find matches. > I'm searching the Quartz user's guide as an example. > > Thanks, > Doug > > > > > >
Re: Wildcard Terms and total word or phrase count
You didn't post your code that creates the index. Make sure you are using a tokenized TextField rather than a single-token StringField. -- Jack Krupansky On Fri, Nov 27, 2015 at 4:06 PM, Kunzman, Douglas * < douglas.kunz...@fda.hhs.gov> wrote: > Hi - > > This is my first Lucene project, my other search projects have used Solr. > I would like to find the total number of WildCard terms in a set of > documents with 0-N matches per document. > I would prefer not have to open each document where a match is found. I > need to be able to support wildcards but my requirements are somewhat > flexible in about phrase search support. > Whatever is easier. > > This is what I have so far. > >public static void main(String args[]) throws IOException, > ParseException { > Directory idx = FSDirectory.open(path); > index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); > > Term term = new Term("Doc", "quar*"); > > WildcardQuery wc = new WildcardQuery(term); > > SpanQuery spanTerm = new > SpanMultiTermQueryWrapper(wc); > IndexReader indexReader = DirectoryReader.open(idx); > > System.out.println("Term freq=" + indexReader.totalTermFreq(term)); > System.out.println("Term freq=" + > indexReader.getSumTotalTermFreq("Doc")); > > IndexSearcher isearcher = new IndexSearcher(indexReader); > > IndexReaderContext indexReaderContext = > isearcher.getTopReaderContext(); > TermContext context = TermContext.build(indexReaderContext, term); > TermStatistics termStatistics = isearcher.termStatistics(term, > context); > System.out.println("termStatics=" + > termStatistics.totalTermFreq()); > } > > Does anyone have any suggestions? totalTermFreq is zero, but when search > using quartz we find matches. > I'm searching the Quartz user's guide as an example. > > Thanks, > Doug > > > > > >
RE: Wildcard Terms and total word or phrase count
Jack - Thanks a lot for taking the time to try and answer my question. From using Solr I knew that it needed to be a TextField. I'm including the entire unit tester as an attachment. Thanks, Doug -Original Message- From: Jack Krupansky [mailto:jack.krupan...@gmail.com] Sent: Sunday, November 29, 2015 12:18 PM To: java-user@lucene.apache.org Subject: Re: Wildcard Terms and total word or phrase count You didn't post your code that creates the index. Make sure you are using a tokenized TextField rather than a single-token StringField. -- Jack Krupansky On Fri, Nov 27, 2015 at 4:06 PM, Kunzman, Douglas * < douglas.kunz...@fda.hhs.gov> wrote: > Hi - > > This is my first Lucene project, my other search projects have used Solr. > I would like to find the total number of WildCard terms in a set of > documents with 0-N matches per document. > I would prefer not have to open each document where a match is found. I > need to be able to support wildcards but my requirements are somewhat > flexible in about phrase search support. > Whatever is easier. > > This is what I have so far. > >public static void main(String args[]) throws IOException, > ParseException { > Directory idx = FSDirectory.open(path); > index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); > > Term term = new Term("Doc", "quar*"); > > WildcardQuery wc = new WildcardQuery(term); > > SpanQuery spanTerm = new > SpanMultiTermQueryWrapper(wc); > IndexReader indexReader = DirectoryReader.open(idx); > > System.out.println("Term freq=" + indexReader.totalTermFreq(term)); > System.out.println("Term freq=" + > indexReader.getSumTotalTermFreq("Doc")); > > IndexSearcher isearcher = new IndexSearcher(indexReader); > > IndexReaderContext indexReaderContext = > isearcher.getTopReaderContext(); > TermContext context = TermContext.build(indexReaderContext, term); > TermStatistics termStatistics = isearcher.termStatistics(term, > context); > System.out.println("termStatics=" + > termStatistics.totalTermFreq()); > } > > Does anyone have any suggestions? totalTermFreq is zero, but when search > using quartz we find matches. > I'm searching the Quartz user's guide as an example. > > Thanks, > Doug > > > > > > - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Wildcard Terms and total word or phrase count
Hi Doug, your attachment is not available (likely security settings). Please put it in github or somewhere else and provide a link to download. MW On Mon, Nov 30, 2015 at 2:29 AM, Kunzman, Douglas * < douglas.kunz...@fda.hhs.gov> wrote: > > Jack - > > Thanks a lot for taking the time to try and answer my question. > > From using Solr I knew that it needed to be a TextField. > > I'm including the entire unit tester as an attachment. > > Thanks, > Doug > > -Original Message- > From: Jack Krupansky [mailto:jack.krupan...@gmail.com] > Sent: Sunday, November 29, 2015 12:18 PM > To: java-user@lucene.apache.org > Subject: Re: Wildcard Terms and total word or phrase count > > You didn't post your code that creates the index. Make sure you are using a > tokenized TextField rather than a single-token StringField. > > -- Jack Krupansky > > On Fri, Nov 27, 2015 at 4:06 PM, Kunzman, Douglas * < > douglas.kunz...@fda.hhs.gov> wrote: > > > Hi - > > > > This is my first Lucene project, my other search projects have used Solr. > > I would like to find the total number of WildCard terms in a set of > > documents with 0-N matches per document. > > I would prefer not have to open each document where a match is found. I > > need to be able to support wildcards but my requirements are somewhat > > flexible in about phrase search support. > > Whatever is easier. > > > > This is what I have so far. > > > >public static void main(String args[]) throws IOException, > > ParseException { > > Directory idx = FSDirectory.open(path); > > index("C:\\Users\\Douglas.Kunzman\\Desktop\\test_index"); > > > > Term term = new Term("Doc", "quar*"); > > > > WildcardQuery wc = new WildcardQuery(term); > > > > SpanQuery spanTerm = new > > SpanMultiTermQueryWrapper(wc); > > IndexReader indexReader = DirectoryReader.open(idx); > > > > System.out.println("Term freq=" + > indexReader.totalTermFreq(term)); > > System.out.println("Term freq=" + > > indexReader.getSumTotalTermFreq("Doc")); > > > > IndexSearcher isearcher = new IndexSearcher(indexReader); > > > > IndexReaderContext indexReaderContext = > > isearcher.getTopReaderContext(); > > TermContext context = TermContext.build(indexReaderContext, > term); > > TermStatistics termStatistics = isearcher.termStatistics(term, > > context); > > System.out.println("termStatics=" + > > termStatistics.totalTermFreq()); > > } > > > > Does anyone have any suggestions? totalTermFreq is zero, but when search > > using quartz we find matches. > > I'm searching the Quartz user's guide as an example. > > > > Thanks, > > Doug > > > > > > > > > > > > > > > - > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org >