Luke's analysis of Trie Dates

2013-07-18 Thread JohnRodey
I have a TrieDateField dynamic field setup in my schema, pretty standard...

  dynamicField name=*_tdt type=tdate  indexed=true  stored=false/

  fieldType name=tdate class=solr.TrieDateField omitNorms=true
precisionStep=6 positionIncrementGap=0/

In my code I only set one field, creation_tdt and I round it to the
nearest second before storing it.  However when I analyze it with Luke I
get:

lst name=fields
lst name=creation_tdt
str name=typetdate/str
str name=schemaIT--OF--/str
str name=dynamicBase*_tdt/str
str name=index(unstored field)/str
int name=docs22404/int
int name=distinct-1/int
lst name=topTerms
  int name=2013-07-18T13:37:33.696Z22404/int
  int name=1970-01-01T00:00:00Z22404/int
  int name=1970-01-01T00:00:00Z22404/int
  int name=2013-07-08T20:36:32.896Z22404/int
  int name=1970-01-01T00:00:00Z22404/int
  int name=2011-05-17T22:07:37.984Z22404/int
  int name=1970-01-01T00:00:00Z22404/int
  int name=2013-07-18T15:09:18.72Z16014/int
  int name=2013-07-18T15:04:56.576Z6390/int
  int name=2013-07-18T15:09:10.528Z1535/int
  int name=2013-07-18T15:09:55.584Z1459/int
  int name=2013-07-18T15:09:14.624Z1268/int
  int name=2013-07-18T15:09:06.432Z1193/int
  int name=2013-07-18T15:09:18.72Z1187/int
  int name=2013-07-18T15:09:51.488Z1152/int
  int name=2013-07-18T15:09:59.68Z1129/int
  int name=2013-07-18T15:09:02.336Z1089/int
  ...


So my questions is, where are all these entries coming from?  They are not
the dates I specified because they have millis, and my field isn't
multivalued, so the term counts dont add up (how could I have more than
22404 terms if I only have 22404 documents).  Why multiple
1970-01-01T00:00:00Z entries?

Is this somehow related to Trie fields and how they are indexed?

Thanks!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Luke-s-analysis-of-Trie-Dates-tp4078885.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: Luke's analysis of Trie Dates

2013-07-18 Thread Yonik Seeley
On Thu, Jul 18, 2013 at 12:53 PM, JohnRodey timothydd...@yahoo.com wrote:
 I have a TrieDateField dynamic field setup in my schema, pretty standard...

   dynamicField name=*_tdt type=tdate  indexed=true  stored=false/

   fieldType name=tdate class=solr.TrieDateField omitNorms=true
 precisionStep=6 positionIncrementGap=0/

 In my code I only set one field, creation_tdt and I round it to the
 nearest second before storing it.  However when I analyze it with Luke I
 get:

 lst name=fields
 lst name=creation_tdt
 str name=typetdate/str
 str name=schemaIT--OF--/str
 str name=dynamicBase*_tdt/str
 str name=index(unstored field)/str
 int name=docs22404/int
 int name=distinct-1/int
 lst name=topTerms
   int name=2013-07-18T13:37:33.696Z22404/int
   int name=1970-01-01T00:00:00Z22404/int
   int name=1970-01-01T00:00:00Z22404/int
   int name=2013-07-08T20:36:32.896Z22404/int
   int name=1970-01-01T00:00:00Z22404/int
   int name=2011-05-17T22:07:37.984Z22404/int
   int name=1970-01-01T00:00:00Z22404/int
   int name=2013-07-18T15:09:18.72Z16014/int
   int name=2013-07-18T15:04:56.576Z6390/int
   int name=2013-07-18T15:09:10.528Z1535/int
   int name=2013-07-18T15:09:55.584Z1459/int
   int name=2013-07-18T15:09:14.624Z1268/int
   int name=2013-07-18T15:09:06.432Z1193/int
   int name=2013-07-18T15:09:18.72Z1187/int
   int name=2013-07-18T15:09:51.488Z1152/int
   int name=2013-07-18T15:09:59.68Z1129/int
   int name=2013-07-18T15:09:02.336Z1089/int
   ...


 So my questions is, where are all these entries coming from?  They are not
 the dates I specified because they have millis, and my field isn't
 multivalued, so the term counts dont add up (how could I have more than
 22404 terms if I only have 22404 documents).  Why multiple
 1970-01-01T00:00:00Z entries?

 Is this somehow related to Trie fields and how they are indexed?

Yes, it's due to how trie fields are indexed (can have multiple
indexed tokens per logical value to speed up range queries).
If you want counts of values (as opposed to tokens), use faceting.

-Yonik
http://lucidworks.com