Chris Hostetter-3 wrote: > > : updating your index between queries, so you may be reloading > : your cache every time. Are you updating or adding documents > : between queries and if so, how? > : > : If this is vaguely on target, have you tried firing up warmup queries > : after you update your index that involve your timestamp? > > based on the usecase, i'm not sure that that will really help -- it sounds > like the range query is alwasy based on the exact timestamp of the most > recent doc from the lat time this particular query was run -- which means > by definition that that timestamp changes every time the query is > run, so caching it is useless. > > skimming the thread, it's seems like the OP isn't using TrieDateField > (when asked if he was, he posted a followup about precision if he did use > it -- implying he is not currently). Switching to TrieDateField is > probably the only thing improvement possibly to make a significant > differnece in speeding up these one time queries. > > i've also seen no explanation of how big the index is, or what the OP's > definition of "slow" is (how fast are the queries with and w/o these > filters?). that type of information is fairly critical to being able to > offer performance suggestions. > > I'm also suspicious of hte entire line of questioning -- it smells like > there might be an XY Problem here. knowing what the ultimate goal that > lead to this timestamp based filter query appraoch might help us suggest > an alternate/better/faster solution. > > > > -Hoss > > > You are correct, first of all i haven't move yet to the TrieDateField, but i am still waiting to find out a bit more information about it, and there's not a lot of info, other then in the xml file. Second, i also think caching is not my problem, as the queries are usually of different time ranges. The index is pretty big, right now we have around 700M documents, the size of it on the disk is about 600GB. More then half of the documents are pretty short, 10-20 words, the others are around 300 words.
I'll explain my use case, so you'll know a bit more. I have an index that's being updated regularly, (every second i have 10 to 50 new documents, most of them are small) Every 30 minutes, i ask the index what are the documents that were added to it, since the last time i queried it, that match a certain criteria. >From time to time, once a week or so, i ask the index for ALL the documents that match that criteria. (i also do this for not only one query, but several) This is why i need the timestamp filter. The queries that don't have any time range, take a few seconds to finish, while the ones with time range, take a few minutes. Hope that helps understanding my situation, and i am open to any suggestion how to change the way things work, if it will improve performance. Thank you guys -- View this message in context: http://lucene.472066.n3.nabble.com/filter-query-on-timestamp-slowing-query-tp977280p980526.html Sent from the Solr - User mailing list archive at Nabble.com.
