Re: DocValues: Which format is better Default or Memory?
So should I use Memory format? With Regards Aman Tandon On Thu, Jul 2, 2015 at 9:20 PM, Toke Eskildsen wrote: > Alessandro Benedetti wrote: > > DocValues is a strategy to store on the disk ( or in memory) the > > Un-inverted index for the field of interests. > > True. > > > This has been done to SPEED UP the faceting calculus using the "fc" > > algorithm, and improve the memory usage. > > Part of the reason was to speed up the _startup_ time for faceting. > > This is not the first time I read about people getting poorer > query-performance with DocValues. It does make sense: DocValues in the > index means that they compete with other files for disk caching and even > when they are fully cached, the UnInverted structure has a speed edge due > to being directly accessible as standard on-heap memory structures. > > The difference is likely to vary a great deal depending on concrete corpus > & hardware. > > - Toke Eskildsen >
Re: DocValues: Which format is better Default or Memory?
Alessandro Benedetti wrote: > DocValues is a strategy to store on the disk ( or in memory) the > Un-inverted index for the field of interests. True. > This has been done to SPEED UP the faceting calculus using the "fc" > algorithm, and improve the memory usage. Part of the reason was to speed up the _startup_ time for faceting. This is not the first time I read about people getting poorer query-performance with DocValues. It does make sense: DocValues in the index means that they compete with other files for disk caching and even when they are fully cached, the UnInverted structure has a speed edge due to being directly accessible as standard on-heap memory structures. The difference is likely to vary a great deal depending on concrete corpus & hardware. - Toke Eskildsen
Re: DocValues: Which format is better Default or Memory?
How are you testing? I'd do a couple of things: 1> turn of your queryResultCache (set its size to 0). 2> run multiple queries through something like jmeter 3> insure you've run enough warmup queries to load all your fields into memory. Basically, if this were always the case, I'd expect a _lot_ of people to be talking about it, I suspect there's something in your test methodology that's giving you innacurate results. On Thu, Jul 2, 2015 at 6:49 AM, Aman Tandon wrote: > Hi, > > I tried to query the without and with docValues, the query with docValues > was taking more time. Does it may be due to IO got involved as some data > will be in some file. > > Are you sure anything else could affect your times ? > > > Yes I am sure. We re-indexed the whole index of 40 Million records, to > implement the docValues to improve the speed. And I somehow managed to do > the simultaneous query for with/without docValues and I am getting higher > time with docValues by approx 200ms. As far as I could see it is increasing > as no of hits are increasing. > > *My configuration for docValue is:* > > "false" omitNorms="true" multiValued="false" /> > > > With Regards > Aman Tandon > > On Thu, Jul 2, 2015 at 3:15 PM, Alessandro Benedetti < > benedetti.ale...@gmail.com> wrote: > >> So first of all, >> DocValues is a strategy to store on the disk ( or in memory) the >> Un-inverted index for the field of interests. >> This has been done to SPEED UP the faceting calculus using the "fc" >> algorithm, and improve the memory usage. >> It is really weird that this is the cause of a degrading of performances. >> >> Building the DocValues should improve the query time to build facets, >> increasing the indexing time. >> Are you sure anything else could affect your times ? >> >> let's try to help you out ! >> >> 2015-07-02 4:19 GMT+01:00 Aman Tandon : >> >> > Hi, >> > >> > I tried to use the docValues to reduce the search time, but when I am >> using >> > the default format for docValues it is taking more time as compared to >> > normal faceting technique (without docValues). >> > >> > Should I go for Memory format or there is something missing? >> > >> > *Note:-* I am doing the indexing at every 10 minutes and I am using solr >> > 4.8.1 >> > >> > With Regards >> > Aman Tandon >> > >> >> >> >> -- >> -- >> >> Benedetti Alessandro >> Visiting card : http://about.me/alessandro_benedetti >> >> "Tyger, tyger burning bright >> In the forests of the night, >> What immortal hand or eye >> Could frame thy fearful symmetry?" >> >> William Blake - Songs of Experience -1794 England >>
Re: DocValues: Which format is better Default or Memory?
Anything wrong? With Regards Aman Tandon On Thu, Jul 2, 2015 at 4:19 PM, Aman Tandon wrote: > Hi, > > I tried to query the without and with docValues, the query with docValues > was taking more time. Does it may be due to IO got involved as some data > will be in some file. > > Are you sure anything else could affect your times ? > > > Yes I am sure. We re-indexed the whole index of 40 Million records, to > implement the docValues to improve the speed. And I somehow managed to do > the simultaneous query for with/without docValues and I am getting higher > time with docValues by approx 200ms. As far as I could see it is increasing > as no of hits are increasing. > > *My configuration for docValue is:* > > ="false" omitNorms="true" multiValued="false" /> > > > With Regards > Aman Tandon > > On Thu, Jul 2, 2015 at 3:15 PM, Alessandro Benedetti < > benedetti.ale...@gmail.com> wrote: > >> So first of all, >> DocValues is a strategy to store on the disk ( or in memory) the >> Un-inverted index for the field of interests. >> This has been done to SPEED UP the faceting calculus using the "fc" >> algorithm, and improve the memory usage. >> It is really weird that this is the cause of a degrading of performances. >> >> Building the DocValues should improve the query time to build facets, >> increasing the indexing time. >> Are you sure anything else could affect your times ? >> >> let's try to help you out ! >> >> 2015-07-02 4:19 GMT+01:00 Aman Tandon : >> >> > Hi, >> > >> > I tried to use the docValues to reduce the search time, but when I am >> using >> > the default format for docValues it is taking more time as compared to >> > normal faceting technique (without docValues). >> > >> > Should I go for Memory format or there is something missing? >> > >> > *Note:-* I am doing the indexing at every 10 minutes and I am using solr >> > 4.8.1 >> > >> > With Regards >> > Aman Tandon >> > >> >> >> >> -- >> -- >> >> Benedetti Alessandro >> Visiting card : http://about.me/alessandro_benedetti >> >> "Tyger, tyger burning bright >> In the forests of the night, >> What immortal hand or eye >> Could frame thy fearful symmetry?" >> >> William Blake - Songs of Experience -1794 England >> > >
Re: DocValues: Which format is better Default or Memory?
Hi, I tried to query the without and with docValues, the query with docValues was taking more time. Does it may be due to IO got involved as some data will be in some file. Are you sure anything else could affect your times ? Yes I am sure. We re-indexed the whole index of 40 Million records, to implement the docValues to improve the speed. And I somehow managed to do the simultaneous query for with/without docValues and I am getting higher time with docValues by approx 200ms. As far as I could see it is increasing as no of hits are increasing. *My configuration for docValue is:* With Regards Aman Tandon On Thu, Jul 2, 2015 at 3:15 PM, Alessandro Benedetti < benedetti.ale...@gmail.com> wrote: > So first of all, > DocValues is a strategy to store on the disk ( or in memory) the > Un-inverted index for the field of interests. > This has been done to SPEED UP the faceting calculus using the "fc" > algorithm, and improve the memory usage. > It is really weird that this is the cause of a degrading of performances. > > Building the DocValues should improve the query time to build facets, > increasing the indexing time. > Are you sure anything else could affect your times ? > > let's try to help you out ! > > 2015-07-02 4:19 GMT+01:00 Aman Tandon : > > > Hi, > > > > I tried to use the docValues to reduce the search time, but when I am > using > > the default format for docValues it is taking more time as compared to > > normal faceting technique (without docValues). > > > > Should I go for Memory format or there is something missing? > > > > *Note:-* I am doing the indexing at every 10 minutes and I am using solr > > 4.8.1 > > > > With Regards > > Aman Tandon > > > > > > -- > -- > > Benedetti Alessandro > Visiting card : http://about.me/alessandro_benedetti > > "Tyger, tyger burning bright > In the forests of the night, > What immortal hand or eye > Could frame thy fearful symmetry?" > > William Blake - Songs of Experience -1794 England >
Re: DocValues: Which format is better Default or Memory?
So first of all, DocValues is a strategy to store on the disk ( or in memory) the Un-inverted index for the field of interests. This has been done to SPEED UP the faceting calculus using the "fc" algorithm, and improve the memory usage. It is really weird that this is the cause of a degrading of performances. Building the DocValues should improve the query time to build facets, increasing the indexing time. Are you sure anything else could affect your times ? let's try to help you out ! 2015-07-02 4:19 GMT+01:00 Aman Tandon : > Hi, > > I tried to use the docValues to reduce the search time, but when I am using > the default format for docValues it is taking more time as compared to > normal faceting technique (without docValues). > > Should I go for Memory format or there is something missing? > > *Note:-* I am doing the indexing at every 10 minutes and I am using solr > 4.8.1 > > With Regards > Aman Tandon > -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti "Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry?" William Blake - Songs of Experience -1794 England
