Mikhail I haven't experimented further yet. I think that the previous experiment of issuing a commit to a specific core proved that all cores get the commit, so I don't think that this approach will work.
Thanks, /Martin On Tue, Nov 27, 2012 at 6:24 PM, Mikhail Khludnev < mkhlud...@griddynamics.com> wrote: > Martin, > > It's still not clear to me whether you solve the problem completely or > partially: > Does reducing number of cores free some resources for searching during > commit? > Does the commiting one-by-one core prevents the "freeze"? > > Thanks > > > On Thu, Nov 22, 2012 at 4:31 PM, Martin Koch <m...@issuu.com> wrote: > >> Mikhail >> >> To avoid freezes we deployed the patches that are now on the 4.1 trunk >> (bug >> 3985). But this wasn't good enough, because SOLR would still take very >> long >> to restart when that was necessary. >> >> I don't see how we could throw more hardware at the problem without making >> it worse, really - the only solution here would be *fewer* shards, not >> >> more. >> >> IMO it would be ideal if the lucene/solr community could come up with a >> good way of updating fields in a document without reindexing. This could >> be >> by linking to some external data store, or in the lucene/solr internals. >> If >> it would make things easier, a good first step would be to have >> dynamically >> updateable numerical fields only. >> >> /Martin >> >> On Wed, Nov 21, 2012 at 8:51 PM, Mikhail Khludnev < >> mkhlud...@griddynamics.com> wrote: >> >> > Martin, >> > >> > I don't think solrconfig.xml shed any light on. I've just found what I >> > didn't get in your setup - the way of how to explicitly assigning core >> to >> > collection. Now, I realized most of details after all! >> > Ball is on your side, let us know whether you have managed your cores to >> > commit one by one to avoid freeze, or could you eliminate pauses by >> > allocating more hardware? >> > Thanks in advance! >> > >> > >> > On Wed, Nov 21, 2012 at 3:56 PM, Martin Koch <m...@issuu.com> wrote: >> > >> > > Mikhail, >> > > >> > > PSB >> > > >> > > On Wed, Nov 21, 2012 at 10:08 AM, Mikhail Khludnev < >> > > mkhlud...@griddynamics.com> wrote: >> > > >> > > > On Wed, Nov 21, 2012 at 11:53 AM, Martin Koch <m...@issuu.com> >> wrote: >> > > > >> > > > > >> > > > > I wasn't aware until now that it is possible to send a commit to >> one >> > > core >> > > > > only. What we observed was the effect of curl >> > > > > localhost:8080/solr/update?commit=true but perhaps we should >> > experiment >> > > > > with solr/coreN/update?commit=true. A quick trial run seems to >> > indicate >> > > > > that a commit to a single core causes commits on all cores. >> > > > > >> > > > You should see something like this in the log: >> > > > ... SolrCmdDistributor .... Distrib commit to: ... >> > > > >> > > > Yup, a commit towards a single core results in a commit on all >> cores. >> > > >> > > >> > > > > >> > > > > >> > > > > Perhaps I should clarify that we are using SOLR as a black box; >> we do >> > > not >> > > > > touch the code at all - we only install the distribution WAR file >> and >> > > > > proceed from there. >> > > > > >> > > > I still don't understand how you deploy/launch Solr. How many jettys >> > you >> > > > start whether you have -DzkRun -DzkHost -DnumShards=2 or you >> specifies >> > > > shards= param for every request and distributes updates yourself? >> What >> > > > collections do you create and with which settings? >> > > > >> > > > We let SOLR do the sharding using one collection with 16 SOLR cores >> > > holding one shard each. We launch only one instance of jetty with the >> > > folllowing arguments: >> > > >> > > -DnumShards=16 >> > > -DzkHost=<zookeeperhost:port> >> > > -Xmx10G >> > > -Xms10G >> > > -Xmn2G >> > > -server >> > > >> > > Would you like to see the solrconfig.xml? >> > > >> > > /Martin >> > > >> > > >> > > > > >> > > > > >> > > > > > Also from my POV such deployments should start at least from >> *16* >> > > 4-way >> > > > > > vboxes, it's more expensive, but should be much better available >> > > during >> > > > > > cpu-consuming operations. >> > > > > > >> > > > > >> > > > > Do you mean that you recommend 16 hosts with 4 cores each? Or 4 >> hosts >> > > > with >> > > > > 16 cores? Or am I misunderstanding something :) ? >> > > > > >> > > > I prefer to start from 16 hosts with 4 cores each. >> > > > >> > > > >> > > > > >> > > > > >> > > > > > Other details, if you use single jetty for all of them, are you >> > sure >> > > > that >> > > > > > jetty's threadpool doesn't limit requests? is it large enough? >> > > > > > You have 60G and set -Xmx=10G. are you sure that total size of >> > cores >> > > > > index >> > > > > > directories is less than 45G? >> > > > > > >> > > > > > The total index size is 230 GB, so it won't fit in ram, but >> we're >> > > using >> > > > > an >> > > > > SSD disk to minimize disk access time. We have tried putting the >> EFF >> > > > onto a >> > > > > ram disk, but this didn't have a measurable effect. >> > > > > >> > > > > Thanks, >> > > > > /Martin >> > > > > >> > > > > >> > > > > > Thanks >> > > > > > >> > > > > > >> > > > > > On Wed, Nov 21, 2012 at 2:07 AM, Martin Koch <m...@issuu.com> >> > wrote: >> > > > > > >> > > > > > > Mikhail >> > > > > > > >> > > > > > > PSB >> > > > > > > >> > > > > > > On Tue, Nov 20, 2012 at 7:22 PM, Mikhail Khludnev < >> > > > > > > mkhlud...@griddynamics.com> wrote: >> > > > > > > >> > > > > > > > Martin, >> > > > > > > > >> > > > > > > > Please find additional question from me below. >> > > > > > > > >> > > > > > > > Simone, >> > > > > > > > >> > > > > > > > I'm sorry for hijacking your thread. The only what I've >> heard >> > > about >> > > > > it >> > > > > > at >> > > > > > > > recent ApacheCon sessions is that Zookeeper is supposed to >> > > > replicate >> > > > > > > those >> > > > > > > > files as configs under solr home. And I'm really looking >> > forward >> > > to >> > > > > > know >> > > > > > > > how it works with huge files in production. >> > > > > > > > >> > > > > > > > Thank You, Guys! >> > > > > > > > >> > > > > > > > 20.11.2012 18:06 пользователь "Martin Koch" <m...@issuu.com> >> > > > написал: >> > > > > > > > > >> > > > > > > > > Hi Mikhail >> > > > > > > > > >> > > > > > > > > Please see answers below. >> > > > > > > > > >> > > > > > > > > On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev < >> > > > > > > > > mkhlud...@griddynamics.com> wrote: >> > > > > > > > > >> > > > > > > > > > Martin, >> > > > > > > > > > >> > > > > > > > > > Thank you for telling your own "war-story". It's really >> > > useful >> > > > > for >> > > > > > > > > > community. >> > > > > > > > > > The first question might seems not really conscious, but >> > > would >> > > > > you >> > > > > > > tell >> > > > > > > > me >> > > > > > > > > > what blocks searching during EFF reload, when it's >> > triggered >> > > by >> > > > > > > handler >> > > > > > > > or >> > > > > > > > > > by listener? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > We continuously index new documents using CommitWithin to >> get >> > > > > regular >> > > > > > > > > commits. However, we observed that the EFFs were not >> re-read, >> > > so >> > > > we >> > > > > > had >> > > > > > > > to >> > > > > > > > > do external commits (curl '.../solr/update?commit=true') >> to >> > > force >> > > > > > > reload. >> > > > > > > > > When this is done, solr blocks. I can't tell you exactly >> why >> > > it's >> > > > > > doing >> > > > > > > > > that (it was related to SOLR-3985). >> > > > > > > > >> > > > > > > > Is there a chance to get a thread dump when they are >> blocked? >> > > > > > > > >> > > > > > > > >> > > > > > > Well I could try to recreate the situation. But the setup is >> > fairly >> > > > > > simple: >> > > > > > > Create a large EFF in a largeish index with many shards. >> Issue a >> > > > > commit, >> > > > > > > and then try to do a search. Solr will not respond to the >> search >> > > > before >> > > > > > the >> > > > > > > commit has completed, and this will take a long time. >> > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > I don't really get the sentence about sequential commits >> > and >> > > > > number >> > > > > > > of >> > > > > > > > > > cores. Do I get right that file is replicated via >> > Zookeeper? >> > > > > > Doesn't >> > > > > > > it >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Again, this is observed behavior. When we issue a commit >> on a >> > > > > system >> > > > > > > with >> > > > > > > > a >> > > > > > > > > system with many solr cores using EFFs, the system blocks >> > for a >> > > > > long >> > > > > > > time >> > > > > > > > > (15 minutes). We do NOT use zookeeper for anything. The >> EFF >> > > is a >> > > > > > > symlink >> > > > > > > > > from each cores index dir to the actual file, which is >> > updated >> > > by >> > > > > an >> > > > > > > > > external process. >> > > > > > > > >> > > > > > > > Hold on, I asked about Zookeeper because the subj mentions >> > > > SolrCloud. >> > > > > > > > >> > > > > > > > Do you use SolrCloud, SolrShards, or these cores are just >> > > replicas >> > > > of >> > > > > > the >> > > > > > > > same index? >> > > > > > > > >> > > > > > > >> > > > > > > Ah - we use solr 4 out of the box, so I guess this is >> SolrCloud. >> > > I'm >> > > > a >> > > > > > bit >> > > > > > > unsure about the terminology here, but we've got a single >> index >> > > > divided >> > > > > > > into 16 shard. Each shard is hosted in a solr core. >> > > > > > > >> > > > > > > >> > > > > > > > Also, about simlink - Don't you share that file via some >> NFS? >> > > > > > > > >> > > > > > > > No, we generate the EFF on the local solr host (there is >> only >> > one >> > > > > > > physical >> > > > > > > host that holds all shards), so there is no need for NFS or >> > copying >> > > > > files >> > > > > > > around. No need for Zookeeper either. >> > > > > > > >> > > > > > > >> > > > > > > > how many cores you run per box? >> > > > > > > > >> > > > > > > This box is a 16-virtual core (8 hyperthreaded cores) with >> 60GB >> > of >> > > > > RAM. >> > > > > > We >> > > > > > > run 16 solr cores on this box in Jetty. >> > > > > > > >> > > > > > > >> > > > > > > > Do boxes has plenty of ram to cache filesystem beside of jvm >> > > heaps? >> > > > > > > > >> > > > > > > > Yes. We've allocated 10GB for jetty, and left the rest for >> the >> > > OS. >> > > > > > > >> > > > > > > >> > > > > > > > I assume you use 64 bit linux and mmap directory. Please >> > confirm >> > > > > that. >> > > > > > > > >> > > > > > > > >> > > > > > > We use 64-bit linux. I'm not sure about the mmap directory or >> > where >> > > > > that >> > > > > > > would be configured in solr - can you explain that? >> > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > causes scalability problem or long time to reload? Will >> it >> > > help >> > > > > if >> > > > > > > > we'll >> > > > > > > > > > have, let's say ExternalDatabaseField which will pull >> > values >> > > > from >> > > > > > > jdbc. >> > > > > > > > ie. >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > I think the possibility of having some fields being >> retrieved >> > > > from >> > > > > an >> > > > > > > > > external, dynamically updatable store would be really >> > > > interesting. >> > > > > > This >> > > > > > > > > could be JDBC, something in-memory like redis, or a NoSql >> > > product >> > > > > > (e.g. >> > > > > > > > > Cassandra). >> > > > > > > > >> > > > > > > > Ok. Let's have it in mind as a possible direction. >> > > > > > > > >> > > > > > > >> > > > > > > Alternatively, an API that would allow updating a single field >> > for >> > > a >> > > > > > > document might be an option. >> > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > why all cores can't read these values simultaneously? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Again, this is a solr implementation detail that I can't >> > answer >> > > > :) >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Can you confirm that IDs in the file is ordered by the >> > index >> > > > term >> > > > > > > > order? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Yes, we sorted the files (standard UNIX sort). >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > AFAIK it can impact load time. >> > > > > > > > > > >> > > > > > > > > Yes, it does >> > > > > > > > >> > > > > > > > Ok, I've got that you aware of it, and your IDs are just >> > strings, >> > > > not >> > > > > > > > integers. >> > > > > > > > >> > > > > > > > >> > > > > > > Yes, ids are strings. >> > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Regarding your post-query solution can you tell me if >> query >> > > > found >> > > > > > > 10000 >> > > > > > > > > > docs, but I need to display only first page with 100 >> rows, >> > > > > whether >> > > > > > I >> > > > > > > > need >> > > > > > > > > > to pull all 10K results to frontend to order them by the >> > > rank? >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > In our architecture, the clients query an API that >> generates >> > > the >> > > > > SOLR >> > > > > > > > > query, retrieves the relevant additional fields that we >> > needs, >> > > > and >> > > > > > > > returns >> > > > > > > > > the relevant JSON to the front-end. >> > > > > > > > > >> > > > > > > > > In our use case, results are returned from SOLR by the >> 10's, >> > > not >> > > > by >> > > > > > the >> > > > > > > > > 1000's, so it is a manageable job. Even so, if solr >> returned >> > > > > > thousands >> > > > > > > of >> > > > > > > > > results, it would be up to the implementation of the api >> to >> > > > augment >> > > > > > > only >> > > > > > > > > the results that needed to be returned to the front-end. >> > > > > > > > > >> > > > > > > > > Even so, patching up a JSON structure with 10000 results >> > should >> > > > be >> > > > > > > > > possible. >> > > > > > > > >> > > > > > > > You are right. I'm concerned anyway because retrieving whole >> > > result >> > > > > is >> > > > > > > > expensive, and not always possible. >> > > > > > > > >> > > > > > > > >> > > > > > > In our case, getting the whole result is almost impossible, >> > because >> > > > > that >> > > > > > > would be millions of documents, and returning the Nth result >> > seems >> > > to >> > > > > be >> > > > > > a >> > > > > > > quadratic (or worse) operation in SOLR. >> > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > I'm really appreciate if you comment on the questions >> > above. >> > > > > > > > > > PS: It's time to pitch, how much >> > > > > > > > > > https://issues.apache.org/jira/browse/SOLR-4085 >> "Commit-free >> > > > > > > > > > ExternalFileField" can help you? >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > It looks very interesting :) Does it make it possible to >> > > avoid >> > > > > > > > re-reading >> > > > > > > > > the EFF on every commit, and only re-read the values that >> > have >> > > > > > actually >> > > > > > > > > changed? >> > > > > > > > >> > > > > > > > >> > > > > > > > You don't need commit (in SOLR-4085) to reload file content, >> > but >> > > > > after >> > > > > > > > commit you need to read whole file and scan all key terms >> and >> > > > > postings. >> > > > > > > > That's because EFF sits on top of top level searcher. it's a >> > > > > Solr-like >> > > > > > > way. >> > > > > > > > In some future we might have per-segment EFF, in this case >> > > adding a >> > > > > > > segment >> > > > > > > > will trigger full file scan, but in the index only that new >> > > segment >> > > > > > will >> > > > > > > be >> > > > > > > > scanned. It should be faster. You know, straightforward >> sharing >> > > > > > internal >> > > > > > > > data structures between different index views/generations is >> > not >> > > > > > > possible. >> > > > > > > > If you are asking about applying delta changes on external >> file >> > > > > that's >> > > > > > > > something what we did ourselves http://goo.gl/P8GFq . This >> > > feature >> > > > > is >> > > > > > > much >> > > > > > > > more doubtful and vague, although it might be the next >> > > contribution >> > > > > > after >> > > > > > > > SOLR-4085. >> > > > > > > > >> > > > > > > > > >> > > > > > > > > /Martin >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch < >> > m...@issuu.com> >> > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Solr 4.0 does support using EFFs, but it might not >> give >> > you >> > > > > what >> > > > > > > > you're >> > > > > > > > > > > hoping fore. >> > > > > > > > > > > >> > > > > > > > > > > We tried using Solr Cloud, and have given up again. >> > > > > > > > > > > >> > > > > > > > > > > The EFF is placed in the parent of the index >> directory in >> > > > each >> > > > > > > core; >> > > > > > > > each >> > > > > > > > > > > core reads the entire EFF and picks out the IDs that >> it >> > is >> > > > > > > > responsible >> > > > > > > > > > for. >> > > > > > > > > > > >> > > > > > > > > > > In the current 4.0.0 release of solr, solr blocks >> > (doesn't >> > > > > answer >> > > > > > > > > > queries) >> > > > > > > > > > > while re-reading the EFF. Even worse, it seems that >> the >> > > time >> > > > to >> > > > > > > > re-read >> > > > > > > > > > the >> > > > > > > > > > > EFF is multiplied by the number of cores in use (i.e. >> the >> > > EFF >> > > > > is >> > > > > > > > re-read >> > > > > > > > > > by >> > > > > > > > > > > each core sequentially). The contents of the EFF >> become >> > > > active >> > > > > > > after >> > > > > > > > the >> > > > > > > > > > > first EXTERNAL commit (commitWithin does NOT work >> here) >> > > after >> > > > > the >> > > > > > > > file >> > > > > > > > > > has >> > > > > > > > > > > been updated. >> > > > > > > > > > > >> > > > > > > > > > > In our case, the EFF was quite large - around 450MB - >> and >> > > we >> > > > > use >> > > > > > 16 >> > > > > > > > > > shards, >> > > > > > > > > > > so when we triggered an external commit to force >> > > re-reading, >> > > > > the >> > > > > > > > whole >> > > > > > > > > > > system would block for several (10-15) minutes. This >> > won't >> > > > work >> > > > > > in >> > > > > > > a >> > > > > > > > > > > production environment. The reason for the size of the >> > EFF >> > > is >> > > > > > that >> > > > > > > we >> > > > > > > > > > have >> > > > > > > > > > > around 7M documents in the index; each document has a >> 45 >> > > > > > character >> > > > > > > > ID. >> > > > > > > > > > > >> > > > > > > > > > > We got some help to try to fix the problem so that the >> > > > re-read >> > > > > of >> > > > > > > the >> > > > > > > > EFF >> > > > > > > > > > > proceeds in the background (see >> > > > > > > > > > > here<https://issues.apache.org/jira/browse/SOLR-3985> >> > for >> > > > > > > > > > > a fix on the 4.1 branch). However, even though the >> > re-read >> > > > > > proceeds >> > > > > > > > in >> > > > > > > > > > the >> > > > > > > > > > > background, the time required to launch solr now >> takes at >> > > > least >> > > > > > as >> > > > > > > > long >> > > > > > > > > > as >> > > > > > > > > > > re-reading the EFFs. Again, this is not good enough >> for >> > our >> > > > > > needs. >> > > > > > > > > > > >> > > > > > > > > > > The next issue is that you cannot sort on EFF fields >> > > (though >> > > > > you >> > > > > > > can >> > > > > > > > > > return >> > > > > > > > > > > them as values using &fl=field(my_eff_field). This is >> > also >> > > > > fixed >> > > > > > in >> > > > > > > > the >> > > > > > > > > > 4.1 >> > > > > > > > > > > branch here < >> > > https://issues.apache.org/jira/browse/SOLR-4022 >> > > > >. >> > > > > > > > > > > >> > > > > > > > > > > So: Even after these fixes, EFF performance is not >> that >> > > > great. >> > > > > > Our >> > > > > > > > > > solution >> > > > > > > > > > > is as follows: The actual value of the popularity >> measure >> > > > (say, >> > > > > > > > reads) >> > > > > > > > > > that >> > > > > > > > > > > we want to report to the user is inserted into the >> search >> > > > > > response >> > > > > > > > > > > post-query by our query front-end. This value will >> then >> > be >> > > > the >> > > > > > > > > > > authoritative value at the time of the query. The >> value >> > of >> > > > the >> > > > > > > > popularity >> > > > > > > > > > > measure that we use for boosting in the ranking of the >> > > search >> > > > > > > results >> > > > > > > > is >> > > > > > > > > > > only updated when the value has changed enough so that >> > the >> > > > > impact >> > > > > > > on >> > > > > > > > the >> > > > > > > > > > > boost will be significant (say, more than 2%). This >> does >> > > > > require >> > > > > > > > frequent >> > > > > > > > > > > re-indexing of the documents that have significant >> > changes >> > > in >> > > > > the >> > > > > > > > number >> > > > > > > > > > of >> > > > > > > > > > > reads, but at least we won't have to update a >> document if >> > > it >> > > > > > moves >> > > > > > > > from, >> > > > > > > > > > > say, 1000000 to 1000001 reads. >> > > > > > > > > > > >> > > > > > > > > > > /Martin Koch - ISSUU - senior systems architect. >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni < >> > > > > > simo...@apache.org >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi all, >> > > > > > > > > > > > I'm planning to move a quite big Solr index to >> > SolrCloud. >> > > > > > > However, >> > > > > > > > in >> > > > > > > > > > > this >> > > > > > > > > > > > index, an external file field is used for popularity >> > > > ranking. >> > > > > > > > > > > > >> > > > > > > > > > > > Does SolrCloud supports external file fields? How >> does >> > it >> > > > > cope >> > > > > > > with >> > > > > > > > > > > > sharding and replication? Where should the external >> > file >> > > be >> > > > > > > placed >> > > > > > > > now >> > > > > > > > > > > that >> > > > > > > > > > > > the index folder is not local but in the cloud? >> > > > > > > > > > > > >> > > > > > > > > > > > Are there otherwise other best practices to deal >> with >> > the >> > > > use >> > > > > > > cases >> > > > > > > > > > > > external file fields were used for, like >> > > > popularity/ranking, >> > > > > in >> > > > > > > > > > > SolrCloud? >> > > > > > > > > > > > Custom ValueSources going to something external? >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance, >> > > > > > > > > > > > Simone >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Sincerely yours >> > > > > > > > > > Mikhail Khludnev >> > > > > > > > > > Principal Engineer, >> > > > > > > > > > Grid Dynamics >> > > > > > > > > > >> > > > > > > > > > <http://www.griddynamics.com> >> > > > > > > > > > <mkhlud...@griddynamics.com> >> > > > > > > > > > >> > > > > > > > 20.11.2012 18:06 пользователь "Martin Koch" <m...@issuu.com >> > >> > > > > написал: >> > > > > > > > >> > > > > > > > > Hi Mikhail >> > > > > > > > > >> > > > > > > > > Please see answers below. >> > > > > > > > > >> > > > > > > > > On Tue, Nov 20, 2012 at 12:28 PM, Mikhail Khludnev < >> > > > > > > > > mkhlud...@griddynamics.com> wrote: >> > > > > > > > > >> > > > > > > > > > Martin, >> > > > > > > > > > >> > > > > > > > > > Thank you for telling your own "war-story". It's really >> > > useful >> > > > > for >> > > > > > > > > > community. >> > > > > > > > > > The first question might seems not really conscious, but >> > > would >> > > > > you >> > > > > > > tell >> > > > > > > > > me >> > > > > > > > > > what blocks searching during EFF reload, when it's >> > triggered >> > > by >> > > > > > > handler >> > > > > > > > > or >> > > > > > > > > > by listener? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > We continuously index new documents using CommitWithin to >> get >> > > > > regular >> > > > > > > > > commits. However, we observed that the EFFs were not >> re-read, >> > > so >> > > > we >> > > > > > had >> > > > > > > > to >> > > > > > > > > do external commits (curl '.../solr/update?commit=true') >> to >> > > force >> > > > > > > reload. >> > > > > > > > > When this is done, solr blocks. I can't tell you exactly >> why >> > > it's >> > > > > > doing >> > > > > > > > > that (it was related to SOLR-3985). >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > I don't really get the sentence about sequential commits >> > and >> > > > > number >> > > > > > > of >> > > > > > > > > > cores. Do I get right that file is replicated via >> > Zookeeper? >> > > > > > Doesn't >> > > > > > > it >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Again, this is observed behavior. When we issue a commit >> on a >> > > > > system >> > > > > > > > with a >> > > > > > > > > system with many solr cores using EFFs, the system blocks >> > for a >> > > > > long >> > > > > > > time >> > > > > > > > > (15 minutes). We do NOT use zookeeper for anything. The >> EFF >> > > is a >> > > > > > > symlink >> > > > > > > > > from each cores index dir to the actual file, which is >> > updated >> > > by >> > > > > an >> > > > > > > > > external process. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > causes scalability problem or long time to reload? Will >> it >> > > help >> > > > > if >> > > > > > > > we'll >> > > > > > > > > > have, let's say ExternalDatabaseField which will pull >> > values >> > > > from >> > > > > > > jdbc. >> > > > > > > > > ie. >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > I think the possibility of having some fields being >> retrieved >> > > > from >> > > > > an >> > > > > > > > > external, dynamically updatable store would be really >> > > > interesting. >> > > > > > This >> > > > > > > > > could be JDBC, something in-memory like redis, or a NoSql >> > > product >> > > > > > (e.g. >> > > > > > > > > Cassandra). >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > why all cores can't read these values simultaneously? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Again, this is a solr implementation detail that I can't >> > answer >> > > > :) >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Can you confirm that IDs in the file is ordered by the >> > index >> > > > term >> > > > > > > > order? >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > Yes, we sorted the files (standard UNIX sort). >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > AFAIK it can impact load time. >> > > > > > > > > > >> > > > > > > > > Yes, it does. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > Regarding your post-query solution can you tell me if >> query >> > > > found >> > > > > > > 10000 >> > > > > > > > > > docs, but I need to display only first page with 100 >> rows, >> > > > > whether >> > > > > > I >> > > > > > > > need >> > > > > > > > > > to pull all 10K results to frontend to order them by the >> > > rank? >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > In our architecture, the clients query an API that >> generates >> > > the >> > > > > SOLR >> > > > > > > > > query, retrieves the relevant additional fields that we >> > needs, >> > > > and >> > > > > > > > returns >> > > > > > > > > the relevant JSON to the front-end. >> > > > > > > > > >> > > > > > > > > In our use case, results are returned from SOLR by the >> 10's, >> > > not >> > > > by >> > > > > > the >> > > > > > > > > 1000's, so it is a manageable job. Even so, if solr >> returned >> > > > > > thousands >> > > > > > > of >> > > > > > > > > results, it would be up to the implementation of the api >> to >> > > > augment >> > > > > > > only >> > > > > > > > > the results that needed to be returned to the front-end. >> > > > > > > > > >> > > > > > > > > Even so, patching up a JSON structure with 10000 results >> > should >> > > > be >> > > > > > > > > possible. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > I'm really appreciate if you comment on the questions >> > above. >> > > > > > > > > > PS: It's time to pitch, how much >> > > > > > > > > > https://issues.apache.org/jira/browse/SOLR-4085 >> "Commit-free >> > > > > > > > > > ExternalFileField" can help you? >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > It looks very interesting :) Does it make it possible to >> > > avoid >> > > > > > > > re-reading >> > > > > > > > > the EFF on every commit, and only re-read the values that >> > have >> > > > > > actually >> > > > > > > > > changed? >> > > > > > > > > >> > > > > > > > > /Martin >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Tue, Nov 20, 2012 at 1:16 PM, Martin Koch < >> > m...@issuu.com> >> > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Solr 4.0 does support using EFFs, but it might not >> give >> > you >> > > > > what >> > > > > > > > you're >> > > > > > > > > > > hoping fore. >> > > > > > > > > > > >> > > > > > > > > > > We tried using Solr Cloud, and have given up again. >> > > > > > > > > > > >> > > > > > > > > > > The EFF is placed in the parent of the index >> directory in >> > > > each >> > > > > > > core; >> > > > > > > > > each >> > > > > > > > > > > core reads the entire EFF and picks out the IDs that >> it >> > is >> > > > > > > > responsible >> > > > > > > > > > for. >> > > > > > > > > > > >> > > > > > > > > > > In the current 4.0.0 release of solr, solr blocks >> > (doesn't >> > > > > answer >> > > > > > > > > > queries) >> > > > > > > > > > > while re-reading the EFF. Even worse, it seems that >> the >> > > time >> > > > to >> > > > > > > > re-read >> > > > > > > > > > the >> > > > > > > > > > > EFF is multiplied by the number of cores in use (i.e. >> the >> > > EFF >> > > > > is >> > > > > > > > > re-read >> > > > > > > > > > by >> > > > > > > > > > > each core sequentially). The contents of the EFF >> become >> > > > active >> > > > > > > after >> > > > > > > > > the >> > > > > > > > > > > first EXTERNAL commit (commitWithin does NOT work >> here) >> > > after >> > > > > the >> > > > > > > > file >> > > > > > > > > > has >> > > > > > > > > > > been updated. >> > > > > > > > > > > >> > > > > > > > > > > In our case, the EFF was quite large - around 450MB - >> and >> > > we >> > > > > use >> > > > > > 16 >> > > > > > > > > > shards, >> > > > > > > > > > > so when we triggered an external commit to force >> > > re-reading, >> > > > > the >> > > > > > > > whole >> > > > > > > > > > > system would block for several (10-15) minutes. This >> > won't >> > > > work >> > > > > > in >> > > > > > > a >> > > > > > > > > > > production environment. The reason for the size of the >> > EFF >> > > is >> > > > > > that >> > > > > > > we >> > > > > > > > > > have >> > > > > > > > > > > around 7M documents in the index; each document has a >> 45 >> > > > > > character >> > > > > > > > ID. >> > > > > > > > > > > >> > > > > > > > > > > We got some help to try to fix the problem so that the >> > > > re-read >> > > > > of >> > > > > > > the >> > > > > > > > > EFF >> > > > > > > > > > > proceeds in the background (see >> > > > > > > > > > > here<https://issues.apache.org/jira/browse/SOLR-3985> >> > for >> > > > > > > > > > > a fix on the 4.1 branch). However, even though the >> > re-read >> > > > > > proceeds >> > > > > > > > in >> > > > > > > > > > the >> > > > > > > > > > > background, the time required to launch solr now >> takes at >> > > > least >> > > > > > as >> > > > > > > > long >> > > > > > > > > > as >> > > > > > > > > > > re-reading the EFFs. Again, this is not good enough >> for >> > our >> > > > > > needs. >> > > > > > > > > > > >> > > > > > > > > > > The next issue is that you cannot sort on EFF fields >> > > (though >> > > > > you >> > > > > > > can >> > > > > > > > > > return >> > > > > > > > > > > them as values using &fl=field(my_eff_field). This is >> > also >> > > > > fixed >> > > > > > in >> > > > > > > > the >> > > > > > > > > > 4.1 >> > > > > > > > > > > branch here < >> > > https://issues.apache.org/jira/browse/SOLR-4022 >> > > > >. >> > > > > > > > > > > >> > > > > > > > > > > So: Even after these fixes, EFF performance is not >> that >> > > > great. >> > > > > > Our >> > > > > > > > > > solution >> > > > > > > > > > > is as follows: The actual value of the popularity >> measure >> > > > (say, >> > > > > > > > reads) >> > > > > > > > > > that >> > > > > > > > > > > we want to report to the user is inserted into the >> search >> > > > > > response >> > > > > > > > > > > post-query by our query front-end. This value will >> then >> > be >> > > > the >> > > > > > > > > > > authoritative value at the time of the query. The >> value >> > of >> > > > the >> > > > > > > > > popularity >> > > > > > > > > > > measure that we use for boosting in the ranking of the >> > > search >> > > > > > > results >> > > > > > > > > is >> > > > > > > > > > > only updated when the value has changed enough so that >> > the >> > > > > impact >> > > > > > > on >> > > > > > > > > the >> > > > > > > > > > > boost will be significant (say, more than 2%). This >> does >> > > > > require >> > > > > > > > > frequent >> > > > > > > > > > > re-indexing of the documents that have significant >> > changes >> > > in >> > > > > the >> > > > > > > > > number >> > > > > > > > > > of >> > > > > > > > > > > reads, but at least we won't have to update a >> document if >> > > it >> > > > > > moves >> > > > > > > > > from, >> > > > > > > > > > > say, 1000000 to 1000001 reads. >> > > > > > > > > > > >> > > > > > > > > > > /Martin Koch - ISSUU - senior systems architect. >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Nov 19, 2012 at 3:22 PM, Simone Gianni < >> > > > > > simo...@apache.org >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi all, >> > > > > > > > > > > > I'm planning to move a quite big Solr index to >> > SolrCloud. >> > > > > > > However, >> > > > > > > > in >> > > > > > > > > > > this >> > > > > > > > > > > > index, an external file field is used for popularity >> > > > ranking. >> > > > > > > > > > > > >> > > > > > > > > > > > Does SolrCloud supports external file fields? How >> does >> > it >> > > > > cope >> > > > > > > with >> > > > > > > > > > > > sharding and replication? Where should the external >> > file >> > > be >> > > > > > > placed >> > > > > > > > > now >> > > > > > > > > > > that >> > > > > > > > > > > > the index folder is not local but in the cloud? >> > > > > > > > > > > > >> > > > > > > > > > > > Are there otherwise other best practices to deal >> with >> > the >> > > > use >> > > > > > > cases >> > > > > > > > > > > > external file fields were used for, like >> > > > popularity/ranking, >> > > > > in >> > > > > > > > > > > SolrCloud? >> > > > > > > > > > > > Custom ValueSources going to something external? >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance, >> > > > > > > > > > > > Simone >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Sincerely yours >> > > > > > > > > > Mikhail Khludnev >> > > > > > > > > > Principal Engineer, >> > > > > > > > > > Grid Dynamics >> > > > > > > > > > >> > > > > > > > > > <http://www.griddynamics.com> >> > > > > > > > > > <mkhlud...@griddynamics.com> >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Sincerely yours >> > > > > > Mikhail Khludnev >> > > > > > Principal Engineer, >> > > > > > Grid Dynamics >> > > > > > >> > > > > > <http://www.griddynamics.com> >> > > > > > <mkhlud...@griddynamics.com> >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Sincerely yours >> > > > Mikhail Khludnev >> > > > Principal Engineer, >> > > > Grid Dynamics >> > > > >> > > > <http://www.griddynamics.com> >> > > > <mkhlud...@griddynamics.com> >> > > > >> > > >> > >> > >> > >> > -- >> > Sincerely yours >> > Mikhail Khludnev >> > Principal Engineer, >> > Grid Dynamics >> > >> > <http://www.griddynamics.com> >> > <mkhlud...@griddynamics.com> >> > >> > > > > -- > Sincerely yours > Mikhail Khludnev > Principal Engineer, > Grid Dynamics > > <http://www.griddynamics.com> > <mkhlud...@griddynamics.com> > >
