Hi, On 11 August 2016 at 09:14, Michael Marth <mma...@adobe.com> wrote:
> To the second part of your mail: > > You touch on a couple of different topics: > A) a design where indexes are not stored in the repo - but specifically a > design that uses an external (shared) indexer like ES > B) indexing latency > C) embedded Lucene vs embedded ES > > I am not entirely sure what you are suggesting, TBH. On one hand you seem > to suggest to use an external indexer like ES (external to the repo) on the > other hand you mention embedded ES (which would lead to the repo and the > embedded ES be colocated in the same JVM IIUC). > ES can be run colocated in the same JVM, colocated in a cluster of the same JVMs or in a dedicated cluster of JVMs. Which mode is determined by the URL given to the Client. The Client provision a ES sever colocated in the same JVM if instructed to do so. > > In order to understand let me ask you this way: > A) Oak does support an external Solr(Cloud) instance as a shared indexer > that is external to the repo. Same could be done with an external ES (in > fact, Tommaso has written a POC for that). On the very high level question > whether the index and the repo should be separate: does this address your > concern? (meaning: if we leave the relative benefits of Solr vs ES aside > for a second) > The External Solr Cloud plugin does not have the same level of functionality as the Lucene plugin from a indexing and querying pov. If it did, and leaving the relative benefits of Solr vs ES aside for a second, yes, it does answer my concern, but see my last point at the end. If Tommaso has ported the Lucene plugin to ES, the has anyone tried to run it co-located or has there always been an assumption it required an external ES cluster ? > B) indexing latency is a very different concern. In the current design it > is relevant in deployments when there is high latency between the > persistence and Oak. OAK-4638 and OAK-4412 are about addressing this. I am > not sure in how far separating out the indexer from the repo persistence > would help. > Oak commit -> Lucene query latency is currently governed the the following steps. 1 Index on the Oak master 2 hard commit the segment 3 write the segment to the repository DS 4 perform an oak commit. 5 wait for the root document to sync (1s). On each slave when the root document is synced 6 transfer the segment from the DS to a local copy. 7 open the segment and merge into the index reader (may require segment merge and optimise) If the index is local a soft commit can be performed with zero latency, leaving only the root document sync to make the revision visible in the query results. On TarMK this would not be an issue as there is only 1 JVM. On DocumentMK, as you point out, every JVM having a complete copy of the index is also wrong. Both Solr Cloud and ES address this by sharding and replicating the indexes, so that all commits are soft, instant and real time. That introduces problems. When a new JVM leaves the cluster for whatever reason, its local copy must have been replicated somewhere. When a new JVM joins the cluster it must become part of a replica set. The soft commit introduces problems, as if the JVM dies everything not hard committed is lost. This is where Solr Cloud and ES diverge (IIUC). Both have a write ahead log containing changes since the last hard commit. Solr's WAL is low level segment information. ES WAL contains the update documents. This gives ES the edge in terms of data latency, but that's irrelevant here as the Oak root document sync imposes a 1s data latency. Making the indexes local and adopting the proven pattern used by both Solr Cloud and ES would. > C) you seem to suggest that an embedded ES has advantages over the > embedded Lucene. What I do not understand in that comparison where you > would store the index. If locally we would be back to the JR2 design. If > somewhere remote then why embed ES at all? > All the research and history says that a Lucene index must be stored locally to the Index Update operation and the Index Search operation and not shipped or shared between instances. For Solr Cloud that history started with Nutch which did segment shipping from a master indexer. IIRC it used NFS as its segment repository. Others used RDBM's, some used scp. Most were in the "nowhere near real time search, trivial scale" domain. IIUC The JR2 design put the index locally but made each index a complete copy of everything, with every JR2 instance performing indexing triggered by events provided by JR2. That is not how ES or Solr Cloud work. JR2 provided no scalability of the index update operation and no resilience in the event of a cluster change. If a JR2 instance died, and it was soft committing, IIRC everything not hard committed was lost. (no WAL, although perhaps the commit log provided that) Backing up the index was hard (but possible, repeated rsyncs worked under light load) New JVMs had to either be provisioned with a backup index or re-index from scratch. Many years ago I ran a JR2 cluster backed by MySQL on 16 machines supporting 25K users, 5K concurrent so, like many JR2 users experienced these issues. Others tried to run larger clusters. I think 200K with about 90K concurrent was the largest attempted, but it failed load tests. Most switched to ES or SolrCloud. When a new ES or Solr Cloud JVM is added it gets segments from other replica set members. No full reindex. No downtime. If the JVM crashed, it replays its WAL to recover to the last update from the last hard commit. AFAIK, although co-locating a standard Solr instance in a JVM is trivial, co-locating Solr Cloud is not trivial as it it requires Zookeeper and is not fully self contained. ES on the other hand is fully self contained and trivial to co-locate. There are some requirements for 100% HA. There must be at least 3 JVMs members to maintain 100% HA and I think ES recommends more if there is to be no reliance on backups at all. I have vastly simplified the details of ES and Solr Cloud, partly to keep the post short and partly because in the case of ES I haven't had to dig too deeply to make ES work for production, because I have not seen production problems. In the case of Solr Cloud I have read the docs and some of the code but not run it in serious production. (TechOps teams favoured ES over Solr Cloud, having had bad experiences with SolrCloud and good with ES. I gave both options.) There is one final point to make. If Oak is "chatty" talking to its lucene index making 100s of sub queries for each Oak query then running a sharded index may not be possible, as each query may require a round trip. I know the ES query language supports complex multistep queries, I don't know about Solr Cloud but I would expect/hope it does also. Best Regards Ian > > Thanks for clarifying > Cheers > Michael > > > > >I am reticent to disagree with you, but I feel I have no option, based on > >research, history and first hand experience over the past 10 years. > > > >Storing indexes in a repo is what Compass did from 2004 onwards, until > >after the third version they gave up trying to build a scalable and near > >real time search engine. Version 4 was a rerwite that became ElasticSearch > >0.4.0. The history is documented here > >https://en.wikipedia.org/wiki/Elasticsearch and was presented at Berlin > >Buzwords in 2010 with a detailed description of why each approach fails. I > >have shared this information before. I am not sharing it to confront. I am > >sharing it because it pains me to see Oak repeating history. I don't feel > I > >can stand by and watch in silence. > > > >If Oak does not want to use ES as a library, then learn from the history > as > >it addresses your concerns (1,2, + brick wall) and those of Davide, and > >satisfies the many of the other issues potentially eliminating property > >indexes completely. It will however, only ever be as NRT as the root > >document commit period (1s), well above the 100ms data latency a model > like > >used by ES delivers under production load. > > > > IMHO, the Hybrid approach being proposed is a step along the same history > >that Compass started treading in 2004. It is an innovative solution to a > >constrained problem space. > > > >Sorry if I sound like a broken record. I did exactly what Oak has done/is > >doing in 2006 onwards but without a vast user base was able to be more > >agile. > > > >Apache is about doing, not standing by, about fact not fiction, about > >evidence and reasoned argument. If there is any interest, I have an Oak > PoC > >somewhere that ports the Lucene index plugin to use embedded ES instances, > >1 per VM as an embedded ES cluster. It's not complete as I gave up on it > >when I realised data latency would be fixed by the Oak root document. My > >interest was proper real time indexing over the cluster. >
