Hi guys! Thanks alot for your suggestions and help - I really appreciate that! As we need e.g. the the price for sorting I think it must be in the index? Thus, I'm not shure that a key-value-store is the thing we are looking for as we need a good searchengine. Currently we are using serveral indices items, bids, etc. I think that ZOIE looks kind of interesting - I will check that out.
BTW do/can you recommend any "newbe" literature or resources for getting a bit more familiar w/ Solr/Lucene ? Thanks a lot again guys! -----Ursprüngliche Nachricht----- Von: Lance Norskog [mailto:goks...@gmail.com] Gesendet: Donnerstag, 18. März 2010 00:38 An: solr-user@lucene.apache.org Betreff: Re: Will Solr fit our needs? Another option is the ExternalFileField: http://www.lucidimagination.com/search/document/CDRG_ch04_4.4.4?q=ExternalFileField This lets you store the current prices for all items in a separate file. You can only use it in a function query, that is. But it does allow you to maintain one Solr index, which is very very worthwhile. On Wed, Mar 17, 2010 at 4:19 AM, Geert-Jan Brits <gbr...@gmail.com> wrote: > If you dont' plan on filtering/ sorting and/or faceting on fast-changing > fields it would be better to store them outside of solr/lucene in my > opinion. > > If you must: for indexing-performance reasons you will probably end up with > maintaining seperate indices (1 for slow-changing/static fields and 1 for > fast-changing-fields) . > You frequently commit the fast-changing -index to incorporate the changes > in current_price. Afterwards you have 2 options I believe: > > 1. use parallelreader to query the seperate indices directly. Afaik, this is > not (completely) integrated in Solr... I wouldn't recommend it. > 2. after you commit the fast-changing-index, merge with the static-index. > You're left with 1 fresh index, which you can push to your slave-servers. > (all this in regular interverals) > > Disadvatages: > - In any way, you must be very careful with maintaining multiple parallel > indexes with the purpose of treating them as one. For instance document > inserts must be done exactly in the same order, otherwise the indices go > 'out-of-sync' and are unusable. > - higher maintenance > - there is always a time-window in which the current_price values are stale. > If that's within reqs that's ok. > > The other path, which I recommend, would be to store the current_price > outside of solr (like you're currently doing) but instead of using a > relational db, try looking into persistent key-value stores. Many of them > exist and a lot of progress has been made in the last couple of years. For > simple key-lookups (what you need as far as I can tell) they really blow > every relational db out of the water (considering the same hardware of > course) > > We're currently using Tokyo Cabinet with the server-frontend Tokyo Tyrant > and seeing almost a 5x increased in lookup performance compared to our > previous kv-store memcachedDB which is based on BerkelyDB. Memcachedb was > already several times faster than our mysql-setup (although not optimally > tuned) . > > to sum things up: use the best tools for what they were meant to do. > > - index/search --> solr/ lucene without a doubt. > > - kv-lookup --> consensus is still forming, and a lot of players (with a lot > of different types of functionality) but if all you need is simple > key-value-lookup, I would go for Tokyo Cabinet (TC) / Tyrant at the moment. > Please note that TC and competitors aren't just some code/ hobby projects > but are usually born out of a real need at huge websites / social networks > such as TC which is born from mixi (big social network in Japan) . So at > least you're in good company.. > > for kv-stores I would suggest to begin your research at: > http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ > (beginning > 2009) > http://randomfoo.net/2009/04/20/some-notes-on-distributed-key-stores (half > 2009) > and get a feel of the kv-playing field. > > Hope this (pretty long) post helps, > Geert-Jan > > > 2010/3/17 Krzysztof Grodzicki <krzysztof.grodzi...@iterate.pl> > >> Hi Mortiz, >> >> You can take a look on the project ZOIE - >> http://code.google.com/p/zoie/. I think it's that what are you looking >> for. >> >> br >> Krzysztof >> >> On Wed, Mar 17, 2010 at 9:49 AM, Moritz Mädler <m...@moritz-maedler.de> >> wrote: >> > Hi List, >> > >> > we are running a marketplace which has about a comparable functionality >> like ebay (auctions, fixed-price items etc). >> > The items are placed on the market by users who want to sell their goods. >> > >> > Currently we are using Sphinx as an indexing engine, but, as Sphinx >> returns only document ids we have to make a >> > database-query to fetch the data to display. This massively decreases >> performance as we have to do two requests to >> > display data. >> > >> > I heard that Solr is able to return a complete dataset and we hope a >> switch to Solr can boost perfomance. >> > A critical question is left and i was not able to find a solution for it >> in the docs: Is it possible to update attributes directly in the >> > index? >> > An example for better illustration: >> > We have an index which holds all the auctions (containing auctionid, >> auction title) with its current prices(field: current_price). When a user >> places a new bid, >> > is it possible to update the attribute 'current_price' directly in the >> index so that we can fetch the current_price from Solr and not from the >> database? >> > >> > I hope you understood my problem. It would be kind if someone can point >> me to the right direction. >> > >> > Thanks alot! >> > >> > Moritz >> > -- Lance Norskog goks...@gmail.com
