RE: Atomic updates with nested documents

://www.qaware.de Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger Registergericht: München Handelsregisternummer: HRB 163761 --- -Ursprüngliche Nachricht- Von: Kaminski, Adi Gesendet: Sonntag, 7. Juni 2020 08:45 An: solr-user@lucene.apache.org Betreff: RE: Atomic updates

Re: Atomic updates with add-distinct in Solr 7 cloud

While checking this, I encountered a bug in master. Since you have mentioned that it works in Solr 8, the bug might be unrelated to your issue. I have raised a ticket to track this https://issues.apache.org/jira/browse/SOLR-14550 to track the bug Regards, Munendra S N On Mon, Jun 8, 2020 at 9:1

RE: Atomic updates with nested documents

Hi Ludger, We had the same issue with Solr 7.6, and after discussing with the community we've found out that this partial update of parent document without "harm" parent-child association can work only on Solr 8.1 or higher, and It also requires some prerequisites. See the below item and it's la

Re: Atomic Updates with PreAnalyzedField

https://issues.apache.org/jira/browse/SOLR-13850 On Wed, Oct 16, 2019 at 11:25 AM Mikhail Khludnev wrote: > Hello, Oleksandr. > It deserves JIRA, please raise one. > > On Tue, Oct 15, 2019 at 8:17 PM Oleksandr Drapushko > wrote: > > > Hello Community, > > > > I've discovered data loss bug and c

Re: Atomic Updates with PreAnalyzedField

Hello, Oleksandr. It deserves JIRA, please raise one. On Tue, Oct 15, 2019 at 8:17 PM Oleksandr Drapushko wrote: > Hello Community, > > I've discovered data loss bug and couldn't find any mention of it. Please > confirm this bug haven't been reported yet. > > > Description: > > If you try to upd

Re: Atomic updates and stored CopyTo destination fields

On 11/14/2018 10:35 AM, Jon Kjær Amundsen wrote: It is not that I want it. I just can't reproduce it even though I read it as an expected behaviour. So I wondered if something has been changed since the warning was written, or if I had misunderstood something. To my knowledge, nothing has chan

Re: Atomic updates and stored CopyTo destination fields

It is not that I want it. I just can't reproduce it even though I read it as an expected behaviour. So I wondered if something has been changed since the warning was written, or if I had misunderstood something. ons. d. 14. nov. 2018 17.09 skrev Erick Erickson : > I'm a little confused on what y

Re: Atomic updates and stored CopyTo destination fields

I'm a little confused on what you're trying. Say your source field is Y and your destination field X. Are you saying that you want your destination field X to contain both the old value of field Y and the new value of field Y when you atomically update that field Y? H, I'm actually not sure wh

Re: Atomic updates using solr-php-client

Sami Why not do the simple case first, with complete document updates. When you have that working, you can decide if you want atomic updates too. Cheers -- Rick On March 6, 2018 2:26:50 AM EST, Sami al Subhi wrote: >Thank you for replying, > >Yes that is the one. Unfortunately there is no docume

Re: Atomic updates using solr-php-client

Thank you for replying, Yes that is the one. Unfortunately there is no documentation for this library. I tried to implement other libraries but I couldn't get them running. This is the easiest library to implement but lacks support and documentation. Thank you and best regards, Sami -- Sent f

Re: Atomic updates using solr-php-client

On 3/5/2018 11:38 PM, Sami al Subhi wrote: I am using solr-php-client to interact with solr. I came this example that shows how to add docs: http://www.ayalon.ch/en/code-samples/solr-php-client-example I checked "service.php" file, m

Re: Atomic Updates : Performance Impact

Thanks Erick for the useful information. Will keep the below points in mind while designing my solution. Thanks, Uday On Sat, Feb 24, 2018 at 12:47 AM, Erick Erickson wrote: > bq: However if i dont have majority of other column data while doing update > operations, is it better to go with atomi

Re: Atomic Updates : Performance Impact

bq: However if i dont have majority of other column data while doing update operations, is it better to go with atomic update? I don't understand what you're asking. To use Atomic Updates, _every_ original field (i.e. any field that is _not_ the destination of a copyField directive) must be stored

Re: Atomic Updates : Performance Impact

Hello Erick, Thanks for the explanation. However if i dont have majority of other column data while doing update operations, is it better to go with atomic update? And also during the update process, if there is a simultaneous search request on the collection, will there be any lag in response?

Re: Atomic Updates : Performance Impact

The approximate amount of work will be very close to what it would take Solr to just index the documents from a client. Actually it puts a little _more_ of a load on Solr. In the case you do an Atomic Update, Solr has to 1> fetch all the stored fields from the index 2> construct a Solr document 3>

Re: Atomic Updates with SolrJ

Hello, as Amrit mentioned, I attached the schema.xml of such an index. Perhaps there is something to find in it. The responses of update and commit look quite normal: {responseHeader={status=0,QTime=2}} {responseHeader={status=0,QTime=14}} After committing the fields fld_BA1F56CD9C87419CB9A271D

Re: Atomic Updates with SolrJ

Hi Martin, I tested the same application SolrJ code on my system, it worked just fine on Solr 6.6.x. My Solrclient is "CloudSolrJClient", which I think doesn't make any difference. Can you show the response and field declarations if you are continuously facing the issue. Amrit Sarkar Search Engin

Re: atomic updates in conjunction with optimistic concurrency

luceneMatchVersion is something that cause differences. You may want to try 6.3 version to be on same page as Amrit's code. On Fri, Jul 21, 2017 at 6:08 PM, Hendrik Haddorp wrote: > Thanks for trying to reproduce my issue. > > I'm using a Solr Cloud, my collection was quite small, only a 50-500

Re: atomic updates in conjunction with optimistic concurrency

Thanks for trying to reproduce my issue. I'm using a Solr Cloud, my collection was quite small, only a 50-500 documents, with one shard and a replication factor of 3. I updated all of the documents in one request. Beside that the flow is pretty much like yours. The goal of my code was to add

Re: atomic updates in conjunction with optimistic concurrency

Hendrik, Ran a little test on 6.3, with infinite atomic updates with optimistic concurrency, cannot *reproduce*: List docs = new ArrayList<>(); > SolrInputDocument document = new SolrInputDocument(); > document.addField("id", String.valueOf(1)); > document.addField("external_version_field_s", Sys

Re: atomic updates in conjunction with optimistic concurrency

Hi, I can't find anything about this in the Solr logs. On the caller side I have this: Error from server at http://x_shard1_replica2: version conflict for x expected=1573538179623944192 actual=1573546159565176832 org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error f

Re: atomic updates in conjunction with optimistic concurrency

Hendrik, Can you list down the error snippet so that we can refer the code where exactly that is happening. Amrit Sarkar Search Engineer Lucidworks, Inc. 415-589-9269 www.lucidworks.com Twitter http://twitter.com/lucidworks LinkedIn: https://www.linkedin.com/in/sarkaramrit2 On Fri, Jul 21, 2017

Re: Atomic Updates

Been there, done that, got the t-shirt. Thanks for closing it out! Erick On Thu, Apr 27, 2017 at 10:29 AM, Chris Ulicny wrote: > While recreating it with a fresh schema, I realized that this was a case of > a very, very stupid user error during configuring the cores. > > I setup the testing core

Re: Atomic Updates

While recreating it with a fresh schema, I realized that this was a case of a very, very stupid user error during configuring the cores. I setup the testing cores with the wrong configset, and then proceeded to edit the schema in the right configset. So, the field was actually stored by default, b

Re: Atomic Updates

I'm sending commit=true with every update while testing. I'll write up the tests and see if someone else can reproduce it. On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson wrote: > bq: but is there any possibility that the values stick around until > there is a segment merge for some strange reas

Re: Atomic Updates

bq: but is there any possibility that the values stick around until there is a segment merge for some strange reason There better not be or it's a bug. Things will stick around until you issue a commit, is there any chance that's the problem? If you can document the exact steps, maybe we can repr

Re: Atomic Updates

Yeah, something's not quite right somewhere. We never even considered in-place updates an option since it requires the fields to be non-indexed and non-stored. Our schemas never have any field that satisfies those two conditions let alone the other necessary ones. I went ahead and tested the atomi

Re: Atomic Updates

@Chris, According to doc-link-above, only INC,SET are in-place-updates. And only when they're not indexed/stored, while your 'integer-field' is. So still shenanigans in there somewhere (docs,your-code,your-test,solr-code). On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny wrote: > That's probably it

Re: Atomic Updates

That's probably it then. None of the atomic updates that I've tried have been on TextFields. I'll give the TextField atomic update to verify that it will clear the other field. Has this functionality been consistent since atomic updates were introduced, or is this a side effect of some other chang

Re: Atomic Updates

> Hmm, interesting. I can imagine that as long as you're updating > docValues fields, the other_text field would be there. But the instant > you updated a non-docValues field (text_field in your example) the > other_text field would disappear I can confirm this. When in-place updates to DV fields

Re: Atomic Updates

Hmm, interesting. I can imagine that as long as you're updating docValues fields, the other_text field would be there. But the instant you updated a non-docValues field (text_field in your example) the other_text field would disappear. I DO NOT KNOW this for a fact, but I'm asking people who do.

Re: Atomic Updates

There are In Place Updates, but according to docs they stll shouldn't work in your case: https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny wrote: > That's the thing I'm curious about though. As I mentioned in the first > p

Re: Atomic Updates

That's the thing I'm curious about though. As I mentioned in the first post, I've already tried a few tests, and the value seems to still be present after an atomic update. I haven't exhausted all possible atomic updates, but 'set' and 'add' seem to preserve the non-stored text field. Thanks, Chr

Re: Atomic Updates

You'll lose the data in that field. Try doing a commit and it should happen. On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny wrote: > Thanks Shawn, I didn't realize docValues were enabled by default now. > That's very convenient and probably makes a lot of the schemas we've been > making excessive

Re: Atomic Updates

Thanks Shawn, I didn't realize docValues were enabled by default now. That's very convenient and probably makes a lot of the schemas we've been making excessively verbose. This is on 6.3.0. Do you know what the first version was that they added the docValues by default for non-Text field? However

Re: Atomic Updates

On 4/25/2017 1:40 PM, Chris Ulicny wrote: > Hello all, > > Suppose I have the following fields in a document and populate all 4 fields > for every document. > > id: uniqueKey, indexed and stored > integer_field: indexed and stored > text_field: indexed and stored > othertext_field: indexed but not

Re: Atomic Updates

All fields are being explicitly populated on the initial document load without copyFields, and the atomic updates come after. This situation actually came up while we were planning removing copyField properties from one of the fields in a schema for a new collection. On Tue, Apr 25, 2017 at 3:54

Re: Atomic Updates

How is "otherText" getting values in the first place? If it's the destination of a copyField directive, it'll be repopulated if the source of the copyField is stored=true. Best, Erick On Tue, Apr 25, 2017 at 12:40 PM, Chris Ulicny wrote: > Hello all, > > Suppose I have the following fields in a

Re: Atomic updates to increase single field bulk updates?

Well, "it depends". The Atomic update has to first go out to disk and decompress the original stored fields in 16K blocks, then overlay the atomic update on the uncompressed doc, then re-index the doc. 40K times in your example. So yes, the stream going to Solr will be smaller if you do atomic upd

Re: Atomic updates to increase single field bulk updates?

> I am aware of the requirements to use atomic updates, but as I understood, > those would not have a big impact on performance and only a slight increase > in index size? AFAIK there won't be a difference in index size between atomic updates and full updates, as the end result is the same. But

RE: Atomic updates to increase single field bulk updates?

: partial update or a complete document. Under the hood a partial update : is a complete object anyway. Using partial updates you gain a little : bandwidth at the expense of additional stored fields. FWIW: once SOLR-5944 lands in a released version, that won't always be true -- atomic updates

RE: Atomic updates to increase single field bulk updates?

Hello Sebastian, Except for the requirement to have all fields stored, there is from Solr/Lucene's point of view not much difference between indexing a partial update or a complete document. Under the hood a partial update is a complete object anyway. Using partial updates you gain a little ban

Re: Atomic updates and "stored"

Thanks Eric! Best, Mark On Mon, May 23, 2016 at 1:35 PM, Erick Erickson wrote: > Yes, currently when using Atomic updates _all_ fields > have to be stored, except the _destinations_ of copyField > directives. > > Yes, it will make your index bigger. The affects on speed are > probably minimal t

Re: Atomic updates and "stored"

Yes, currently when using Atomic updates _all_ fields have to be stored, except the _destinations_ of copyField directives. Yes, it will make your index bigger. The affects on speed are probably minimal though. The stored data is in your *.fdt and *.fdx segments files and are not referenced only t

Re: Atomic updates on multiple documents

Thank you everyone for the useful comments and observations. I’ll start using atomic updates in the way they’re intended to and, if I don’t get the performance we need, I’ll try with a custom RequestUpdateProcessor. On 17/09/2015 20:08, Alexandre Rafalovitch wrote: You could probably do this

Re: Atomic updates on multiple documents

You could probably do this as a RequestUpdateProcessor (a custom one) that would take your submitted document, run a query and expand it to a bunch of documents. So, do the ID mapping internally. But you would need the ID/uniqueKeys. Definitely nothing out of the box, that I can think of. Regards

Re: Atomic updates on multiple documents

We’re using SolrJ as well, but if I understood correctly I would need to have the uniqueKey values anyway, right? There’s no way to do what I want with one request. A simple outline is: 1. Get the uniqueKey values that match my query 2. Create a set of SolrInputDocument 3. Add the id and the ad

Re: Atomic updates on multiple documents

You need to do that programmatically. Using SolrJ would be not so difficult to do that in few line of codes. Be careful to the stored fields if you don't want to lose anything. Cheers 2015-09-17 17:48 GMT+01:00 Alfonso Muñoz-Pomer Fuentes : > You’re right, we’re not working with a uniqueKey and

Re: Atomic updates on multiple documents

Hi Highlight Shawn answer : " Solr does not have anything like the SQL syntax that lets you update all rows that match a WHERE clause." In particular when adding a document to Solr, you are actually adding a new Document to the index ( under the hood) . The atomic update is only a user friendly w

Re: Atomic updates on multiple documents

You’re right, we’re not working with a uniqueKey and I wasn’t aware of that requirement. What I’d like is to update the documents without having to retrieve all of them (or their unique ids). Basically, there are some data that all documents that match a query will share; for the sake of the e

Re: Atomic updates on multiple documents

On 9/17/2015 10:14 AM, Shawn Heisey wrote: > This assumes that the uniqueKey field is "id". Unless your uniqueKey > field is "author_s" (which is highly unlikely), the JSON that you used > will not work. Chances are that the request failed, that nothing happened. On my first reading, I did not cat

Re: Atomic updates on multiple documents

On 9/17/2015 9:48 AM, Alfonso Muñoz-Pomer Fuentes wrote: > Hi, I have a question regarding atomic updates on multiple documents. > > We’re using Solr 5.1.0, and I’m wondering if it’s possible to perform > an atomic update on multiple documents with one request. > > After having a look at http://yon

Re: Atomic Updates in SOLR

dated for 4.5 yet, but the gist of the examples is the same.) -- Jack Krupansky -Original Message- From: Jack Krupansky Sent: Wednesday, October 30, 2013 9:03 AM To: solr-user@lucene.apache.org Subject: Re: Atomic Updates in SOLR Unfortunately, atomic "add" is add to a "l

Re: Atomic Updates in SOLR

Unfortunately, atomic "add" is add to a "list" (append) rather than add to a "set" (only unique values). But, you can use the unique fields update processor (solr.UniqFieldsUpdateProcessorFactory) to de-dupe specified multivalued fields. See: http://lucene.apache.org/solr/4_5_1/solr-core/org/a

Re: Atomic Updates in SOLR

Think it'll be a good thing to have. I just created a JIRA for that. https://issues.apache.org/jira/browse/SOLR-5403 Will try and get to it soon. On Wed, Oct 30, 2013 at 4:28 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Ah I misread your email. You are actually sending the updat

Re: Atomic Updates in SOLR

Ah I misread your email. You are actually sending the update twice and asking about how to dedup the multi-valued field values. No I don't think we have an update processor which can do that. On Wed, Oct 30, 2013 at 4:18 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Perhaps you a

Re: Atomic Updates in SOLR

I am not sure if optimistic concurrency would help in deduplicating but yes, as Shalin points out, you'll be able to spot issues with your client code. On Wed, Oct 30, 2013 at 4:18 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Perhaps you are running the update request more than

Re: Atomic Updates in SOLR

Perhaps you are running the update request more than once accidentally? Can you try using optimistic update with _version_ while sending the update? This way, if some part of your code is making a duplicate request then Solr would throw an error. See https://cwiki.apache.org/confluence/display/so

Re: Atomic updates with solr cloud in solr 4.4

The field variant_count is stored and is not the target of a copyfield. However I did notice that we were setting the same coreNodeName on both the shards in core.properties. Removing this property fixed the issue and updates succeed. What role does this play in handling updates and why were oth

Re: Atomic updates with solr cloud in solr 4.4

I can't reproduce this. I tried starting up a 2 shard cluster and then followed the example here: http://yonik.com/solr/atomic-updates/ "book1" was on shard2 (port 7574) and everything still worked fine. > missing required field: variant_count Perhaps the problem is document specific... What can

Re: Atomic updates with solr cloud in solr 4.4

curl http://localhost:8983/solr/search/update -H 'Content-type:application/json' -d ' [ { "id": "c8cce27c1d8129d733a3df3de68dd675!c8cce27c1d8129d733a3df3de68dd675", "link_id_45454" : {"set":"abcdegff"} } ]' I have two collections search and meta. I want to do an update in the sea

Re: Atomic updates with solr cloud in solr 4.4

On Tue, Sep 17, 2013 at 10:47 AM, Sesha Sendhil Subramanian wrote: > I am using solr 4.4 in solr cloud configuration. When i try to 'set' a > field in a document using the update request handler, I get a 'missing > required field' error. Can you show the exact error message you get, and the updat

Re: Atomic updates and indexed fields

0:22 AM To: solr-user@lucene.apache.org Subject: Re: Atomic updates and indexed fields see: https://issues.apache.org/jira/browse/LUCENE-4258 I'm sure the people working on this would gladly get all the help they can. WARNING: I suspect (although I haven't looked myself) that this is v

Re: Atomic updates and indexed fields

see: https://issues.apache.org/jira/browse/LUCENE-4258 I'm sure the people working on this would gladly get all the help they can. WARNING: I suspect (although I haven't looked myself) that this is very hairy code . Ah excellent! Thanks! Exactly what I was looking for. Looks like this has been in

Re: Atomic updates and indexed fields

Bram: see: https://issues.apache.org/jira/browse/LUCENE-4258 I'm sure the people working on this would gladly get all the help they can. WARNING: I suspect (although I haven't looked myself) that this is very hairy code . bq: Making all fields stored is simply not possible from a performance poin

Re: atomic updates w/ double field

: I'm using solr 4.0 and I'm using an atomic update to increment a tdouble : 3 times with the same value (99.4). The third time it is incremented the : values comes out to 298.25. Has anyone seen this error or : how to fix it? Maybe I should use the regular double instead of a : td

Re: atomic updates fail with solrcloud , and real time get throwing NPE

Sounds like you should file a JIRA so this can be looked into before 4.2 comes out shortly. Mark Sent from my iPhone On Mar 3, 2013, at 10:54 PM, "mike st. john" wrote: > atomic updates are failing in solrcloud , unless the update is sent to the > shard where the doc resides. Real time get

Re: Atomic Updates, Payloads, Non-stored data

Yeah, there are a bunch, some of which have been around for years. It's not an easy problem given the structure of the index. Can't come up with the JIRAs right now, sorry. Best Erick On Sun, Dec 2, 2012 at 8:13 AM, yriveiro wrote: > Hi, > > Exists some issue open in the Solr Project about t

Re: Atomic Updates, Payloads, Non-stored data

Hi, Exists some issue open in the Solr Project about this issue? Thanks - Best regards -- View this message in context: http://lucene.472066.n3.nabble.com/Atomic-Updates-Payloads-Non-stored-data-tp4006678p4023789.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Atomic Updates, Payloads, Non-stored data

You can't. This is the whole "stacked segment" discussion But you might consider external file fields, sometimes they can work out... Best Erick On Mon, Sep 10, 2012 at 1:19 PM, jimtronic wrote: > Hi, > > I'm using payloads to tie a value to an attribute for a document -- eg a > user's rati

Re: Atomic updates behavior

> > Hi, > > We've been evaluating the atomic update feature in Solr4.0. It would be > great if anyone can shed some light on the following. > > #) There seems to be more than one way to pass data to UpdateJSON for > add/update. What would be the recommended way. > https://gist.github.com/3256587 OR