While recreating it with a fresh schema, I realized that this was a case of a very, very stupid user error during configuring the cores.
I setup the testing cores with the wrong configset, and then proceeded to edit the schema in the right configset. So, the field was actually stored by default, but I wasn't attempting to retrieve it so I never realized it was being stored since I ended up looking at the wrong schema. I switched the config sets and everything works as expected. Any atomic updates clear out the indexed values for the non-stored field. Thanks for bearing with me. Chris On Thu, Apr 27, 2017 at 11:23 AM Chris Ulicny <culicny@iq.media> wrote: > I'm sending commit=true with every update while testing. I'll write up the > tests and see if someone else can reproduce it. > > On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson <erickerick...@gmail.com> > wrote: > >> bq: but is there any possibility that the values stick around until >> there is a segment merge for some strange reason >> >> There better not be or it's a bug. Things will stick around until >> you issue a commit, is there any chance that's the problem? >> >> If you can document the exact steps, maybe we can reproduce >> the issue and raise a JIRA. >> >> Best, >> Erick >> >> On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <culicny@iq.media> wrote: >> > Yeah, something's not quite right somewhere. We never even considered >> > in-place updates an option since it requires the fields to be >> non-indexed >> > and non-stored. Our schemas never have any field that satisfies those >> two >> > conditions let alone the other necessary ones. >> > >> > I went ahead and tested the atomic updates on different textfields, and >> I >> > still can't get the indexed but not-stored othertext_field to >> disappear. So >> > far set, add, and remove updates do not change it regardless of what the >> > fields are in the atomic update. >> > >> > It would be extraordinarily useful if this update behavior is now >> expected >> > (but not currently documented) functionality. >> > >> > I'm not too familiar with the nitty-gritty details of merging segment >> > files, but is there any possibility that the values stick around until >> > there is a segment merge for some strange reason? >> > >> > On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <dorian.ho...@gmail.com> >> > wrote: >> > >> >> @Chris, >> >> According to doc-link-above, only INC,SET are in-place-updates. And >> only >> >> when they're not indexed/stored, while your 'integer-field' is. So >> still >> >> shenanigans in there somewhere (docs,your-code,your-test,solr-code). >> >> >> >> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <culicny@iq.media> >> wrote: >> >> >> >> > That's probably it then. None of the atomic updates that I've tried >> have >> >> > been on TextFields. I'll give the TextField atomic update to verify >> that >> >> it >> >> > will clear the other field. >> >> > >> >> > Has this functionality been consistent since atomic updates were >> >> > introduced, or is this a side effect of some other change? It'd be >> very >> >> > convenient for us to use this functionality as it currently works, >> but if >> >> > it's something that prevents us from upgrading versions in the >> future, we >> >> > should probably avoid expecting it to work. >> >> > >> >> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya < >> >> > ichattopadhy...@gmail.com> wrote: >> >> > >> >> > > > Hmm, interesting. I can imagine that as long as you're updating >> >> > > > docValues fields, the other_text field would be there. But the >> >> instant >> >> > > > you updated a non-docValues field (text_field in your example) >> the >> >> > > > other_text field would disappear >> >> > > >> >> > > I can confirm this. When in-place updates to DV fields are done, >> the >> >> rest >> >> > > of the fields remain as they were. >> >> > > >> >> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson < >> >> erickerick...@gmail.com >> >> > > >> >> > > wrote: >> >> > > >> >> > > > Hmm, interesting. I can imagine that as long as you're updating >> >> > > > docValues fields, the other_text field would be there. But the >> >> instant >> >> > > > you updated a non-docValues field (text_field in your example) >> the >> >> > > > other_text field would disappear. >> >> > > > >> >> > > > I DO NOT KNOW this for a fact, but I'm asking people who do. >> >> > > > >> >> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha < >> >> dorian.ho...@gmail.com> >> >> > > > wrote: >> >> > > > > There are In Place Updates, but according to docs they stll >> >> shouldn't >> >> > > > work >> >> > > > > in your case: >> >> > > > > https://cwiki.apache.org/confluence/display/solr/ >> >> > > > Updating+Parts+of+Documents >> >> > > > > >> >> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny >> <culicny@iq.media> >> >> > > wrote: >> >> > > > > >> >> > > > >> That's the thing I'm curious about though. As I mentioned in >> the >> >> > first >> >> > > > >> post, I've already tried a few tests, and the value seems to >> still >> >> > be >> >> > > > >> present after an atomic update. >> >> > > > >> >> >> > > > >> I haven't exhausted all possible atomic updates, but 'set' and >> >> 'add' >> >> > > > seem >> >> > > > >> to preserve the non-stored text field. >> >> > > > >> >> >> > > > >> Thanks, >> >> > > > >> Chris >> >> > > > >> >> >> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha < >> >> > dorian.ho...@gmail.com> >> >> > > > >> wrote: >> >> > > > >> >> >> > > > >> > You'll lose the data in that field. Try doing a commit and >> it >> >> > should >> >> > > > >> > happen. >> >> > > > >> > >> >> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny >> <culicny@iq.media >> >> > >> >> > > > wrote: >> >> > > > >> > >> >> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by >> >> default >> >> > > > now. >> >> > > > >> > > That's very convenient and probably makes a lot of the >> schemas >> >> > > we've >> >> > > > >> been >> >> > > > >> > > making excessively verbose. >> >> > > > >> > > >> >> > > > >> > > This is on 6.3.0. Do you know what the first version was >> that >> >> > they >> >> > > > >> added >> >> > > > >> > > the docValues by default for non-Text field? >> >> > > > >> > > >> >> > > > >> > > However, that shouldn't apply to this since I'm concerned >> >> with a >> >> > > > >> > non-stored >> >> > > > >> > > TextField without docValues enabled. >> >> > > > >> > > >> >> > > > >> > > Best, >> >> > > > >> > > Chris >> >> > > > >> > > >> >> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey < >> >> > apa...@elyograg.org >> >> > > > >> >> > > > >> > wrote: >> >> > > > >> > > >> >> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote: >> >> > > > >> > > > > Hello all, >> >> > > > >> > > > > >> >> > > > >> > > > > Suppose I have the following fields in a document and >> >> > populate >> >> > > > all >> >> > > > >> 4 >> >> > > > >> > > > fields >> >> > > > >> > > > > for every document. >> >> > > > >> > > > > >> >> > > > >> > > > > id: uniqueKey, indexed and stored >> >> > > > >> > > > > integer_field: indexed and stored >> >> > > > >> > > > > text_field: indexed and stored >> >> > > > >> > > > > othertext_field: indexed but not stored >> >> > > > >> > > > > >> >> > > > >> > > > > No default values, multivalues, docvalues, >> copyfields, or >> >> > any >> >> > > > other >> >> > > > >> > > > > properties set. >> >> > > > >> > > > >> >> > > > >> > > > You didn't indicate the Solr version. In recent Solr >> >> > versions, >> >> > > > most >> >> > > > >> > > > field classes other than TextField have docValues >> enabled by >> >> > > > default, >> >> > > > >> > > > even if the config is not mentioned on the field, and in >> >> those >> >> > > > >> > versions, >> >> > > > >> > > > docValues will take the place of stored if stored is >> false. >> >> > > > >> > > > >> >> > > > >> > > > Thanks, >> >> > > > >> > > > Shawn >> >> > > > >> > > > >> >> > > > >> > > > >> >> > > > >> > > >> >> > > > >> > >> >> > > > >> >> >> > > > >> >> > > >> >> > >> >> >> >
