In high throughput situations that can be a problem. The entire packet has to be assembled and transmitted over the network. This can cause grief in many situations.
Not to mention that for “regular” queries, say using the /select or /query handlers and assuming you’re getting one or more stored-but-not-docValues fields, that means 2+m seeks of the disk, decompressing 2+m 16K blocks, creating the entier 2M+ packet in memory and transmitting it to the client. This doesn’t apply to, say, the /export handler upon which streaming is built. For a low-query-volume situations where there are just a few simultaneous queries you can get away with it. But it’s still an anti-pattern. Again, though, none of that is relevant if you’re using anything built on the /export handler, which includes almost all of the streaming capabilities. Best, Erick > On Jul 29, 2020, at 4:59 PM, David Hastings <hastings.recurs...@gmail.com> > wrote: > > "Oh, and returning 100K docs is an anti-pattern, if you really need that > many docs consider cursorMark and/or Streaming." > > er, i routinely ask for 2+ million records into a single file based on a > query. I mean not into a web application or anything, its meant to be > processed after the fact, but solr has no issue doing this > > > > On Wed, Jul 29, 2020 at 4:53 PM Erick Erickson <erickerick...@gmail.com> > wrote: > >> I don’t think there’s really a canned way to do what you’re asking. A >> custom DocTransformer would probably do the trick though. >> >> You could also create a custom QueryComponent that examined the docs being >> returned and inserted a blank field for a selected number of fields >> (possibly configurable in solrconfig.xml). >> >> Oh, and returning 100K docs is an anti-pattern, if you really need that >> many docs consider cursorMark and/or Streaming. >> >> Best, >> Erick >> >>> On Jul 29, 2020, at 2:55 PM, Teresa McMains <ter...@t14-consulting.com> >> wrote: >>> >>> Thanks so much. Is there any other way to return the data value if it >> exists, otherwise an empty string? I'm integrating this with a 3rd party >> app which I can't change. When the field is null it isn't showing up in the >> output. >>> >>> -----Original Message----- >>> From: Erick Erickson <erickerick...@gmail.com> >>> Sent: Wednesday, July 29, 2020 12:49 PM >>> To: solr-user@lucene.apache.org >>> Subject: Re: solr query returns items with spaces removed >>> >>> The “def” function goes after the _indexed_ value, so that’s what you’re >> getting back. Try just specifying “fl=INSTRUCTIONS”, and if the value is >> stored that should return the original field value before any analysis is >> done. >>> >>> Why are you using the def function? If the field is absent from the doc, >> nothing will be returned for that field, not even the name. Are you trying >> to insure that a blank field is returned if the field isn’t in the >> document? You can handle that on the client side if so… >>> >>> Best, >>> Erick >>> >>>> On Jul 29, 2020, at 10:34 AM, Teresa McMains <ter...@t14-consulting.com> >> wrote: >>>> >>>> _20_Instructions_And_Notes:def(INSTRUCTIONS,%22%22) >>> >> >>
