Hi all, I’ve prepared the pull request and submitted the issue.
https://issues.apache.org/jira/browse/SOLR-13863 <https://issues.apache.org/jira/browse/SOLR-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984249#comment-16984249> If anyone is interested in the feature, please write your opinion the thread. Ciao, Vincenzo -- mobile: 3498513251 skype: free.dev On 24 Oct 2019, at 13:59, Vincenzo D'Amore <v.dam...@gmail.com> wrote: Hi all, just to let you know that we started using spayload function in our quality environment for testing. Within a couple of weeks the feature will be deployed in production. Best regards, Vincenzo On Wed, Oct 23, 2019 at 4:31 PM Vincenzo D'Amore <v.dam...@gmail.com> wrote: > Hi Erick, yes, absolutely, it's a great pleasure for me contribute. > > On Wed, Oct 23, 2019 at 2:25 PM Erick Erickson <erickerick...@gmail.com> > wrote: > >> Bookmarked. Do you intend that this should be incorporated into Solr? If >> so, please raise a JIRA and link your PR in…. >> >> Thanks! >> Erick >> >> > On Oct 22, 2019, at 6:56 PM, Vincenzo D'Amore <v.dam...@gmail.com> >> wrote: >> > >> > Hi all, >> > >> > this evening I had some spare hour to spend in order to put everything >> > together in a repository. >> > >> > https://github.com/freedev/solr-payload-string-function-query >> > >> > >> > >> > On Tue, Oct 22, 2019 at 5:54 PM Vincenzo D'Amore <v.dam...@gmail.com> >> wrote: >> > >> >> Hi all, >> >> >> >> thanks for supporting. And many thanks whom have implemented >> >> the integration of the github Solr repository with the intellij IDE. >> >> To configure the environment and run the debugger I spent less than one >> >> hour, (and most of the time I had to wait the compilation). >> >> Solr and you guys really rocks together. >> >> >> >> What I've done: >> >> >> >> I was looking at the original payload function is defined into >> >> the ValueSourceParser, this function uses a FloatPayloadValueSource to >> >> return the value found. >> >> >> >> As said I wrote a new version of payload function that handles >> strings, I >> >> named it spayload, and basically is able to extract the string value >> from >> >> the payload. >> >> >> >> Given the former example where I have a multivalue field >> payloadCurrency >> >> >> >> payloadCurrency: [ >> >> "store1|USD", >> >> "store2|EUR", >> >> "store3|GBP" >> >> ] >> >> >> >> executing spayload(payloadCurrency,store2) returns "EUR", and so on for >> >> the remaining key/value in the field. >> >> >> >> To implement the spayload function, I've added a new ValueSourceParser >> >> instance to the list of defined functions and which returns >> >> a StringPayloadValueSource with the value inside (does the same thing >> of >> >> former FloatPayloadValueSource). >> >> >> >> That's all. As said, always beware of your code when works at first >> run. >> >> And really there was something wrong, initially I messed up in the >> >> conversion of the payload into String (bytes, offset, etc). >> >> Now it is fixed, or at least it seems to me. >> >> I see this function cannot be used in the sort, very likely the simple >> >> implementation of the StringPayloadValueSource miss something. >> >> >> >> As far as I understand I'm scratching the surface of this solution, >> there >> >> are few things I'm worried about. I have a bunch of questions, please >> be >> >> patient. >> >> This function returns an empty string "" when does not match any key, >> or >> >> should return an empty value? not sure about, what's the correct way to >> >> return an empty value? >> >> I wasn't able to find a test unit for the payload function in the >> tests. >> >> Could you give me few suggestion in order to test properly the >> >> implementation? >> >> In case the spayload is used on a different field type (i.e. the use >> >> spayload on a float payload) the behaviour is not handled. Can this >> >> function check the type of the payload content? >> >> And at last, what do you think, can this simple fix be interesting for >> the >> >> Solr community, may I try to submit a pull request or add a feature to >> JIRA? >> >> >> >> Best regards, >> >> Vincenzo >> >> >> >> >> >> On Mon, Oct 21, 2019 at 9:12 PM Erik Hatcher <erik.hatc...@gmail.com> >> >> wrote: >> >> >> >>> Yes. The decoding of a payload based on its schema type is what the >> >>> payload() function does. Your Payloader won't currently work >> well/legibly >> >>> for fields encoded numerically: >> >>> >> >>> >> >>> >> https://github.com/o19s/payload-component/blob/master/src/main/java/com/o19s/payloads/Payloader.java#L130 >> >>> < >> >>> >> https://github.com/o19s/payload-component/blob/master/src/main/java/com/o19s/payloads/Payloader.java#L130 >> >>>> >> >>> >> >>> I think that code could probably be slightly enhanced to leverage >> >>> PayloadUtils.getPayloadDecoder(fieldType) and use bytes if the field >> type >> >>> doesn't have a better decoder. >> >>> >> >>> Erik >> >>> >> >>> >> >>>> On Oct 21, 2019, at 2:55 PM, Eric Pugh < >> ep...@opensourceconnections.com> >> >>> wrote: >> >>>> >> >>>> Have you checked out >> >>>> https://github.com/o19s/payload-component >> >>>> >> >>>> On Mon, Oct 21, 2019 at 2:47 PM Erik Hatcher <erik.hatc...@gmail.com >> > >> >>> wrote: >> >>>> >> >>>>> How about a single field, with terms like: >> >>>>> >> >>>>> store1_USD|125.0 store2_EUR|220.0 store3_GBP|225.0 >> >>>>> >> >>>>> Would that do the trick? >> >>>>> >> >>>>> And yeah, payload decoding is currently limited to float and int >> with >> >>> the >> >>>>> built-in payload() function. We'd need a new way to pull out >> >>>>> textual/bytes payloads - like maybe a DocTransformer? >> >>>>> >> >>>>> Erik >> >>>>> >> >>>>> >> >>>>>> On Oct 21, 2019, at 9:59 AM, Vincenzo D'Amore <v.dam...@gmail.com> >> >>>>> wrote: >> >>>>>> >> >>>>>> Hi Erick, >> >>>>>> >> >>>>>> thanks for getting back to me. We started to use payloads because >> we >> >>> have >> >>>>>> the classical per-store pricing problem. >> >>>>>> Thousands of stores across and different prices. >> >>>>>> Then we found the payloads very useful started to use it for many >> >>>>> reasons, >> >>>>>> like enabling/disabling the product for such store, save the stock >> >>>>>> availability, or save the other info like buy/sell price, discount >> >>> rates, >> >>>>>> and so on. >> >>>>>> All those information are numbers, but stores can also be in >> different >> >>>>>> countries, I mean would be useful also have the currency and other >> >>>>>> attributes related to the store. >> >>>>>> >> >>>>>> Thinking about an alternative for payloads maybe I could use the >> >>> dynamic >> >>>>>> fields, well, I know it is ugly. >> >>>>>> >> >>>>>> Consider this hypothetical case where I have two field payload : >> >>>>>> >> >>>>>> payloadPrice: [ >> >>>>>> "store1|125.0", >> >>>>>> "store2|220.0", >> >>>>>> "store3|225.0" >> >>>>>> ] >> >>>>>> >> >>>>>> payloadCurrency: [ >> >>>>>> "store1|USD", >> >>>>>> "store2|EUR", >> >>>>>> "store3|GBP" >> >>>>>> ] >> >>>>>> >> >>>>>> with dynamic fields I could have different fields for each >> document. >> >>>>>> >> >>>>>> currency_store1_s: "USD" >> >>>>>> currency_store2_s: "EUR" >> >>>>>> currency_store3_s: "GBP" >> >>>>>> >> >>>>>> But how many dynamic fields like this can I have? more than >> thousands? >> >>>>>> >> >>>>>> Again, I've just started to look at solr-ocrhighlighting github >> >>> project >> >>>>> you >> >>>>>> suggested. >> >>>>>> Those seems have written their own payload object type where store >> ocr >> >>>>>> highlighting information. >> >>>>>> It seems interesting, I'll take a look immediately. >> >>>>>> >> >>>>>> Thanks again for your time. >> >>>>>> >> >>>>>> Best regards, >> >>>>>> Vincenzo >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Oct 21, 2019 at 2:55 PM Erick Erickson < >> >>> erickerick...@gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> This is one of those situations where I know a client did it, but >> >>> didn’t >> >>>>>>> see the code myself. >> >>>>>>> >> >>>>>>> So I can’t help much. >> >>>>>>> >> >>>>>>> Perhaps a good question at this point, though, is “why do you >> want to >> >>>>> add >> >>>>>>> string payloads anyway”? >> >>>>>>> >> >>>>>>> This isn’t the client, but it might give you some pointers: >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> >>> >> https://github.com/dbmdz/solr-ocrpayload-plugin/blob/master/src/main/java/de/digitalcollections/solr/plugin/components/ocrhighlighting/OcrHighlighting.java >> >>>>>>> >> >>>>>>> Best, >> >>>>>>> Erick >> >>>>>>> >> >>>>>>>> On Oct 21, 2019, at 6:37 AM, Vincenzo D'Amore < >> v.dam...@gmail.com> >> >>>>>>> wrote: >> >>>>>>>> >> >>>>>>>> Hi Erick, >> >>>>>>>> >> >>>>>>>> It seems I've reached a dead-point, or at least it seems looking >> at >> >>> the >> >>>>>>>> code, it seems I can't easily add a custom decoder: >> >>>>>>>> >> >>>>>>>> Looking at PayloadUtils class there is getPayloadDecoder method >> >>> invoked >> >>>>>>> to >> >>>>>>>> return the PayloadDecoder : >> >>>>>>>> >> >>>>>>>> public static PayloadDecoder getPayloadDecoder(FieldType >> fieldType) >> >>> { >> >>>>>>>> PayloadDecoder decoder = null; >> >>>>>>>> >> >>>>>>>> String encoder = getPayloadEncoder(fieldType); >> >>>>>>>> >> >>>>>>>> if ("integer".equals(encoder)) { >> >>>>>>>> decoder = (BytesRef payload) -> payload == null ? 1 : >> >>>>>>>> PayloadHelper.decodeInt(payload.bytes, payload.offset); >> >>>>>>>> } >> >>>>>>>> if ("float".equals(encoder)) { >> >>>>>>>> decoder = (BytesRef payload) -> payload == null ? 1 : >> >>>>>>>> PayloadHelper.decodeFloat(payload.bytes, payload.offset); >> >>>>>>>> } >> >>>>>>>> // encoder could be "identity" at this point, in the case of >> >>>>>>>> DelimitedTokenFilterFactory encoder="identity" >> >>>>>>>> >> >>>>>>>> // TODO: support pluggable payload decoders? >> >>>>>>>> >> >>>>>>>> return decoder; >> >>>>>>>> } >> >>>>>>>> >> >>>>>>>> Any advice to work around this situation? >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Mon, Oct 21, 2019 at 1:51 AM Erick Erickson < >> >>>>> erickerick...@gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> You’d need to write one. Payloads are generally intended to hold >> >>>>>>> numerics >> >>>>>>>>> you can then use in a function query to factor into the score… >> >>>>>>>>> >> >>>>>>>>> Best, >> >>>>>>>>> Erick >> >>>>>>>>> >> >>>>>>>>>> On Oct 20, 2019, at 4:57 PM, Vincenzo D'Amore < >> v.dam...@gmail.com >> >>>> >> >>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Sorry, I just realized that I was wrong in how I'm using the >> >>> payload >> >>>>>>>>>> function. >> >>>>>>>>>> Give that the payload function only handles a numeric (integer >> or >> >>>>>>> float) >> >>>>>>>>>> payload, could you suggest me an alternative function that >> handles >> >>>>>>>>> strings? >> >>>>>>>>>> If not, should I write one? >> >>>>>>>>>> >> >>>>>>>>>> On Sun, Oct 20, 2019 at 10:43 PM Vincenzo D'Amore < >> >>>>> v.dam...@gmail.com> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hi all, >> >>>>>>>>>>> >> >>>>>>>>>>> I'm trying to understand what I did wrong with a payload query >> >>> that >> >>>>>>>>>>> returns >> >>>>>>>>>>> >> >>>>>>>>>>> error: { >> >>>>>>>>>>> metadata: [ "error-class", >> >>> "org.apache.solr.common.SolrException", >> >>>>>>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], >> >>>>>>>>>>> msg: "No payload decoder found for field: colorCode", >> >>>>>>>>>>> code: 400 >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> I have reduced my problem in a little sample to show what >> >>> happens to >> >>>>>>> me. >> >>>>>>>>>>> Basically I have a document with a couple of payload fields >> one >> >>>>>>>>>>> delimited_payloads_string and one delimited_payloads_integer >> >>>>>>>>>>> >> >>>>>>>>>>> { >> >>>>>>>>>>> field_dps: "key|data", >> >>>>>>>>>>> field_dpi: "key|1", >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> When I execute this query solr returns as expected the payload >> >>> for >> >>>>> the >> >>>>>>>>> key >> >>>>>>>>>>> >> >>>>>>>>>>> q=*:*&fl=payload(field_dpi,key) >> >>>>>>>>>>> >> >>>>>>>>>>> { >> >>>>>>>>>>> payload(field_dpi,key): 1 >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> But for the strings there have to be something of different to >> >>> do, >> >>>>>>>>> because >> >>>>>>>>>>> I'm unable receive the payload value back. Executing this >> query, >> >>> as >> >>>>> in >> >>>>>>>>> the >> >>>>>>>>>>> short introduction of this post, I receive an error. >> >>>>>>>>>>> >> >>>>>>>>>>> ?q=*:*&fl=payload(field_dps,key) >> >>>>>>>>>>> >> >>>>>>>>>>> error: { >> >>>>>>>>>>> metadata: [ "error-class", >> >>> "org.apache.solr.common.SolrException", >> >>>>>>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], >> >>>>>>>>>>> msg: "No payload decoder found for field: colorCode", >> >>>>>>>>>>> code: 400 >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> Am I doing something wrong? How can I read strings payload >> data? >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks in advance for your time, >> >>>>>>>>>>> Vincenzo >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> Vincenzo D'Amore >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Vincenzo D'Amore >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Vincenzo D'Amore >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Vincenzo D'Amore >> >>>>> >> >>>>> >> >>> >> >>> >> >> >> >> -- >> >> Vincenzo D'Amore >> >> >> >> >> > >> > -- >> > Vincenzo D'Amore >> >> > > -- > Vincenzo D'Amore > > -- Vincenzo D'Amore
