Hi Hem, The negation detection problem within the clinical or medical context has been tackled for many researchers. At https://github.com/chapmanbe/negex you can find different implementations using different approaches. I have not tested all of them, but I have widely use this concrete implementation which is not even in github (actually I will upload it, it is licensed under Apache License): https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/negex/JavaConText.zip
This last algorithm usage is quite simple at the end. First of all, you need to define a set of Triggers terms of different types (negation terms, termination terms...). There is a good set of them for English already defined (totally hardcoded ) in the main Java Class. Then, negation detection is based on those terms, a target concept and a sentence (context). For instance: Patient denies cough but complains of headache. Your targets could be 'cough' and 'headache'. The main method contract is the following: public ArrayList<String> applyContext(String concept, String sentence) So you need to call twice, one for cough and one for headache with the same sentence context. This algorithm, under my experience, performs reasonably well. Anyway, the challenge here is basically to extract the context. If I have understood correctly, the target concept is basically defined through the user query. Specially if the queries could return a large number of results, you shouldn't be detecting the negation as a sort of results filtering. On contrary, I would suggest to detect the negations at indexing time and store the results into Solr Fields. That leads you to a different problem which basically is to detect the concepts first in the text Hope that helps On Wed, Nov 23, 2016 at 4:37 PM Alessandro Benedetti <abenede...@apache.org> wrote: > I add to the thread a friend of mine, > Rafa just presented at the Apachecon a medical system which deal exactly > with a negation engine ( he will publish the slides soon) > Hope it helps ! > > Cheers > > > On Wed, Nov 23, 2016 at 3:03 PM, Alexandre Rafalovitch <arafa...@gmail.com > > wrote: > > Well, then 'no' becomes a signal token. So, the question is how many > tokens after that it affects in its circle of negation? > > You could probably use something like > > https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-SurroundQueryParser > to say (if user said 'headache'). > -{!surround} 3w(not, headache) > > But I am not sure how this would work in terms of multi-term queries. > > Alternatively, you could transform your input with custom token filter > that, after seeing the term 'no', 'not', will just eat that and next > n? tokens. > > Or you could run the sentences through natural language recognition > and remove/mark noun phrases that are negative. > > What I am trying to say is that Solr can do a bunch of different > things for you. But you first need to translate your domain problem > into a much lower level pseudo-language problem that addresses your > needs. Including the edge-cases, which none of us can guess from your > description. Then you can implement it in Solr. > > Hope this helps, > Alex. > > ---- > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 24 November 2016 at 01:43, Hem Naidu > <hem.na...@teschglobal.com.invalid> wrote: > > Correct Alex. The use case is when provider searches on patient medical > information for certain symptoms, the mentions likes "no headache" , "no > blood loss", "not diabetic" should not show up in the search results. > > > > Thanks > > > > > > -----Original Message----- > > From: Alessandro Benedetti [mailto:benedetti.ale...@gmail.com] > > Sent: Wednesday, November 23, 2016 8:22 AM > > To: solr-user@lucene.apache.org > > Subject: Re: negation search help > > > > Now that I read better, do you mean that at indexing time those > negations must be recognized, in the way that they are no match ? > > > > Cheers > > > > On Wed, Nov 23, 2016 at 2:20 PM, Alessandro Benedetti < > benedetti.ale...@gmail.com> wrote: > > > >> Hi Hem, > >> are you expecting Solr to parse your natural language query out of the > >> box ? > >> Are you using any custom query parser ? > >> > >> If not, you need to follow the lucene Syntax to define engative queries. > >> > >> And be careful to the edge cases [1] . > >> > >> Cheers > >> > >> [1] https://wiki.apache.org/solr/NegativeQueryProblems > >> > >> On Wed, Nov 23, 2016 at 1:54 PM, Hem Naidu <hem.na...@teschglobal.com. > >> invalid> wrote: > >> > >>> Alex > >>> > >>> Whenever the keywords or sentence followed by "no", "not", etc should > >>> be excluded from the search results. Does solr support this feature? > >>> > >>> Thanks > >>> > >>> > >>> Sent from my iPhone > >>> > >>> > >>> > On Nov 23, 2016, at 12:09 AM, Alexandre Rafalovitch > >>> > <arafa...@gmail.com> > >>> wrote: > >>> > > >>> > How do you _know_ it is not 'apparent' ? Is it because it is > >>> > preceded by the keyword 'no'? Just that keyword? At what maximum > distance? > >>> > > >>> > Regards, > >>> > Alex > >>> > > >>> > On 23 Nov 2016 2:59 PM, "Hem Naidu" > >>> > <hem.na...@teschglobal.com.invalid> > >>> > wrote: > >>> > > >>> >> Gurus, > >>> >> > >>> >> I am new to Solr, I have a requirement to index entire pdf/word > >>> documents > >>> >> using Solr Tika. Which was successful and able to get the search > >>> results > >>> >> displayed. Now I need to fine tune the results or adjust index so > >>> >> the negative statements should be filtered out the results like my > >>> >> input > >>> text > >>> >> for index from the documents would be > >>> >> ----------------------------------- > >>> >> Fortunately no concurrent trauma was found In no apparent distress > >>> >> -------------------------------------- > >>> >> > >>> >> If user searches for concurrent trauma or distress the search > >>> >> engine > >>> should > >>> >> filter out the results as it not apparent symptom. > >>> >> > >>> >> Any help on whether Solr can do this? > >>> >> If so, do I need to adjust the index or build custom queries? > >>> >> > >>> >> Any help on this would be greatly appreciated ! > >>> >> > >>> >> Thanks > >>> >> > >>> >> > >>> >> > >>> > >> > >> > >> > >> -- > >> -------------------------- > >> > >> Benedetti Alessandro > >> Visiting card - http://about.me/alessandro_benedetti > >> Blog - http://alexbenedetti.blogspot.co.uk > >> > >> "Tyger, tyger burning bright > >> In the forests of the night, > >> What immortal hand or eye > >> Could frame thy fearful symmetry?" > >> > >> William Blake - Songs of Experience -1794 England > >> > > > > > > > > -- > > -------------------------- > > > > Benedetti Alessandro > > Visiting card - http://about.me/alessandro_benedetti > > Blog - http://alexbenedetti.blogspot.co.uk > > > > "Tyger, tyger burning bright > > In the forests of the night, > > What immortal hand or eye > > Could frame thy fearful symmetry?" > > > > William Blake - Songs of Experience -1794 England > > > > > > > -- > -------------------------- > > Benedetti Alessandro > Visiting card : http://about.me/alessandro_benedetti > > "Tyger, tyger burning bright > In the forests of the night, > What immortal hand or eye > Could frame thy fearful symmetry?" > > William Blake - Songs of Experience -1794 England >
