Thanks Sean. I actually think I figured out what is causing the difference. When I create the UMLS install on my machine, I only install RxNorm and SNOMEDCT_US, so when I use the dictionaryCreator GUI, there are only those two sources on the left. I noticed in the screenshots on the wiki page for the dictionary creator GUI that many sources were installed, but only SNOMEDCT_US and RxNorm were selected. So, I tried installing all of the active UMLS set (but still only selecting RxNorm and SNOMEDCT_US in the dictionaryCreator GUI) and it made a difference as to which terms appeared in the final cTAKES dictionary. As an example, I now get the "DM" entry for diabetes. I don't know why this should make a difference, but it appears that it does.
Another odd observation related to this. In the sno_rx_2016ab file, I noticed there seems to be an error: INSERT INTO CUI_TERMS VALUES(11849,0,2,'diabete mellitus','diabete') The 's' is missing from diabetes. When I created my dictionary (from the restricted UMLS install, but still 2016ab) the cTAKES dictionary entry for that term is correct: INSERT INTO CUI_TERMS VALUES(11849,1,2,'diabetes mellitus','mellitus') When I created the dictionary from the full cTAKES install tonight, that error appeared again. Jeff On Mon, Jun 17, 2019 at 8:08 PM Finan, Sean < sean.fi...@childrens.harvard.edu> wrote: > Hi Jeff, > > Thanks for doing the research. Since the sno_rx_16ab was made 3+ years > ago I can't swear to any of those filter sets being exactly what was used. > > I think that the key to working with any project is to check the > dictionary against a project's needs. Fill in the gaps by either editing > the sql (.script) file or by adding a second dictionary. In smaller > "focus" projects I usually end up augmenting the default dictionary with a > small custom bsv dictionary to catch any known synonyms or terms that > aren't represented in the default. In projects requiring larger nets I > have built dictionaries that are horribly inclusive - 2 to 3 times the > sno_rx_16ab. > > Sean > ________________________________________ > From: Jeffrey Miller <jeff...@gmail.com> > Sent: Monday, June 17, 2019 4:39 PM > To: dev@ctakes.apache.org > Subject: Re: Differences in dictionary built with dictionaryBuilder and > sno_rx16ab from sourceforge [EXTERNAL] > > Thanks for following up Sean. I've looked into the links you sent along. > There are different groups of filters and it appears that the > dictionaryBuilder GUI is hardcoded to use the files in the "tiny" > directory. I don't think this is the set of filters used to make > sno_rx_16ab because the 'tiny' filter group contains "today" (today brand > veterinary product. 310367) in "UnwantedTexts.txt", but the > sno_rx_16ab.script file has "today" still in there. If you create a > dictionary with the dictionary builder, it does not include that term. > > I thought maybe the set of files under the "default" filter directory might > be the one used for the sno_rx_16ab package so I recompiled the > dictionaryCreator GUI to use the "default" filter files and created a new > snomed rxnorm dictionary from the 2016ab umls release, but the output is > still quite different that the packaged sno_rx_16ab dictionary. From > looking at diffs, it looks like there are a substantial number of additions > to the sno_rx_16ab, so much so that I really must be missing something. For > example, for CUI 12169 which describes a low sodium diet, there are about > 27 CUI terms in sno_rx_16ab.script, but in the script generated by the > dictionaryGUI there are only 7 (with the "tiny" or "default" filter > groups). > > On Sun, Jun 16, 2019 at 3:27 PM Remy Sanouillet <re...@foreseemed.com> > wrote: > > > Thanks for the clarifications, Sean. That was very enlightening. I look > > forward to the documentation (even if it entails some suffering on your > > part.) > > > > If/when you stumble on some idle time allowing you to implement the > manual > > edit panel, it would be nice to have it allow for re-partitioning the > > ontology. As you are very aware, UMLS CUIs and SNOMED do not always have > a > > one-to-one correspondence resulting in a CUI matching multiples SNOMEDs > or > > a SNOMED being mapped to several CUIs. > > > > In some cases, clinicians don't agree with that partitioning in > specialized > > contexts and the inheritance that ensues and would like to re-assign > them. > > > > Not holding my breath, but just something to keep in mind. > > > > Remy > > > > On Sun, Jun 16, 2019 at 7:16 AM Finan, Sean < > > sean.fi...@childrens.harvard.edu> wrote: > > > > > Hi Jeff, > > > > > > >1) ... > > > There are several collections of filter sets here: > > > > ctakes-gui-res\src\main\resources\org\apache\ctakes\gui\dictionary\data\ > > > > > > 2) ... > > > There is additional logic within the dictionary creator code: > > > ctakes-gui\src\main\java\org\apache\ctakes\gui\dictionary\ > > > > > > I haven't gone through it in a really long time, and without doing so > now > > > I can't enumerate the filters. I have family visiting, otherwise my > > > curiosity would force me to do so and get back to you. Honestly, it > > > should be documented somewhere, but writing (especially technical) is > > > pretty much my least favorite activity. > > > > > > Sean > > > > > > > > > p.s. > > > Please don't wait for it, but I am currently working on new dictionary > > > code and plan to introduce that in ctakes. Again, please don't wait > for > > it > > > as it is mixed in with other work and will not be available for several > > > months (if at all). > > > > > > > > > ________________________________________ > > > From: Jeffrey Miller <jeff...@gmail.com> > > > Sent: Sunday, June 16, 2019 9:49 AM > > > To: dev@ctakes.apache.org > > > Subject: Re: Differences in dictionary built with dictionaryBuilder and > > > sno_rx16ab from sourceforge [EXTERNAL] > > > > > > Hi Sean, > > > > > > Thanks for your response. I had two follow-up questions that would be > > very > > > helpful to understand if you have a few moments: > > > > > > 1) Are the specific filters used in the official sno_rx_16ab codified > > > anywhere so that I could reproduce them? > > > > > > 2) Do these filters explain all the changes? For example, when I use > the > > > dictionary creator to export sno_med and rx_norm, I only get "diabetes > > > mellitus" where as sno_rx_16ab contains both "diabetes" and "dm". > > > Especially with the addition of "dm" it feels like I must be missing a > > step > > > or a setting somewhere. > > > > > > Thanks! > > > Jeff > > > > > > On Sun, Jun 16, 2019 at 8:55 AM Finan, Sean < > > > sean.fi...@childrens.harvard.edu> wrote: > > > > > > > Hi all, > > > > > > > > The contents of the sno_rx_16ab are a dump of the umls 2016AB snomed > > and > > > > rxnorm terms with certain symantic types. Nothing was added, but > > > synonyms > > > > are filtered based upon various rules. For instance, unnecessary > > > suffixes > > > > are removed ("Wart (Finding)" -> "Wart"), really long terms are > > excluded > > > > ("can walk straight line with only minimal assistance"), terms with > > dose > > > or > > > > form are ignored and so forth. > > > > > > > > Some filters can be changed by adding/removing from > > > prefix/suffix/contains > > > > lists in plaintext files or by modifying the dictionary creator code. > > > > > > > > There was no manual curation (or nothing major). As Remy mentioned > > that > > > > requires a lot of attention and time. The dictionary database was > not > > > > intended to be perfect, just as good as possible without major > > > investment - > > > > and reproducible with updates to the umls. > > > > > > > > As the dictionary is released as a sql database, you should be able > to > > > add > > > > and remove fairly easily if sql savvy. I have long wanted to add a > > > "manual > > > > edit" panel to the dictionary gui, but haven't had the time. If > > anybody > > > > else would like to work on such a tool that would be tonic. > > > > > > > > Sean > > > > > > > > > > > > ________________________________________ > > > > From: Harish Kulkarni <harish.m.kulka...@gmail.com> > > > > Sent: Saturday, June 15, 2019 5:16 PM > > > > To: dev@ctakes.apache.org > > > > Subject: Re: Differences in dictionary built with dictionaryBuilder > and > > > > sno_rx16ab from sourceforge [EXTERNAL] > > > > > > > > unsubscribe > > > > > > > > On Sat, Jun 15, 2019 at 1:40 PM Remy Sanouillet < > re...@foreseemed.com> > > > > wrote: > > > > > > > > > Yes, I agree it would be nice because the tokenization that occurs > > when > > > > > creating the dictionaries from the releases make comparisons a bit > > > tricky > > > > > and is not 100% reversible. I would love to hear an answer to your > > > > > quandary. > > > > > > > > > > Remy > > > > > > > > > > On Sat, Jun 15, 2019 at 1:23 PM Jeffrey Miller <jeff...@gmail.com> > > > > wrote: > > > > > > > > > > > Thanks, I was curious if the cTAKES devs that created the > > sno_rx_16ab > > > > > > dictionary had put the differences applied to the default UMLS > > output > > > > > into > > > > > > version control in some form. I imagine the > > > > > > additions/synonyms/abbreviations that were added manually must > have > > > > been > > > > > > collected over time somewhere prior to merging them with 2016ab > > UMLS > > > > > > release? I basically want to recreate the default cTAKES 4.0.0 > > > release > > > > > with > > > > > > an additional ontology and the latest terms. I can likely come up > > > with > > > > a > > > > > > diff myself but was wondering if this was already maintained as > > part > > > of > > > > > > cTAKES. > > > > > > > > > > > > On Sat, Jun 15, 2019 at 12:24 PM Remy Sanouillet < > > > re...@foreseemed.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Yes, that's pretty much what we do too. Not only to enhance the > > > > > > dictionary, > > > > > > > but to put in corrections because, lo and behold, there are > some > > > > errors > > > > > > in > > > > > > > there!. As you know, an ontology is a constant curation job and > > > that > > > > > > > script, under SCM, allows you to isolate those changes and, if > > > > > necessary, > > > > > > > re-apply them to new versions. > > > > > > > > > > > > > > Remy > > > > > > > > > > > > > > On Sat, Jun 15, 2019 at 8:36 AM gandhi rajan < > > > > gandhiraja...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Jeff, > > > > > > > > > > > > > > > > As far as I know, maintaining a separate SQL script to add > > > > additional > > > > > > > > entries should work seamlessly. > > > > > > > > > > > > > > > > On Saturday, June 15, 2019, Jeffrey Miller < > jeff...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Remy. Does anyone know if these manually curated > > > > > > > > > modifications/synonyms are tracked anywhere (aside from the > > > > > > dictionary > > > > > > > > > itself) so they can be carried forward in future dictionary > > > > > updates? > > > > > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 4:28 PM Remy Sanouillet < > > > > > > re...@foreseemed.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > From my experience, it seems pretty obvious that > > sno_rx_16ab > > > > is a > > > > > > > > curated > > > > > > > > > > dictionary based on the SNOMED 2016AB release. It does > not > > > > > contain > > > > > > > the > > > > > > > > > full > > > > > > > > > > set but it has additional edits and synonyms that are > > pretty > > > > > useful > > > > > > > > > > (including 'dm'). > > > > > > > > > > > > > > > > > > > > We have had to manage those mods as an adjunct. > > > > > > > > > > > > > > > > > > > > Remy > > > > > > > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 1:03 PM Jeffrey Miller < > > > > > jeff...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > I have created a custom dictionary from the latest UMLS > > > > release > > > > > > > with > > > > > > > > > > > SNOMEDCT_US and RxNorm and I've noticed it seems to be > > > > > > generating > > > > > > > > > > .script > > > > > > > > > > > file with unexpected differences as compared to the > > > > sno_rx_16ab > > > > > > > file > > > > > > > > > > > available as part of the cTAKES release. Specifically, > > for > > > > > > > diabetes, > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > missing these two rows: > > > > > > > > > > > INSERT INTO CUI_TERMS VALUES(11849,0,1,'dm','dm') > > > > > > > > > > > INSERT INTO CUI_TERMS > > > VALUES(11849,0,1,'diabetes','diabetes') > > > > > > > > > > > > > > > > > > > > > > and only has this one: > > > > > > > > > > > INSERT INTO CUI_TERMS VALUES(11849,1,2,'diabetes > > > > > > > > mellitus','mellitus') > > > > > > > > > > > > > > > > > > > > > > The end result is that "diabetes" is not being picked > up > > in > > > > the > > > > > > > test > > > > > > > > > > text I > > > > > > > > > > > am running through- it requires the full 'diabetes > > > mellitus'. > > > > > > > > > > > > > > > > > > > > > > Is there any setting on the UMLS install side or the > > > ctTAKES > > > > > > > > dictionary > > > > > > > > > > > creator that could account for missing alternative > forms > > > like > > > > > > this? > > > > > > > > > I've > > > > > > > > > > > tried downloading the 2016AB release (which I think is > > the > > > > one > > > > > > used > > > > > > > > to > > > > > > > > > > > create the bundled sno_rx_16ab package?) and I am not > > > getting > > > > > the > > > > > > > > > > alternate > > > > > > > > > > > forms in that dictionary either. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Jeff > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > Gandhi > > > > > > > > > > > > > > > > "The best way to find urself is to lose urself in the service > > of > > > > > others > > > > > > > > !!!" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >