Thank you Sean for the detailed response. I think there was miscommunication from our end with the requirement. Your solution of adding spaces between the entries worked but it required the input text also to have the spaces. If the text comes in as 'F84.1' cTAKES didn't reckon the token but if the text came as 'F84 . 1' then cTAKES was recognizing the tokens for the below INSERT scripts.
INSERT INTO CUI_TERMS VALUES(4352,0,3, ‘F84 . 1’,’F84’) But we encountered a similar issue when we configured an INSERT entry as below for CPT codes, INSERT INTO CUI_TERMS VALUES(41154,0,1, ‘97112’,’97112’) Where 97112 is a CPT code(which usually doesn’t have decimals or '.'). We expected cTAKES to recognize the CPT code '97112' as a separate token but it didn't. Could you pls. advise us on why this issue came up. Is there something wrong in the configuration. Do we need to have something additional for cTAKES to recognize the code alone as a separate token Is there any other way in which we can try to get the respective ICD/CPT code of the identified annotation from cTAKES, like querying the CPT/ICD table using the fetched CUI? Kindly advise. Thanks & Regards Abad Ayyub Vnet: 406170 | Cell : +91-9447379028 -----Original Message----- From: Finan, Sean <sean.fi...@childrens.harvard.edu> Sent: Monday, September 14, 2020 9:35 PM To: dev@ctakes.apache.org Subject: Re: Building a new custom dictionary or Updating/Adding values to the existing dictionary in cTAKES [EXTERNAL] [External] Hi Abad, I think that you need to make only one minor change. ctakes uses "tokens" for identification and not the actual text. Tokenization turns text such as "F84.1" into "F84 . 1" The first token being F84, followed by a token encompassing '.' and another with '1'. The manner in which this is indicated in the .script file is by adding a space between each token. This makes the full entry: INSERT INTO CUI_TERMS VALUES(4352,0,3, ‘F84 . 1’,’F84’) Notice that the token length is now 3 and the full text contains the between-token spaces. This would carry forward for the other entries, such as: INSERT INTO CUI_TERMS VALUES(4352,3,4, ‘F84 . 1 pdd’, ‘pdd’) Sean ________________________________ From: abad.ay...@cognizant.com <abad.ay...@cognizant.com> Sent: Monday, September 14, 2020 11:32 AM To: dev@ctakes.apache.org Subject: RE: Building a new custom dictionary or Updating/Adding values to the existing dictionary in cTAKES [EXTERNAL] * External Email - Caution * Hi Team, I hope you all are doing good. With your support ,We were able to successfully add our required synonyms into existing dictionary and could see that it was getting successfully picked up by cTAKES. Now we have a requirement to configure the ICD and CPT also, where we followed the steps as mentioned in cTAKES wiki and generated the respective .script file. The newly created dictionary which comprises of SNOMEDCT_US,RxNORM,ICD10,CPT are identifying the descriptions as expected but we have a requirement to extract the ICD code for the respective description . so the scenario would be like for a text like below ‘F84.1 pervasive developmental disorders’ We would need cTAKES to reckon F84.1 as a token or at least as an attribute in any of the ‘IdentifiedAnnotation’. So for achieving the same based on our prior experience we tried to tweak the dictionary where we added a synonym for the existing CUI as below INSERT INTO CUI_TERMS VALUES(4352,1,4, ‘F84.1 pervasive developmental disorders’, ‘pervasive’) INSERT INTO CUI_TERMS VALUES(4352,1,2, ‘F84.1 pdd’, ‘pdd’) INSERT INTO CUI_TERMS VALUES(4352,0,1, ‘F84.1’,’F84.1’) Though we have seen cTAKES can identify ‘F84’ alone as a token but it won’t consider whenever a ‘.’ Has been encountered. As an end result cTAKES won’t be able to give the ICD codes like F84.1,M25.6 as separate tokens. Since almost all of the ICD codes have a ‘.’ Associated with it, this way of tweaking the dictionary is not working. Infact cTAKES is recognizing the digit after decimal within the ‘FractionAnnotation’ Does cTAKES have the capability to return the code like ICD code while retrieving the token as an individual token or as an attribute in any of the tokens Is there any other way in which the dictionary can be tweaked , so that a synonym addition as below will recognize the ICD code as a token and will be returned from cTAKES INSERT INTO CUI_TERMS VALUES(4352,0,1, ‘F84.1’,’F84.1’) Kindly check and advise us on how to proceed on this situation Thanks & Regards [cid:D3145E69-CD94-48C1-877F-5134EEAFB598] Abad Ayyub Vnet: 406170 | Cell : +91-9447379028 From: Remy Sanouillet <re...@foreseemed.com> Sent: Tuesday, June 2, 2020 7:23 AM To: dev@ctakes.apache.org Subject: Re: Building a new custom dictionary or Updating/Adding values to the existing dictionary in cTAKES [External] Hi Abad, • How can we point cTAKES application to multiple dictionaries. Currently only sno_rx_16ab is pointed to the application, how can I tweak it to point that to multiple dictionary simultaneously. Or you meant to say create a fresh dictionary with all the vocabularies and point just that in cTAKES. If you go back in the archive a bit, you should find a thread where I went into detail on how to add multiple dictionaries. Combining all dictionaries into a fresh dictionary is not recommended for obvious reasons. If you can't find the thread, I will dig it up. • So for these edits I will have to add INSERT queries to respective tables in the sno_rx_16ab.script file right? Do I need to make any more changes for these tokens to get reflected in cTAKES. Nope! That is all that is needed and next time you launch cTakes, it should recognize your new entries. • If it is a non-existing CUI , I can get the respective CUI,TUI from here https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Futs.nlm.nih.gov%2F%2Fmetathesaurus.html&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=2tgzGJUzWdtDSTyT7MI93e2i17aeFW8Nqp3s4D1cj8g%3D&reserved=0<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__apc01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Futs.nlm.nih.gov-252Fmetathesaurus.html-26data-3D02-257C01-257CAbad.Ayyub-2540cognizant.com-257Cc8b0b69302014cff91ac08d80697c6a7-257Cde08c40719b9427d9fe8edf254300ca7-257C0-257C0-257C637266596246493365-26sdata-3DhNixbxffJ9-252Fx-252Bho9J41gjonaT9IGLsxIqABKq1dpzG8-253D-26reserved-3D0%26d%3DDwMGaQ%26c%3DqS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU%26r%3Dfs67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao%26m%3DYKrpBhJM7sqCBY3Ow1jSUhu5QBdlnoqFGZbsVZIHH8U%26s%3DAks7ZCfU7hTRPTyJJdrrdupKbd1n1TpuFdf-10yQtrA%26e%3D&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=%2BC1T6ynAPci%2FvDGvWZV6uSE4gKYC5cjaUyszZRYGY34%3D&reserved=0> right? Correct! Remember that the ontology has multiple-inheritance so you need to grab all the TUIs for a given CUI. • Based on the source I will have to add entry to respective table right? Like SNOMED,RxNORM,ICD 10 and a CUI will belong to either one of it and not in all. Correct me if am wrong on this understanding That is also correct. And most of the time, the dictionaries only contain one CODE table so it is not even a question. However, sno_rx_16ab is an exception with both a CODE table for SNOMEDCT_US and RXNORM. They mostly do not overlap. I do remember that there were a couple of exceptions but, in the case where that happens, the metathesaurus will show it. For example: 'Acebutolol' (CUI: C0000946) has two SNOMEDCT_US codes (372815001 and 68088000) *and* an RXNORM of 149. • PREFTERM table will be having only one entry for each CUI right? Basically it’s a one-to-one mapping between CUI and PREFTERM . Correct me if am wrong on this understanding. You are correct here also. It is a one-to-one mapping although the system appears to tolerate when the PREFTERM is missing. Rémy Sanouillet NLP Engineer re...@foreseemed.com<mailto:xx...@foreseemed.com> [image.png] ForeSee Medical, Inc. 12555 High Bluff Drive, Suite 100 San Diego, CA 92130 NOTICE: This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. On Mon, Jun 1, 2020 at 7:56 AM <abad.ay...@cognizant.com<mailto:abad.ay...@cognizant.com>> wrote: Thank you Remy and Peter for your responses. I hope you guys are doing good and safe in this lock down period. Could you pls. help me on my below queries in creating an additional dictionary. • How to create additional dictionary. You meant to say using the UMLS tool , so that using that tool we create .script files from .RRF files? • How can we point cTAKES application to multiple dictionaries. Currently only sno_rx_16ab is pointed to the application, how can I tweak it to point that to multiple dictionary simultaneously. Or you meant to say create a fresh dictionary with all the vocabularies and point just that in cTAKES. I hope Remy was explaining editing the existing dictionary where I would deal with two scenarios where one was with existing CUI and other was with Non-existing CUI. Could you pls. resolve the below queries regarding the same. • So for these edits I will have to add INSERT queries to respective tables in the sno_rx_16ab.script file right? Do I need to make any more changes for these tokens to get reflected in cTAKES. • If it is a non-existing CUI , I can get the respective CUI,TUI from here https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Futs.nlm.nih.gov%2F%2Fmetathesaurus.html&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=2tgzGJUzWdtDSTyT7MI93e2i17aeFW8Nqp3s4D1cj8g%3D&reserved=0<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__apc01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Futs.nlm.nih.gov-252Fmetathesaurus.html-26data-3D02-257C01-257CAbad.Ayyub-2540cognizant.com-257Cc8b0b69302014cff91ac08d80697c6a7-257Cde08c40719b9427d9fe8edf254300ca7-257C0-257C0-257C637266596246503352-26sdata-3DbbpLuRz7gcbSopU7kFxTJrlsAiqZY4TiK15eq1l4qVs-253D-26reserved-3D0%26d%3DDwMGaQ%26c%3DqS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU%26r%3Dfs67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao%26m%3DYKrpBhJM7sqCBY3Ow1jSUhu5QBdlnoqFGZbsVZIHH8U%26s%3D3BlK-CxQfaf_mvf6rMZ7MK1GJIEnflO1MlbEZ1oTsEM%26e%3D&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=w%2B4UeSVqPRQktC%2FMoCGQhRbzn7xSuMoA3znLM86Uk8M%3D&reserved=0> right? • Based on the source I will have to add entry to respective table right? Like SNOMED,RxNORM,ICD 10 and a CUI will belong to either one of it and not in all. Correct me if am wrong on this understanding • PREFTERM table will be having only one entry for each CUI right? Basically it’s a one-to-one mapping between CUI and PREFTERM . Correct me if am wrong on this understanding. Thanks & Regards Abad Ayyub Vnet: 406170 | Cell : +91-9447379028 From: Remy Sanouillet <re...@foreseemed.com<mailto:re...@foreseemed.com>> Sent: Friday, May 29, 2020 9:25 PM To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org> Cc: u...@ctakes.apache.org<mailto:u...@ctakes.apache.org> Subject: Re: Building a new custom dictionary or Updating/Adding values to the existing dictionary in cTAKES [External] Hello Abad, The short answer is, yes, the sno_rx_16ab can be "hacked". A couple of caveats are that any mistake can stop all recognition and you will lose all your mods on updates. So an additional dictionary is a recommended approach. There are two cases. EIther the CUI you are adding already exists and you are just adding a synonym. In that case, you only need to add one line: INSERT INTO CUI_TERMS VALUES(CUI,RINDEX,TCOUNT,TEXT,RWORD) where: * CUI is the cui, nuf'said * TEXT is the tokenized lowercase string for the entry. In your case 'pap smear'. Most punctuation is a separate token. Single quotes are escaped by doubling them * RWORD is the one token in TEXT that is the most indicative (least common) which will be used as the index in the lookup. In your case probably 'pap' since it is not as common as 'smear' * RINDEX is the index of RWORD in TEXT. First token is 0 which is the case for 'pap' * TCOUNT is the token count for TEXT. In your case, 2 So you would want to add: INSERT INTO CUI_TERMS VALUES(200845,0,2,'pap smear','pap') If the entry is a non-existing one, you will need to add a few more lines. Their positions are unimportant as long as they are below the header lines (below the final "SET SCHEMA PUBLIC" line). 1. INSERT INTO TUI VALUES(CUI,TUI) One line for each TUI in the taxonomy 2. INSERT INTO SNOMEDCT_US VALUES(CUI,SNOMED) assuming you are adding a SNOMED 3. INSERT INTO PREFTERM VALUES(CUI,PREFTERM) where PREFTERM is the pretty string to describe the entry. It need not correspond to any indexed entry. It is used for display once the lookup has been successful. That's it. Use at your own discretion. No guarantees. Rémy Sanouillet NLP Engineer re...@foreseemed.com<mailto:xx...@foreseemed.com> ForeSee Medical, Inc. 12555 High Bluff Drive, Suite 100 San Diego, CA 92130 NOTICE: This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. On Fri, May 29, 2020 at 7:34 AM <abad.ay...@cognizant.com<mailto:abad.ay...@cognizant.com>> wrote: Hi Team, We set up cTAKES4.0.0 as our NLP engine for our profile recently . We have faced situations where some of the expected tokens are not picked up by cTAKES during clinical text extraction. So our first thought process was to identify where the dictionary is configured and how that can be updated. After some code analysis it was found that the dictionary is configured in the below path under ctakes/resources for sources RxNorm and SNOMEDCT_US We were able to open the hsqldb using the hsql db gui and found out that some of our required entries are already there . So if I come specifically to our current problem. The Pap Smear and Mamogram are two clinical terms which are not currently recognized by cTAKES in our profile. • If I look into the .script file , Pap Smear and Mammogram/Mammography is already present in the .script file and in the respective tables. PFB a snapshot as below But still this was not recogonised by cTAKES. I see there are some filters working on top of the available entries in dictionary(ctakes-gui and ctake-gui-res). Will that be because of these filters the tokens are not recognized as expected. Could you pls. share us what exactly these filters do. This will help us in future also when we are trying to add new terms into the dictionary • What are the steps to do if we need to add/edit entries into the existing dictionaries. I see we can add/edit the existing values in .scripts files but our primary doubt is if suppose I have a term ‘xyz’ to be added to dictionary how can I get the CUI and other values like TUI,RINDEX,TCOUNT and PREFTERM. Is it fine if I can give any random value for the TUI/CUI/RINDEX/TCOUNT. I could also see options to create custom bsv dictionaries but couldn’t see much documentation for it. Kindly advise which is the better option from the below 3. o Generate a custom dictionary using METAMORPHOSYS UML installation tool(where we provide sources as ICD10,RxNORM,SNOMEDCT_US) and leverage the full set of .rrf files in the meta folder . Is this approach better if the entries to be populated are maximal? o Add/edit the available dictionary sno_rx_16ab and in that case how to provide valid values for each columns like CUI, TUI,RINDEX,TCOUNT and PREFTERM. If the entries to be populated are minimal is this approach would be better?. o Use a custom bsv , in that case how should we add values to custom bsv. Could you also provide a sample in that case. I found a Metathesaurus browser in the below url , where I can search for the terms and get the CUI and the respective source like ICD/CPT/MDR. But still I was unable to get the other required attributes to be populated like TUI,RINDEX,TCOUNT and PREFTERM. Could you pls. brief what these attributes signifies https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Futs.nlm.nih.gov%2F%2Fmetathesaurus.html&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=2tgzGJUzWdtDSTyT7MI93e2i17aeFW8Nqp3s4D1cj8g%3D&reserved=0<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__apc01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Futs.nlm.nih.gov-252Fmetathesaurus.html-26data-3D02-257C01-257CAbad.Ayyub-2540cognizant.com-257Cc8b0b69302014cff91ac08d80697c6a7-257Cde08c40719b9427d9fe8edf254300ca7-257C0-257C0-257C637266596246513622-26sdata-3DCYHTv-252B8qE9VFAz1mzW2XP18B8EsdrhpchPQKuEDHlBU-253D-26reserved-3D0%26d%3DDwMGaQ%26c%3DqS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU%26r%3Dfs67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao%26m%3DYKrpBhJM7sqCBY3Ow1jSUhu5QBdlnoqFGZbsVZIHH8U%26s%3D8AfoyzMZC6lva419TTWLPVYtTCWEZOmAiRxvgSn6cxM%26e%3D&data=02%7C01%7CAbad.Ayyub%40cognizant.com%7C28b35c064d474a289fbd08d858c7ea90%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637356962959976394&sdata=gqff4%2BWMgqDHHXLv%2FJw3f7x6GqLqQJ3b67IkjBl8QaI%3D&reserved=0> Kindly advise us on how to proceed on this and correct us if we went wrong somewhere. This would be of great help for us P.S : We comply with UMLS license Thanks & Regards Abad Ayyub Vnet: 406170 | Cell : +91-9447379028 This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
