Thanks for the details Sean. I had assumed the conversion to Long was related to sort/search efficiency but that makes sense.
I had been thinking of something similar with parsing out the non-numerals and
converting them to 2-digit numeral values. i.e. “a” = 01, “z” = 26. Ultimately
CN123456 would become 0314123456 but I don’t think its sophisticated enough to
avoid issues with leading zeros. We could prepend a 9 to it to avoid losing
digits and use something like:
if(length>8 && begins with 9)
discard 9
while (length > 8)
convert first 2 numbers to a letter
I think your suggestion sounds good to me. To run the example through:
“NLM300" gets parsed to “NLM” + “300”
Store Pair<Integer,String>(3, NLM) at Pair[0]
Produce a Long of 0x10000000 + 300 = 300L
Backtrack to the actual “CUI” floor(300/10000000) = 0L
300L - 0L = 300L
Pair[0] = NLM
CUI = NLM + 300
In that case, do we need to store it as a Pair at all or is just storing the
prefix in a String[] sufficient?
I’m happy to start working on this unless you have a preference for splitting
it out into multiple tasks.
Britt Fitch
Wired Informatics
265 Franklin St Ste 1702
Boston, MA 02110
http://wiredinformatics.com
britt.fi...@wiredinformatics.com
> On Jul 8, 2015, at 2:54 PM, Finan, Sean <sean.fi...@childrens.harvard.edu>
> wrote:
>
> By the way, in case you are wondering why it does this … the umls database
> that we use has roughly half a million cuis. Storing cuis in the various
> tables as longs takes up a lot less space than storing them as 8 character
> strings.
>
> From: britt fitch [mailto:britt.fi...@wiredinformatics.com]
> Sent: Wednesday, July 08, 2015 2:23 PM
> To: dev@ctakes.apache.org
> Subject: dictionary-look-fast fails to handle alternative CUIs
>
> This is largely directed to Sean but open to other feedback as well.
>
> The current fast lookup using a BSV parses the first field as “C” and up to 7
> numerals, padding with “0" as needed to reach that length when applicable
> [see CuiCodeUtil.getCuiCode(String)]
>
> The CUI string is then substring’d from 1 to len and parsed as a Long.
>
> This is producing issues with other related, but separate, ontologies
> (MedGen) where the bulk of concepts use UMLS CUIs but some additional
> concepts were created by the NCBI where no CUI previously existed.
> These MedGen-specific concepts are created with a prefix “CN” + 6 numerals,
> resulting in “N123456” failing to produce a Long.
>
> I wanted Sean’s thoughts on this and to get some feedback on if others are
> running into this issue and if the community wants a solution to providing a
> CUI format beyond the standard C + 7 numerals.
>
> I’m happy to make these edits and check them in whether that means updating
> the CuiCodeUtil class or creating an entirely new BSVConceptFactory if thats
> what makes the most sense.
>
> Thoughts?
>
>
>
>
>
>
>
>
>
> Britt Fitch
> Wired Informatics
> 265 Franklin St Ste 1702
> Boston, MA 02110
> http://wiredinformatics.com
> britt.fi...@wiredinformatics.com
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
