Re: HSQLDB out of memory with custom dictionary

2017-10-10 Thread Kathy Ferro
Gandhi,

My name is Kathy Ferro.

Matthew and I are trying to accomplish the thing.  I got the scripts loaded
into both SQL server and MySQL.  I did it in two ways.
1. Manually modifier the scripts for DB specific and run them in query
analyzer window as you described.  Works find if the data is small enough.
For bigger file, it looks up.
2. I wrote c# program to read the scripts and insert records one by one I
re-load them.

My question for you are:

1. Does it matter what the name of the database?
2. What configuration file do I change to switch to use the new database?
3. Do you think I can use SQL server instead of MySQL?  My SQL seems to run
faster.

Thank
Kathy




On Tue, Oct 10, 2017 at 2:34 AM, Gandhi Rajan Natarajan <
gandhi.natara...@arisglobal.com> wrote:

> Hi Matthew,
>
> The SQLs looks fine. The only additional table I'm using apart from the
> tables mentioned below is MDR table (MEDDRA related) and I don’t use AIR
> table.
>
> Do you really think you need a JAVA program to convert those insert
> statements to work with MySQL? I just opened the script file in text editor
> like Editplus and did a find for `[\)]\n` and replaced it with `);\n` using
> find and replace all option with REGEX and we are done with the scripts.
>
> But only thing is you can load the data in parallel by splitting the
> script files as mentioned earlier which saves times for you and may be you
> can write a JAVA program to split the file. This is the easiest approach I
> feel.
>
> Regards,
> Gandhi
>
>
> -Original Message-
> From: Matthew Vita [mailto:matthewvit...@gmail.com]
> Sent: Tuesday, October 10, 2017 10:47 AM
> To: dev@ctakes.apache.org
> Subject: Re: HSQLDB out of memory with custom dictionary
>
> Gandhi,
>
> I really appreciate this information. I have started working out the
> schema and plan on writing a program that will automatically prepare a
> script to work with MySQL. Work in progress. Can you do a quick review of
> my MySQL schema so far?
>
> CREATE SCHEMA CTAKES_DATA;
>
> use CTAKES_DATA;
>
> CREATE TABLE CUI_TERMS (
>   CUI BIGINT NOT NULL,
>   RINDEX INT(128) NOT NULL,
>   TCOUNT INT(128) NOT NULL,
>   TEXT VARCHAR(255) NOT NULL,
>   RWORD VARCHAR(48) NOT NULL
> );
> CREATE INDEX IDX_CUI_TERMS ON CUI_TERMS (RWORD);
>
> CREATE TABLE TUI (
>   CUI BIGINT NOT NULL,
>   TUI INT(128) NOT NULL
> );
> CREATE INDEX IDX_TUI ON TUI (CUI);
>
> CREATE TABLE PREFTERM (
>   CUI BIGINT NOT NULL,
>   PREFTERM VARCHAR(511) NOT NULL
> );
> CREATE INDEX IDX_PREFTERM ON PREFTERM (CUI);
>
> CREATE TABLE RXNORM (
>   CUI BIGINT NOT NULL,
>   RXNORM BIGINT NOT NULL
> );
> CREATE INDEX IDX_RXNORM ON RXNORM (CUI);
>
> CREATE TABLE SNOMEDCT_US (
>   CUI BIGINT NOT NULL,
>   SNOMEDCT_US BIGINT NOT NULL
> );
> CREATE INDEX IDX_SNOMEDCT_US ON SNOMEDCT_US (CUI);
>
> Quick question: do you use the AIR table?
>
> Thanks,
>
> Matthew Vita
> www.matthewvita.com
>
> On Mon, Oct 9, 2017 at 1:14 AM, Gandhi Rajan Natarajan <
> gandhi.natara...@arisglobal.com> wrote:
>
> > Hi Mathew,
> >
> > First I would like to tell you that even I m a newbie in cTAKES.
> > Unfortunately I don’t find any documentation on this. I have followed
> > a crude way to accomplish as this is an one time activity. This is what
> I did:
> >
> > 1) Used dictionary generator GUI to generate Snomed, RxNorm and MEDDRA
> > dictionary data that resulted in '.script' file under my
> > \resources\org\apache\ctakes\dictionary\lookup\fast\ > ect_name>
> > folder
> > 2) The '.script' file has HSQLDB specific queries. I have removed the
> > unwanted statements for me pertaining to HSQLDB from the file and
> > converted them to mysql specific queries manually.
> > 3) I have added semicolons at the end of each line in the script using
> > text editor and splitted the file in to five parts. Then I ran those
> > five sctipr files  in five different mysql command lines. It took me
> > approximately 4 hours to pump all the data in to MySQL DB.
> >
> > I'm not sure whether it is the right way to proceed as I mentioned
> > earlier. But with no documentation available for MySQL DB with
> > cTAKES, this is the approached that worked for me. Hope it will be
> helpful.
> >
> > Regards,
> > Gandhi
> >
> >
> > -Original Message-
> > From: Matthew Vita [mailto:matthewvit...@gmail.com]
> > Sent: Monday, October 09, 2017 10:41 AM
> > To: dev@ctakes.apache.org
> > Subject: Re: HSQLDB out of memory with custom dictionary
> >
> > Gandhi,
> >
> > Thank you

Re: HSQLDB out of memory with custom dictionary

2017-10-11 Thread Kathy Ferro
Gandhi and Matthew,

Thank you for the information.

Kathy

On Wed, Oct 11, 2017 at 1:35 AM, Gandhi Rajan Natarajan <
gandhi.natara...@arisglobal.com> wrote:

> Hi Matthew,
>
> Please check out my response to Kathy. If feel that has the required info
> to start off. Please let me know if you are looking for any specific
> additional info.
>
> Regards,
> Gandhi
>
>
> -Original Message-
> From: Matthew Vita [mailto:matthewvit...@gmail.com]
> Sent: Wednesday, October 11, 2017 11:00 AM
> To: dev@ctakes.apache.org
> Subject: Re: HSQLDB out of memory with custom dictionary
>
> Hi Kathy and Gandhi,
>
> I started to put together a more formal solution for this here:
> https://github.com/GoTeamEpsilon/cTAKES-HSQLDB-to-MySQL-Dictionary - It
> is not perfect but it makes things a bit easier. I was able to load in
> millions of records into MySQL, which is awesome!
>
> *If you have a non-trivial dictionary, chances are you will exhaust
> HSQLDB's capabilities. By using this solution, you will have a MySQL schema
> filled up with what would have been the HSQLDB data.*
>
> *This solution uses lazy lists and streams to keep memory usage low when
> the script files are huge.*
>
> I have not got it working with the XML jdbc configuration yet so if you
> (or anyone else) could share an example that would be amazing.
>
> Thanks,
>
> Matthew Vita
> www.matthewvita.com
>
> On Tue, Oct 10, 2017 at 9:57 PM, Gandhi Rajan Natarajan <
> gandhi.natara...@arisglobal.com> wrote:
>
> > Hi Kathy,
> >
> > Good to hear from you. Please find the response below.
> >
> > NOTE: This is based on my experience with cTAKES so far. Please
> > correct me if someone find the answers to be wrong.
> >
> > 1. Does it matter what the name of the database?
> >
> > Name of the database really don’t matter. But the name you have
> > created should be mapped in the Dictionary GUI generated XML file's
> 'jdbcurl'
> > property.
> >
> > 2. What configuration file do I change to switch to use the new database?
> >
> > If you are using the example downloaded from
> > https://github.com/healthnlp/
> > examples/tree/master/ctakes-temporal-demo , then in Pipeline.java you
> > gotta map the XML file name generated using the Dictionary GUI instead
> of 'sno_rx_16ab.xml'
> >
> > If you want to use the new database for CVD, then you got to change '
> > DEFAULT_DICT_DESC_PATH' to point to the new XML file in
> > JCasTermAnnotator.java and rebuild ctakes-dictionary-lookup-fast
> > module and use the jar file.
> >
> > 3) Do you think I can use SQL server instead of MySQL?  My SQL seems
> > to run faster.
> >
> > This choice is user specific and I can't comment on performance
> > comparison as I have no clue on this.
> >
> >
> >
> > Regards,
> > Gandhi
> >
> >
> > -Original Message-
> > From: Kathy Ferro [mailto:healthcare1...@gmail.com]
> > Sent: Tuesday, October 10, 2017 9:26 PM
> > To: dev@ctakes.apache.org
> > Subject: Re: HSQLDB out of memory with custom dictionary
> >
> > Gandhi,
> >
> > My name is Kathy Ferro.
> >
> > Matthew and I are trying to accomplish the thing.  I got the scripts
> > loaded into both SQL server and MySQL.  I did it in two ways.
> > 1. Manually modifier the scripts for DB specific and run them in query
> > analyzer window as you described.  Works find if the data is small
> enough.
> > For bigger file, it looks up.
> > 2. I wrote c# program to read the scripts and insert records one by
> > one I re-load them.
> >
> > My question for you are:
> >
> > 2. What configuration file do I change to switch to use the new database?
> > 3. Do you think I can use SQL server instead of MySQL?  My SQL seems
> > to run faster.
> >
> > Thank
> > Kathy
> >
> >
> >
> >
> > On Tue, Oct 10, 2017 at 2:34 AM, Gandhi Rajan Natarajan <
> > gandhi.natara...@arisglobal.com> wrote:
> >
> > > Hi Matthew,
> > >
> > > The SQLs looks fine. The only additional table I'm using apart from
> > > the tables mentioned below is MDR table (MEDDRA related) and I don’t
> > > use AIR table.
> > >
> > > Do you really think you need a JAVA program to convert those insert
> > > statements to work with MySQL? I just opened the script file in text
> > > editor like Editplus and did a find for `[\)]\n` and replaced it
> > > with `);\n` using find and replace all option with

multipleReferencesAllowed=false warning

2017-10-11 Thread Kathy Ferro
Good Evening,

I got this warning message when running ctakes clinical pipeline.  It
doesn't give me the warning on every documents.

What file do I turn the setting to true?

WARNING: Message count: 128; Feature
org.apache.ctakes.typesystem.type.textsem.Predicate:relations is marked
multipleReferencesAllowed=false, but it has multiple references.  These
will be serialized in duplicate. Message count indicates messages skipped
to avoid potential flooding. Set FINE logging level for stacktrace.


Thank you
Kathy


Re: HSQLDB out of memory with custom dictionary

2017-10-13 Thread Kathy Ferro
Gandhi,

Thanks again for your response.

I am pretty new with ctakes myself and my Java knowledge is not up to
dated.

I am looking at the sample source code from https://github.com/healthnlp/
examples/tree/master/ctakes-temporal-demo.  In pipeline.java, it looks like
it changes the dictionary name only.

   builder.add( AnalysisEngineFactory.createEngineDescription(
DefaultJCasTermAnnotator.class,
AbstractJCasTermAnnotator.PARAM_WINDOW_ANNOT_KEY,
"org.apache.ctakes.typesystem.type.textspan.Sentence",
JCasTermAnnotator.DICTIONARY_DESCRIPTOR_KEY,
"org/apache/ctakes/dictionary/lookup/fast/sno_rx_16ab.xml")
   );


1. Do I change to MySQL driver in (dictionary).xml? Below is the code
snip.
2, What do I do with the blue highlight?
3. If I leave hsqldb, would that just use the hsqldb script file?
4. If I change it, do you have sample?

Right now, I run the pipeline using the new dictionary with this option "-l
org/apache/ctakes/dictionary/lookup/fast/(dictionary name).xml" which loads
the dictionary into hsqldb memory.


 
 


I'm very appreciated your help.
Kathy



On Wed, Oct 11, 2017 at 5:14 PM, Kathy Ferro 
wrote:

> Gandhi and Matthew,
>
> Thank you for the information.
>
> Kathy
>
> On Wed, Oct 11, 2017 at 1:35 AM, Gandhi Rajan Natarajan <
> gandhi.natara...@arisglobal.com> wrote:
>
>> Hi Matthew,
>>
>> Please check out my response to Kathy. If feel that has the required info
>> to start off. Please let me know if you are looking for any specific
>> additional info.
>>
>> Regards,
>> Gandhi
>>
>>
>> -Original Message-
>> From: Matthew Vita [mailto:matthewvit...@gmail.com]
>> Sent: Wednesday, October 11, 2017 11:00 AM
>> To: dev@ctakes.apache.org
>> Subject: Re: HSQLDB out of memory with custom dictionary
>>
>> Hi Kathy and Gandhi,
>>
>> I started to put together a more formal solution for this here:
>> https://github.com/GoTeamEpsilon/cTAKES-HSQLDB-to-MySQL-Dictionary - It
>> is not perfect but it makes things a bit easier. I was able to load in
>> millions of records into MySQL, which is awesome!
>>
>> *If you have a non-trivial dictionary, chances are you will exhaust
>> HSQLDB's capabilities. By using this solution, you will have a MySQL schema
>> filled up with what would have been the HSQLDB data.*
>>
>> *This solution uses lazy lists and streams to keep memory usage low when
>> the script files are huge.*
>>
>> I have not got it working with the XML jdbc configuration yet so if you
>> (or anyone else) could share an example that would be amazing.
>>
>> Thanks,
>>
>> Matthew Vita
>> www.matthewvita.com
>>
>> On Tue, Oct 10, 2017 at 9:57 PM, Gandhi Rajan Natarajan <
>> gandhi.natara...@arisglobal.com> wrote:
>>
>> > Hi Kathy,
>> >
>> > Good to hear from you. Please find the response below.
>> >
>> > NOTE: This is based on my experience with cTAKES so far. Please
>> > correct me if someone find the answers to be wrong.
>> >
>> > 1. Does it matter what the name of the database?
>> >
>> > Name of the database really don’t matter. But the name you have
>> > created should be mapped in the Dictionary GUI generated XML file's
>> 'jdbcurl'
>> > property.
>> >
>> > 2. What configuration file do I change to switch to use the new
>> database?
>> >
>> > If you are using the example downloaded from
>> > https://github.com/healthnlp/
>> > examples/tree/master/ctakes-temporal-demo , then in Pipeline.java you
>> > gotta map the XML file name generated using the Dictionary GUI instead
>> of 'sno_rx_16ab.xml'
>> >
>> > If you want to use the new database for CVD, then you got to change '
>> > DEFAULT_DICT_DESC_PATH' to point to the new XML file in
>> > JCasTermAnnotator.java and rebuild ctakes-dictionary-lookup-fast
>> > module and use the jar file.
>> >
>> > 3) Do you think I can use SQL server instead of MySQL?  My SQL seems
>> > to run faster.
>> >
>> > This choice is user specific and I can't comment on performance
>> > comparison as I have no clue on this.
>> >
>> >
>> >
>> > Regards,
>> > Gandhi
>> >
>> >
>> > -Original Message-
>> > From: Kathy Ferro [mailto:healthcare1...@gmail.com]
>> > Sent: Tuesday, October 10, 2017 9:26 PM
>> > To: dev@ctakes.apache.org
>>

polarity tag in output for mention/concept.

2017-11-27 Thread Kathy Ferro
Good evening,

I ran a few sentences through default clinical pipeline.

It really reliable if it's only one term after negative, but I am get
in-consistent value for polarity for the list of terms.  Please see example
below.

1.   denies fatigue, malaise, fever, weight loss
SignSymthomMention:
polarity = -1:  fatigue, malaise,fever
polarity = 1: weight loss.
Why does weight loss got single out?

2.   denies ear pain or discharge, nasal obstruction or discharge, sore
throat
polarity = -1: ear pain or discharge
polarity = 1: nasal obstruction or discharge, obstruction, sore throat
Doesn't even acknowledge the list.

3.   denies back pain, joint swelling, joint stiffness, joint pain
polarity = -1: back pain, Swelling
polarity = 1: Joint swelling, Stiffness, pain
What! totally messy the pattern.

4.   denied back pain, joint swelling, joint stiffness, joint pain
Ok, may be it doesn't like the word denies; I changed to denied, deny, etc..
polarity = -1 : Swelling
everything else is 1.


My question is:
How do I handle the negative claims in the document?

Thank you
Kathy


Re: polarity tag in output for mention/concept. [EXTERNAL]

2017-11-28 Thread Kathy Ferro
Sean,

Thank you for information.

I was reading the document.  So, the MaxLeftScopeSize and MaxRightScopeSize
are limit up to 10?  Is there anyway to adjust it without modify the source
code?

https://cwiki.apache.org/confluence/display/CTAKES/cTAKES+4.0+-+NE+Contexts


Thanks again,
Kathy



On Tue, Nov 28, 2017 at 9:31 AM, Finan, Sean <
sean.fi...@childrens.harvard.edu> wrote:

> Hi Kathy,
>
> The negation annotator used in the default clinical pipeline is based upon
> machine learning and trained on real data.  It is possible that such
> "denies" lists were underrepresented in the training data.  One thing that
> you can try is adding another negation annotator.  The ContextAnnotator in
> ctakes-ne-contexts will add negation to terms without removing existing
> negation.  It also has configurable scope/distance that may be helpful.
>
> To use this, create a new piper file containing the two lines
>
> load DefaultFastPipeline
> add ContextAnnotator MaxLeftScopeSize=10 MaxRightScopeSize=10
>
> The default scope sizes are 7, but increasing  the MaxRight* might help
> with your "denies" discoveries.  7 might be ok for the left, so feel free
> to remove "MaxLeftScopeSize=10" from the line.
>
> Then run your piper file (command line, gui, maven profile, etc.)
> https://cwiki.apache.org/confluence/display/CTAKES/Piper+Files
>
> Sean
>
> -Original Message-
> From: Kathy Ferro [mailto:healthcare1...@gmail.com]
> Sent: Monday, November 27, 2017 8:10 PM
> To: dev@ctakes.apache.org
> Subject: polarity tag in output for mention/concept. [EXTERNAL]
>
> Good evening,
>
> I ran a few sentences through default clinical pipeline.
>
> It really reliable if it's only one term after negative, but I am get
> in-consistent value for polarity for the list of terms.  Please see example
> below.
>
> 1.   denies fatigue, malaise, fever, weight loss
> SignSymthomMention:
> polarity = -1:  fatigue, malaise,fever
> polarity = 1: weight loss.
> Why does weight loss got single out?
>
> 2.   denies ear pain or discharge, nasal obstruction or discharge, sore
> throat
> polarity = -1: ear pain or discharge
> polarity = 1: nasal obstruction or discharge, obstruction, sore throat
> Doesn't even acknowledge the list.
>
> 3.   denies back pain, joint swelling, joint stiffness, joint pain
> polarity = -1: back pain, Swelling
> polarity = 1: Joint swelling, Stiffness, pain What! totally messy the
> pattern.
>
> 4.   denied back pain, joint swelling, joint stiffness, joint pain
> Ok, may be it doesn't like the word denies; I changed to denied, deny,
> etc..
> polarity = -1 : Swelling
> everything else is 1.
>
>
> My question is:
> How do I handle the negative claims in the document?
>
> Thank you
> Kathy
>


Re: polarity tag in output for mention/concept. [EXTERNAL] [SUSPICIOUS] [SUSPICIOUS]

2017-11-28 Thread Kathy Ferro
Tim and Sean,

I think I spoke too soon. Don't know why it didn't work the 1st few times I
ran it.

I changed both to 20.  Looks like we are safe here; looks like it's 20 to
the left/right within the same sentence because the term "right breast" is
positive.








*my sample text:A Regional Med Center This should not interfere with ROS
section: denies fatigue, malaise, fever, weight lossMDM/ED
CourseImaging:The patient underwent an ultrasound-guilded core needle
biopsy with clip pacement of the 2.3cm mass in the upper outer quadrant of
the right breast.ROS: Heme/Lymphatic: denies easy or excessive bruising,
history of blood transfusions, anemia, bleeding disorders, adenopathy,
chills, sweatsAllergic/Immunologic: denies urticaria, hay fever, frequent
UTIs; denies HIV high risk behaviors*
Here's how I run it.  Is there a better way to run it?

I made a copy of runPipeFile.bat and added:

@rem -
set *FAST_PIPER*
=resources\org\apache\ctakes\clinical\pipeline\FastPipeline1.piper
java -Dctakes.umlsuser="myUser" -Dctakes.umlspw="myPW" -cp "%CLASS_PATH%"
%LOG4J_PARM% -Xms512M -Xmx3g %PIPE_RUNNER% -p %*FAST_PIPER*% %* -i
C:\Projects\NLPinbound --xmiOut C:\Projects\NLPoutbound  -l
org/apache/ctakes/dictionary/lookup/fast/aa_5.xml
:end


I am thrill with this discovery.
Thanks for your help.
Kathy


On Tue, Nov 28, 2017 at 2:05 PM, Finan, Sean <
sean.fi...@childrens.harvard.edu> wrote:

> Thanks Tim,
>
> "Negation's Not Solved" :^)
>
> -Original Message-
> From: Miller, Timothy [mailto:timothy.mil...@childrens.harvard.edu]
> Sent: Tuesday, November 28, 2017 2:01 PM
> To: dev@ctakes.apache.org
> Subject: Re: polarity tag in output for mention/concept. [EXTERNAL]
> [SUSPICIOUS] [SUSPICIOUS]
>
> I'll just point out -- the kind of examples Kathy gave were the bane of
> our existence while working on the ML-based assertion system. Even though
> it is obvious what is going on to a human it was hard to encode as a
> feature in a way that was learnable. But I think most rule-based algorithms
> will also run into problems with this type of example eventually if they
> have a hard-coded scoping mechanism (e.g., scope extends up to 10 words to
> the right). If you make it larger than you may increase the number of false
> positives your algorithm finds (confusingly, here a false positive is an
> example the algorithm calls negated that is not actually being negated).
> Tim
>
>
> On Tue, 2017-11-28 at 17:22 +, Finan, Sean wrote:
> > Hi Kathy,
> >
> > I am glad that you checked the wiki!  I should have pointed to it ...
> >
> > In the example I sent the "relevant distance" between trigger terms
> > and events would be 10.  There isn't any maximum as far as I know, but
> > I think that 10 is the most that I've ever used.  The default is 7,
> > and you can try with that (remove "*=*") before increasing the
> > number(s).
> >
> > The piper files aren't source code, they are just plain text and don't
> > require compiling, etc.  How are you running the pipeline right now?
> > From a binary with a bin/run* script?
> >
> > Sean
> >
> >
> > -Original Message-
> > From: Kathy Ferro [mailto:healthcare1...@gmail.com]
> > Sent: Tuesday, November 28, 2017 12:11 PM
> > To: dev@ctakes.apache.org
> > Subject: Re: polarity tag in output for mention/concept. [EXTERNAL]
> >
> > Sean,
> >
> > Thank you for information.
> >
> > I was reading the document.  So, the MaxLeftScopeSize and
> > MaxRightScopeSize are limit up to 10?  Is there anyway to adjust it
> > without modify the source code?
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org
> > _confluence_display_CTAKES_cTAKES-2B4.0-2B-2D-2BNE-
> > 2BContexts&d=DwIBaQ&c=qS4goWBT7poplM69zy_3xhKwEW14JZMSdioCoppxeFU&r=f
> > s67GvlGZstTpyIisCYNYmQCP6r0bcpKGd4f7d4gTao&m=4K9fxMmBiI0QZB0UhriFp_Yv
> > XDL8rmXtGRiKVgxMCPE&s=hsCB9xPXLC8fpiwrGXuEW9snw_WZbY0e-E-mhPOO9N8&e=
> >
> >
> > Thanks again,
> > Kathy
> >
> >
> >
> > On Tue, Nov 28, 2017 at 9:31 AM, Finan, Sean < Sean.Finan@childrens.h
> > arvard.edu> wrote:
> >
> > >
> > > Hi Kathy,
> > >
> > > The negation annotator used in the default clinical pipeline is
> > > based upon machine learning and trained on real data.  It is
> > > possible that such "denies" lists were underrepresented in the
> > > training data.  One thing that you can try is adding another
> > > negation annotator.  The ContextAnnotator in ct

exact match to CUI_TERM table question.

2017-11-29 Thread Kathy Ferro
Good Morning,

1. I have a term for x-ray that has different spelling such as x.ray,
x.rays, xray, xrays, etc...
I see several files in resources\org\apache\ctakes\assertion\semantic_classes
folder.
I created x-ray.txt with all the terms above and hoping it will do the
trick.  No luck.
Is there a way to link all this term to x-ray without have to modify fast
dictionary for every x-ray entries?

2. This might not have solution, but I'll ask anyway.  Looks like the terms
has to be exact match to terms in cut_terms table.  Example document has
"x-ray right elbow" or "elbow x-ray".  In the dictionary, I have "x-ray of
elbow" and "x-ray of the elbow".  Is there a way to pick up both of entries
in the dictionary without using black box (list)?  The term "left" and
"right" might be important in some instance.

3. This sample is kinda related to #2.  Document has term "diabetes" in one
sentence.  Down several pages, it has more specific term such as "
retinopathy" and  "controlled with insulin".
What is the best way to handle this?  Do you suggest I add "'retinopathy".
Does cTakes has term dependency?

It picks up.  (E08-E13) is wide range of codes.
PREFTERM VALUES(11849,'Diabetes Mellitus').
ICD10CM VALUES(11849,'E08-E13').
PREFTERM VALUES(11860,'Diabetes Mellitus, Non-Insulin-Dependent')
ICD10CM VALUES(11849,'E11').

I should also have pick up these, but didn't because of the exact match.
INSERT INTO CUI_TERMS VALUES(11884,0,3,'retinopathy ;
diabetic','retinopathy')
INSERT INTO CUI_TERMS VALUES(11884,3,6,'retina abnormal - diabet -
relat','diabet')
INSERT INTO CUI_TERMS VALUES(11884,1,2,'diabetic retinopathy','retinopathy')
INSERT INTO CUI_TERMS VALUES(11884,0,2,'retinopathy diabetic','retinopathy')


Snip of Sample text:
chief complaint: Patient came in complaining of having chest pain.
Procedure: chest xrays.
Problems:
Type 2 diabetes
depression
retinopathy
patient controlled with insulin.

Sincerely appreciated you help.
Kathy


Re: exact match to CUI_TERM table question. [EXTERNAL] [SUSPICIOUS]

2017-11-29 Thread Kathy Ferro
Time, Sean,

Thank you for information.  I'll try it out.

Kathy






On Wed, Nov 29, 2017 at 10:10 AM, Finan, Sean <
sean.fi...@childrens.harvard.edu> wrote:

> Excellent answers Tim!
>
> > Is there a way to link all this term to x-ray without have to modify
> > fast dictionary for every x-ray entries?
>
> Unfortunately no.  The dictionary can only match (give or take) the data
> that it contains.  I have long had an idea on how to improve this by
> trickery, but I'm not sure how well it would pan out in the end ...  Plus
> there is the investment of time in implementation.
> Anyway, you would need to add every text that you would like to match to a
> dictionary.  This doesn't need to be the hsqldb dictionary.  ctakes can
> also read plain-text files as dictionary sources.  But it requires a
> certain amount of crystal ball prediction on your part as you have to
> provide every permutation of a term that isn't already in the hsqldb
> dictionary.
>
>
> If you are interested in anatomic locations then I would try what Tim
> suggested and at the end of your piper file add:
>
> load RelationSubPipe
>
> That should add location relations (and degree-of) to your pipeline and
> would be easier than trying to rely on the dictionary to pick up every
> nuance.
>
> Sean
>
> -Original Message-
> From: Miller, Timothy [mailto:timothy.mil...@childrens.harvard.edu]
> Sent: Wednesday, November 29, 2017 9:48 AM
> To: dev@ctakes.apache.org
> Subject: Re: exact match to CUI_TERM table question. [EXTERNAL]
> [SUSPICIOUS]
>
> On Wed, 2017-11-29 at 09:36 -0500, Kathy Ferro wrote:
> > Good Morning,
> >
> > 1. I have a term for x-ray that has different spelling such as x.ray,
> > x.rays, xray, xrays, etc...
> > I see several files in
> > resources\org\apache\ctakes\assertion\semantic_classes
> > folder.
> > I created x-ray.txt with all the terms above and hoping it will do the
> > trick.  No luck.
> > Is there a way to link all this term to x-ray without have to modify
> > fast dictionary for every x-ray entries?
>
> No, these files are not for the dictionary lookup and will not add
> concepts to the CAS.
>
> >
> > 2. This might not have solution, but I'll ask anyway.  Looks like the
> > terms has to be exact match to terms in cut_terms table.  Example
> > document has "x-ray right elbow" or "elbow x-ray".  In the dictionary,
> > I have "x- ray of elbow" and "x-ray of the elbow".  Is there a way to
> > pick up both of entries in the dictionary without using black box
> > (list)?  The term "left"
> > and
> > "right" might be important in some instance.
> >
>
> How much is found really depends on the granularity of the source
> resource (UMLS/SNOMED) and whatever tricks Sean's import tool applies.
> UMLS often represents relations as concepts (elbow x-ray is in there).
> But as the modifiers get added it sometimes is easier to model as
> relations. For example, if you can detect "left" as a modifier, "elbow"
> as AnatomicalSite, and "x-ray" as procedure, then a relation extractor
> should find with "left" is modifying "elbow" and x-ray modifies
> "elbow," to give a complete picture. cTAKES can do relations between
> anatomical sites and other arguments, but I don't know if the default
> release does body side (left,right).
>
> > 3. This sample is kinda related to #2.  Document has term "diabetes"
> > in one
> > sentence.  Down several pages, it has more specific term such as "
> > retinopathy" and  "controlled with insulin".
> > What is the best way to handle this?  Do you suggest I add
> > "'retinopathy".
> > Does cTakes has term dependency?
> >
> > It picks up.  (E08-E13) is wide range of codes.
> > PREFTERM VALUES(11849,'Diabetes Mellitus').
> > ICD10CM VALUES(11849,'E08-E13').
> > PREFTERM VALUES(11860,'Diabetes Mellitus, Non-Insulin-Dependent')
> > ICD10CM VALUES(11849,'E11').
> >
> > I should also have pick up these, but didn't because of the exact
> > match.
> > INSERT INTO CUI_TERMS VALUES(11884,0,3,'retinopathy ;
> > diabetic','retinopathy')
> > INSERT INTO CUI_TERMS VALUES(11884,3,6,'retina abnormal - diabet -
> > relat','diabet')
> > INSERT INTO CUI_TERMS VALUES(11884,1,2,'diabetic
> > retinopathy','retinopathy')
> > INSERT INTO CUI_TERMS VALUES(11884,0,2,'retinopathy
> > diabetic','retinopathy')
> >
> >
> > Snip of Sample text:
> > chief complaint: Patient came in complaining of having chest pain.
> > Procedure: chest xrays.
> > Problems:
> > Type 2 diabetes
> > depression
> > retinopathy
> > patient controlled with insulin.
> >
>
> It should definitely get "retinopathy" since that's in snomed. The
> first thing I check when dictionary misses something is whether the
> linguistic annotations around it are correct (sentence, token, part of
> speech).
>
> > Sincerely appreciated you help.
> > Kathy
>


adding terms to cui_terms table

2018-05-21 Thread Kathy Ferro
Hello,

I know I had asked similar question.  I am welling to keep adding the terms
into dictionary; but I don't seen to get cTakes to pick them up.


example 1:
In the dictionary
INSERT INTO CUI_TERMS VALUES(2865275,0,14,'strain of musc / tend at lower
leg level , left leg , init','strain')
INSERT INTO PREFTERM VALUES(2865275,'Strain of other muscle(s) and
tendon(s) at lower leg level, left leg, initial encounter')
INSERT INTO ICD10CM VALUES(2865275,'S86.812A')

I added
INSERT INTO CUI_TERMS VALUES(2865275,0,22,'Strain of other muscle ( s ) and
tendon ( s ) at lower leg level , left leg , initial encounter','Strain')
INSERT INTO CUI_TERMS VALUES(2865275,0,22,'Strain of other muscle(s) and
tendon(s) at lower leg level, left leg, initial encounter','Strain')

The text has exactly term that matches prefer term.  Does cTakes not do the
lookup on prefer term?
Strain of other muscle(s) and tendon(s) at lower leg level, left leg,
initial encounter

example 2:
In dictionary
INSERT INTO CUI_TERMS VALUES(203273,0,2,'x-ray knee','x-ray')
INSERT INTO CUI_TERMS VALUES(203273,1,2,'knee x-ray','x-ray')
INSERT INTO CUI_TERMS VALUES(203273,0,4,'radiologic examination ,
knee','radiologic')
INSERT INTO CUI_TERMS VALUES(203273,0,3,'x-ray of knee','x-ray')
INSERT INTO CUI_TERMS VALUES(203273,0,4,'radiologic examination of
knee','radiologic')
INSERT INTO CUI_TERMS VALUES(203273,0,4,'x-ray of the knee','x-ray')
INSERT INTO CUI_TERMS VALUES(203273,0,3,'x-ray ; knee','x-ray')
INSERT INTO PREFTERM VALUES(203273,'Radiologic examination of knee')
INSERT INTO CPT VALUES(203273,'73564')
INSERT INTO CPT VALUES(203273,'73562')
INSERT INTO CPT VALUES(203273,'73560')

I added
INSERT INTO CUI_TERMS VALUES(203273,2,3,'XR , KNEE','KNEE')
INSERT INTO CUI_TERMS VALUES(203273,1,2,'XR, KNEE','KNEE')


The text this term:
XR, KNEE


Why does cTakes not pick them up?

Unfortunately, we have no control of how DR/Nurse document.

Thanks for the help.


misleading end of sentence when abbreviation in text.

2018-08-16 Thread Kathy Ferro
Good Afternoon,

As we know, providerd love to abbreviate their terms.  Is there a set up to
look for abbreviations and avoid it as the end of sentence.

Example:
Pt. denies any leg pain.

cTakes parses out as two sentences.  It really need to be one.

Thank you
Kathy


ctake web service

2018-08-16 Thread Kathy Ferro
Hi,

Just want to see if anybody has experience this issue.

If the web service had been up for a day or two, it will drop the
dictionary lookup.  The only result it returns are ConllDependencyNode tag
in the xmi file;  no mention, no concept, etc...

I haven't have a chance to investigate it, yet.

Kathy


Re: ctake web service

2019-03-03 Thread Kathy Ferro
Gandhi,

Do you get any response to this issue?  Does it try to keep the connection
open while WS is up? Or does it open and close after it's done?

We are still getting this error.
"ERROR JdbcRareWordDictionary - No operations allowed after statement
closed."

Thanks
Kathy



On Fri, Aug 17, 2018 at 9:43 AM Gandhi Rajan Natarajan <
gandhi.natara...@arisglobal.com> wrote:

> Hi Kathy,
>
> Sometime back we encountered this issue and the problem seems to be DB
> connections getting timed out.
>
> Currently we are using the following implementations:
> "org.apache.ctakes.dictionary.lookup2.dictionary.JdbcRareWordDictionary"
> and "org.apache.ctakes.dictionary.lookup2.concept.JdbcConceptFactory"
>
> Does anybody aware of any timeout settings that needs to be done in these
> implementations to avoid DB connection timeout issue?
>
> -Original Message-
> From: Kathy Ferro 
> Sent: Thursday, August 16, 2018 11:07 PM
> To: dev@ctakes.apache.org
> Subject: ctake web service
>
> Hi,
>
> Just want to see if anybody has experience this issue.
>
> If the web service had been up for a day or two, it will drop the
> dictionary lookup.  The only result it returns are ConllDependencyNode tag
> in the xmi file;  no mention, no concept, etc...
>
> I haven't have a chance to investigate it, yet.
>
> Kathy
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you are not the named addressee you should not disseminate, distribute
> or copy this e-mail. Please notify the sender or system manager by email
> immediately if you have received this e-mail by mistake and delete this
> e-mail from your system. If you are not the intended recipient you are
> notified that disclosing, copying, distributing or taking any action in
> reliance on the contents of this information is strictly prohibited and
> against the law.
>


Re: ctake web service [EXTERNAL]

2019-03-04 Thread Kathy Ferro
Sean,

What machine do you hosting the WS and mysql on?  I am on window 10
server.  mySQL ini file looks fine.  I'm wondering window and mysql are not
being friend.

Thanks
Kathy

On Mon, Mar 4, 2019 at 10:52 AM Miller, Timothy <
timothy.mil...@childrens.harvard.edu> wrote:

> I don't know what the solution was, but I leave my ctakes REST server
> running basically full time and haven't seen time outs yet.
> Tim
>
> 
> From: gandhi rajan 
> Sent: Monday, March 4, 2019 10:43 AM
> To: dev@ctakes.apache.org
> Subject: Re: ctake web service [EXTERNAL]
>
> Hi Kathy, Sean did respond that there is no timeout happening from cTAKES
> end. You might probably have to look at database settings for this closed
> connection issue.
>
> Does someone have any clue on this?
>
> On Monday, March 4, 2019, Kathy Ferro  wrote:
>
> > Gandhi,
> >
> > Do you get any response to this issue?  Does it try to keep the
> connection
> > open while WS is up? Or does it open and close after it's done?
> >
> > We are still getting this error.
> > "ERROR JdbcRareWordDictionary - No operations allowed after statement
> > closed."
> >
> > Thanks
> > Kathy
> >
> >
> >
> > On Fri, Aug 17, 2018 at 9:43 AM Gandhi Rajan Natarajan <
> > gandhi.natara...@arisglobal.com> wrote:
> >
> > > Hi Kathy,
> > >
> > > Sometime back we encountered this issue and the problem seems to be DB
> > > connections getting timed out.
> > >
> > > Currently we are using the following implementations:
> > >
> "org.apache.ctakes.dictionary.lookup2.dictionary.JdbcRareWordDictionary"
> > > and "org.apache.ctakes.dictionary.lookup2.concept.JdbcConceptFactory"
> > >
> > > Does anybody aware of any timeout settings that needs to be done in
> these
> > > implementations to avoid DB connection timeout issue?
> > >
> > > -Original Message-
> > > From: Kathy Ferro 
> > > Sent: Thursday, August 16, 2018 11:07 PM
> > > To: dev@ctakes.apache.org
> > > Subject: ctake web service
> > >
> > > Hi,
> > >
> > > Just want to see if anybody has experience this issue.
> > >
> > > If the web service had been up for a day or two, it will drop the
> > > dictionary lookup.  The only result it returns are ConllDependencyNode
> > tag
> > > in the xmi file;  no mention, no concept, etc...
> > >
> > > I haven't have a chance to investigate it, yet.
> > >
> > > Kathy
> > > This email and any files transmitted with it are confidential and
> > intended
> > > solely for the use of the individual or entity to whom they are
> > addressed.
> > > If you are not the named addressee you should not disseminate,
> distribute
> > > or copy this e-mail. Please notify the sender or system manager by
> email
> > > immediately if you have received this e-mail by mistake and delete this
> > > e-mail from your system. If you are not the intended recipient you are
> > > notified that disclosing, copying, distributing or taking any action in
> > > reliance on the contents of this information is strictly prohibited and
> > > against the law.
> > >
> >
>
>
> --
> Regards,
> Gandhi
>
> "The best way to find urself is to lose urself in the service of others
> !!!"
>


Re: ctake web service [EXTERNAL] [SUSPICIOUS]

2019-03-05 Thread Kathy Ferro
Thank you, Tim.
I don't know where I get Sean from.

On Tue, Mar 5, 2019 at 11:11 AM Miller, Timothy <
timothy.mil...@childrens.harvard.edu> wrote:

> The custom dictionary lookup descriptor I use is the one I checked into
> the svn repo:
>
> http://svn.apache.org/viewvc/ctakes/trunk/ctakes-web-rest/docker/customDictionary.xml?view=markup
>
> for this to work you need to have the snomed/rxnorm dictionary somewhere
> that the dictionary module looks for dictionaries (like in
> $CTAKES_HOME/resources/)
>
> You can also check out the Docker build file for a step-by-step of setting
> up and building the war file with that setup.
>
> http://svn.apache.org/viewvc/ctakes/trunk/ctakes-web-rest/docker/Dockerfile?view=markup
>
> Tim
>
>
> -Original Message-
> From: "Miller, Timothy"  %22Miller,%20timothy%22%20%3ctimothy.mil...@childrens.harvard.edu%3e>>
> Reply-to: 
> To: dev@ctakes.apache.org  22...@ctakes.apache.org%22%20%3c...@ctakes.apache.org%3e>>
> Subject: Re: ctake web service [EXTERNAL] [SUSPICIOUS]
> Date: Tue, 5 Mar 2019 15:34:18 +
>
>
> Assuming you meant me, I'm hosting on an ubuntu linux machine, and I'm
> using the hsql dictionary instead of the mysql dictionary.
> Tim
>
>
> -Original Message-
> From: Kathy Ferro  healthcare1...@gmail.com> kathy%20ferro%20%3chealthcare1...@gmail.com%3e>>
> Reply-to: mailto:dev@ctakes.apache.org>>
> To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org> dev@ctakes.apache.org>
> Subject: Re: ctake web service [EXTERNAL]
> Date: Mon, 4 Mar 2019 22:57:22 -0500
>
>
> Sean,
>
> What machine do you hosting the WS and mysql on?  I am on window 10
> server.  mySQL ini file looks fine.  I'm wondering window and mysql are not
> being friend.
>
> Thanks
> Kathy
>
> On Mon, Mar 4, 2019 at 10:52 AM Miller, Timothy <
> timothy.mil...@childrens.harvard.edu timothy.mil...@childrens.harvard.edu> timothy.mil...@childrens.harvard.edu>> wrote:
>
>
>
> I don't know what the solution was, but I leave my ctakes REST server
> running basically full time and haven't seen time outs yet.
> Tim
>
> 
> From: gandhi rajan mailto:gandhiraja...@gmail.com
> ><mailto:gandhiraja...@gmail.com>>
> Sent: Monday, March 4, 2019 10:43 AM
> To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org> dev@ctakes.apache.org>
> Subject: Re: ctake web service [EXTERNAL]
>
> Hi Kathy, Sean did respond that there is no timeout happening from cTAKES
> end. You might probably have to look at database settings for this closed
> connection issue.
>
> Does someone have any clue on this?
>
> On Monday, March 4, 2019, Kathy Ferro  healthcare1...@gmail.com><mailto:healthcare1...@gmail.com>> wrote:
>
>
>
> Gandhi,
>
> Do you get any response to this issue?  Does it try to keep the
>
>
> connection
>
>
> open while WS is up? Or does it open and close after it's done?
>
> We are still getting this error.
> "ERROR JdbcRareWordDictionary - No operations allowed after statement
> closed."
>
> Thanks
> Kathy
>
>
>
> On Fri, Aug 17, 2018 at 9:43 AM Gandhi Rajan Natarajan <
> gandhi.natara...@arisglobal.com<mailto:gandhi.natara...@arisglobal.com
> ><mailto:gandhi.natara...@arisglobal.com>> wrote:
>
>
>
> Hi Kathy,
>
> Sometime back we encountered this issue and the problem seems to be DB
> connections getting timed out.
>
> Currently we are using the following implementations:
>
>
>
>
>
> "org.apache.ctakes.dictionary.lookup2.dictionary.JdbcRareWordDictionary"
>
>
>
>
> and "org.apache.ctakes.dictionary.lookup2.concept.JdbcConceptFactory"
>
> Does anybody aware of any timeout settings that needs to be done in
>
>
>
>
> these
>
>
>
>
> implementations to avoid DB connection timeout issue?
>
> -Original Message-
> From: Kathy Ferro  healthcare1...@gmail.com><mailto:healthcare1...@gmail.com>>
> Sent: Thursday, August 16, 2018 11:07 PM
> To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org> dev@ctakes.apache.org>
> Subject: ctake web service
>
> Hi,
>
> Just want to see if anybody has experience this issue.
>
> If the web service had been up for a day or two, it will drop the
> dictionary lookup.  The only result it returns are ConllDependencyNode
>
>
> tag
>
>
> in the xmi file;  no mention, no concept, etc...
>
> I haven't have a chance to investigate it, yet.
>
> Kathy
> This email and any files transmitted with it

Re: ctake web service [EXTERNAL]

2019-03-07 Thread Kathy Ferro
Tim,

Does docker solution handle multiple instances?  I tested the Rest Web
Service with 2 requests at the same time, it errors out.  I removed the
part that write the result xml file to the disc; it still error out.

Best,
Kathy

On Mon, Mar 4, 2019 at 10:52 AM Miller, Timothy <
timothy.mil...@childrens.harvard.edu> wrote:

> I don't know what the solution was, but I leave my ctakes REST server
> running basically full time and haven't seen time outs yet.
> Tim
>
> 
> From: gandhi rajan 
> Sent: Monday, March 4, 2019 10:43 AM
> To: dev@ctakes.apache.org
> Subject: Re: ctake web service [EXTERNAL]
>
> Hi Kathy, Sean did respond that there is no timeout happening from cTAKES
> end. You might probably have to look at database settings for this closed
> connection issue.
>
> Does someone have any clue on this?
>
> On Monday, March 4, 2019, Kathy Ferro  wrote:
>
> > Gandhi,
> >
> > Do you get any response to this issue?  Does it try to keep the
> connection
> > open while WS is up? Or does it open and close after it's done?
> >
> > We are still getting this error.
> > "ERROR JdbcRareWordDictionary - No operations allowed after statement
> > closed."
> >
> > Thanks
> > Kathy
> >
> >
> >
> > On Fri, Aug 17, 2018 at 9:43 AM Gandhi Rajan Natarajan <
> > gandhi.natara...@arisglobal.com> wrote:
> >
> > > Hi Kathy,
> > >
> > > Sometime back we encountered this issue and the problem seems to be DB
> > > connections getting timed out.
> > >
> > > Currently we are using the following implementations:
> > >
> "org.apache.ctakes.dictionary.lookup2.dictionary.JdbcRareWordDictionary"
> > > and "org.apache.ctakes.dictionary.lookup2.concept.JdbcConceptFactory"
> > >
> > > Does anybody aware of any timeout settings that needs to be done in
> these
> > > implementations to avoid DB connection timeout issue?
> > >
> > > -Original Message-
> > > From: Kathy Ferro 
> > > Sent: Thursday, August 16, 2018 11:07 PM
> > > To: dev@ctakes.apache.org
> > > Subject: ctake web service
> > >
> > > Hi,
> > >
> > > Just want to see if anybody has experience this issue.
> > >
> > > If the web service had been up for a day or two, it will drop the
> > > dictionary lookup.  The only result it returns are ConllDependencyNode
> > tag
> > > in the xmi file;  no mention, no concept, etc...
> > >
> > > I haven't have a chance to investigate it, yet.
> > >
> > > Kathy
> > > This email and any files transmitted with it are confidential and
> > intended
> > > solely for the use of the individual or entity to whom they are
> > addressed.
> > > If you are not the named addressee you should not disseminate,
> distribute
> > > or copy this e-mail. Please notify the sender or system manager by
> email
> > > immediately if you have received this e-mail by mistake and delete this
> > > e-mail from your system. If you are not the intended recipient you are
> > > notified that disclosing, copying, distributing or taking any action in
> > > reliance on the contents of this information is strictly prohibited and
> > > against the law.
> > >
> >
>
>
> --
> Regards,
> Gandhi
>
> "The best way to find urself is to lose urself in the service of others
> !!!"
>


Re: ctake web service [EXTERNAL]

2019-03-07 Thread Kathy Ferro
Tim,

Thanks for reply.  I'm continuing the research.  With all the layers that
wrap around this, you would think we can handle this suggestion.





On Thu, Mar 7, 2019 at 8:01 PM Miller, Timothy <
timothy.mil...@childrens.harvard.edu> wrote:

> That's a good question that I've also heard from others, and unfortunately
> I don't know the answer. My use cases are typically a single job at a time
> making sequential calls, so I wasn't stressing it with multiple
> asynchronous calls. I would've thought that the Tomcat container would have
> some ability to manage that though!
> Tim
>
> 
> From: Kathy Ferro 
> Sent: Thursday, March 7, 2019 6:10 PM
> To: dev@ctakes.apache.org
> Subject: Re: ctake web service [EXTERNAL]
>
> Tim,
>
> Does docker solution handle multiple instances?  I tested the Rest Web
> Service with 2 requests at the same time, it errors out.  I removed the
> part that write the result xml file to the disc; it still error out.
>
> Best,
> Kathy
>
> On Mon, Mar 4, 2019 at 10:52 AM Miller, Timothy <
> timothy.mil...@childrens.harvard.edu> wrote:
>
> > I don't know what the solution was, but I leave my ctakes REST server
> > running basically full time and haven't seen time outs yet.
> > Tim
> >
> > 
> > From: gandhi rajan 
> > Sent: Monday, March 4, 2019 10:43 AM
> > To: dev@ctakes.apache.org
> > Subject: Re: ctake web service [EXTERNAL]
> >
> > Hi Kathy, Sean did respond that there is no timeout happening from cTAKES
> > end. You might probably have to look at database settings for this closed
> > connection issue.
> >
> > Does someone have any clue on this?
> >
> > On Monday, March 4, 2019, Kathy Ferro  wrote:
> >
> > > Gandhi,
> > >
> > > Do you get any response to this issue?  Does it try to keep the
> > connection
> > > open while WS is up? Or does it open and close after it's done?
> > >
> > > We are still getting this error.
> > > "ERROR JdbcRareWordDictionary - No operations allowed after statement
> > > closed."
> > >
> > > Thanks
> > > Kathy
> > >
> > >
> > >
> > > On Fri, Aug 17, 2018 at 9:43 AM Gandhi Rajan Natarajan <
> > > gandhi.natara...@arisglobal.com> wrote:
> > >
> > > > Hi Kathy,
> > > >
> > > > Sometime back we encountered this issue and the problem seems to be
> DB
> > > > connections getting timed out.
> > > >
> > > > Currently we are using the following implementations:
> > > >
> > "org.apache.ctakes.dictionary.lookup2.dictionary.JdbcRareWordDictionary"
> > > > and "org.apache.ctakes.dictionary.lookup2.concept.JdbcConceptFactory"
> > > >
> > > > Does anybody aware of any timeout settings that needs to be done in
> > these
> > > > implementations to avoid DB connection timeout issue?
> > > >
> > > > -Original Message-
> > > > From: Kathy Ferro 
> > > > Sent: Thursday, August 16, 2018 11:07 PM
> > > > To: dev@ctakes.apache.org
> > > > Subject: ctake web service
> > > >
> > > > Hi,
> > > >
> > > > Just want to see if anybody has experience this issue.
> > > >
> > > > If the web service had been up for a day or two, it will drop the
> > > > dictionary lookup.  The only result it returns are
> ConllDependencyNode
> > > tag
> > > > in the xmi file;  no mention, no concept, etc...
> > > >
> > > > I haven't have a chance to investigate it, yet.
> > > >
> > > > Kathy
> > > > This email and any files transmitted with it are confidential and
> > > intended
> > > > solely for the use of the individual or entity to whom they are
> > > addressed.
> > > > If you are not the named addressee you should not disseminate,
> > distribute
> > > > or copy this e-mail. Please notify the sender or system manager by
> > email
> > > > immediately if you have received this e-mail by mistake and delete
> this
> > > > e-mail from your system. If you are not the intended recipient you
> are
> > > > notified that disclosing, copying, distributing or taking any action
> in
> > > > reliance on the contents of this information is strictly prohibited
> and
> > > > against the law.
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Gandhi
> >
> > "The best way to find urself is to lose urself in the service of others
> > !!!"
> >
>


Re: ctake web service [EXTERNAL]

2019-03-08 Thread Kathy Ferro
I thought about creating a queue that acts as traffic cop.  Only the
traffic cop calls the WS.  I also want to test multiple WS running on
different port.  Traffic cop calls which every WS is available and keep
track of WS statuses.  With all this processing going, it might kill the
power for blocks.

On Fri, Mar 8, 2019 at 10:34 AM Finan, Sean <
sean.fi...@childrens.harvard.edu> wrote:

> Hi all,
>
> I guess that a quick test could be run with a multi-threaded pipeline.
> Tim, for some reason I recall you checking in one with a dockerfile.  Maybe
> not, and it might not be the default in the service.  Anyway, you could set
> the procs to something like 50 and throw 50 users at it.  It definitely
> does not scale anything close to linearly.  ctakes aes aren't build for
> thread-safety, so they are all wrapped with locks and there is a lot of
> thread contention.  However, running such a test might indicate the source
> of the problem.
>
> The other option is to create a queue that collects post calls and doles
> them out serially to a single pipeline.  User #50 would probably not
> appreciate it though ...
> 
> From: gandhi rajan 
> Sent: Friday, March 8, 2019 10:02 AM
> To: dev@ctakes.apache.org
> Subject: Re: ctake web service [EXTERNAL]
>
> Hi Kathy,
>
> I guess the initializations happens in post construct method. So if we
> could synchronize that I feel we can get away from the problem.
> Unfortunately I m not able to tet this as my setup is gone with my old job.
> Try it out.
>
> Regards,
> Gandhi.
>
> On Friday, March 8, 2019, Kathy Ferro  wrote:
>
> > Tim,
> >
> > Thanks for reply.  I'm continuing the research.  With all the layers that
> > wrap around this, you would think we can handle this suggestion.
> >
> >
> >
> >
> >
> > On Thu, Mar 7, 2019 at 8:01 PM Miller, Timothy <
> > timothy.mil...@childrens.harvard.edu> wrote:
> >
> > > That's a good question that I've also heard from others, and
> > unfortunately
> > > I don't know the answer. My use cases are typically a single job at a
> > time
> > > making sequential calls, so I wasn't stressing it with multiple
> > > asynchronous calls. I would've thought that the Tomcat container would
> > have
> > > some ability to manage that though!
> > > Tim
> > >
> > > 
> > > From: Kathy Ferro 
> > > Sent: Thursday, March 7, 2019 6:10 PM
> > > To: dev@ctakes.apache.org
> > > Subject: Re: ctake web service [EXTERNAL]
> > >
> > > Tim,
> > >
> > > Does docker solution handle multiple instances?  I tested the Rest Web
> > > Service with 2 requests at the same time, it errors out.  I removed the
> > > part that write the result xml file to the disc; it still error out.
> > >
> > > Best,
> > > Kathy
> > >
> > > On Mon, Mar 4, 2019 at 10:52 AM Miller, Timothy <
> > > timothy.mil...@childrens.harvard.edu> wrote:
> > >
> > > > I don't know what the solution was, but I leave my ctakes REST server
> > > > running basically full time and haven't seen time outs yet.
> > > > Tim
> > > >
> > > > 
> > > > From: gandhi rajan 
> > > > Sent: Monday, March 4, 2019 10:43 AM
> > > > To: dev@ctakes.apache.org
> > > > Subject: Re: ctake web service [EXTERNAL]
> > > >
> > > > Hi Kathy, Sean did respond that there is no timeout happening from
> > cTAKES
> > > > end. You might probably have to look at database settings for this
> > closed
> > > > connection issue.
> > > >
> > > > Does someone have any clue on this?
> > > >
> > > > On Monday, March 4, 2019, Kathy Ferro 
> > wrote:
> > > >
> > > > > Gandhi,
> > > > >
> > > > > Do you get any response to this issue?  Does it try to keep the
> > > > connection
> > > > > open while WS is up? Or does it open and close after it's done?
> > > > >
> > > > > We are still getting this error.
> > > > > "ERROR JdbcRareWordDictionary - No operations allowed after
> statement
> > > > > closed."
> > > > >
> > > > > Thanks
> > > > > Kathy
> > > > >
> > > > >
> > > > >
> > > > &g