Hi Jerry, thanks a lot for your answer! I’m sorry that I didn’t make myself clearer. I will try again! :-) Here comes a lot of text, sorry for that. The post actually has two parts: The first explaining my issue, the second responding to the pointer to UIMA-AS.
First: Yes, I use a CPE. I process text documents. Tens of millions of them. So, I have the following components to my issue, running with the CPE. 1. A CAS-Consumer (just an AnalysisEngine internally, of course). This consumer is responsible to serialise the document CAS into XMI and send the XMI to a database. It is a XMI-to-database consumer. For performance reasons, the XMI of multiple CASes is buffered and then sent as a batch, lets say, 50 CAS XMIs at a time. 2. A CPE StatusCallbackListener which also writes to the same database, but in another table. It logs into the database which documents have been successfully processed by the CPE. It also works on a batch basis. The goal: The CallbackListener should only mark those documents as successfully processed (i.e. as “finished”) where the CAS-Consumer actually has sent the XMI data to the database. Reason: I don’t want documents marked as “finished” where the XMI data is not in the database but still in the CAS buffer. Because when now the pipeline crashes, the XMI data never gets sent to the database. Then, the processing state is inconsistent: Documents that have not been written into the database are marked as successfully processed. But their data is missing. Also, not each XMI data is stored. There is a condition in the consumer to decide whether the XMI is to be stored or not. Thus, I cannot “create consistency” by checking which XMI made it into the database. Is this better understandable? Regarding UIMA-AS: I tried it out a few years back when it was rather new, UIMA 2.3.1 or something. Back then, it was like the following: 1. Install a broker (or something - ActiveMQ was it called?) 2. Configure it. 3. Start it. 4. For each AE you want to use, deploy the AE on some server in your cluster (multiple AEs can be bundled into an AAE). 5. Start a reader process that will then fill the broker queue. 6. Wait until processing is finished. 7. Stop all the AE services deployed to the cluster, if you want to save the resources. 8. Stop the broker. This was quite a while back so perhaps this is not exactly how it was. But it seemed overly complex to me. I had to login into each server where I wanted work to be done. We have like 20 nodes or something. Perhaps I could write a script for that, but then I would have to keep track of the servers that are free to use at a current time. Because I am not the only one using the cluster. And then I have to stop all AE “services”. Until then, they will use memory because they just idle when there is nothing more to do. In contrast, CPEs are self-contained projects in my case which I can distribute easily through our job system (SLURM). I thought all the setup for UIMA-AS would pay out in better performance. But in my - admittedly limited - tests there was not much of a performance difference. CPEs seemed to be a bit faster due to the lack of CAS serialization between reader and AEs. Of course, this was years in the past. Is the process a bit simpler today? Or perhaps I got it wrong to begin with, that’s possible. But I read the documentation back then and couldn’t see how to do things much simpler. BUT if CPEs can’t solve my issue and UIMA-AS can, then perhaps I will try it again. Another question: You said “CPE was replaced by UIMA-AS”. Does that mean that CPEs will eventually be removed from UIMA? Are they still a part of UIMA 3? Sorry for all the text! Best regards and thanks! Erik > On 20 Apr 2017, at 20:31, Jaroslaw Cwiklik <uim...@gmail.com> wrote: > > Hi Erik, sorry for a delay responding to your question. This seems like a > CPE question is this right? I am not quite following what is the issue you > are running into. Could you explain this better? With a clearer problem > description perhaps others will jump in with an answer :) > > Just FYI, the CPE was replaced by the UIMA-AS quite a long time ago. > Perhaps UIMA-AS can work better for you. You can read about it here: > https://uima.apache.org/d/uima-as-2.9.0/uima_async_scaleout.html > > Jerry > UIMA Team > > On Tue, Apr 18, 2017 at 5:56 AM, Erik Fäßler <erik.faess...@uni-jena.de> > wrote: > >> Hi all, >> >> I have a use case where a consumer of mine sends CAS XMI data into a >> database in batchProcessComplete(). I also use a StatusCallbackListener >> that logs into the database whether a document has been completed >> processing, this is also done batch wise. >> Now the issue is, if the pipeline crashes for any reason, I must start >> over because the “completion” flag from the CallbackListener and the data >> actually sent by the XMI consumer is not synchronised, i.e. I don’t know if >> the data has actually been sent for a document that has completed >> processing because everything is done batch-wise and not immediately for >> performance reasons. I also cannot just look into the database which XMI >> data is there because it only gets sent on a met condition. >> >> I would like to somehow communicate between the consumer and the >> CallbackListener to send their data for the same documents in agreement. Is >> there anything I can do to achieve this? >> >> Best, >> >> Erik
