Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
I don't know why it doesn't work because I have explicitly set the brokerUrl to
the deployment descriptors, ran the UIMA_Service with argument
-defaultBrokerURL and set also the system property -DdefaultBrokerURL. All were
set with value:
tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0wireFormat.maxFrameSize=209715200jms.useCompression=true.
The pipeline has the following components: one main aggregate with a remote RDF
Consumer and one remote aggregate. The remote aggregate has one remote
primitive annotator and one local primitive annotator.
I don't know exactly how many connections are created for the services to
communicate with each other but in the log file of each of the components I
noticed that there is ONLY ONE connection which is built with default frame
size of 100 MB instead of 200 MB as the others.
In the aggregators I see a line like the following:
02/13/2014 05:36:37.081 [DEBUG] [main]
org.apache.activemq.transport.WireFormatNegotiator - Sending: WireFormatInfo {
version=9, properties={MaxFrameSize=104857600, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true},
magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.082 [TRACE] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.tcp.TcpTransport
- TCP consumer thread for tcp://localhost/127.0.0.1:61616 starting
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616]
org.apache.activemq.transport.InactivityMonitor - Using min of local:
WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true},
magic=[A,c,t,i,v,e,M,Q]} and remote: WireFormatInfo { version=9,
properties={CacheSize=1024, MaxFrameSize=209715200, CacheEnabled=true,
SizePrefixDisabled=false, TcpNoDelayEnabled=true,
MaxInactivityDurationInitalDelay=1, MaxInactivityDuration=0,
TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616]
org.apache.activemq.transport.WireFormatNegotiator - Received WireFormat:
WireFormatInfo { version=9, properties={CacheSize=1024, MaxFrameSize=209715200,
CacheEnabled=true, SizePrefixDisabled=false, TcpNoDelayEnabled=true,
MaxInactivityDurationInitalDelay=1, MaxInactivityDuration=0,
TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616]
org.apache.activemq.transport.WireFormatNegotiator -
tcp://localhost/127.0.0.1:61616 before negotiation: OpenWireFormat{version=9,
cacheEnabled=false, stackTraceEnabled=false, tightEncodingEnabled=false,
sizePrefixDisabled=false, maxFrameSize=104857600}
02/13/2014 05:36:37.083 [DEBUG] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616]
org.apache.activemq.transport.WireFormatNegotiator -
tcp://localhost/127.0.0.1:61616 after negotiation: OpenWireFormat{version=9,
cacheEnabled=true, stackTraceEnabled=true, tightEncodingEnabled=true,
sizePrefixDisabled=false, maxFrameSize=104857600}
and in the primitive annotator I see the following:
02/13/2014 05:36:37.122 [DEBUG]
[primitive_input_msgLsnrCntnr_TokenizerQueue_1_getMeta - Thread - 18]
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer -
Received message of type [class
org.apache.activemq.command.ActiveMQTextMessage] from consumer
[ActiveMQMessageConsumer { value=ID:PCCS000630WS-56652-1392287793445-1:2:1:1,
started=true }] of session [ActiveMQSession
{id=ID:PCCS000630WS-56652-1392287793445-1:2:1,started=true}]
02/13/2014 05:36:37.143
[DEBUG][primitive_input_msgLsnrCntnr_TokenizerQueue_1_getMeta - Thread - 18]
org.apache.activemq.transport.WireFormatNegotiator - Sending: WireFormatInfo {
version=9, properties={MaxFrameSize=104857600, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=0, TightEncodingEnabled=true, StackTraceEnabled=true},
magic=[A,c,t,i,v,e,M,Q]}
02/13/2014 05:36:37.146 [TRACE] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616] org.apache.activemq.transport.tcp.TcpTransport
- TCP consumer thread for tcp://localhost/127.0.0.1:61616 starting
02/13/2014 05:36:37.146 [DEBUG] [ActiveMQ Transport:
tcp://localhost/127.0.0.1:61616]
org.apache.activemq.transport.InactivityMonitor - Using min of local:
WireFormatInfo { version=9, properties={MaxFrameSize=104857600, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=1, TcpNoDelayEnabled=true,
MaxInactivityDuration=0, TightEncodingEnabled=true,
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
Hello, I have upgraded uima-as to version 2.4.2 but I still encounter an issue with the wireFormat.maxFrameSize setting for the ActiveMQ broker. 1. I have updated the configuration for transport connector in activemq.xml file: transportConnectors transportConnector name=openwire uri=tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0amp;wireFormat.maxFrameSize=209715200amp;jms.useCompression=true/ /transportConnectors 2. I have set the brokerURL attribute in uima-as deployment descriptors to value: tcp://127.0.0.1:61616?wireFormat.maxInactivityDuration=0amp;wireFormat.maxFrameSize=209715200amp;jms.useCompression=true 3. I have set the TRACE level for logger org.apache.activemq.transport After performing all the above settings I noticed that when I started the pipeline, for each remote delegate, multiple negotiations are performed by org.apache.activemq.transport.WireFormatNegotiator. All use the maxFrameSize of 200 MB that I specified, except one negotiation that is done using maxFrameSize of 100 MB. I do not understand from where does come this limitation of 100 MB. Does exist in the UIMA client? By default I saw that ActiveMQ is using MAX_LONG for maxFrameSize so I really don't know from where does come this 100 MB setting for maxFrameSize. Does anyone have an idea why is happening this? Could somebody tell me a starting point for looking in the uima code? On the other hand does anybody know whether there are some limitations when using the binary serializer for remote delegates instead of xmi serializer? I found in one jira issue (https://issues.apache.org/jira/browse/UIMA-1196) that for the binary serializer is mandatory that all uima AS services use a common type system. Is this still an issue in uima-as 2.4.2? Thank you! Mihaela On Monday, January 27, 2014 4:30 PM, Eddie Epstein eaepst...@gmail.com wrote: On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter thomas.gin...@utah.eduwrote: It is likely then that your expansion is happening after the remote service is called or else is not yet big enough to be over the 100MB limit. Also note that by default UIMA-AS [Java] services use a delta-CAS interface. Only changes to the CAS are returned from a service. Besides deleting unnecessary FS from the final CAS to be returned, another option to consider is to use compression on JMS messages: jms.useCompression=true This decoration can be added to the broker configuration file, $UIMA_HOME/amq/conf/activemq-nojournal.xml as transportConnector name=openwire uri=tcp:// 0.0.0.0:61616?jms.useCompression=true/ which will cause messages in all queues to be compressed. Eddie
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
It seems like the ActimeMQ documentation (
http://activemq.apache.org/configuring-wire-formats.html)
is wrong with respect to the default maxFrameSize being MAX_LONG. I checked
ActiveMQ source code and the default is 100 MB:
public final class OpenWireFormat implements WireFormat {public
static final int DEFAULT_VERSION =
CommandTypes.PROTOCOL_STORE_VERSION;public static final int
DEFAULT_WIRE_VERSION = CommandTypes.PROTOCOL_VERSION;public static
final int *DEFAULT_MAX_FRAME_SIZE* = 100 * 1024 * 1024; //100 MB
-static final byte NULL_TYPE
= CommandTypes.NULL;private static final int MARSHAL_CACHE_SIZE =
Short.MAX_VALUE / 2;private static final int
MARSHAL_CACHE_FREE_SPACE = 100;
The UIMA-AS doesnt set this value so the default is being used unless
overriden. It seems to me that
either your service or a client is not overriding the default. Please check
your deployment descriptors to make sure
that you changing the default in the brokerURL.
Jerry
On Wed, Feb 12, 2014 at 9:21 AM, Mihaela M mmihaela1...@yahoo.com wrote:
Hello,
I have upgraded uima-as to version 2.4.2 but I still encounter an issue
with the wireFormat.maxFrameSize setting for the ActiveMQ broker.
1. I have updated the configuration for transport connector in
activemq.xml file:
transportConnectors
transportConnector name=openwire uri=tcp://
127.0.0.1:61616?wireFormat.maxInactivityDuration=0wireFormat.maxFrameSize=209715200jms.useCompression=true
/
/transportConnectors
2. I have set the brokerURL attribute in uima-as deployment descriptors to
value: tcp://
127.0.0.1:61616?wireFormat.maxInactivityDuration=0wireFormat.maxFrameSize=209715200jms.useCompression=true
3. I have set the TRACE level for logger org.apache.activemq.transport
After performing all the above settings I noticed that when I started the
pipeline, for each remote delegate, multiple negotiations are performed
by org.apache.activemq.transport.WireFormatNegotiator. All use the
maxFrameSize of 200 MB that I specified, except one negotiation that is
done using maxFrameSize of 100 MB.
I do not understand from where does come this limitation of 100 MB. Does
exist in the UIMA client? By default I saw that ActiveMQ is using MAX_LONG
for maxFrameSize so I really don't know from where does come this 100 MB
setting for maxFrameSize.
Does anyone have an idea why is happening this? Could somebody tell me a
starting point for looking in the uima code?
On the other hand does anybody know whether there are some limitations
when using the binary serializer for remote delegates instead of xmi
serializer? I found in one jira issue (
https://issues.apache.org/jira/browse/UIMA-1196) that for the binary
serializer is mandatory that all uima AS services use a common type system.
Is this still an issue in uima-as 2.4.2?
Thank you!
Mihaela
On Monday, January 27, 2014 4:30 PM, Eddie Epstein eaepst...@gmail.com
wrote:
On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter thomas.gin...@utah.edu
wrote:
It is likely then that your expansion is happening after the remote
service is called or else is not yet big enough to be over the 100MB
limit.
Also note that by default UIMA-AS [Java] services use a delta-CAS
interface. Only changes to the CAS
are returned from a service.
Besides deleting unnecessary FS from the final CAS to be returned, another
option to consider is to use compression on JMS messages:
jms.useCompression=true
This decoration can be added to the broker configuration file,
$UIMA_HOME/amq/conf/activemq-nojournal.xml
as
transportConnector name=openwire uri=tcp://
0.0.0.0:61616?jms.useCompression=true/
which will cause messages in all queues to be compressed.
Eddie
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
On Thu, Jan 23, 2014 at 9:28 AM, Thomas Ginter thomas.gin...@utah.eduwrote: It is likely then that your expansion is happening after the remote service is called or else is not yet big enough to be over the 100MB limit. Also note that by default UIMA-AS [Java] services use a delta-CAS interface. Only changes to the CAS are returned from a service. Besides deleting unnecessary FS from the final CAS to be returned, another option to consider is to use compression on JMS messages: jms.useCompression=true This decoration can be added to the broker configuration file, $UIMA_HOME/amq/conf/activemq-nojournal.xml as transportConnector name=openwire uri=tcp:// 0.0.0.0:61616?jms.useCompression=true/ which will cause messages in all queues to be compressed. Eddie
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
1. Your annotators can remove as well as add annotations. Perhaps if there is a large number of annotations that you don’t really need you could have a clean up annotator that removes the extra stuff, or else just don’t generate it in the first place, whatever works best for your algorithm. 2. Remote services in your pipeline are serialized the same way as the serialization with the client. In fact the framework essentially creates a client interface for sending and receiving CAS objects and then passing them to/from your pipeline. It is likely then that your expansion is happening after the remote service is called or else is not yet big enough to be over the 100MB limit. Thanks, Thomas Ginter 801-448-7676 thomas.gin...@utah.edu On Jan 23, 2014, at 12:53 AM, Mihaela M mmihaela1...@yahoo.com wrote: 1. I will upgrade uima-as and review the annotations gathered in the CAS, but is it a way to have the CAS reset before sending it to the client? In my case I only want to get the status of the processing, not all the annotations found, because they were handled by the consumers configured in the pipeline anyway. 2. Do you know whether the aggregates communicate with the clients the same as with the remote CAS consumers? I wonder why it did not complain while sending the exploded CAS to the remote consumer, but it did when communicating with the client. Thank you! Mihaela On Wednesday, January 22, 2014 7:07 PM, Thomas Ginter thomas.gin...@utah.edu wrote: Mihaela, There are two things that you should probably do in order to get started with these issues. 1. Upgrade to UIMA-AS 2.4.2 which uses a newer version of ActiveMQ and contains numerous bug fixes for UIMA-AS related to how the JMS queues are handled. 2. The UIMA-AS framework adds very little as far as overhead space for the CAS objects which means the vast majority of the size expansion from 48KB to 147MB is coming from annotations/metadata being added by your service. Increasing the frame size in ActiveMQ may allow your CAS objects to be transferred in JMS but it is more important to find out what is causing this dramatic expansion and whether or not the service can be written differently so that the expansion is much smaller. Thanks, Thomas Ginter 801-448-7676 thomas.gin...@utah.edu On Jan 22, 2014, at 9:44 AM, Mihaela M mmihaela1...@yahoo.com wrote: Hello, I have a uima pipeline that uses uima-as 2.3.1 which has one aggregator with one local annotator, one remote consumer and one remote annotator. It actually has more components but I will get into exactly the configuration only if needed. I have developed also a UIMA client for it using class: UimaAsynchronousEngine, method sendCas (async as far I understood) and a callback listener that waits for the processing to complete. 1. I have noticed that the CAS returned, in general is quite big. Is it a way to send, at least to the client, a CAS that does not contain all the types that the various annotators added? When could I remove those things from the CAS? 2. I send a text message for processing which has 48 KB - it gets processed successfully by the pipeline, but the pipeline fails to send a reply to the client. The exception that I get is: 01/21/2014 07:36:02.978 [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] [DEBUG] org.apache.activemq.ActiveMQConnection - Async exception with no exception listener: java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:229) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:221) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) ~[activemq-core-5.6.0.jar:5.6.0] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30] 01/21/2014 07:36:03.093 [ActiveMQ Connection Executor: tcp://localhost/127.0.0.1:61616] [DEBUG] org.apache.activemq.transport.tcp.TcpTransport - Stopping transport tcp://localhost/127.0.0.1:61616 As far as I understood, the client connects via JMS to the uima pipeline and a temporary reply queue gets created where the reply from the pipeline should be sent and then consumed by the client. After the above exception is thrown, the connection to the pipeline gets closed and automatically the temp queue gets deleted hence the client does not receive anymore the reply. I am wondering why the error I was mentioning is not thrown while the aggregator sends the CAS to the consumer, because the consumer is
Re: uima-as 2.3.1 - java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB
1. I will upgrade uima-as and review the annotations gathered in the CAS, but is it a way to have the CAS reset before sending it to the client? In my case I only want to get the status of the processing, not all the annotations found, because they were handled by the consumers configured in the pipeline anyway. 2. Do you know whether the aggregates communicate with the clients the same as with the remote CAS consumers? I wonder why it did not complain while sending the exploded CAS to the remote consumer, but it did when communicating with the client. Thank you! Mihaela On Wednesday, January 22, 2014 7:07 PM, Thomas Ginter thomas.gin...@utah.edu wrote: Mihaela, There are two things that you should probably do in order to get started with these issues. 1. Upgrade to UIMA-AS 2.4.2 which uses a newer version of ActiveMQ and contains numerous bug fixes for UIMA-AS related to how the JMS queues are handled. 2. The UIMA-AS framework adds very little as far as overhead space for the CAS objects which means the vast majority of the size expansion from 48KB to 147MB is coming from annotations/metadata being added by your service. Increasing the frame size in ActiveMQ may allow your CAS objects to be transferred in JMS but it is more important to find out what is causing this dramatic expansion and whether or not the service can be written differently so that the expansion is much smaller. Thanks, Thomas Ginter 801-448-7676 thomas.gin...@utah.edu On Jan 22, 2014, at 9:44 AM, Mihaela M mmihaela1...@yahoo.com wrote: Hello, I have a uima pipeline that uses uima-as 2.3.1 which has one aggregator with one local annotator, one remote consumer and one remote annotator. It actually has more components but I will get into exactly the configuration only if needed. I have developed also a UIMA client for it using class: UimaAsynchronousEngine, method sendCas (async as far I understood) and a callback listener that waits for the processing to complete. 1. I have noticed that the CAS returned, in general is quite big. Is it a way to send, at least to the client, a CAS that does not contain all the types that the various annotators added? When could I remove those things from the CAS? 2. I send a text message for processing which has 48 KB - it gets processed successfully by the pipeline, but the pipeline fails to send a reply to the client. The exception that I get is: 01/21/2014 07:36:02.978 [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] [DEBUG] org.apache.activemq.ActiveMQConnection - Async exception with no exception listener: java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB java.io.IOException: Frame size of 147 MB larger than max allowed 100 MB at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:229) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:221) ~[activemq-core-5.6.0.jar:5.6.0] at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) ~[activemq-core-5.6.0.jar:5.6.0] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30] 01/21/2014 07:36:03.093 [ActiveMQ Connection Executor: tcp://localhost/127.0.0.1:61616] [DEBUG] org.apache.activemq.transport.tcp.TcpTransport - Stopping transport tcp://localhost/127.0.0.1:61616 As far as I understood, the client connects via JMS to the uima pipeline and a temporary reply queue gets created where the reply from the pipeline should be sent and then consumed by the client. After the above exception is thrown, the connection to the pipeline gets closed and automatically the temp queue gets deleted hence the client does not receive anymore the reply. I am wondering why the error I was mentioning is not thrown while the aggregator sends the CAS to the consumer, because the consumer is remote, hence the communication between them is done through JMS queue as well, and I think the aggregator has a reply queue as well for the consumer... I want to mention that I tried to increase the maxFrameSize on AMQ broker but without success. It seems to be a bug in AMQ 5.6 that uima as 2.3.1 is using. Any feedback is appreciated. Thank you, Mihaela
