RE: Rép. : RE: camel-mail IMAP mime-multipart splitAttachment

[Siano, Stephan]

Hi,

the name of the xml-element is "mime-multipart", not "mimeMultipart". Can you 
try that?

Best regards
Stephan

-Original Message-
From: Daniel Langevin [mailto:daniel.lange...@shq.gouv.qc.ca] 
Sent: Freitag, 8. Dezember 2017 19:56
To: users@camel.apache.org
Subject: Rép. : RE: camel-mail IMAP mime-multipart splitAttachment

Hi,

i try with 


   
  


 

but receive an error into deployement.

org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 147 
in XML document from URL 
[bundle://456.1:0/META-INF/spring/bdl.routes_XXX-0.0.1.xml] is invalid; nested 
exception is org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid 
content was found starting with element 'mimeMultipart'.

I don't understand i user Camel 2.17.7, it supposed to be there since 2.17.1  
?


Daniel Langevin








>>> "Siano, Stephan"  2017-12-08 01:50 >>>

Hi,

well, the error message means that the JMS endpoints cannot handle binary 
MIME-Messages. It tries to serialize the message in order to store it, but the 
data type is not (Java) serializable.

I see two ways how you could proceed. You could either marshal the MIME-Message 
with attachments into a serialized MIME-Multipart (either with the 
MIME-Multipart data type or with some settings in the imap-endpoint). This will 
give you one JMS message containing a MIME-Multipart with message and 
attachments (similar to the one that goes over the wire in the mail). Or you 
could convert each of the split parts into something serializable (a byte array 
or a String if it is text content) before sending it the the JMS queue 
(convertBodyTo...).

Best regards
Stephan

-Original Message-
From: Daniel Langevin [mailto:daniel.lange...@shq.gouv.qc.ca] 

Sent: Donnerstag, 7. Dezember 2017 19:55
To: users@camel.apache.org 

Subject: camel-mail IMAP mime-multipart splitAttachment

Hi,

i try to implement a MAIL service In OSGI Karaf 4.0.9  with Camel 2.17.7, 
Camel-Mail and Active-MQ and SQL
i want to read MAIL from IMAP server, then keep It into JMS Queue an then 
process each mail with mime-multipart with attachment and insert it into sql 
database

I stuck at the first Step process read MAIL and insert it into JMS.

My Case.
I have a MAIL with 2 attachments, i read the MAIL, split it into 2 messages 
into JMS Queue,  each message have a diiferent CamelSplitAttachment Id, 
corresponding the name of the attachment file, 
but i don't have the content of my MAIL or of my attachment

The body of each split contain: 

Unknown message type [org.apache.activemq.command.ActiveMQMessage] 
ActiveMQMessage {commandId = 5, responseRequired = true, messageId = .
...
...
...


I think i'm the good track, do you have an idea, what is missing ?

Only one error/warning in the log message:
JmsBinding| 46 - 
org.apache.camel.camel-jms - 2.17.7 | Cannot determine specific JmsMessage type 
to use from body class. Will use generic JmsMessage. Body class: 
javax.mail.internet.MimeMultipart. If you want to send a POJO then your class 
might need to implement java.io.Serializable, or you can force a specific type 
by setting the jmsMessageType option on the JMS endpoint.


Thanks!


Daniel








Here is my bundle.


http://www.springframework.org/schema/beans;

   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;

   xmlns:camel="http://camel.apache.org/schema/spring;

   xmlns:osgi="http://www.springframework.org/schema/osgi;

   xmlns:cxf="http://camel.apache.org/schema/cxf;

   xsi:schemaLocation="
  http://www.springframework.org/schema/beans  
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd 

  http://camel.apache.org/schema/cxf  
http://camel.apache.org/schema/cxf/camel-cxf.xsd 

  http://camel.apache.org/schema/spring  
http://camel.apache.org/schema/spring/camel-spring-2.17.7.xsd 

  http://www.springframework.org/schema/osgi 
http://www.springframework.org/schema/osgi/spring-osgi.xsd;
( 'http://www.springframework.org/schema/osgi/spring-osgi.xsd;') >




  
 
 
  



http://camel.apache.org/schema/spring; trace="false">





























"Le pr?sent courriel peut contenir des renseignements confidentiels et ne 
s'adresse qu'au destinataire dont le nom appara?t ci-dessus. Si ce courriel 
vous est parvenu par m?garde, veuillez le supprimer et nous en aviser aussit?t."


"Le pr?sent courriel peut contenir des renseignements confidentiels et ne 
s'adresse qu'au destinataire dont le nom appara?t ci-dessus. Si ce courriel 
vous est parvenu par m?garde, veuillez le supprimer et nous en aviser aussit?t."

Exchange stats

[Mihai Glonț]

Hi,

In our application we have a route from a SEDA queue to an Exec endpoint.

I know how to retrieve the number of pending exchanges, but is there a
way to tell how many are currently in progress (waiting for the external
process to finish), or have been processed by the current CamelContext
since the application started?

Thanks,

Mihai.

RE: MIME-Mutipart DataFormat streaming?

[Siano, Stephan]

Hi,

I actually don't understand that issue. Can you provide a simple test case for 
it? The stream involved in the code lines you are citing is actually the output 
stream, not the input stream. The multipart data format should also work on 
streams, so if there is an issue with that, that is probably a bug. If you want 
to open a JIRA issue for it, a unit test reproducing the issue would be great.

Best regards
Stephan

-Original Message-
From: Tim Dudgeon [mailto:tdudgeon...@gmail.com] 
Sent: Montag, 11. Dezember 2017 12:45
To: users@camel.apache.org
Subject: Re: MIME-Mutipart DataFormat streaming?

Following up on this again (after a long delay).

I tried using the MIME-Multipart DataFormat to send and hit a problem 
when the input stream for an attachment cannot be read multiple times. 
It looks like in the MimeMultipartDataFormat.marshal( ... ) method [1] 
this InputStream for an attachment is read on both of these lines:

Line 140: mm.saveChanges();
Line 154: mm.writeTo(stream, headers.toArray(new String[0]));

When using input that can be read repeatedly (e.g. ByteArrayInputStream) 
it works fine. When using an InputStream that can only be consumed once 
then it blows up the second time it is read.

I recognise that when reading a mulitpart message there can be 
complications, but when writing one it should be possible to do this by 
only reading the input once, so I believe this should be considered a bug.

Shall I raise an issue for this?

[1] 
https://github.com/apache/camel/blob/master/components/camel-mail/src/main/java/org/apache/camel/dataformat/mime/multipart/MimeMultipartDataFormat.java#L92:L160


On 26/10/16 06:43, Siano, Stephan wrote:
> Hi Tim,
>
> The MIME-Multipart DataFormat uses javamail as the underlying technology. The 
> DataFormat is using streaming as far as the DataFormat is concerned, but I 
> fear that javamail as such keeps a lot of stuff in memory so there is very 
> likely no full streaming with that component available. For marshalling this 
> might work with streaming as long as the component providing the Camel 
> Message with its attachments is stream capbable, but for unmarshalling, I 
> think that the javamail component will create ByteArrayDataSources at least 
> for the attachments.
>
> Best regards
> Stephan
>
> -Original Message-
> From: Tim Dudgeon [mailto:tdudgeon...@gmail.com]
> Sent: Dienstag, 25. Oktober 2016 14:21
> To: users@camel.apache.org
> Subject: MIME-Mutipart DataFormat streaming?
>
> I was looking at the MIME-Mutipart DataFormat and it looks very useful:
> https://camel.apache.org/mime-multipart.html
>
> But I was not sure if it fully streams the data for the body and
> attachments, or whether it holds the content in memory (this is for both
> marshalling and unmarshalling).
>
> Thanks
> Tim
>

netty4-http send chunked response

[Sharples, Colin]

I have a route that sends to a service running on DataPower, which by default 
sends responses as Transfer-Encoding: chunked.

I need to be able to set up a route to use in a test case that simulates this 
service, but I can't see any way to tell a netty4-http consumer to send a 
chunked response.

I've seen some old pages that suggest there is an option chunked=Boolean on the 
netty4-http component, but it is not mentioned in the 2.18 documentation. I 
tried setting it anyway, and it didn't seem to make a difference. At least, the 
message came back with a header chunked: true, but not Transfer-Encoding: 
chunked.

When I hook up the route to the DataPower service, when I get the response body 
as a string, it still has the chunked encoding in it - a hex integer indicating 
the length at the start, and a zero on the end to indicate the last chunk. I'm 
expecting the body to be a JSON string, so first I have to chop of the chunked 
encoding parts to get the proper JSON. Seems weird that I have to do this 
manually.

Is there another setting to tell it to use the right Transfer-Encoding?

Colin Sharples

"This e-mail and any attachments to it (the "Communication") is, unless 
otherwise stated, confidential, may contain copyright material and is for the 
use only of the intended recipient. If you receive the Communication in error, 
please notify the sender immediately by return e-mail, delete the Communication 
and the return e-mail, and do not read, copy, retransmit or otherwise deal with 
it. Any views expressed in the Communication are those of the individual sender 
only, unless expressly stated to be those of Australia and New Zealand Banking 
Group Limited ABN 11 005 357 522, or any of its related entities including ANZ 
Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in 
connection with the integrity of or errors in the Communication, computer 
virus, data corruption, interference or delay arising from or in respect of the 
Communication."

Writing to message body

[Tim Dudgeon]

I have a payload that is generated in a Camel Processor that allows its 
data to be obtained with a writeTo(OutputStream out) method.


Is there an easy way to use this to write to the output Message's body 
using this method?
e.g. Normally you might have an InputStream and and set it using 
something like exchange.getOut().setBody(inputStream), but here I have 
something that writes to an OutputSteam.


I know I can used a java.io.PipedInputStream if necessary but thinking 
there should be an easier way.

Re: Most efficient way to unmarshall a large web service response

[bob anderson]

Hi,

I'm not sure if this is helpful.
I use Camel Spring XML DSL to update a Master Data table from a suppliers
database. The Master Data is accessible to us via a webservice. The server
this runs on has only 500Mb of heap memory allocated. I use CXF to handle
the Webservice call and had issues with the payload size - its
approximately 180,000 records with approx size of 200Mb (so smaller than
yours). I am not technical and cobbled the solution from reading Camel
documentation, examples and copying snippets from the web..

I found a reference on stack overflow (can't remember the link) that
suggested disabling the payload limits






org.apache.cxf.stax.maxChildElements
100




AND I stream the the data splitting on the individual record within the
response, also increased the continuation timeout value - as I was timing
out too.
I set the exchange body to the soap request and then make the SOAP call to
the web service as follows..



 http://MyWebServiceURL/GetMasterdataAction
  

 http://MyWebServiceURL/Latest/Data.svc?
wsdlURL=http://MyWebServiceURL/Latest/Data.svc?Wsdl;
serviceName={
http://wcf.isolutions.it/Their.API.Data/1.0/Data;

dataFormat=PAYLOADallowStreaming=truecontinuationTimeout=8
defaultOperationName=GetMasterdataAction"/>


  

  //b:Record
 
   

The solution works. takes some time though 





On Fri, Dec 8, 2017 at 11:27 PM, Gokul J  wrote:

> Hi All,
>
>   So we use Apache camel in order to consume web services via spring WA
> and recently we are being plagued with OOM issues relating to heap size .
> The response is around 675 MB in size and we had to increase the heap by 5
> GB to successfully unmarshall it . We use vanilla jaxb data format . Is
> there anything we can do to reduce the heap usage even if it costs us some
> time ? Any ideas would be highly appreciated.
>
> Sent from my iPhone




-- 
Bob Anderson
+27 (0) 82 389 0335
[image: View my profile on LinkedIn]

Re: MIME-Mutipart DataFormat streaming?

[Tim Dudgeon]

Following up on this again (after a long delay).

I tried using the MIME-Multipart DataFormat to send and hit a problem 
when the input stream for an attachment cannot be read multiple times. 
It looks like in the MimeMultipartDataFormat.marshal( ... ) method [1] 
this InputStream for an attachment is read on both of these lines:


Line 140: mm.saveChanges();
Line 154: mm.writeTo(stream, headers.toArray(new String[0]));

When using input that can be read repeatedly (e.g. ByteArrayInputStream) 
it works fine. When using an InputStream that can only be consumed once 
then it blows up the second time it is read.


I recognise that when reading a mulitpart message there can be 
complications, but when writing one it should be possible to do this by 
only reading the input once, so I believe this should be considered a bug.


Shall I raise an issue for this?

[1] 
https://github.com/apache/camel/blob/master/components/camel-mail/src/main/java/org/apache/camel/dataformat/mime/multipart/MimeMultipartDataFormat.java#L92:L160



On 26/10/16 06:43, Siano, Stephan wrote:

Hi Tim,

The MIME-Multipart DataFormat uses javamail as the underlying technology. The 
DataFormat is using streaming as far as the DataFormat is concerned, but I fear 
that javamail as such keeps a lot of stuff in memory so there is very likely no 
full streaming with that component available. For marshalling this might work 
with streaming as long as the component providing the Camel Message with its 
attachments is stream capbable, but for unmarshalling, I think that the 
javamail component will create ByteArrayDataSources at least for the 
attachments.

Best regards
Stephan

-Original Message-
From: Tim Dudgeon [mailto:tdudgeon...@gmail.com]
Sent: Dienstag, 25. Oktober 2016 14:21
To: users@camel.apache.org
Subject: MIME-Mutipart DataFormat streaming?

I was looking at the MIME-Mutipart DataFormat and it looks very useful:
https://camel.apache.org/mime-multipart.html

But I was not sure if it fully streams the data for the body and
attachments, or whether it holds the content in memory (this is for both
marshalling and unmarshalling).

Thanks
Tim