Senaka Fernando wrote:
Hi Thilina,
What is the upper limit?
And, we also can use a buffer of say 1MB and keep reading the content into
it until we find the end, and in the mean time write to a file, if we
exceed this buffer. This way, we can do the necessary tracking in real
time. Meaning that the MIME parts will be parsed while it is being cached
to the file.
You always can jump to a known location of the file, so no need to
re-parse. This is just a matter of altering the file pointer.
Also, Manjula, isn't the whole file read to memory at once? When we use
fileopen() I mean. In that case you will have to cache to multiple files
rather.
I do not think this is an issue because once open you can keep on
writing to it (and flush) till the whole file is read. That is in the
inflow.
We also have to be aware that flushing too often too slows down the
system, because it is an IO to the disk. This is where the optimal
buffer size comes into play, if we keep on flushing too small buffer, it
gets too slow.
BTW, this whole discussion is about in path, that is reading an
incomming message. How about the out path? We have the same problems
when sending attachments. Right now, we read the whole file into memory
and then only we send over the wire.
Samisa...
Regards,
Senaka
Hi,
> In Axis2/Java case we do write the attachment content directly from
> the InputStream to the File when the attachment size is larger than
> the threshold. This avoids loading the whole attachment to the
memory
> at all.
In this case to find out the attachment size don't you need to do any
mime parsing? How do you find the attachment size with out searching
for
the mime boundaries ?
Yes.. MIME is a boundary based packaging mechanism and you does not
need to specify the length for each of the parts...Even the HTTP
content length is not there if the message is chunked.
What we did in Axis2/Java to overcome this is to read the data to a
byte[] buffer of up to a certain size (the size threshold). If there
are more data available in the mime part (if we have not encountered
the boundary yet) then we know this attachment is bigger than the
threshold. So we create the temp file, pump the content in the buffer
to the file, then pump the rest of the stream to the file.. In this
way we do not need to know the size of the attachment upfront.. BTW we
do all of the above while we are parsing the MIME message at the MIME
parser level..
> This has the plus point that the attachment size will be
> limited only by the available free space in the Temp Directory..
> Will that be possible in Axis2/C.. Or is that wat you have in mind
:)..
Yes this is possible.
But in Axis2/JAVA we will get a OutOfMemory if we parse a large MIME
part upfront, since it reads the attachment to memory. May be you can
have a larger limit with C than in Java, but ultimately you'll come to
a situation where you will not have enough memory to store that MIME
part in memory in the parsing time, unless you write in to a File
while parsing,..
thanks,
Thilina
>
> thanks,
> Thilina
>
> >and keeping the file name inside
> > data_handler instead of the whole buffer. So the service or the
client
> > will get the file name instead of the buffered stream, when it
receives
> > an attachment. This will not prevent buffering the attachment at
the
> > transport but will prevent keeping it inside the om_tree till it
reaches
> > the receiver.
> >
> > Before implementing this I would like to know your suggestions
regarding
> > this.
> >
> > [1] https://issues.apache.org/jira/browse/AXIS2C-672
> >
> > Thanks,
> > -Manjula
> >
> > --
> > Manjula Peiris: http://manjula-peiris.blogspot.com/
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
> >
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Thilina Gunarathne - http://thilinag.blogspot.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
