Thilina Gunarathne wrote:
Hi,
With your latest changes you always read the file in one go (i.e.
not in chunks) and you don't need a loop. The attached compact
code does
the same thing.
My suggestion,
Don't read the file in to the memory at all...Wrap the file with
something like a DataHandler which we use on Axis2/java...You can get
sizes and other information using the wrapper.. Stream the
file directly to the output stream when we are serializing the message
with No or very low buffering in the memory..
Well that is exactly what we have done in the current implementation.
Identical design to Java implementation.
We have the data handler, and it does the streaming stuff. We do not
read the file until it is time to send it out.
Only thing is that we had to write everything, including the data
handler. We are not so lucky as Java ;-)
Samisa...
Our policy in Axis2/java is to read it only when need it...
HTH,
~Thilina
Suggestion:
When we need to attach *very* large file, we may not be keep the
whole file in memory. IMO, this where chunk reading is really
useful. If
you are to implement this, you'll have to change the code from
data_handler -> mime_body_part -> mime_output to om_output so that
you'll keep only chunk_size amount of bytes in memory at any given
time.
-Nabeel
Samisa Abeysinghe wrote:
> Cool, so there sure is an improvement when using stat on read
side of
> data handler. Hence lets use this solution.
> Still we need to figure out what goes wrong with large files
beyond 1M
>
> Samisa...
>
> Kapila Dissanayake wrote:
>
>> Hi
>>
>> I did some modification to the data handler. Insted of using fixed
>> buffer size to read files
>> 'stat' is used.
>>
>> Please see the perfomance details.
>>
>> Thank You
>>
>> Kapila
>>
>>
>> Environment
>>
>>
>> ----------
>>
>>
>>
>> Processor - Intel Celeron M 1.40GHz
>> RAM - 256 MB
>>
>>
>> OS - Fedora Core 4.
>>
>> Java Server - axis2 Java
>>
>> C Server - axis2
>>
>>
>> mtom C and java test clients were used (in axis2 c
>> data_handler.c, different buffer sizes were used to read files)
>> Time Taken in milli seconds
>>
>>
>>
>> File Size (kb) Java Server (ms) Java Server
>> (ms) C Server (ms) C Server (ms)
>> Java Client (ms) C Client (ms) C
>> Client (ms) Java Client (ms)
>> *Buffer Size 1024 * *Buffer
>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024 *
>> *Buffer Size '1024 X 1024 * *Using stat *
>> ----------------- ----------------- -----------------
>> ----------------- ----------------- -----------------
>> ----------------- ----------------- -----------------
>> ----------------- ----------------- -----------------
>> -----------------
>> 13.8 147 158 73 1884
>> 78 188 143 136
>> 28.2 515 488 78 1912
>> 78 222 413 364
>> 37.8 538 694 131 1917
>> 81 232 474 377
>> 132 571 807 212 2177
>> 102 263 420 381
>> 154 547 672 239 2215
>> 117 347 389 383
>> 165 688 683 397 2380
>> 205 364 400 393
>> 561 1340 1024 995 2537
>> 777 637 899 775
>> 762 1766 1336 1903 2890
>> 1458 1248 1218 1051
>> 1162 3464 1970 4859 3602
>> 3405 1702 1718 1712
>> 1761 4666 3274 10928 8028
>> 7927 2780 3249 2691
>> 2081 6777 2993 15066 11008
>> 10864 3595 3281 2859
>> 4241 23038 Could not transfer (Java client
>> Broken) 52853 33351 32957 16234
>> 115777 7991
>>
>>
>>
>>
>> On 5/10/06, *Samisa Abeysinghe* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>> wrote:
>>
>> Well, from this I am certain that we should not change the
buffer
>> size
>> to 1024*1024, rather keep it at 1024.
>> Please try with stat, using a dynamic buffer size based on
the file
>> size, my gut feel is that it is going to be the best solution.
>>
>> Thanks,
>> Samisa...
>>
>> Kapila Dissanayake wrote:
>>
>> >
>> > Hi,
>> >
>> > I did some testing agian with to measure the binary file
transfer
>> > times. In axis2 c data_handler.c, different buffer sizes
(1024 and
>> > 1024*1024) were used to read files.
>> >
>> > I think using fixed size buffers to read binary files in
axis2-C
>> has
>> > been affected in some cases to slow down the transfer.
>> >
>> > Thanks
>> >
>> > Kapila
>> >
>> > Please see the following test results
>> >
>> > Environment
>> >
>> >
>> >
>> >
>> >
>> >
>> > ----------
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Processor - Intel Celeron M 1.40GHz
>> >
>> >
>> >
>> >
>> > RAM - 256 MB
>> >
>> >
>> >
>> >
>> >
>> >
>> > OS - Fedora Core 4.
>> >
>> >
>> >
>> >
>> >
>> >
>> > Java Server - axis2 Java
>> >
>> >
>> >
>> >
>> >
>> > C Server - axis2
>> >
>> >
>> >
>> >
>> >
>> >
>> > mtom C and java test clients were used
>> >
>> >
>> >
>> > Time Taken in milli seconds
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > File Size (kb) Java Server (ms) Java
>> Server (ms) C
>> > Server (ms) C Server (ms)
>> > Java Client (ms) C
>> Client (ms) C Client
(ms) Java
>> > Client (ms)
>> > *Buffer Size*
>> *1024 * *1024 X 1024* * * *1024*
>> > *1024 X 1024 *
>> > ----------------- -----------------
-----------------
>> > ----------------- -----------------
-----------------
>> > ----------------- -----------------
-----------------
>> > ----------------- -----------------
-----------------
>> > 13.8 147 158
>> 73 1884 188 143
>> > 28.2 515 488
>> 78 1912 222 413
>> > 37.8 538 694 131
>> 1917 232 474
>> > 132 571 807 212
>> 2177 263 420
>> > 154 547 672 239
>> 2215 347 389
>> > 165 688 683 397
>> 2380 364 400
>> > 561 1340 1024 995
>> 2537 637 899
>> > 762 1766 1336
>> 1903 2890 1248 1218
>> >
>> 1162 3464 1970 4859
>> 3602 1702 1718
>>
>> > 1761 4666 3274 10928
>> 8028 2780 3249
>> > 2081 6777 2993 15066
>> 11008 3595 3281
>> > 4241 23038 Could not transfer (Java client
>> Broken)
>> > 52853 33351 16234 115777
>> >
>> >
>> >
>> >
>> > On 5/8/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>> > <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>> wrote:
>> >
>> > Hmmm, Interesting. There seem to be a problem in C
client in
>> case of
>> > large attachments.
>> > This must be due to the fact that we are using a loop to
>> read the
>> > file.
>> > If we stat and read the file at once, there could be an
>> improvement.
>> >
>> > Why is it so slow, in C vs C case? (Third column) This is
>> worth
>> > investigating.
>> >
>> > Thanks,
>> > Samisa...
>> >
>> > Kapila Dissanayake wrote:
>> >
>> > > Hi,
>> > >
>> > > I did some testing on binary attachment transfer
times,
>> with axis2
>> > > 'java' and 'C' servers with sample mtom clients. It
took
>> very high
>> > > transfer times for large size attachments. Some minor
>> modificaions
>> > > were done to the data_handler.c and it was sent to the
>> mailing list.
>> > >
>> > >
>> > > (File Size) vs (Time Taken in milli seconds) to
transfer
>> binary
>> > > attachments
>> > >
>> > > Environment
>> > > ----------
>> > > Processor - Intel Celeron M 1.40GHz
>> > > RAM - 256 MB
>> > > OS - Fedora Core 4.
>> > > Java Server - axis2 Java
>> > > C Server - axis2
>> > > mtom C and java test clients were used
>> > > Time Taken in milli seconds
>> > >
>> > > File Size (kb)
>> > >
>> > > Java Server
>> > >
>> > > (ms)
>> > >
>> > > Java Server
>> > >
>> > > (ms)
>> > >
>> > > C Server
>> > >
>> > > (ms)
>> > >
>> > > C Server
>> > >
>> > > (ms)
>> > >
>> > >
>> > >
>> > >
>> > > Java Client
>> > >
>> > > (ms)
>> > >
>> > > C Client
>> > >
>> > > (ms)
>> > >
>> > > C Client
>> > >
>> > > (ms)
>> > >
>> > > Java Client
>> > >
>> > > (ms)
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > >
>> > > ---------------
>> > > 28
>> > >
>> > >
>> > >
>> > > 445
>> > >
>> > >
>> > >
>> > > 274
>> > >
>> > >
>> > >
>> > > 2004
>> > >
>> > >
>> > >
>> > > 1009
>> > >
>> > >
>> > > 38
>> > >
>> > >
>> > >
>> > > 617
>> > >
>> > >
>> > >
>> > > 297
>> > >
>> > >
>> > >
>> > > 1957
>> > >
>> > >
>> > >
>> > > 842
>> > >
>> > >
>> > > 132
>> > >
>> > >
>> > >
>> > > 593
>> > >
>> > >
>> > >
>> > > 527
>> > >
>> > >
>> > >
>> > > 2181
>> > >
>> > >
>> > >
>> > > 726
>> > >
>> > >
>> > > 154
>> > >
>> > >
>> > >
>> > > 551
>> > >
>> > >
>> > >
>> > > 447
>> > >
>> > >
>> > >
>> > > 2081
>> > >
>> > >
>> > >
>> > > 677
>> > >
>> > >
>> > > 165
>> > >
>> > >
>> > >
>> > > 440
>> > >
>> > >
>> > >
>> > > 743
>> > >
>> > >
>> > >
>> > > 2147
>> > >
>> > >
>> > >
>> > > 676
>> > >
>> > >
>> > > 561
>> > >
>> > >
>> > >
>> > > 1029
>> > >
>> > >
>> > >
>> > > 418
>> > >
>> > >
>> > >
>> > > 2581
>> > >
>> > >
>> > >
>> > > 1075
>> > >
>> > >
>> > > 762
>> > >
>> > >
>> > >
>> > > 2137
>> > >
>> > >
>> > >
>> > > 1572
>> > >
>> > >
>> > >
>> > > 3231
>> > >
>> > >
>> > >
>> > > 2356
>> > >
>> > >
>> > > 1162
>> > >
>> > >
>> > >
>> > > 3172
>> > >
>> > >
>> > >
>> > > 2962
>> > >
>> > >
>> > >
>> > > 3981
>> > >
>> > >
>> > >
>> > > 3100
>> > >
>> > >
>> > > 1761
>> > >
>> > >
>> > >
>> > > 7672
>> > >
>> > >
>> > >
>> > > 5494
>> > >
>> > >
>> > >
>> > > 8186
>> > >
>> > >
>> > >
>> > > 3041
>> > >
>> > >
>> > > 2081
>> > >
>> > >
>> > >
>> > > 7768
>> > >
>> > >
>> > >
>> > > 6837
>> > >
>> > >
>> > >
>> > > 11757
>> > >
>> > >
>> > >
>> > > 3499
>> > >
>> > >
>> > > 4241
>> > >
>> > >
>> > >
>> > > 14465
>> > >
>> > >
>> > >
>> > > 24060
>> > >
>> > >
>> > >
>> > > 36168
>> > >
>> > >
>> > >
>> > > Could not transfer (Error Occured in Java client)
>> > >
>> > >
>> > >
>> > >
>> > > Thank You
>> > >
>> > > Kapila
>> >
>> >
>> >
>>
>>
>
>
#include <stdio.h>
#include <sys/stat.h>
int read_from_file(char *file_name, char** output_stream, int
*output_stream_size);
int main(int argc, char **argv)
{
int size;
char *output;
if (argc != 2)
{
printf("usage: read_binary <file_name>\n");
return 0;
}
if (read_from_file(argv[1], &output, &size))
printf("read successful\n");
if (output)
free(output);
return 1;
}
int read_from_file(char *file_name, char** output_stream, int
*output_stream_size)
{
FILE *f = NULL;
struct stat stat_p;
int count = 0;
if (!file_name)
{
printf("error: file not given \n");
return 0;
}
f = fopen(file_name, "rb");
if (!f)
{
printf("error:cannot open file %s\n", file_name);
return ;
}
if ( -1 == stat (file_name, &stat_p))
{
printf("error:error reading stat\n");
return 0;
}
*output_stream_size = stat_p.st_size;
printf("size = %d\n", *output_stream_size);
*output_stream = malloc((*output_stream_size) * sizeof(char));
if (!(*output_stream))
{
printf("error: cannot create stream\n");
return 0;
}
count = fread(*output_stream, 1, *output_stream_size, f);
if (ferror(f) != 0)
{
printf("error:cannot read file\n");
if (*output_stream)
{
free(*output_stream);
*output_stream = NULL;
}
return 0;
}
fclose(f);
return 1;
}
--
"May the SourcE be with u"
http://webservices.apache.org/~thilina/
<http://webservices.apache.org/%7Ethilina/>
http://thilinag.blogspot.com/
http://www.bloglines.com/blog/Thilina
<http://www.bloglines.com/blog/Thilina>
