Hi ,
I did another modification to improve the messege transfer speed.
http_transport_utils.c was modified. (Buffer size was changed from
1024 to 1024 * 1024)
Now the transfer speed has been changed significantly. Please see the
table below.
The attached file includes the svn diff of the change.
Thank You
Kapila
(File Size) vs (Time Taken in milli seconds) to send binary attachments
Environment
----------
Processor - Intel Celeron M 1.40GHz
RAM - 256 MB
OS - Fedora Core 4.
Java Server - axis2 Java
C Server - axis2
mtom C and java test clients were used
http_transport_utils.c was modified. (Buffer size was changed from
1024 to 1024 * 1024)
Time Taken in milli seconds
File Size (kb) Java Server Java Server C Server C
Server C Server C Server C Server C Server C Server C
Server
Java Client (ms) C Client (ms) C Client (ms)
Java Client (ms)
*data_handler.c Buffer Size 1024 * *data_handler.c
'Buffer Size 1024 X 1024 * *Using stat* *http_transport_utils.c
'Buffer Size 1024 X 1024 * * * *Buffer Size '1024 * *Buffer Size
'1024 X 1024 * *Using stat* *http_transport_utils.c 'Buffer Size
1024 X 1024 *
----------------- ----------------- -----------------
----------------- ----------------- -----------------
----------------- ----------------- -----------------
--------------------------------- -----------------
----------------- ----------------- -----------------
---------------------------------
13.8 147 158 73 1884 78 76
188 143 136 136
28.2 515 488 78 1912 78 78
222 413 364 387
37.8 538 694 131 1917 81 81
232 474 377 396
132 571 807 212 2177 102 83
263 420 381 432
154 547 672 239 2215 117 84
347 389 383 517
165 688 683 397 2380 205 85
364 400 393 602
561 1340 1024 995 2537 777 100
637 899 775 766
762 1766 1336 1903 2890 1458 111 1248 1218 1051
1055
1162 3464 1970 4859 3602 3405 135 1702 1718
1712 1092
1761 4666 3274 10928 8028 7927 188 2780 3249
2691 2596
2081 6777 2993 15066 11008 10864 219 3595 3281
2859 3185
4241 23038 Could not transfer (Java client Broken) 52853
33351 32957 519 16234 115777
7991 6499
On 5/12/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
BTW,
If we are going to use chunking at some point in time, then the
data
handler logic for reading file too has to change to keep track of the
amount of chunks already read, and current position of read pointer.
There is no provision for such reading in the code as of now,
neither in
the data handler nor in the mime_output that uses the data handler
API.
So there is considerable amount of work on the part of the data hander
and the places that uses the API of data handler to read to turn on
chunking as far as I can see.
Thanks,
Samisa...
Samisa Abeysinghe wrote:
> Nabeel wrote:
>
>> Kapila,
>> With your latest changes you always read the file in one go
>> (i.e. not in chunks) and you don't need a loop. The attached
compact
>> code does the same thing.
>>
>> Suggestion:
>> When we need to attach *very* large file, we may not be keep the
>> whole file in memory. IMO, this where chunk reading is really
useful.
>> If you are to implement this, you'll have to change the code from
>> data_handler -> mime_body_part -> mime_output to om_output so that
>> you'll keep only chunk_size amount of bytes in memory at any
given time.
>
>
> Chunking is a very good idea, in fact that is the perfect
solution for
> large attachments.
> However, AFAIK, we have chunking capability only on the client side
> transport, not on the server side transport.
> Hence, we got to implement chunking capability into simple axis
server
> transport first. (May be we can raise a Jira on this). Or we have to
> see if we can piggyback on Apache for chunking.
> Secondly, we got to research and find out the ideal chunk size.
As we
> have seen through the results sent by Kapila, too small a chunk
size
> makes too large file transmission very slow and too large chunk size
> makes small file transmission very slow. Best is to not use chunking
> for small files and use chunking, with a reasonable chunk size, for
> large files. So we have several parameters to figure out.
> i. Max size we can read a file in one go without chunking.
> ii. When chunking is turned on, ideal chunk size.
> As I mentioned earlier, we need some research around this to
decide on
> the figures.
>
> Thanks,
> Samisa...
>
>>
>> -Nabeel
>>
>> Samisa Abeysinghe wrote:
>>
>>> Cool, so there sure is an improvement when using stat on read
side
>>> of data handler. Hence lets use this solution.
>>> Still we need to figure out what goes wrong with large files
beyond 1M
>>>
>>> Samisa...
>>>
>>> Kapila Dissanayake wrote:
>>>
>>>> Hi
>>>>
>>>> I did some modification to the data handler. Insted of using
fixed
>>>> buffer size to read files
>>>> 'stat' is used.
>>>>
>>>> Please see the perfomance details.
>>>>
>>>> Thank You
>>>>
>>>> Kapila
>>>>
>>>>
>>>> Environment ---------- Processor -
Intel
>>>> Celeron M 1.40GHz
>>>> RAM - 256 MB OS - Fedora Core 4. Java Server -
>>>> axis2 Java C Server - axis2 mtom C and java
test
>>>> clients were used (in axis2 c data_handler.c, different
buffer
>>>> sizes were used to read files)
>>>> Time Taken in milli seconds
>>>> File Size (kb) Java Server (ms) Java Server
>>>> (ms) C Server (ms) C Server (ms)
>>>> Java Client (ms) C Client (ms) C
>>>> Client (ms) Java Client (ms)
>>>> *Buffer Size 1024 *
*Buffer
>>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024
>>>> * *Buffer Size '1024 X 1024 * *Using stat *
>>>> ----------------- ----------------- -----------------
>>>> ----------------- ----------------- -----------------
>>>> ----------------- ----------------- -----------------
>>>> ----------------- ----------------- -----------------
>>>> -----------------
>>>> 13.8 147 158 73 1884
>>>> 78 188 143 136
>>>> 28.2 515 488 78 1912
>>>> 78 222 413 364
>>>> 37.8 538 694 131 1917
>>>> 81 232 474 377
>>>> 132 571 807 212 2177
>>>> 102 263 420 381
>>>> 154 547 672 239 2215
>>>> 117 347 389 383
>>>> 165 688 683 397 2380
>>>> 205 364 400 393
>>>> 561 1340 1024 995 2537
>>>> 777 637 899 775
>>>> 762 1766 1336 1903 2890
>>>> 1458 1248 1218 1051
>>>> 1162 3464 1970 4859 3602
>>>> 3405 1702 1718 1712
>>>> 1761 4666 3274 10928 8028
>>>> 7927 2780 3249 2691
>>>> 2081 6777 2993 15066 11008
>>>> 10864 3595 3281 2859
>>>> 4241 23038 Could not transfer (Java client
>>>> Broken) 52853 33351 32957 16234
>>>> 115777 7991
>>>>
>>>>
>>>>
>>>>
>>>> On 5/10/06, *Samisa Abeysinghe* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>>>> <mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>>> wrote:
>>>>
>>>> Well, from this I am certain that we should not change the
>>>> buffer size
>>>> to 1024*1024, rather keep it at 1024.
>>>> Please try with stat, using a dynamic buffer size based
on the
>>>> file
>>>> size, my gut feel is that it is going to be the best
solution.
>>>>
>>>> Thanks,
>>>> Samisa...
>>>>
>>>> Kapila Dissanayake wrote:
>>>>
>>>> >
>>>> > Hi,
>>>> >
>>>> > I did some testing agian with to measure the binary file
>>>> transfer
>>>> > times. In axis2 c data_handler.c, different buffer sizes
>>>> (1024 and
>>>> > 1024*1024) were used to read files.
>>>> >
>>>> > I think using fixed size buffers to read binary files
in axis2-C
>>>> has
>>>> > been affected in some cases to slow down the transfer.
>>>> >
>>>> > Thanks
>>>> >
>>>> > Kapila
>>>> >
>>>> > Please see the following test results
>>>> >
>>>> > Environment
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > ----------
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Processor - Intel Celeron M 1.40GHz
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > RAM - 256 MB
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > OS - Fedora Core 4.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Java Server - axis2 Java
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > C Server - axis2
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > mtom C and java test clients were used
>>>> >
>>>> >
>>>> >
>>>> > Time Taken in milli seconds
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > File Size (kb) Java Server (ms) Java
>>>> Server (ms) C
>>>> > Server (ms) C Server (ms)
>>>> > Java Client (ms) C
>>>> Client (ms) C Client (ms)
>>>> Java
>>>> > Client (ms)
>>>> > *Buffer Size*
>>>> *1024 * *1024 X 1024* * * *1024*
>>>> > *1024 X 1024 *
>>>> > ----------------- -----------------
-----------------
>>>> > ----------------- -----------------
-----------------
>>>> > ----------------- -----------------
-----------------
>>>> > ----------------- -----------------
-----------------
>>>> > 13.8 147 158
>>>> 73 1884 188 143
>>>> > 28.2 515 488
>>>> 78 1912 222 413
>>>> > 37.8 538 694 131
>>>> 1917 232 474
>>>> > 132 571 807 212
>>>> 2177 263 420
>>>> > 154 547 672 239
>>>> 2215 347 389
>>>> > 165 688 683 397
>>>> 2380 364 400
>>>> > 561 1340 1024 995
>>>> 2537 637 899
>>>> > 762 1766 1336
>>>> 1903 2890 1248 1218
>>>> >
>>>> 1162 3464 1970 4859
>>>> 3602 1702 1718
>>>>
>>>> > 1761 4666 3274 10928
>>>> 8028 2780 3249
>>>> >
2081 6777 2993 15066
>>>> 11008 3595 3281
>>>> > 4241 23038 Could not transfer (Java
client
>>>> Broken)
>>>> > 52853 33351 16234 115777
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On 5/8/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
>>>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>>>> > <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
<mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>> wrote:
>>>> >
>>>> > Hmmm, Interesting. There seem to be a problem in C
client in
>>>> case of
>>>> > large attachments.
>>>> > This must be due to the fact that we are using a
loop to
>>>> read the
>>>> > file.
>>>> > If we stat and read the file at once, there could
be an
>>>> improvement.
>>>> >
>>>> > Why is it so slow, in C vs C case? (Third column)
This is
>>>> worth
>>>> > investigating.
>>>> >
>>>> > Thanks,
>>>> > Samisa...
>>>> >
>>>> > Kapila Dissanayake wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > I did some testing on binary attachment transfer
times,
>>>> with axis2
>>>> > > 'java' and 'C' servers with sample mtom clients.
It took
>>>> very high
>>>> > > transfer times for large size attachments. Some minor
>>>> modificaions
>>>> > > were done to the data_handler.c and it was sent
to the
>>>> mailing list.
>>>> > >
>>>> > >
>>>> > > (File Size) vs (Time Taken in milli seconds) to
transfer
>>>> binary
>>>> > > attachments
>>>> > >
>>>> > > Environment
>>>> > > ----------
>>>> > > Processor - Intel Celeron M 1.40GHz
>>>> > > RAM - 256 MB
>>>> > > OS - Fedora Core 4.
>>>> > > Java Server - axis2 Java
>>>> > > C Server - axis2
>>>> > > mtom C and java test clients were used
>>>> > > Time Taken in milli seconds
>>>> > >
>>>> > > File Size (kb)
>>>> > >
>>>> > > Java Server
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > Java Server
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > C Server
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > C Server
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > Java Client
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > C Client
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > C Client
>>>> > >
>>>> > > (ms)
>>>> > >
>>>> > > Java Client
>>>> > >
>>>> > > (ms)
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > >
>>>> > > ---------------
>>>> > > 28
>>>> > >
>>>> > >
>>>> > >
>>>> > > 445
>>>> > >
>>>> > >
>>>> > >
>>>> > > 274
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2004
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1009
>>>> > >
>>>> > >
>>>> > > 38
>>>> > >
>>>> > >
>>>> > >
>>>> > > 617
>>>> > >
>>>> > >
>>>> > >
>>>> > > 297
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1957
>>>> > >
>>>> > >
>>>> > >
>>>> > > 842
>>>> > >
>>>> > >
>>>> > > 132
>>>> > >
>>>> > >
>>>> > >
>>>> > > 593
>>>> > >
>>>> > >
>>>> > >
>>>> > > 527
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2181
>>>> > >
>>>> > >
>>>> > >
>>>> > > 726
>>>> > >
>>>> > >
>>>> > > 154
>>>> > >
>>>> > >
>>>> > >
>>>> > > 551
>>>> > >
>>>> > >
>>>> > >
>>>> > > 447
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2081
>>>> > >
>>>> > >
>>>> > >
>>>> > > 677
>>>> > >
>>>> > >
>>>> > > 165
>>>> > >
>>>> > >
>>>> > >
>>>> > > 440
>>>> > >
>>>> > >
>>>> > >
>>>> > > 743
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2147
>>>> > >
>>>> > >
>>>> > >
>>>> > > 676
>>>> > >
>>>> > >
>>>> > > 561
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1029
>>>> > >
>>>> > >
>>>> > >
>>>> > > 418
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2581
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1075
>>>> > >
>>>> > >
>>>> > > 762
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2137
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1572
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3231
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2356
>>>> > >
>>>> > >
>>>> > > 1162
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3172
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2962
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3981
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3100
>>>> > >
>>>> > >
>>>> > > 1761
>>>> > >
>>>> > >
>>>> > >
>>>> > > 7672
>>>> > >
>>>> > >
>>>> > >
>>>> > > 5494
>>>> > >
>>>> > >
>>>> > >
>>>> > > 8186
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3041
>>>> > >
>>>> > >
>>>> > > 2081
>>>> > >
>>>> > >
>>>> > >
>>>> > > 7768
>>>> > >
>>>> > >
>>>> > >
>>>> > > 6837
>>>> > >
>>>> > >
>>>> > >
>>>> > > 11757
>>>> > >
>>>> > >
>>>> > >
>>>> > > 3499
>>>> > >
>>>> > >
>>>> > > 4241
>>>> > >
>>>> > >
>>>> > >
>>>> > > 14465
>>>> > >
>>>> > >
>>>> > >
>>>> > > 24060
>>>> > >
>>>> > >
>>>> > >
>>>> > > 36168
>>>> > >
>>>> > >
>>>> > >
>>>> > > Could not transfer (Error Occured in Java client)
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thank You
>>>> > >
>>>> > > Kapila
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>
>>>
>>
>>
------------------------------------------------------------------------
>>
>> #include <stdio.h>
>> #include <sys/stat.h>
>>
>> int read_from_file(char *file_name, char** output_stream, int
>> *output_stream_size);
>>
>> int main(int argc, char **argv)
>> {
>> int size;
>> char *output;
>>
>> if (argc != 2)
>> {
>> printf("usage: read_binary <file_name>\n");
>> return 0;
>> }
>> if (read_from_file(argv[1], &output, &size))
>> printf("read successful\n");
>> if (output)
>> free(output);
>> return 1;
>> }
>>
>> int read_from_file(char *file_name, char** output_stream, int
>> *output_stream_size)
>> {
>> FILE *f = NULL;
>> struct stat stat_p;
>> int count = 0;
>>
>> if (!file_name)
>> {
>> printf("error: file not given \n");
>> return 0;
>> }
>> f = fopen(file_name, "rb");
>> if (!f)
>> {
>> printf("error:cannot open file %s\n", file_name);
>> return ;
>> }
>> if ( -1 == stat (file_name, &stat_p))
>> {
>> printf("error:error reading stat\n");
>> return 0;
>> }
>>
>> *output_stream_size = stat_p.st_size;
>> printf("size = %d\n", *output_stream_size);
>> *output_stream = malloc((*output_stream_size) * sizeof(char));
>>
>> if (!(*output_stream))
>> {
>> printf("error: cannot create stream\n");
>> return 0;
>> }
>> count = fread(*output_stream, 1, *output_stream_size, f);
>> if (ferror(f) != 0)
>> {
>> printf("error:cannot read file\n");
>> if (*output_stream)
>> {
>> free(*output_stream);
>> *output_stream = NULL;
>> }
>> return 0;
>>
>> }
>> fclose(f);
>> return 1;
>> }
>>
>>
>
>
------------------------------------------------------------------------
Index: modules/xml/attachments/data_handler.c
===================================================================
--- modules/xml/attachments/data_handler.c (revision 398104)
+++ modules/xml/attachments/data_handler.c (working copy)
@@ -16,6 +16,7 @@
#include <axis2_data_handler.h>
#include <stdio.h>
+#include <sys/stat.h>
typedef struct axis2_data_handler_impl
{
@@ -185,7 +186,12 @@
return AXIS2_FAILURE;
do {
- read_stream_size = 1024;
+ struct stat stat_p;
+ if ( -1 == stat (data_handler_impl->file_name, &stat_p))
+ {
+ return AXIS2_FAILURE;
+ }
+ read_stream_size = stat_p.st_size;
read_stream = AXIS2_MALLOC((*env)->allocator, (read_stream_size) *
sizeof(axis2_byte_t));
if (!read_stream)
{
@@ -197,7 +203,7 @@
}
return AXIS2_FAILURE;
}
- count = fread(read_stream, 1, 1024, f);
+ count = fread(read_stream, 1, read_stream_size, f);
if (ferror(f) != 0)
{
/*TODO : need to set the correct error code */
@@ -354,3 +360,4 @@
return AXIS2_SUCCESS;
}
+
Index: modules/core/transport/http/http_transport_utils.c
===================================================================
--- modules/core/transport/http/http_transport_utils.c (revision 398104)
+++ modules/core/transport/http/http_transport_utils.c (working copy)
@@ -199,7 +199,7 @@
if (mime_boundary)
{
axis2_char_t *buffer = NULL;
- int size = 1024;
+ int size = 1024 * 1024;
int len = 0;
axis2_char_t *root_mime = NULL;
int root_mime_len = 0;
Index: build.sh
===================================================================
--- build.sh (revision 398104)
+++ build.sh (working copy)
@@ -2,7 +2,7 @@
./autogen.sh
AXIS2C_HOME=`pwd`/deploy
export AXIS2C_HOME
-./configure --prefix=${AXIS2C_HOME} --enable-tests=yes --enable-trace=yes
--with-apache2=/usr/include/apache2 --with-apr=/usr/include/apr-0
--enable-diclient=no --enable-static=no
+./configure --prefix=${AXIS2C_HOME} --enable-tests=yes --enable-trace=yes
--enable-diclient=no --enable-static=no
make
make install
cd samples
