Re: [AXIS2] MTOM performance Measuring

Fri, 12 May 2006 07:06:13 -0700 

BTW,
If we are going to use chunking at some point in time, then the data handler logic for reading file too has to change to keep track of the amount of chunks already read, and current position of read pointer. There is no provision for such reading in the code as of now, neither in the data handler nor in the mime_output that uses the data handler API. So there is considerable amount of work on the part of the data hander and the places that uses the API of data handler to read to turn on chunking as far as I can see. 
Thanks,
Samisa...

Samisa Abeysinghe wrote:


Nabeel wrote:


Kapila,
With your latest changes you always read the file in one go (i.e. not in chunks) and you don't need a loop. The attached compact code does the same thing. 

Suggestion:
When we need to attach *very* large file, we may not be keep the whole file in memory. IMO, this where chunk reading is really useful. If you are to implement this, you'll have to change the code from data_handler -> mime_body_part -> mime_output to om_output so that you'll keep only chunk_size amount of bytes in memory at any given time.
Chunking is a very good idea, in fact that is the perfect solution for large attachments. However, AFAIK, we have chunking capability only on the client side transport, not on the server side transport. Hence, we got to implement chunking capability into simple axis server transport first. (May be we can raise a Jira on this). Or we have to see if we can piggyback on Apache for chunking. Secondly, we got to research and find out the ideal chunk size. As we have seen through the results sent by Kapila, too small a chunk size makes too large file transmission very slow and too large chunk size makes small file transmission very slow. Best is to not use chunking for small files and use chunking, with a reasonable chunk size, for large files. So we have several parameters to figure out. 
   i.  Max size we can read a file in one go without chunking.
   ii. When chunking is turned on, ideal chunk size.
As I mentioned earlier, we need some research around this to decide on the figures. 

Thanks,
Samisa...



-Nabeel

Samisa Abeysinghe wrote:
Cool, so there sure is an improvement when using stat on read side of data handler. Hence lets use this solution. 
Still we need to figure out what goes wrong with large files beyond 1M

Samisa...

Kapila Dissanayake wrote:


Hi
I did some modification to the data handler. Insted of using fixed buffer size to read files 
'stat' is used.

Please see the perfomance details.

Thank You

Kapila
Environment ---------- Processor - Intel Celeron M 1.40GHz RAM - 256 MB OS - Fedora Core 4. Java Server - axis2 Java C Server - axis2 mtom C and java test clients were used (in axis2 c data_handler.c, different buffer sizes were used to read files) Time Taken in milli seconds File Size (kb) Java Server (ms) Java Server (ms) C Server (ms) C Server (ms) Java Client (ms) C Client (ms) C Client (ms) Java Client (ms) *Buffer Size 1024 * *Buffer Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024 * *Buffer Size '1024 X 1024 * *Using stat * ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- ----------------- 13.8 147 158 73 1884 78 188 143 136 28.2 515 488 78 1912 78 222 413 364 37.8 538 694 131 1917 81 232 474 377 132 571 807 212 2177 102 263 420 381 154 547 672 239 2215 117 347 389 383 165 688 683 397 2380 205 364 400 393 561 1340 1024 995 2537 777 637 899 775 762 1766 1336 1903 2890 1458 1248 1218 1051 1162 3464 1970 4859 3602 3405 1702 1718 1712 1761 4666 3274 10928 8028 7927 2780 3249 2691 2081 6777 2993 15066 11008 10864 3595 3281 2859 4241 23038 Could not transfer (Java client Broken) 52853 33351 32957 16234 115777 7991
On 5/10/06, *Samisa Abeysinghe* <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
Well, from this I am certain that we should not change the buffer size 
    to 1024*1024, rather keep it at 1024.
Please try with stat, using a dynamic buffer size based on the file 
    size, my gut feel is that it is going to be the best solution.

    Thanks,
    Samisa...

    Kapila Dissanayake wrote:

    >
    > Hi,
    >
> I did some testing agian with to measure the binary file transfer > times. In axis2 c data_handler.c, different buffer sizes (1024 and 
    > 1024*1024) were used to read files.
    >
    > I think using fixed size buffers to read binary files in axis2-C
    has
    > been affected in some cases to slow down the transfer.
    >
    > Thanks
    >
    > Kapila
    >
    > Please see the following test results
    >
    > Environment
    >
    >
    >
    >
    >
    >
    > ----------
    >
    >
    >
    >
    >
    >
    >
    > Processor - Intel Celeron M 1.40GHz
    >
    >
    >
    >
    > RAM - 256 MB
    >
    >
    >
    >
    >
    >
    > OS - Fedora Core 4.
    >
    >
    >
    >
    >
    >
    > Java Server - axis2 Java
    >
    >
    >
    >
    >
    > C Server - axis2
    >
    >
    >
    >
    >
    >
    > mtom C and java test clients were used
    >
    >
    >
    > Time Taken in milli seconds
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > File Size (kb)                Java Server     (ms)    Java
    Server     (ms)             C
    > Server        (ms)            C Server        (ms)
    >               Java Client     (ms)    C
Client (ms) C Client (ms) Java 
    > Client        (ms)
> *Buffer Size* *1024 * *1024 X 1024* * * *1024* 
    > *1024 X 1024 *
    > -----------------     -----------------       -----------------
    > -----------------     -----------------       -----------------
    > -----------------     -----------------       -----------------
    > -----------------     -----------------       -----------------
> 13.8 147 158 73 1884 188 143 > 28.2 515 488 78 1912 222 413 > 37.8 538 694 131 1917 232 474 > 132 571 807 212 2177 263 420 > 154 547 672 239 2215 347 389 > 165 688 683 397 2380 364 400 > 561 1340 1024 995 2537 637 899 > 762 1766 1336 1903 2890 1248 1218 
    >
1162 3464 1970 4859 3602 1702 1718
> 1761 4666 3274 10928 8028 2780 3249 > 2081 6777 2993 15066 11008 3595 3281 
    > 4241          23038           Could not transfer (Java client
    Broken)
    > 52853         33351           16234   115777
    >
    >
    >
    >
    > On 5/8/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    > <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
    >
    >     Hmmm, Interesting. There seem to be a problem in C client in
    case of
    >     large attachments.
    >     This must be due to the fact that we are using a loop to
    read the
    >     file.
    >     If we stat and read the file at once, there could be an
    improvement.
    >
> Why is it so slow, in C vs C case? (Third column) This is worth 
    >     investigating.
    >
    >     Thanks,
    >     Samisa...
    >
    >     Kapila Dissanayake wrote:
    >
    >     > Hi,
    >     >
    >     > I did some testing on binary attachment transfer times,
    with axis2
    >     > 'java' and 'C' servers with sample mtom clients. It took
    very high
    >     > transfer times for large size attachments. Some minor
    modificaions
    >     > were done to the data_handler.c and it was sent to the
    mailing list.
    >     >
    >     >
    >     > (File Size) vs (Time Taken in milli seconds) to transfer
    binary
    >     > attachments
    >     >
    >     > Environment
    >     > ----------
    >     > Processor - Intel Celeron M 1.40GHz
    >     > RAM - 256 MB
    >     > OS - Fedora Core 4.
    >     > Java Server - axis2 Java
    >     > C Server - axis2
    >     > mtom C and java test clients were used
    >     > Time Taken in milli seconds
    >     >
    >     > File Size (kb)
    >     >
    >     > Java Server
    >     >
    >     > (ms)
    >     >
    >     > Java Server
    >     >
    >     > (ms)
    >     >
    >     >  C Server
    >     >
    >     > (ms)
    >     >
    >     > C Server
    >     >
    >     > (ms)
    >     >
    >     >
    >     >
    >     >
    >     > Java Client
    >     >
    >     > (ms)
    >     >
    >     > C Client
    >     >
    >     > (ms)
    >     >
    >     >  C Client
    >     >
    >     > (ms)
    >     >
    >     >  Java Client
    >     >
    >     > (ms)
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     >
    >     > ---------------
    >     > 28
    >     >
    >     >
    >     >
    >     > 445
    >     >
    >     >
    >     >
    >     > 274
    >     >
    >     >
    >     >
    >     > 2004
    >     >
    >     >
    >     >
    >     > 1009
    >     >
    >     >
    >     > 38
    >     >
    >     >
    >     >
    >     > 617
    >     >
    >     >
    >     >
    >     > 297
    >     >
    >     >
    >     >
    >     > 1957
    >     >
    >     >
    >     >
    >     > 842
    >     >
    >     >
    >     > 132
    >     >
    >     >
    >     >
    >     > 593
    >     >
    >     >
    >     >
    >     > 527
    >     >
    >     >
    >     >
    >     > 2181
    >     >
    >     >
    >     >
    >     > 726
    >     >
    >     >
    >     > 154
    >     >
    >     >
    >     >
    >     > 551
    >     >
    >     >
    >     >
    >     > 447
    >     >
    >     >
    >     >
    >     > 2081
    >     >
    >     >
    >     >
    >     > 677
    >     >
    >     >
    >     > 165
    >     >
    >     >
    >     >
    >     > 440
    >     >
    >     >
    >     >
    >     > 743
    >     >
    >     >
    >     >
    >     > 2147
    >     >
    >     >
    >     >
    >     > 676
    >     >
    >     >
    >     > 561
    >     >
    >     >
    >     >
    >     > 1029
    >     >
    >     >
    >     >
    >     > 418
    >     >
    >     >
    >     >
    >     > 2581
    >     >
    >     >
    >     >
    >     > 1075
    >     >
    >     >
    >     > 762
    >     >
    >     >
    >     >
    >     > 2137
    >     >
    >     >
    >     >
    >     > 1572
    >     >
    >     >
    >     >
    >     > 3231
    >     >
    >     >
    >     >
    >     > 2356
    >     >
    >     >
    >     > 1162
    >     >
    >     >
    >     >
    >     > 3172
    >     >
    >     >
    >     >
    >     > 2962
    >     >
    >     >
    >     >
    >     > 3981
    >     >
    >     >
    >     >
    >     > 3100
    >     >
    >     >
    >     > 1761
    >     >
    >     >
    >     >
    >     > 7672
    >     >
    >     >
    >     >
    >     > 5494
    >     >
    >     >
    >     >
    >     > 8186
    >     >
    >     >
    >     >
    >     > 3041
    >     >
    >     >
    >     > 2081
    >     >
    >     >
    >     >
    >     > 7768
    >     >
    >     >
    >     >
    >     > 6837
    >     >
    >     >
    >     >
    >     > 11757
    >     >
    >     >
    >     >
    >     > 3499
    >     >
    >     >
    >     > 4241
    >     >
    >     >
    >     >
    >     > 14465
    >     >
    >     >
    >     >
    >     > 24060
    >     >
    >     >
    >     >
    >     > 36168
    >     >
    >     >
    >     >
    >     > Could not transfer (Error Occured in Java client)
    >     >
    >     >
    >     >
    >     >
    >     > Thank You
    >     >
    >     > Kapila
    >
    >
    >







------------------------------------------------------------------------

#include <stdio.h>
#include <sys/stat.h>
int read_from_file(char *file_name, char** output_stream, int *output_stream_size); 

int main(int argc, char **argv)
{
    int size;
    char *output;

    if (argc != 2)
    {
        printf("usage: read_binary <file_name>\n");
        return 0;
    }
    if (read_from_file(argv[1], &output, &size))
            printf("read successful\n");
    if (output)
        free(output);
    return 1;
}
int read_from_file(char *file_name, char** output_stream, int *output_stream_size) 
{
    FILE *f = NULL;
    struct stat stat_p;
    int count = 0;
if (!file_name) 
   {
        printf("error: file not given \n");
        return 0;
    }
          f = fopen(file_name, "rb");
   if (!f)
    {
        printf("error:cannot open file %s\n", file_name);
       return ;
    }
           if ( -1 ==  stat (file_name, &stat_p))
    {
        printf("error:error reading stat\n");
        return 0;
    }
*output_stream_size = stat_p.st_size; 
    printf("size = %d\n",  *output_stream_size);
    *output_stream = malloc((*output_stream_size) * sizeof(char));
if (!(*output_stream)) 
   {
       printf("error: cannot create stream\n");
       return 0;
    }
   count = fread(*output_stream, 1, *output_stream_size, f);
   if (ferror(f) != 0)
   {
       printf("error:cannot read file\n");
       if (*output_stream)
       {
           free(*output_stream);
           *output_stream = NULL;
       }
       return 0;
} 
   fclose(f);
   return 1;
}

Reply via email to