Hi folks, Sorry abt being late to join the thread See some of my inline comments.
Samisa Abeysinghe wrote: > Nabeel wrote: > >> Kapila, >> With your latest changes you always read the file in one go >> (i.e. not in chunks) and you don't need a loop. The attached compact >> code does the same thing. >> >> Suggestion: >> When we need to attach *very* large file, we may not be keep the >> whole file in memory. IMO, this where chunk reading is really useful. >> If you are to implement this, you'll have to change the code from >> data_handler -> mime_body_part -> mime_output to om_output so that >> you'll keep only chunk_size amount of bytes in memory at any given time. > > > Chunking is a very good idea, in fact that is the perfect solution for > large attachments. > However, AFAIK, we have chunking capability only on the client side > transport, not on the server side transport. We do support chunking at the server side. The simple axis server switches to chunking mode as soon as it sees a "Transer-Encoding: chunked" header. For the Apache2 httpd module we get chunking automatically (by the core of httpd). > Hence, we got to implement chunking capability into simple axis server > transport first. (May be we can raise a Jira on this). Or we have to > see if we can piggyback on Apache for chunking. > Secondly, we got to research and find out the ideal chunk size. As we > have seen through the results sent by Kapila, too small a chunk size > makes too large file transmission very slow and too large chunk size > makes small file transmission very slow. Best is to not use chunking > for small files and use chunking, with a reasonable chunk size, for > large files. So we have several parameters to figure out. > i. Max size we can read a file in one go without chunking. > ii. When chunking is turned on, ideal chunk size. > As I mentioned earlier, we need some research around this to decide on > the figures. > This is the hard part. The ideal chunk size is higly dependnet on the OS and the machine architecture. Some of the parameters are: optimal hard disk read/write size, optimal memory write/read size, optimal read/write size of the transport stack (For example 1500 byte writes would be ideal for TCP/IP on a 100 Mbps LAN). May be we can look at some APR or Apache httpd stuff to get an idea. - Sahan > Thanks, > Samisa... > >> >> -Nabeel >> >> Samisa Abeysinghe wrote: >> >>> Cool, so there sure is an improvement when using stat on read side >>> of data handler. Hence lets use this solution. >>> Still we need to figure out what goes wrong with large files beyond 1M >>> >>> Samisa... >>> >>> Kapila Dissanayake wrote: >>> >>>> Hi >>>> >>>> I did some modification to the data handler. Insted of using fixed >>>> buffer size to read files >>>> 'stat' is used. >>>> >>>> Please see the perfomance details. >>>> >>>> Thank You >>>> >>>> Kapila >>>> >>>> >>>> Environment ---------- Processor - Intel >>>> Celeron M 1.40GHz >>>> RAM - 256 MB OS - Fedora Core 4. Java Server - >>>> axis2 Java C Server - axis2 mtom C and java test >>>> clients were used (in axis2 c data_handler.c, different buffer >>>> sizes were used to read files) >>>> Time Taken in milli seconds >>>> File Size (kb) Java Server (ms) Java Server >>>> (ms) C Server (ms) C Server (ms) >>>> Java Client (ms) C Client (ms) C >>>> Client (ms) Java Client (ms) >>>> *Buffer Size 1024 * *Buffer >>>> Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024 >>>> * *Buffer Size '1024 X 1024 * *Using stat * >>>> ----------------- ----------------- ----------------- >>>> ----------------- ----------------- ----------------- >>>> ----------------- ----------------- ----------------- >>>> ----------------- ----------------- ----------------- >>>> ----------------- >>>> 13.8 147 158 73 1884 >>>> 78 188 143 136 >>>> 28.2 515 488 78 1912 >>>> 78 222 413 364 >>>> 37.8 538 694 131 1917 >>>> 81 232 474 377 >>>> 132 571 807 212 2177 >>>> 102 263 420 381 >>>> 154 547 672 239 2215 >>>> 117 347 389 383 >>>> 165 688 683 397 2380 >>>> 205 364 400 393 >>>> 561 1340 1024 995 2537 >>>> 777 637 899 775 >>>> 762 1766 1336 1903 2890 >>>> 1458 1248 1218 1051 >>>> 1162 3464 1970 4859 3602 >>>> 3405 1702 1718 1712 >>>> 1761 4666 3274 10928 8028 >>>> 7927 2780 3249 2691 >>>> 2081 6777 2993 15066 11008 >>>> 10864 3595 3281 2859 >>>> 4241 23038 Could not transfer (Java client >>>> Broken) 52853 33351 32957 16234 >>>> 115777 7991 >>>> >>>> >>>> >>>> >>>> On 5/10/06, *Samisa Abeysinghe* <[EMAIL PROTECTED] >>>> <mailto:[EMAIL PROTECTED]>> wrote: >>>> >>>> Well, from this I am certain that we should not change the >>>> buffer size >>>> to 1024*1024, rather keep it at 1024. >>>> Please try with stat, using a dynamic buffer size based on the >>>> file >>>> size, my gut feel is that it is going to be the best solution. >>>> >>>> Thanks, >>>> Samisa... >>>> >>>> Kapila Dissanayake wrote: >>>> >>>> > >>>> > Hi, >>>> > >>>> > I did some testing agian with to measure the binary file >>>> transfer >>>> > times. In axis2 c data_handler.c, different buffer sizes >>>> (1024 and >>>> > 1024*1024) were used to read files. >>>> > >>>> > I think using fixed size buffers to read binary files in axis2-C >>>> has >>>> > been affected in some cases to slow down the transfer. >>>> > >>>> > Thanks >>>> > >>>> > Kapila >>>> > >>>> > Please see the following test results >>>> > >>>> > Environment >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > ---------- >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > Processor - Intel Celeron M 1.40GHz >>>> > >>>> > >>>> > >>>> > >>>> > RAM - 256 MB >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > OS - Fedora Core 4. >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > Java Server - axis2 Java >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > C Server - axis2 >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > mtom C and java test clients were used >>>> > >>>> > >>>> > >>>> > Time Taken in milli seconds >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > File Size (kb) Java Server (ms) Java >>>> Server (ms) C >>>> > Server (ms) C Server (ms) >>>> > Java Client (ms) C >>>> Client (ms) C Client (ms) >>>> Java >>>> > Client (ms) >>>> > *Buffer Size* >>>> *1024 * *1024 X 1024* * * *1024* >>>> > *1024 X 1024 * >>>> > ----------------- ----------------- ----------------- >>>> > ----------------- ----------------- ----------------- >>>> > ----------------- ----------------- ----------------- >>>> > ----------------- ----------------- ----------------- >>>> > 13.8 147 158 >>>> 73 1884 188 143 >>>> > 28.2 515 488 >>>> 78 1912 222 413 >>>> > 37.8 538 694 131 >>>> 1917 232 474 >>>> > 132 571 807 212 >>>> 2177 263 420 >>>> > 154 547 672 239 >>>> 2215 347 389 >>>> > 165 688 683 397 >>>> 2380 364 400 >>>> > 561 1340 1024 995 >>>> 2537 637 899 >>>> > 762 1766 1336 >>>> 1903 2890 1248 1218 >>>> > >>>> 1162 3464 1970 4859 >>>> 3602 1702 1718 >>>> >>>> > 1761 4666 3274 10928 >>>> 8028 2780 3249 >>>> > 2081 6777 2993 15066 >>>> 11008 3595 3281 >>>> > 4241 23038 Could not transfer (Java client >>>> Broken) >>>> > 52853 33351 16234 115777 >>>> > >>>> > >>>> > >>>> > >>>> > On 5/8/06, *Samisa Abeysinghe* <[EMAIL PROTECTED] >>>> <mailto:[EMAIL PROTECTED]> >>>> > <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote: >>>> > >>>> > Hmmm, Interesting. There seem to be a problem in C client in >>>> case of >>>> > large attachments. >>>> > This must be due to the fact that we are using a loop to >>>> read the >>>> > file. >>>> > If we stat and read the file at once, there could be an >>>> improvement. >>>> > >>>> > Why is it so slow, in C vs C case? (Third column) This is >>>> worth >>>> > investigating. >>>> > >>>> > Thanks, >>>> > Samisa... >>>> > >>>> > Kapila Dissanayake wrote: >>>> > >>>> > > Hi, >>>> > > >>>> > > I did some testing on binary attachment transfer times, >>>> with axis2 >>>> > > 'java' and 'C' servers with sample mtom clients. It took >>>> very high >>>> > > transfer times for large size attachments. Some minor >>>> modificaions >>>> > > were done to the data_handler.c and it was sent to the >>>> mailing list. >>>> > > >>>> > > >>>> > > (File Size) vs (Time Taken in milli seconds) to transfer >>>> binary >>>> > > attachments >>>> > > >>>> > > Environment >>>> > > ---------- >>>> > > Processor - Intel Celeron M 1.40GHz >>>> > > RAM - 256 MB >>>> > > OS - Fedora Core 4. >>>> > > Java Server - axis2 Java >>>> > > C Server - axis2 >>>> > > mtom C and java test clients were used >>>> > > Time Taken in milli seconds >>>> > > >>>> > > File Size (kb) >>>> > > >>>> > > Java Server >>>> > > >>>> > > (ms) >>>> > > >>>> > > Java Server >>>> > > >>>> > > (ms) >>>> > > >>>> > > C Server >>>> > > >>>> > > (ms) >>>> > > >>>> > > C Server >>>> > > >>>> > > (ms) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > Java Client >>>> > > >>>> > > (ms) >>>> > > >>>> > > C Client >>>> > > >>>> > > (ms) >>>> > > >>>> > > C Client >>>> > > >>>> > > (ms) >>>> > > >>>> > > Java Client >>>> > > >>>> > > (ms) >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > >>>> > > --------------- >>>> > > 28 >>>> > > >>>> > > >>>> > > >>>> > > 445 >>>> > > >>>> > > >>>> > > >>>> > > 274 >>>> > > >>>> > > >>>> > > >>>> > > 2004 >>>> > > >>>> > > >>>> > > >>>> > > 1009 >>>> > > >>>> > > >>>> > > 38 >>>> > > >>>> > > >>>> > > >>>> > > 617 >>>> > > >>>> > > >>>> > > >>>> > > 297 >>>> > > >>>> > > >>>> > > >>>> > > 1957 >>>> > > >>>> > > >>>> > > >>>> > > 842 >>>> > > >>>> > > >>>> > > 132 >>>> > > >>>> > > >>>> > > >>>> > > 593 >>>> > > >>>> > > >>>> > > >>>> > > 527 >>>> > > >>>> > > >>>> > > >>>> > > 2181 >>>> > > >>>> > > >>>> > > >>>> > > 726 >>>> > > >>>> > > >>>> > > 154 >>>> > > >>>> > > >>>> > > >>>> > > 551 >>>> > > >>>> > > >>>> > > >>>> > > 447 >>>> > > >>>> > > >>>> > > >>>> > > 2081 >>>> > > >>>> > > >>>> > > >>>> > > 677 >>>> > > >>>> > > >>>> > > 165 >>>> > > >>>> > > >>>> > > >>>> > > 440 >>>> > > >>>> > > >>>> > > >>>> > > 743 >>>> > > >>>> > > >>>> > > >>>> > > 2147 >>>> > > >>>> > > >>>> > > >>>> > > 676 >>>> > > >>>> > > >>>> > > 561 >>>> > > >>>> > > >>>> > > >>>> > > 1029 >>>> > > >>>> > > >>>> > > >>>> > > 418 >>>> > > >>>> > > >>>> > > >>>> > > 2581 >>>> > > >>>> > > >>>> > > >>>> > > 1075 >>>> > > >>>> > > >>>> > > 762 >>>> > > >>>> > > >>>> > > >>>> > > 2137 >>>> > > >>>> > > >>>> > > >>>> > > 1572 >>>> > > >>>> > > >>>> > > >>>> > > 3231 >>>> > > >>>> > > >>>> > > >>>> > > 2356 >>>> > > >>>> > > >>>> > > 1162 >>>> > > >>>> > > >>>> > > >>>> > > 3172 >>>> > > >>>> > > >>>> > > >>>> > > 2962 >>>> > > >>>> > > >>>> > > >>>> > > 3981 >>>> > > >>>> > > >>>> > > >>>> > > 3100 >>>> > > >>>> > > >>>> > > 1761 >>>> > > >>>> > > >>>> > > >>>> > > 7672 >>>> > > >>>> > > >>>> > > >>>> > > 5494 >>>> > > >>>> > > >>>> > > >>>> > > 8186 >>>> > > >>>> > > >>>> > > >>>> > > 3041 >>>> > > >>>> > > >>>> > > 2081 >>>> > > >>>> > > >>>> > > >>>> > > 7768 >>>> > > >>>> > > >>>> > > >>>> > > 6837 >>>> > > >>>> > > >>>> > > >>>> > > 11757 >>>> > > >>>> > > >>>> > > >>>> > > 3499 >>>> > > >>>> > > >>>> > > 4241 >>>> > > >>>> > > >>>> > > >>>> > > 14465 >>>> > > >>>> > > >>>> > > >>>> > > 24060 >>>> > > >>>> > > >>>> > > >>>> > > 36168 >>>> > > >>>> > > >>>> > > >>>> > > Could not transfer (Error Occured in Java client) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > Thank You >>>> > > >>>> > > Kapila >>>> > >>>> > >>>> > >>>> >>>> >>> >>> >> >> ------------------------------------------------------------------------ >> >> #include <stdio.h> >> #include <sys/stat.h> >> >> int read_from_file(char *file_name, char** output_stream, int >> *output_stream_size); >> >> int main(int argc, char **argv) >> { >> int size; >> char *output; >> >> if (argc != 2) >> { >> printf("usage: read_binary <file_name>\n"); >> return 0; >> } >> if (read_from_file(argv[1], &output, &size)) >> printf("read successful\n"); >> if (output) >> free(output); >> return 1; >> } >> >> int read_from_file(char *file_name, char** output_stream, int >> *output_stream_size) >> { >> FILE *f = NULL; >> struct stat stat_p; >> int count = 0; >> >> if (!file_name) >> { >> printf("error: file not given \n"); >> return 0; >> } >> f = fopen(file_name, "rb"); >> if (!f) >> { >> printf("error:cannot open file %s\n", file_name); >> return ; >> } >> if ( -1 == stat (file_name, &stat_p)) >> { >> printf("error:error reading stat\n"); >> return 0; >> } >> >> *output_stream_size = stat_p.st_size; >> printf("size = %d\n", *output_stream_size); >> *output_stream = malloc((*output_stream_size) * sizeof(char)); >> >> if (!(*output_stream)) >> { >> printf("error: cannot create stream\n"); >> return 0; >> } >> count = fread(*output_stream, 1, *output_stream_size, f); >> if (ferror(f) != 0) >> { >> printf("error:cannot read file\n"); >> if (*output_stream) >> { >> free(*output_stream); >> *output_stream = NULL; >> } >> return 0; >> >> } >> fclose(f); >> return 1; >> } >> >> > >
