Re: [MTT users] Can not find my testing results in OMPI MTT DB
On Wed, May/21/2008 09:53:11PM, Pavel Shamis (Pasha) wrote: > Oops, in the "MTT server side problem" we discussed other issue. > > But anyway I did not see the problem on my server after the upgrade :) We took *some* steps to alleviate the PHP memory overload problem (e.g., r668, and then r1119), but evidently there's more work to do :-) > > Pasha > > Pavel Shamis (Pasha) wrote: >> I had similar problem on my server. I upgraded the server to latest trunk >> and the problem disappear. >> (see "MTT server side problem" thread). >> >> Pasha >> >> Jeff Squyres wrote: >> >>> FWIW: I think we have at least one open ticket on this issue (break up >>> submits so that we don't overflow PHP and/or apache). https://svn.open-mpi.org/trac/mtt/ticket/221 -Ethan >>> >>> On May 21, 2008, at 2:36 PM, Ethan Mallove wrote: >>> >>> On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote: > I sent it directly to your email. Please check. > Thanks, > Pasha > Got it. Thanks. It's a PHP memory overload issue. (Apparently I didn't look far back enough in the httpd error_logs.) See below. > Ethan Mallove wrote: > >> On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote: >> >> >>> Jeff Squyres wrote: >>> >>> Are we running into http max memory problems or http max upload size problems again? >>> I guess it is some server side issue, you need to check the >>> /var/log/httpd/* log on the server. >>> >>> >> The only thing I found in the httpd logs >> (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP >> warning, which I don't think would result in lost results: >> >> PHP Warning: array_shift(): The argument should be an array in >> .../submit/index.php on line 1683 >> >> I haven't received any emailed Postgres errors either. When >> were these results submitted? I searched for "mellanox" over >> the past four days. It seem the results aren't buried in >> here, because I see no test run failures ... >> >> http://www.open-mpi.org/mtt/index.php?do_redir=659 >> >> I'm assuming you're running with two Reporter INI sections: >> Textfile and MTTDatabase? Can you send some MTT client >> --verbose/--debug output from the below runs? >> >> Thanks, >> Ethan >> >> >> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: > Hi, > > Here is test result from my last mtt run: > +-++--+--+--+--+ > | Phase | Section| Pass | Fail | Time out | Skip | > +-++--+--+--+--+ > | MPI install | ompi/gcc | 1| 0| 0| 0| > | MPI install | ompi/intel-9.0 | 1| 0| 0| 0| > | Test Build | trivial| 1| 0| 0| 0| > | Test Build | trivial| 1| 0| 0| 0| > | Test Build | intel-suite| 1| 0| 0| 0| > | Test Build | intel-suite| 1| 0| 0| 0| > | Test Build | imb| 1| 0| 0| 0| > | Test Build | imb| 1| 0| 0| 0| > | Test Build | presta | 1| 0| 0| 0| > | Test Build | presta | 1| 0| 0| 0| > | Test Build | osu_benchmarks | 1| 0| 0| 0| > | Test Build | osu_benchmarks | 1| 0| 0| 0| > | Test Build | netpipe| 1| 0| 0| 0| > | Test Build | netpipe| 1| 0| 0| 0| > | Test Run| trivial| 64 | 0| 0| 0| > | Test Run| trivial| 64 | 0| 0| 0| > | Test Run| intel-suite| 3179 | 165 | 400 | 0| > | Test Run| intel-suite| 492 | 0| 0| 0| > +-++--+--+--+--+ > > In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow > "test run" results: > | Test Run| trivial| 64 | 0| 0| 0| > | Test Run| trivial| 64 | 0| 0| 0| > | Test Run| intel-suite| 492 | 0| 0| 0| > > And I can not find this one: > | Test Run| intel-suite| 3179 | 165 | 400 | 0| > Some missing results are in mttdb_debug_file.16.txt (and 17.txt), which are the largest .txt files of the bunch. 8 variants isn't that
Re: [MTT users] Can not find my testing results in OMPI MTT DB
Oops, in the "MTT server side problem" we discussed other issue. But anyway I did not see the problem on my server after the upgrade :) Pasha Pavel Shamis (Pasha) wrote: I had similar problem on my server. I upgraded the server to latest trunk and the problem disappear. (see "MTT server side problem" thread). Pasha Jeff Squyres wrote: FWIW: I think we have at least one open ticket on this issue (break up submits so that we don't overflow PHP and/or apache). On May 21, 2008, at 2:36 PM, Ethan Mallove wrote: On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote: I sent it directly to your email. Please check. Thanks, Pasha Got it. Thanks. It's a PHP memory overload issue. (Apparently I didn't look far back enough in the httpd error_logs.) See below. Ethan Mallove wrote: On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote: Jeff Squyres wrote: Are we running into http max memory problems or http max upload size problems again? I guess it is some server side issue, you need to check the /var/log/httpd/* log on the server. The only thing I found in the httpd logs (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP warning, which I don't think would result in lost results: PHP Warning: array_shift(): The argument should be an array in .../submit/index.php on line 1683 I haven't received any emailed Postgres errors either. When were these results submitted? I searched for "mellanox" over the past four days. It seem the results aren't buried in here, because I see no test run failures ... http://www.open-mpi.org/mtt/index.php?do_redir=659 I'm assuming you're running with two Reporter INI sections: Textfile and MTTDatabase? Can you send some MTT client --verbose/--debug output from the below runs? Thanks, Ethan On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: Hi, Here is test result from my last mtt run: +-++--+--+--+--+ | Phase | Section| Pass | Fail | Time out | Skip | +-++--+--+--+--+ | MPI install | ompi/gcc | 1| 0| 0| 0| | MPI install | ompi/intel-9.0 | 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 3179 | 165 | 400 | 0| | Test Run| intel-suite| 492 | 0| 0| 0| +-++--+--+--+--+ In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow "test run" results: | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 492 | 0| 0| 0| And I can not find this one: | Test Run| intel-suite| 3179 | 165 | 400 | 0| Some missing results are in mttdb_debug_file.16.txt (and 17.txt), which are the largest .txt files of the bunch. 8 variants isn't that much, but maybe it causes a problem when there's lots of stderr/stdout? I'm surprised submit/index.php barfs on files this size: $ ls -l ... -rw-r--r--1 em162155 staff 956567 May 21 14:21 mttdb_debug_file.16.inc.gz -rw-r--r--1 em162155 staff 9603132 May 21 14:09 mttdb_debug_file.16.txt ... $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -z -u sun -p sun4sun -d LWP::UserAgent::new: () LWP::UserAgent::proxy: http Filelist: $VAR1 = 'mttdb_debug_file.16.txt'; LWP::MediaTypes::read_media_types: Reading media types from /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /usr/perl5/site_perl/5.8.4/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /home/em162155/.mime.types LWP::UserAgent::request: () LWP::UserAgent::send_request: POST http://www.open-mpi.org/mtt/submit/index.php LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: ()
Re: [MTT users] Can not find my testing results in OMPI MTT DB
I had similar problem on my server. I upgraded the server to latest trunk and the problem disappear. (see "MTT server side problem" thread). Pasha Jeff Squyres wrote: FWIW: I think we have at least one open ticket on this issue (break up submits so that we don't overflow PHP and/or apache). On May 21, 2008, at 2:36 PM, Ethan Mallove wrote: On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote: I sent it directly to your email. Please check. Thanks, Pasha Got it. Thanks. It's a PHP memory overload issue. (Apparently I didn't look far back enough in the httpd error_logs.) See below. Ethan Mallove wrote: On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote: Jeff Squyres wrote: Are we running into http max memory problems or http max upload size problems again? I guess it is some server side issue, you need to check the /var/log/httpd/* log on the server. The only thing I found in the httpd logs (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP warning, which I don't think would result in lost results: PHP Warning: array_shift(): The argument should be an array in .../submit/index.php on line 1683 I haven't received any emailed Postgres errors either. When were these results submitted? I searched for "mellanox" over the past four days. It seem the results aren't buried in here, because I see no test run failures ... http://www.open-mpi.org/mtt/index.php?do_redir=659 I'm assuming you're running with two Reporter INI sections: Textfile and MTTDatabase? Can you send some MTT client --verbose/--debug output from the below runs? Thanks, Ethan On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: Hi, Here is test result from my last mtt run: +-++--+--+--+--+ | Phase | Section| Pass | Fail | Time out | Skip | +-++--+--+--+--+ | MPI install | ompi/gcc | 1| 0| 0| 0| | MPI install | ompi/intel-9.0 | 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 3179 | 165 | 400 | 0| | Test Run| intel-suite| 492 | 0| 0| 0| +-++--+--+--+--+ In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow "test run" results: | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 492 | 0| 0| 0| And I can not find this one: | Test Run| intel-suite| 3179 | 165 | 400 | 0| Some missing results are in mttdb_debug_file.16.txt (and 17.txt), which are the largest .txt files of the bunch. 8 variants isn't that much, but maybe it causes a problem when there's lots of stderr/stdout? I'm surprised submit/index.php barfs on files this size: $ ls -l ... -rw-r--r--1 em162155 staff 956567 May 21 14:21 mttdb_debug_file.16.inc.gz -rw-r--r--1 em162155 staff 9603132 May 21 14:09 mttdb_debug_file.16.txt ... $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -z -u sun -p sun4sun -d LWP::UserAgent::new: () LWP::UserAgent::proxy: http Filelist: $VAR1 = 'mttdb_debug_file.16.txt'; LWP::MediaTypes::read_media_types: Reading media types from /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /usr/perl5/site_perl/5.8.4/LWP/media.types LWP::MediaTypes::read_media_types: Reading media types from /home/em162155/.mime.types LWP::UserAgent::request: () LWP::UserAgent::send_request: POST http://www.open-mpi.org/mtt/submit/index.php LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::UserAgent::request: Simple response: OK $ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit ... [client 192.18.128.5] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 14 bytes) in /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php on line
Re: [MTT users] Can not find my testing results in OMPI MTT DB
Jeff Squyres wrote: Are we running into http max memory problems or http max upload size problems again? I guess it is some server side issue, you need to check the /var/log/httpd/* log on the server. On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: Hi, Here is test result from my last mtt run: +-++--+--+--+--+ | Phase | Section| Pass | Fail | Time out | Skip | +-++--+--+--+--+ | MPI install | ompi/gcc | 1| 0| 0| 0| | MPI install | ompi/intel-9.0 | 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 3179 | 165 | 400 | 0| | Test Run| intel-suite| 492 | 0| 0| 0| +-++--+--+--+--+ In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow "test run" results: | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 492 | 0| 0| 0| And I can not find this one: | Test Run| intel-suite| 3179 | 165 | 400 | 0| From the log I see that all tests results were submitted successfully. Can you please check ? Thanks, Pasha ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
Re: [MTT users] Can not find my testing results in OMPI MTT DB
Are we running into http max memory problems or http max upload size problems again? On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote: Hi, Here is test result from my last mtt run: +-++--+--+--+--+ | Phase | Section| Pass | Fail | Time out | Skip | +-++--+--+--+--+ | MPI install | ompi/gcc | 1| 0| 0| 0| | MPI install | ompi/intel-9.0 | 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | trivial| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | intel-suite| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | imb| 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | presta | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | osu_benchmarks | 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Build | netpipe| 1| 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 3179 | 165 | 400 | 0| | Test Run| intel-suite| 492 | 0| 0| 0| +-++--+--+--+--+ In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow "test run" results: | Test Run| trivial| 64 | 0| 0| 0| | Test Run| trivial| 64 | 0| 0| 0| | Test Run| intel-suite| 492 | 0| 0| 0| And I can not find this one: | Test Run| intel-suite| 3179 | 165 | 400 | 0| From the log I see that all tests results were submitted successfully. Can you please check ? Thanks, Pasha ___ mtt-users mailing list mtt-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users -- Jeff Squyres Cisco Systems
