Bug #55438 [Com]: race condition: curlwapper is not sending http header randomly
Edit report at https://bugs.php.net/bug.php?id=55438&edit=1 ID: 55438 Comment by: phpnet at lostreality dot org Reported by:xuefer at gmail dot com Summary:race condition: curlwapper is not sending http header randomly Status: Assigned Type: Bug Package:cURL related Operating System: gentoo PHP Version:5.3.6 Assigned To:pierrick Block user comment: N Private report: N New Comment: Ok, thanks for the info. I was mostly wondering if it would make 5.3.x at all. Well now it seems I just have to wait for 5.3.21 to be released, and CPanel to upgrade, and so on :) Previous Comments: [2012-12-19 16:57:15] pierr...@php.net Unfortunately, it's to late for 5.3.20 and 5.4.10, sorry. It will only be available in 5.3.21 and 5.4.11 [2012-12-19 15:17:12] phpnet at lostreality dot org Great. Any chance this can make it to 5.3.20 also? [2012-12-19 07:51:36] pierr...@php.net Ok, I finally reproduced the problem. I was trying the code snippet on my local network and everything was fine, once I modified the code to fetch an URL on a slower network I had the problem. Since curl multi is used, it sometime happen that the resource is freed before the curl multi really execute the query. The patch looks good, I'll have a second look tomorrow and will commit it. Thanks for your help on this one :) [2012-12-19 06:01:04] phpnet at lostreality dot org I have curl-7.15.5-15.el5 according to rpm -q, but I can only locate /usr/lib/libcurl.so.3.0.0 and /usr/lib64/libcurl.so.3.0.0 on my machine I'm testing on (CentOS 5.8). The binary says: /usr/bin/curl -V curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Normally, I use the stock RPMs for PHP, but a recent project I was working on failed to run properly on another machine, where the owners also use CentOS 5.8, but use CPanel/WHM instead of the CentOS RPMs for PHP. CPanel uses the --with-curlwrappers option, where as the stock CentOS and RHEL RPMs have never used that option on any of their builds. It took a lot of digging before I realized that it was the --with-curlwrappers option that caused the scripts to fail on that machine while working perfectly on mine. To verify if the headers were actually sent, I used: tcpdump -i eth1 -Als0 host www.example.com I had two PuTTY windows open, one with tcpdump, the other running the test script I mentioned before with: ./php-5.4.9/sapi/cli/php ./test.php It was pretty clear to me that the headers were never sent before the patch, and always sent after the patch. [2012-12-19 05:50:16] pierr...@php.net I tried to reproduce this bug but wasn't able to do it. Could you give me more details on the libcurl version used by your PHP instance ? And also, how do you make sure that the headers are not properly sent ? The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=55438 -- Edit this bug report at https://bugs.php.net/bug.php?id=55438&edit=1
Bug #55438 [Com]: race condition: curlwapper is not sending http header randomly
Edit report at https://bugs.php.net/bug.php?id=55438&edit=1 ID: 55438 Comment by: phpnet at lostreality dot org Reported by:xuefer at gmail dot com Summary:race condition: curlwapper is not sending http header randomly Status: Assigned Type: Bug Package:cURL related Operating System: gentoo PHP Version:5.3.6 Assigned To:pierrick Block user comment: N Private report: N New Comment: Great. Any chance this can make it to 5.3.20 also? Previous Comments: [2012-12-19 07:51:36] pierr...@php.net Ok, I finally reproduced the problem. I was trying the code snippet on my local network and everything was fine, once I modified the code to fetch an URL on a slower network I had the problem. Since curl multi is used, it sometime happen that the resource is freed before the curl multi really execute the query. The patch looks good, I'll have a second look tomorrow and will commit it. Thanks for your help on this one :) [2012-12-19 06:01:04] phpnet at lostreality dot org I have curl-7.15.5-15.el5 according to rpm -q, but I can only locate /usr/lib/libcurl.so.3.0.0 and /usr/lib64/libcurl.so.3.0.0 on my machine I'm testing on (CentOS 5.8). The binary says: /usr/bin/curl -V curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Normally, I use the stock RPMs for PHP, but a recent project I was working on failed to run properly on another machine, where the owners also use CentOS 5.8, but use CPanel/WHM instead of the CentOS RPMs for PHP. CPanel uses the --with-curlwrappers option, where as the stock CentOS and RHEL RPMs have never used that option on any of their builds. It took a lot of digging before I realized that it was the --with-curlwrappers option that caused the scripts to fail on that machine while working perfectly on mine. To verify if the headers were actually sent, I used: tcpdump -i eth1 -Als0 host www.example.com I had two PuTTY windows open, one with tcpdump, the other running the test script I mentioned before with: ./php-5.4.9/sapi/cli/php ./test.php It was pretty clear to me that the headers were never sent before the patch, and always sent after the patch. [2012-12-19 05:50:16] pierr...@php.net I tried to reproduce this bug but wasn't able to do it. Could you give me more details on the libcurl version used by your PHP instance ? And also, how do you make sure that the headers are not properly sent ? [2012-12-19 04:33:11] phpnet at lostreality dot org I submitted a patch that moves the slist from a local variable in php_curl_stream_opener() into the php_curl_stream struct. The headers are no longer cleared and freed at the end of php_curl_stream_opener(). The code to free the slist is moved into php_curl_stream_close() instead. I'm not sure if this is the best approach, but it clearly gives me a 100% success rate with having headers get sent, where as I had a literal 0% success rate before (not sure if there is really a race condition or not, just that the headers get cleared and the slist freed before they get used.) The test code I used was as follows (Actual cookie and URL redacted) array('method' => 'GET', 'header' => 'Cookie: foo=bar')); $ctx = stream_context_create($opt); $f = fopen('http://www.example.com/', 'r', false, $ctx); fread($f, 1); //work-around curl-wrappers bug where meta_data doesn't exist until the stream is read $data = stream_get_meta_data($f); fclose($f); var_dump($data); ?> I compiled PHP with the following flags (not that I think anything matters to this bug other than --with-curlwrappers): --enable-static --with-mcrypt --with-ldap --with-iconv --enable-mbstring --with-gd --enable-mbregex --with-zlib --with-imap --enable-ftp --with-gettext --enable-sockets --with-mysql=/usr --enable-cgi --with-imap-ssl --enable-sockets --with-pdo-mysql --with-openssl --with-kerberos --with-curl --with-curlwrappers --with-tidy --with-pcre-regex --with-bz2 --enable-zip --with-libdir=/lib64 -------------------- [2012-12-18 19:29:56] phpnet at lostreality dot org I think I am seeing this same problem too (On 5.3.10, but nothing has changed in the source in 5.4.9 either). Can you explain how this is happening, or suggest a work-around? I was digging into the PHP source, expecting that the --with-curlwrappers option was basically broken and incomplete. I was surprised to find the line: curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEA
Bug #55438 [Com]: race condition: curlwapper is not sending http header randomly
Edit report at https://bugs.php.net/bug.php?id=55438&edit=1 ID: 55438 Comment by: phpnet at lostreality dot org Reported by:xuefer at gmail dot com Summary:race condition: curlwapper is not sending http header randomly Status: Assigned Type: Bug Package:cURL related Operating System: gentoo PHP Version:5.3.6 Assigned To:pierrick Block user comment: N Private report: N New Comment: I have curl-7.15.5-15.el5 according to rpm -q, but I can only locate /usr/lib/libcurl.so.3.0.0 and /usr/lib64/libcurl.so.3.0.0 on my machine I'm testing on (CentOS 5.8). The binary says: /usr/bin/curl -V curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Normally, I use the stock RPMs for PHP, but a recent project I was working on failed to run properly on another machine, where the owners also use CentOS 5.8, but use CPanel/WHM instead of the CentOS RPMs for PHP. CPanel uses the --with-curlwrappers option, where as the stock CentOS and RHEL RPMs have never used that option on any of their builds. It took a lot of digging before I realized that it was the --with-curlwrappers option that caused the scripts to fail on that machine while working perfectly on mine. To verify if the headers were actually sent, I used: tcpdump -i eth1 -Als0 host www.example.com I had two PuTTY windows open, one with tcpdump, the other running the test script I mentioned before with: ./php-5.4.9/sapi/cli/php ./test.php It was pretty clear to me that the headers were never sent before the patch, and always sent after the patch. Previous Comments: [2012-12-19 05:50:16] pierr...@php.net I tried to reproduce this bug but wasn't able to do it. Could you give me more details on the libcurl version used by your PHP instance ? And also, how do you make sure that the headers are not properly sent ? [2012-12-19 04:33:11] phpnet at lostreality dot org I submitted a patch that moves the slist from a local variable in php_curl_stream_opener() into the php_curl_stream struct. The headers are no longer cleared and freed at the end of php_curl_stream_opener(). The code to free the slist is moved into php_curl_stream_close() instead. I'm not sure if this is the best approach, but it clearly gives me a 100% success rate with having headers get sent, where as I had a literal 0% success rate before (not sure if there is really a race condition or not, just that the headers get cleared and the slist freed before they get used.) The test code I used was as follows (Actual cookie and URL redacted) array('method' => 'GET', 'header' => 'Cookie: foo=bar')); $ctx = stream_context_create($opt); $f = fopen('http://www.example.com/', 'r', false, $ctx); fread($f, 1); //work-around curl-wrappers bug where meta_data doesn't exist until the stream is read $data = stream_get_meta_data($f); fclose($f); var_dump($data); ?> I compiled PHP with the following flags (not that I think anything matters to this bug other than --with-curlwrappers): --enable-static --with-mcrypt --with-ldap --with-iconv --enable-mbstring --with-gd --enable-mbregex --with-zlib --with-imap --enable-ftp --with-gettext --enable-sockets --with-mysql=/usr --enable-cgi --with-imap-ssl --enable-sockets --with-pdo-mysql --with-openssl --with-kerberos --with-curl --with-curlwrappers --with-tidy --with-pcre-regex --with-bz2 --enable-zip --with-libdir=/lib64 -------------------- [2012-12-18 19:29:56] phpnet at lostreality dot org I think I am seeing this same problem too (On 5.3.10, but nothing has changed in the source in 5.4.9 either). Can you explain how this is happening, or suggest a work-around? I was digging into the PHP source, expecting that the --with-curlwrappers option was basically broken and incomplete. I was surprised to find the line: curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, slist); Because that code all seems to indicate that the headers should be sent, but no matter what I try, nothing I put in headers ever appears in the actual request. I keep running tcpdump but I never see the headers I put in http->header. [2011-08-17 11:45:54] xuefer at gmail dot com sorry for the mismatch http url string. i was trying to remove some string for privacy [2011-08-17 11:38:50] xuefer at gmail dot com Description: background: php is configured with curl wrapper, which make file_get_contents use curl. i haven't tested wit
Bug #55438 [Com]: race condition: curlwapper is not sending http header randomly
Edit report at https://bugs.php.net/bug.php?id=55438&edit=1 ID: 55438 Comment by: phpnet at lostreality dot org Reported by:xuefer at gmail dot com Summary:race condition: curlwapper is not sending http header randomly Status: Open Type: Bug Package:cURL related Operating System: gentoo PHP Version:5.3.6 Block user comment: N Private report: N New Comment: I submitted a patch that moves the slist from a local variable in php_curl_stream_opener() into the php_curl_stream struct. The headers are no longer cleared and freed at the end of php_curl_stream_opener(). The code to free the slist is moved into php_curl_stream_close() instead. I'm not sure if this is the best approach, but it clearly gives me a 100% success rate with having headers get sent, where as I had a literal 0% success rate before (not sure if there is really a race condition or not, just that the headers get cleared and the slist freed before they get used.) The test code I used was as follows (Actual cookie and URL redacted) array('method' => 'GET', 'header' => 'Cookie: foo=bar')); $ctx = stream_context_create($opt); $f = fopen('http://www.example.com/', 'r', false, $ctx); fread($f, 1); //work-around curl-wrappers bug where meta_data doesn't exist until the stream is read $data = stream_get_meta_data($f); fclose($f); var_dump($data); ?> I compiled PHP with the following flags (not that I think anything matters to this bug other than --with-curlwrappers): --enable-static --with-mcrypt --with-ldap --with-iconv --enable-mbstring --with-gd --enable-mbregex --with-zlib --with-imap --enable-ftp --with-gettext --enable-sockets --with-mysql=/usr --enable-cgi --with-imap-ssl --enable-sockets --with-pdo-mysql --with-openssl --with-kerberos --with-curl --with-curlwrappers --with-tidy --with-pcre-regex --with-bz2 --enable-zip --with-libdir=/lib64 Previous Comments: -------------------- [2012-12-18 19:29:56] phpnet at lostreality dot org I think I am seeing this same problem too (On 5.3.10, but nothing has changed in the source in 5.4.9 either). Can you explain how this is happening, or suggest a work-around? I was digging into the PHP source, expecting that the --with-curlwrappers option was basically broken and incomplete. I was surprised to find the line: curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, slist); Because that code all seems to indicate that the headers should be sent, but no matter what I try, nothing I put in headers ever appears in the actual request. I keep running tcpdump but I never see the headers I put in http->header. [2011-08-17 11:45:54] xuefer at gmail dot com sorry for the mismatch http url string. i was trying to remove some string for privacy [2011-08-17 11:38:50] xuefer at gmail dot com Description: background: php is configured with curl wrapper, which make file_get_contents use curl. i haven't tested with calling curl functions directly php unset curl header too soon before curl make the request expected order: set header, build and send request. unset header actual order 1: set header, build and send request. unset header (good) actual order 2: set header, unset header, build and send request (bad) "send request" comes after php "unset header" curl behavior randomly, by sending request before or after php unset the header Test script: --- #!/usr/lib/php5.3/bin/php http://localhost/";; $context = stream_context_create(array( 'http' => array( 'header' => "Authorization: Basic " . base64_encode("$username:$password") ) )); $http_response_header = array(); $data = file_get_contents($authUrl, false, $context); sleep(1); } ?> tcpdump -nilo dst port 80 -w- -s0 Expected result: GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u
Bug #55438 [Com]: race condition: curlwapper is not sending http header randomly
Edit report at https://bugs.php.net/bug.php?id=55438&edit=1 ID: 55438 Comment by: phpnet at lostreality dot org Reported by:xuefer at gmail dot com Summary:race condition: curlwapper is not sending http header randomly Status: Open Type: Bug Package:cURL related Operating System: gentoo PHP Version:5.3.6 Block user comment: N Private report: N New Comment: I think I am seeing this same problem too (On 5.3.10, but nothing has changed in the source in 5.4.9 either). Can you explain how this is happening, or suggest a work-around? I was digging into the PHP source, expecting that the --with-curlwrappers option was basically broken and incomplete. I was surprised to find the line: curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, slist); Because that code all seems to indicate that the headers should be sent, but no matter what I try, nothing I put in headers ever appears in the actual request. I keep running tcpdump but I never see the headers I put in http->header. Previous Comments: [2011-08-17 11:45:54] xuefer at gmail dot com sorry for the mismatch http url string. i was trying to remove some string for privacy [2011-08-17 11:38:50] xuefer at gmail dot com Description: background: php is configured with curl wrapper, which make file_get_contents use curl. i haven't tested with calling curl functions directly php unset curl header too soon before curl make the request expected order: set header, build and send request. unset header actual order 1: set header, build and send request. unset header (good) actual order 2: set header, unset header, build and send request (bad) "send request" comes after php "unset header" curl behavior randomly, by sending request before or after php unset the header Test script: --- #!/usr/lib/php5.3/bin/php http://localhost/";; $context = stream_context_create(array( 'http' => array( 'header' => "Authorization: Basic " . base64_encode("$username:$password") ) )); $http_response_header = array(); $data = file_get_contents($authUrl, false, $context); sleep(1); } ?> tcpdump -nilo dst port 80 -w- -s0 Expected result: GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= Actual result: -- GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6MTQzNjEuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* Authorization: Basic dGVzdDE6NTY3MDQuLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4u Li4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4uLi4= GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* GET / HTTP/1.1 User-Agent: PHP/5.3.6-pl0-gentoo Host: localhost Accept: */* gdb --args /usr/lib/php5.3/bin/php.debug test.php (gdb) l /usr/src/debug/dev-lang/php-5.3.6/sapis-build/cli/ext/curl/streams.c:367 360 slist = curl_slist_append(slist, trimmed); 361 efree(trimmed); 362 p = php_strtok_r(NULL, "\r\n", &token); 363 } 364 efree(copy_ctx_opt); 365 } 366 if (slist) { 367 curl_easy_setopt(curlstream->curl, CURLOPT_HTTPHEADER, slist); 368 } 369 }