Re: Problem with git-http-backend.exe as iis cgi
Hi Junio, On Tue, 21 Jun 2016, Junio C Hamano wrote: > Konstantin Khomoutovwrites: [purportedly on 10 Mar 13:55 2016, see $gmane/297739] > Isn't this responding to a stale thread? I was puzzled, too. I think the mail somehow got re-sent (and another one by Konst, too). Ciao, Dscho -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem with git-http-backend.exe as iis cgi
Konstantin Khomoutovwrites: > On Thu, 10 Mar 2016 07:28:50 + > Florian Manschwetus wrote: > >> I tried to setup git-http-backend with iis, as iis provides proper >> impersonation for cgi under windows, which leads to have the >> filesystem access performed with the logon user, therefore the >> webserver doesn't need generic access to the files. I stumbled across >> a problem, ending up with post requests hanging forever. After some >> investigation I managed to get it work by wrapping the http-backend >> into a bash script, giving a lot of control about the environmental >> things, I was unable to solve within IIS configuration. The >> workaround, I use currently, is to use "/bin/head -c >> ${CONTENT_LENGTH} | ./git-http-backend.exe", which directly shows the >> issue. Git http-backend should check if CONTENT_LENGTH is set to >> something reasonable (e.g. >0) and should in this case read only >> CONTENT_LENGTH bytes from stdin, instead of reading till EOF what I >> suspect it is doing currently. > ... > So yes, if Git currently reads until EOF, it's an error. This sounded vaguely familiar. Isn't this responding to a stale thread? http://thread.gmane.org/gmane.comp.version-control.git/290114 proposed a patch along the line, and corrections to the patch was suggested in the review, but it was not followed through, it seems. -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem with git-http-backend.exe as iis cgi
On Thu, 10 Mar 2016 07:28:50 + Florian Manschwetuswrote: > I tried to setup git-http-backend with iis, as iis provides proper > impersonation for cgi under windows, which leads to have the > filesystem access performed with the logon user, therefore the > webserver doesn't need generic access to the files. I stumbled across > a problem, ending up with post requests hanging forever. After some > investigation I managed to get it work by wrapping the http-backend > into a bash script, giving a lot of control about the environmental > things, I was unable to solve within IIS configuration. The > workaround, I use currently, is to use "/bin/head -c > ${CONTENT_LENGTH} | ./git-http-backend.exe", which directly shows the > issue. Git http-backend should check if CONTENT_LENGTH is set to > something reasonable (e.g. >0) and should in this case read only > CONTENT_LENGTH bytes from stdin, instead of reading till EOF what I > suspect it is doing currently. The rfc [1] states in its section 4.2: | A request-body is supplied with the request if the CONTENT_LENGTH is | not NULL. The server MUST make at least that many bytes available | for the script to read. The server MAY signal an end-of-file | condition after CONTENT_LENGTH bytes have been read or it MAY supply | extension data. Therefore, the script MUST NOT attempt to read more | than CONTENT_LENGTH bytes, even if more data is available. However, | it is not obliged to read any of the data. So yes, if Git currently reads until EOF, it's an error. The correct way would be: 1) Check to see if the CONTENT_LENGTH variable is available in the environment. If no, read nothing. 2) Otherwise read as many bytes it specifies, and no more. 1. https://www.ietf.org/rfc/rfc3875 -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Fix http-backend reading till EOF, ignoring CONTENT_LENGTH, violating rfc3875 -- WAS: Problem with git-http-backend.exe as iis cgi
On Wed, Mar 30, 2016 at 09:08:56AM +, Florian Manschwetus wrote: > After additional analysis it turned out, that in the case you > mentioned, at least IIS, sets CONTENT_LENGTH to -1 resulting in the > current behavior of git-http-backend being sufficient in this > situation. > Therefore I refactored the code again a bit, to match up the behavior > I currently fake by using some bash magic... OK, so I'd agree it makes sense to catch "-1", and read to EOF in that case (or if CONTENT_LENGTH is NULL). > From ccd6c88e39a850b253979b785463719cdc0fa1e2 Mon Sep 17 00:00:00 2001 > From: manschwetus> Date: Tue, 29 Mar 2016 12:16:21 +0200 > Subject: [PATCH 1/2] Fix http-backend reading till EOF, ignoring > CONTENT_LENGTH, violating rfc3875 Please send one patch per email, and these header bits should be the header of your email. Though we also generally revise and re-send patches, rather than presenting one patch that has problems and then tacking fixes on top. I'll ignore the problems in this patch 1, as it looks like it's just the original one repeated. > From 4b2aac3dfd4954098190745a9e4fa17f254cd6a1 Mon Sep 17 00:00:00 2001 > From: Florian Manschwetus > Date: Wed, 30 Mar 2016 10:54:21 +0200 > Subject: [PATCH 2/2] restored old behavior as read_request_eof(...) and moved > new variant to read_request_fix_len(...) and introduced read_request(...) as > wrapper, which decides based on value retrieved from CONTENT_LENGTH which > variant to use Please use a short subject for your commit message, followed by a blank line and then a more explanatory body. Also, don't just describe _what_ is happening (we can see that from the diff), but _why_. You can find more similar tips in SubmittingPatches. > +/** > + * wrapper function, whcih determines based on CONTENT_LENGTH value, > + * to > + * - use old behaviour of read_request, to read until EOF > + * => read_request_eof(...) > + * - just read CONTENT_LENGTH-bytes, when provided > + * => read_request_fix_len(...) > + */ > +static ssize_t read_request(int fd, unsigned char **out) > +{ > +/* get request size */ > +size_t req_len = git_env_ulong("CONTENT_LENGTH", > + -1); > +if (req_len < 0){ > + read_request_eof(fd, out); > +} else { > + read_request_fix_len(fd, req_len, out); > +} > +} I don't think "if (req_len < 0)" can ever trigger, because size_t is an unsigned type (and I do not recall what kind of integer overflow validation we do in git_env_ulong, but I suspect it may complain about "-1"). You may have to parse the variable manually rather than using git_env_ulong (i.e., pick out the NULL and "-1" cases, and then feed the rest to git_parse_ulong()). Also, a few style nits. We usually omit braces for one-line conditionals, and please make sure there is whitespace between the closing parenthesis and the opening brace. There's some discussing in CodingGuidelines. -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: [PATCH] Fix http-backend reading till EOF, ignoring CONTENT_LENGTH, violating rfc3875 -- WAS: Problem with git-http-backend.exe as iis cgi
> -Ursprüngliche Nachricht- > Von: Jeff King [mailto:p...@peff.net] > Gesendet: Dienstag, 29. März 2016 22:14 > An: Florian Manschwetus > Cc: Chris Packham; Konstantin Khomoutov; git@vger.kernel.org > Betreff: Re: [PATCH] Fix http-backend reading till EOF, ignoring > CONTENT_LENGTH, violating rfc3875 -- WAS: Problem with git-http- > backend.exe as iis cgi > > On Tue, Mar 29, 2016 at 10:38:23AM +, Florian Manschwetus wrote: > > > > | A request-body is supplied with the request if the CONTENT_LENGTH > > > | is not NULL. The server MUST make at least that many bytes > > > | available for the script to read. The server MAY signal an > > > | end-of-file condition after CONTENT_LENGTH bytes have been read or > > > | it MAY supply extension data. Therefore, the script MUST NOT > > > | attempt to read more than CONTENT_LENGTH bytes, even if more data > > > | is available. However, it is not obliged to read any of the data. > > > > > > So yes, if Git currently reads until EOF, it's an error. > > > The correct way would be: > > > > > > 1) Check to see if the CONTENT_LENGTH variable is available in the > > >environment. If no, read nothing. > > > > > > 2) Otherwise read as many bytes it specifies, and no more. > > > > > > 1. https://www.ietf.org/rfc/rfc3875 > > I don't think the second part of (1) will work very well if the client sends a > chunked transfer-encoding (which git will do if the input is large). In such a > case the server would either have to buffer the entire input to find its > length, > or stream the data to the CGI without setting $CONTENT_LENGTH. At least > some servers choose the latter (including Apache). > > > diff --git a/http-backend.c b/http-backend.c index 8870a26..94976df > > 100644 > > --- a/http-backend.c > > +++ b/http-backend.c > > @@ -277,16 +277,32 @@ static struct rpc_service *select_service(const > char *name) > > */ > > static ssize_t read_request(int fd, unsigned char **out) { > > - size_t len = 0, alloc = 8192; > > - unsigned char *buf = xmalloc(alloc); > > + unsigned char *buf = null; > > ... > > git-am complained that your patch did not apply, but after writing something > similar locally, I found that t5551.25 hangs indefinitely. > Which is not surprising. Most tests are doing very limited ref negotiation, so > the content that hits read_request() here is small, and we send it in a single > write with a content-length header. But t5551.25 uses a much bigger > workload, which causes the client to use a chunked transfer-encoding, and > this code to refuse to read anything (and then the protocol stalls, as we are > waiting for the client to say something). > > So I think you'd want to take a missing CONTENT_LENGTH as a hint to read > until EOF. > > That also raises another issue: what happens in the paths that don't hit > read_request()? We may also process input via: > > - inflate_request(), if the client gzipped it; for well-formed input, > I think we'll stop reading when the gzip stream tells us there is no > more data, but a malformed one would have us reading until EOF, > regardless of what $CONTENT_LENGTH says. > > - for input which we expect to be large (like incoming packfiles for a > push), buffer_input will be unset, and we will pass the descriptor > directly to a sub-program like git-index-pack. Again, for > well-formed input it would read just the packfile, but it may > actually continue to EOF. > > So I don't think your patch is covering all cases. > > -Peff After additional analysis it turned out, that in the case you mentioned, at least IIS, sets CONTENT_LENGTH to -1 resulting in the current behavior of git-http-backend being sufficient in this situation. Therefore I refactored the code again a bit, to match up the behavior I currently fake by using some bash magic... From ccd6c88e39a850b253979b785463719cdc0fa1e2 Mon Sep 17 00:00:00 2001 From: manschwetus <manschwe...@cs-software-gmbh.de> Date: Tue, 29 Mar 2016 12:16:21 +0200 Subject: [PATCH 1/2] Fix http-backend reading till EOF, ignoring CONTENT_LENGTH, violating rfc3875 Signed-off-by: Florian Manschwetus <manschwe...@cs-software-gmbh.de> --- http-backend.c | 48 +++- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/http-backend.c b/http-backend.c index 8870a26..94976df 100644 --- a/http-backend.c +++ b/http-backend.c @@ -277,16 +277,32 @@ static struct rpc_service *select_service(const char *name) */ static ssize_t read_request(int fd, unsigned char **out) { - size_t len = 0, alloc =
Re: [PATCH] Fix http-backend reading till EOF, ignoring CONTENT_LENGTH, violating rfc3875 -- WAS: Problem with git-http-backend.exe as iis cgi
On Tue, Mar 29, 2016 at 10:38:23AM +, Florian Manschwetus wrote: > > | A request-body is supplied with the request if the CONTENT_LENGTH is > > | not NULL. The server MUST make at least that many bytes available > > | for the script to read. The server MAY signal an end-of-file > > | condition after CONTENT_LENGTH bytes have been read or it MAY supply > > | extension data. Therefore, the script MUST NOT attempt to read more > > | than CONTENT_LENGTH bytes, even if more data is available. However, > > | it is not obliged to read any of the data. > > > > So yes, if Git currently reads until EOF, it's an error. > > The correct way would be: > > > > 1) Check to see if the CONTENT_LENGTH variable is available in the > >environment. If no, read nothing. > > > > 2) Otherwise read as many bytes it specifies, and no more. > > > > 1. https://www.ietf.org/rfc/rfc3875 I don't think the second part of (1) will work very well if the client sends a chunked transfer-encoding (which git will do if the input is large). In such a case the server would either have to buffer the entire input to find its length, or stream the data to the CGI without setting $CONTENT_LENGTH. At least some servers choose the latter (including Apache). > diff --git a/http-backend.c b/http-backend.c > index 8870a26..94976df 100644 > --- a/http-backend.c > +++ b/http-backend.c > @@ -277,16 +277,32 @@ static struct rpc_service *select_service(const char > *name) > */ > static ssize_t read_request(int fd, unsigned char **out) > { > - size_t len = 0, alloc = 8192; > - unsigned char *buf = xmalloc(alloc); > + unsigned char *buf = null; > + size_t len = 0; > + /* get request size */ > + size_t req_len = git_env_ulong("CONTENT_LENGTH", > +0); > + > + /* check request size */ > + if (max_request_buffer < req_len) { > + die("request was larger than our maximum size (%lu);" > + " try setting GIT_HTTP_MAX_REQUEST_BUFFER", > + max_request_buffer); > + } > + > + if (req_len <= 0) { > + *out = null; > + return 0; > + } git-am complained that your patch did not apply, but after writing something similar locally, I found that t5551.25 hangs indefinitely. Which is not surprising. Most tests are doing very limited ref negotiation, so the content that hits read_request() here is small, and we send it in a single write with a content-length header. But t5551.25 uses a much bigger workload, which causes the client to use a chunked transfer-encoding, and this code to refuse to read anything (and then the protocol stalls, as we are waiting for the client to say something). So I think you'd want to take a missing CONTENT_LENGTH as a hint to read until EOF. That also raises another issue: what happens in the paths that don't hit read_request()? We may also process input via: - inflate_request(), if the client gzipped it; for well-formed input, I think we'll stop reading when the gzip stream tells us there is no more data, but a malformed one would have us reading until EOF, regardless of what $CONTENT_LENGTH says. - for input which we expect to be large (like incoming packfiles for a push), buffer_input will be unset, and we will pass the descriptor directly to a sub-program like git-index-pack. Again, for well-formed input it would read just the packfile, but it may actually continue to EOF. So I don't think your patch is covering all cases. -Peff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Fix http-backend reading till EOF, ignoring CONTENT_LENGTH, violating rfc3875 -- WAS: Problem with git-http-backend.exe as iis cgi
-Ursprüngliche Nachricht- Von: Chris Packham [mailto:judge.pack...@gmail.com] Gesendet: Dienstag, 29. März 2016 11:28 An: Florian Manschwetus Cc: Konstantin Khomoutov; git@vger.kernel.org Betreff: Re: Problem with git-http-backend.exe as iis cgi Hi Florian On Tue, Mar 29, 2016 at 7:01 PM, Florian Manschwetus <manschwe...@cs-software-gmbh.de> wrote: > Hi, > I put together a first patch for the issue. > > Mit freundlichen Grüßen / With kind regards Florian Manschwetus > > E-Mail: manschwe...@cs-software-gmbh.de > Tel.: +49-(0)611-8908534 > > CS Software Concepts and Solutions GmbH Geschäftsführer / Managing > director: Dr. Werner Alexi Amtsgericht Wiesbaden HRB 10004 (Commercial > registry) Schiersteiner Straße 31 > D-65187 Wiesbaden > Germany > Tel.: 0611/8908555 > > > -Ursprüngliche Nachricht- > Von: Konstantin Khomoutov [mailto:kostix+...@007spb.ru] > Gesendet: Donnerstag, 10. März 2016 13:55 > An: Florian Manschwetus > Cc: git@vger.kernel.org > Betreff: Re: Problem with git-http-backend.exe as iis cgi > > On Thu, 10 Mar 2016 07:28:50 + > Florian Manschwetus <manschwe...@cs-software-gmbh.de> wrote: > >> I tried to setup git-http-backend with iis, as iis provides proper >> impersonation for cgi under windows, which leads to have the >> filesystem access performed with the logon user, therefore the >> webserver doesn't need generic access to the files. I stumbled across >> a problem, ending up with post requests hanging forever. After some >> investigation I managed to get it work by wrapping the http-backend >> into a bash script, giving a lot of control about the environmental >> things, I was unable to solve within IIS configuration. The >> workaround, I use currently, is to use "/bin/head -c >> ${CONTENT_LENGTH} >> | ./git-http-backend.exe", which directly shows the issue. Git >> http-backend should check if CONTENT_LENGTH is set to something >> reasonable (e.g. >0) and should in this case read only CONTENT_LENGTH >> bytes from stdin, instead of reading till EOF what I suspect it is >> doing currently. > > The rfc [1] states in its section 4.2: > > | A request-body is supplied with the request if the CONTENT_LENGTH is > | not NULL. The server MUST make at least that many bytes available > | for the script to read. The server MAY signal an end-of-file > | condition after CONTENT_LENGTH bytes have been read or it MAY supply > | extension data. Therefore, the script MUST NOT attempt to read more > | than CONTENT_LENGTH bytes, even if more data is available. However, > | it is not obliged to read any of the data. > > So yes, if Git currently reads until EOF, it's an error. > The correct way would be: > > 1) Check to see if the CONTENT_LENGTH variable is available in the >environment. If no, read nothing. > > 2) Otherwise read as many bytes it specifies, and no more. > > 1. https://www.ietf.org/rfc/rfc3875 Your patch description seems well thought out but if you want someone to notice it you should have a read of https://git.kernel.org/cgit/git/git.git/tree/Documentation/SubmittingPatches Moin, I have cloned git and created a more clean patch... Signed-off-by: Florian Manschwetus <manschwe...@cs-software-gmbh.de> --- http-backend.c | 48 +++- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/http-backend.c b/http-backend.c index 8870a26..94976df 100644 --- a/http-backend.c +++ b/http-backend.c @@ -277,16 +277,32 @@ static struct rpc_service *select_service(const char *name) */ static ssize_t read_request(int fd, unsigned char **out) { - size_t len = 0, alloc = 8192; - unsigned char *buf = xmalloc(alloc); + unsigned char *buf = null; + size_t len = 0; + /* get request size */ + size_t req_len = git_env_ulong("CONTENT_LENGTH", + 0); + + /* check request size */ + if (max_request_buffer < req_len) { + die("request was larger than our maximum size (%lu);" + " try setting GIT_HTTP_MAX_REQUEST_BUFFER", + max_request_buffer); + } + + if (req_len <= 0) { + *out = null; + return 0; + } + + /* allocate buffer */ + buf = xmalloc(req_len) - if (max_request_buffer < alloc) - max_request_buffer = alloc; while (1) { ssize_t cnt; - cnt = read_in_full(fd, buf + len, alloc - len); + cnt = read_in_full(fd, buf + len, req_len - len); if (cnt < 0) { free(buf);
Re: Problem with git-http-backend.exe as iis cgi
Hi Florian On Tue, Mar 29, 2016 at 7:01 PM, Florian Manschwetus <manschwe...@cs-software-gmbh.de> wrote: > Hi, > I put together a first patch for the issue. > > Mit freundlichen Grüßen / With kind regards > Florian Manschwetus > > E-Mail: manschwe...@cs-software-gmbh.de > Tel.: +49-(0)611-8908534 > > CS Software Concepts and Solutions GmbH > Geschäftsführer / Managing director: Dr. Werner Alexi > Amtsgericht Wiesbaden HRB 10004 (Commercial registry) > Schiersteiner Straße 31 > D-65187 Wiesbaden > Germany > Tel.: 0611/8908555 > > > -Ursprüngliche Nachricht- > Von: Konstantin Khomoutov [mailto:kostix+...@007spb.ru] > Gesendet: Donnerstag, 10. März 2016 13:55 > An: Florian Manschwetus > Cc: git@vger.kernel.org > Betreff: Re: Problem with git-http-backend.exe as iis cgi > > On Thu, 10 Mar 2016 07:28:50 + > Florian Manschwetus <manschwe...@cs-software-gmbh.de> wrote: > >> I tried to setup git-http-backend with iis, as iis provides proper >> impersonation for cgi under windows, which leads to have the >> filesystem access performed with the logon user, therefore the >> webserver doesn't need generic access to the files. I stumbled across >> a problem, ending up with post requests hanging forever. After some >> investigation I managed to get it work by wrapping the http-backend >> into a bash script, giving a lot of control about the environmental >> things, I was unable to solve within IIS configuration. The >> workaround, I use currently, is to use "/bin/head -c ${CONTENT_LENGTH} >> | ./git-http-backend.exe", which directly shows the issue. Git >> http-backend should check if CONTENT_LENGTH is set to something >> reasonable (e.g. >0) and should in this case read only CONTENT_LENGTH >> bytes from stdin, instead of reading till EOF what I suspect it is >> doing currently. > > The rfc [1] states in its section 4.2: > > | A request-body is supplied with the request if the CONTENT_LENGTH is > | not NULL. The server MUST make at least that many bytes available for > | the script to read. The server MAY signal an end-of-file condition > | after CONTENT_LENGTH bytes have been read or it MAY supply extension > | data. Therefore, the script MUST NOT attempt to read more than > | CONTENT_LENGTH bytes, even if more data is available. However, it is > | not obliged to read any of the data. > > So yes, if Git currently reads until EOF, it's an error. > The correct way would be: > > 1) Check to see if the CONTENT_LENGTH variable is available in the >environment. If no, read nothing. > > 2) Otherwise read as many bytes it specifies, and no more. > > 1. https://www.ietf.org/rfc/rfc3875 Your patch description seems well thought out but if you want someone to notice it you should have a read of https://git.kernel.org/cgit/git/git.git/tree/Documentation/SubmittingPatches -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: Problem with git-http-backend.exe as iis cgi
Hi, I put together a first patch for the issue. Mit freundlichen Grüßen / With kind regards Florian Manschwetus E-Mail: manschwe...@cs-software-gmbh.de Tel.: +49-(0)611-8908534 CS Software Concepts and Solutions GmbH Geschäftsführer / Managing director: Dr. Werner Alexi Amtsgericht Wiesbaden HRB 10004 (Commercial registry) Schiersteiner Straße 31 D-65187 Wiesbaden Germany Tel.: 0611/8908555 -Ursprüngliche Nachricht- Von: Konstantin Khomoutov [mailto:kostix+...@007spb.ru] Gesendet: Donnerstag, 10. März 2016 13:55 An: Florian Manschwetus Cc: git@vger.kernel.org Betreff: Re: Problem with git-http-backend.exe as iis cgi On Thu, 10 Mar 2016 07:28:50 + Florian Manschwetus <manschwe...@cs-software-gmbh.de> wrote: > I tried to setup git-http-backend with iis, as iis provides proper > impersonation for cgi under windows, which leads to have the > filesystem access performed with the logon user, therefore the > webserver doesn't need generic access to the files. I stumbled across > a problem, ending up with post requests hanging forever. After some > investigation I managed to get it work by wrapping the http-backend > into a bash script, giving a lot of control about the environmental > things, I was unable to solve within IIS configuration. The > workaround, I use currently, is to use "/bin/head -c ${CONTENT_LENGTH} > | ./git-http-backend.exe", which directly shows the issue. Git > http-backend should check if CONTENT_LENGTH is set to something > reasonable (e.g. >0) and should in this case read only CONTENT_LENGTH > bytes from stdin, instead of reading till EOF what I suspect it is > doing currently. The rfc [1] states in its section 4.2: | A request-body is supplied with the request if the CONTENT_LENGTH is | not NULL. The server MUST make at least that many bytes available for | the script to read. The server MAY signal an end-of-file condition | after CONTENT_LENGTH bytes have been read or it MAY supply extension | data. Therefore, the script MUST NOT attempt to read more than | CONTENT_LENGTH bytes, even if more data is available. However, it is | not obliged to read any of the data. So yes, if Git currently reads until EOF, it's an error. The correct way would be: 1) Check to see if the CONTENT_LENGTH variable is available in the environment. If no, read nothing. 2) Otherwise read as many bytes it specifies, and no more. 1. https://www.ietf.org/rfc/rfc3875 http-backend-content-length.patch Description: http-backend-content-length.patch