Re: HTTP and HTTP/1.x separation

[Stefan Eissing]

just a heads up: I will replace this PR with a series of smaller ones. 

The accumulated changes are too large to make someone review and have
an opinion about it in reasonable time. I will therefore introduce the
changes in small steps, trying to make them each easier to read.

I have the first one sitting here locally and will publish that once
our CI is working again (and I can be sure that they do as well).

Kind Regards,
Stefan


> Am 31.03.2022 um 09:40 schrieb jean-frederic clere :
> 
> On 30/03/2022 11:11, Stefan Eissing wrote:
>>> Am 28.03.2022 um 15:52 schrieb jean-frederic clere :
>>> 
>>> On 24/03/2022 13:21, Stefan Eissing wrote:
 You are invited to have a look at my PR for separating HTTP/1.x processing 
 from
 generic HTTP protocol handling and verification:
 https://github.com/apache/httpd/pull/291
 I made a description of the changes in the PR that helps reviewing it (I 
 hope).
 "Changes appear larger than they really are"
 A lot is code split+move from mod_http to mod_http1. In mod_http2, changes 
 are
 mainly removals of quirks necessary so far.
 Kind Regards,
 Stefan
>>> 
>>> Something fishy:
>>> http/1.1:
>>> +++
>>> 
>>> < HTTP/1.1 200 OK
>>> < Date: Mon, 28 Mar 2022 13:48:23 GMT
>>> < Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
>>> < Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
>>> < ETag: "bf-5db0ce1e1e93e"
>>> < Accept-Ranges: bytes
>>> < Content-Length: 191
>>> < Content-Type: text/html
>>> 
>>> +++
>>> http/2:
>>> +++
>>> < HTTP/2 200
>>> < last-modified: Fri, 25 Mar 2022 15:47:39 GMT
>>> < etag: "bf-5db0ce1e1e93e"
>>> < accept-ranges: bytes
>>> < content-length: 191
>>> < content-type: text/html
>>> +++
>>> 
>>> Did I miss something?
>> Just added the fix to the PR:
>> *) core, mod_http1, mod_http: moved the handling of the standard
>> response headers `Date` and `Server` from mod_http1 into the
>> generic HTTP protocol handling.
>> Response buckets not always carry those headers (values preserved
>> from proxied responses), irregardless of the HTTP protocol
>> versions involved.
>> mod_http1: the serialization of response header into HTTP/1.x
>> format always writes `Date` and `Server` first if present. This
>> assured backward compatibility with clients who are accustomed
>> to this order.
> 
> Thanks my tests are passing now.
> 
>> Kind Regards,
>> Stefan
>>> 
>>> -- 
>>> Cheers
>>> 
>>> Jean-Frederic
>>> 
> 
> 
> -- 
> Cheers
> 
> Jean-Frederic

Re: HTTP and HTTP/1.x separation

[jean-frederic clere]

On 30/03/2022 11:11, Stefan Eissing wrote:




Am 28.03.2022 um 15:52 schrieb jean-frederic clere :

On 24/03/2022 13:21, Stefan Eissing wrote:

You are invited to have a look at my PR for separating HTTP/1.x processing from
generic HTTP protocol handling and verification:
https://github.com/apache/httpd/pull/291
I made a description of the changes in the PR that helps reviewing it (I hope).
"Changes appear larger than they really are"
A lot is code split+move from mod_http to mod_http1. In mod_http2, changes are
mainly removals of quirks necessary so far.
Kind Regards,
Stefan


Something fishy:
http/1.1:
+++

< HTTP/1.1 200 OK
< Date: Mon, 28 Mar 2022 13:48:23 GMT
< Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
< Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
< ETag: "bf-5db0ce1e1e93e"
< Accept-Ranges: bytes
< Content-Length: 191
< Content-Type: text/html

+++
http/2:
+++
< HTTP/2 200
< last-modified: Fri, 25 Mar 2022 15:47:39 GMT
< etag: "bf-5db0ce1e1e93e"
< accept-ranges: bytes
< content-length: 191
< content-type: text/html
+++

Did I miss something?


Just added the fix to the PR:

   *) core, mod_http1, mod_http: moved the handling of the standard
  response headers `Date` and `Server` from mod_http1 into the
  generic HTTP protocol handling.
  Response buckets not always carry those headers (values preserved
  from proxied responses), irregardless of the HTTP protocol
  versions involved.
  mod_http1: the serialization of response header into HTTP/1.x
  format always writes `Date` and `Server` first if present. This
  assured backward compatibility with clients who are accustomed
  to this order.


Thanks my tests are passing now.



Kind Regards,

Stefan



--
Cheers

Jean-Frederic






--
Cheers

Jean-Frederic

Re: HTTP and HTTP/1.x separation

[Stefan Eissing]

> Am 28.03.2022 um 15:52 schrieb jean-frederic clere :
> 
> On 24/03/2022 13:21, Stefan Eissing wrote:
>> You are invited to have a look at my PR for separating HTTP/1.x processing 
>> from
>> generic HTTP protocol handling and verification:
>> https://github.com/apache/httpd/pull/291
>> I made a description of the changes in the PR that helps reviewing it (I 
>> hope).
>> "Changes appear larger than they really are"
>> A lot is code split+move from mod_http to mod_http1. In mod_http2, changes 
>> are
>> mainly removals of quirks necessary so far.
>> Kind Regards,
>> Stefan
> 
> Something fishy:
> http/1.1:
> +++
> 
> < HTTP/1.1 200 OK
> < Date: Mon, 28 Mar 2022 13:48:23 GMT
> < Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
> < Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
> < ETag: "bf-5db0ce1e1e93e"
> < Accept-Ranges: bytes
> < Content-Length: 191
> < Content-Type: text/html
> 
> +++
> http/2:
> +++
> < HTTP/2 200
> < last-modified: Fri, 25 Mar 2022 15:47:39 GMT
> < etag: "bf-5db0ce1e1e93e"
> < accept-ranges: bytes
> < content-length: 191
> < content-type: text/html
> +++
> 
> Did I miss something?

Just added the fix to the PR:

  *) core, mod_http1, mod_http: moved the handling of the standard
 response headers `Date` and `Server` from mod_http1 into the
 generic HTTP protocol handling.
 Response buckets not always carry those headers (values preserved
 from proxied responses), irregardless of the HTTP protocol
 versions involved.
 mod_http1: the serialization of response header into HTTP/1.x
 format always writes `Date` and `Server` first if present. This
 assured backward compatibility with clients who are accustomed
 to this order.

Kind Regards,

Stefan

> 
> -- 
> Cheers
> 
> Jean-Frederic
> 

Re: HTTP and HTTP/1.x separation

[jean-frederic clere]

On 28/03/2022 17:25, Stefan Eissing wrote:

Correct. When I first tried, the perl http2 framework was not very mature


Yep the perl http2 module looks abandoned :-(

--
Cheers

Jean-Frederic

Re: HTTP and HTTP/1.x separation

[Stefan Eissing]

> Am 28.03.2022 um 17:15 schrieb jean-frederic clere :
> 
> On 28/03/2022 16:03, Stefan Eissing wrote:
>>> Am 28.03.2022 um 15:52 schrieb jean-frederic clere :
>>> 
>>> On 24/03/2022 13:21, Stefan Eissing wrote:
 You are invited to have a look at my PR for separating HTTP/1.x processing 
 from
 generic HTTP protocol handling and verification:
 https://github.com/apache/httpd/pull/291
 I made a description of the changes in the PR that helps reviewing it (I 
 hope).
 "Changes appear larger than they really are"
 A lot is code split+move from mod_http to mod_http1. In mod_http2, changes 
 are
 mainly removals of quirks necessary so far.
 Kind Regards,
 Stefan
>>> 
>>> Something fishy:
>>> http/1.1:
>>> +++
>>> 
>>> < HTTP/1.1 200 OK
>>> < Date: Mon, 28 Mar 2022 13:48:23 GMT
>>> < Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
>>> < Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
>>> < ETag: "bf-5db0ce1e1e93e"
>>> < Accept-Ranges: bytes
>>> < Content-Length: 191
>>> < Content-Type: text/html
>>> 
>>> +++
>>> http/2:
>>> +++
>>> < HTTP/2 200
>>> < last-modified: Fri, 25 Mar 2022 15:47:39 GMT
>>> < etag: "bf-5db0ce1e1e93e"
>>> < accept-ranges: bytes
>>> < content-length: 191
>>> < content-type: text/html
>>> +++
>>> 
>>> Did I miss something?
>> No, you found something. The generic server headers are not applied, it 
>> seems. Will look into this and add a test.
>>> -- 
>>> Cheers
>>> 
>>> Jean-Frederic
>>> 
> 
> I think we have NO http/2 tests in the httpd-framework test, correct?

Correct. When I first tried, the perl http2 framework was not very mature and I 
turned to what ultimately became the pytest suite.

> 
> -- 
> Cheers
> 
> Jean-Frederic
> 

Re: HTTP and HTTP/1.x separation

[jean-frederic clere]

On 28/03/2022 16:03, Stefan Eissing wrote:




Am 28.03.2022 um 15:52 schrieb jean-frederic clere :

On 24/03/2022 13:21, Stefan Eissing wrote:

You are invited to have a look at my PR for separating HTTP/1.x processing from
generic HTTP protocol handling and verification:
https://github.com/apache/httpd/pull/291
I made a description of the changes in the PR that helps reviewing it (I hope).
"Changes appear larger than they really are"
A lot is code split+move from mod_http to mod_http1. In mod_http2, changes are
mainly removals of quirks necessary so far.
Kind Regards,
Stefan


Something fishy:
http/1.1:
+++

< HTTP/1.1 200 OK
< Date: Mon, 28 Mar 2022 13:48:23 GMT
< Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
< Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
< ETag: "bf-5db0ce1e1e93e"
< Accept-Ranges: bytes
< Content-Length: 191
< Content-Type: text/html

+++
http/2:
+++
< HTTP/2 200
< last-modified: Fri, 25 Mar 2022 15:47:39 GMT
< etag: "bf-5db0ce1e1e93e"
< accept-ranges: bytes
< content-length: 191
< content-type: text/html
+++

Did I miss something?


No, you found something. The generic server headers are not applied, it seems. 
Will look into this and add a test.


--
Cheers

Jean-Frederic





I think we have NO http/2 tests in the httpd-framework test, correct?

--
Cheers

Jean-Frederic

Re: HTTP and HTTP/1.x separation

[Stefan Eissing]

> Am 28.03.2022 um 15:52 schrieb jean-frederic clere :
> 
> On 24/03/2022 13:21, Stefan Eissing wrote:
>> You are invited to have a look at my PR for separating HTTP/1.x processing 
>> from
>> generic HTTP protocol handling and verification:
>> https://github.com/apache/httpd/pull/291
>> I made a description of the changes in the PR that helps reviewing it (I 
>> hope).
>> "Changes appear larger than they really are"
>> A lot is code split+move from mod_http to mod_http1. In mod_http2, changes 
>> are
>> mainly removals of quirks necessary so far.
>> Kind Regards,
>> Stefan
> 
> Something fishy:
> http/1.1:
> +++
> 
> < HTTP/1.1 200 OK
> < Date: Mon, 28 Mar 2022 13:48:23 GMT
> < Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
> < Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
> < ETag: "bf-5db0ce1e1e93e"
> < Accept-Ranges: bytes
> < Content-Length: 191
> < Content-Type: text/html
> 
> +++
> http/2:
> +++
> < HTTP/2 200
> < last-modified: Fri, 25 Mar 2022 15:47:39 GMT
> < etag: "bf-5db0ce1e1e93e"
> < accept-ranges: bytes
> < content-length: 191
> < content-type: text/html
> +++
> 
> Did I miss something?

No, you found something. The generic server headers are not applied, it seems. 
Will look into this and add a test.

> -- 
> Cheers
> 
> Jean-Frederic
> 

Re: HTTP and HTTP/1.x separation

[jean-frederic clere]

On 24/03/2022 13:21, Stefan Eissing wrote:

You are invited to have a look at my PR for separating HTTP/1.x processing from
generic HTTP protocol handling and verification:

https://github.com/apache/httpd/pull/291

I made a description of the changes in the PR that helps reviewing it (I hope).

"Changes appear larger than they really are"

A lot is code split+move from mod_http to mod_http1. In mod_http2, changes are
mainly removals of quirks necessary so far.


Kind Regards,
Stefan



Something fishy:
http/1.1:
+++

< HTTP/1.1 200 OK
< Date: Mon, 28 Mar 2022 13:48:23 GMT
< Server: Apache/2.5.1-dev (Unix) OpenSSL/1.1.1n
< Last-Modified: Fri, 25 Mar 2022 15:47:39 GMT
< ETag: "bf-5db0ce1e1e93e"
< Accept-Ranges: bytes
< Content-Length: 191
< Content-Type: text/html

+++
http/2:
+++
< HTTP/2 200
< last-modified: Fri, 25 Mar 2022 15:47:39 GMT
< etag: "bf-5db0ce1e1e93e"
< accept-ranges: bytes
< content-length: 191
< content-type: text/html
+++

Did I miss something?

--
Cheers

Jean-Frederic

HTTP and HTTP/1.x separation

[Stefan Eissing]

You are invited to have a look at my PR for separating HTTP/1.x processing from
generic HTTP protocol handling and verification:

https://github.com/apache/httpd/pull/291

I made a description of the changes in the PR that helps reviewing it (I hope).

"Changes appear larger than they really are"

A lot is code split+move from mod_http to mod_http1. In mod_http2, changes are
mainly removals of quirks necessary so far.


Kind Regards,
Stefan

HTTP and HTTP/1.x separation

[Stefan Eissing]

My PR regarding separation of HTTP/1.1 and generic HTTP filters:

https://github.com/apache/httpd/pull/291

There is more to be done, but I think this is a good start.

Kind Regards,
Stefan

Re: http and http/1.x separation

[Graham Leggett]

On 24 Jan 2022, at 16:53, Stefan Eissing  wrote:

> Yes, it's 0 length like ERROR and has a struct as that holds:
> - int status, opt 0 
> - const char *reason, opt NULL
> - apr_table_t *headers
> - apr_table_t *notes
> 
> on responses, shallow copied from the request_rec, etc.
> 
> The nice thing is that the HTTP/1.x transcode out filter can
> now check in one place that
> - there is indeed a HEADERS before data buckets
> - there is only one HEADERS with status >= 200
> 
> And "above" the transcode layer, a data bucket is content, period.

I have a module that parses multipart/mime headers and puts them into a 
metadata bucket, that has a lot of similarity to what you’re after:

https://github.com/minfrin/mod_multipart/blob/main/mod_multipart.c#L514

Some of the functions above would be useful in httpd.

Regards,
Graham
—

Re: http and http/1.x separation

[Stefan Eissing]

> Am 25.01.2022 um 17:42 schrieb Roy T. Fielding :
> 
>> On Jan 24, 2022, at 6:53 AM, Stefan Eissing  wrote:
>> 
>> 
>> 
>>> Am 24.01.2022 um 15:40 schrieb Yann Ylavic :
>>> 
>>> On Mon, Jan 24, 2022 at 3:28 PM Stefan Eissing  wrote:
 
 FYI: I am busy hacking away at the separation between our HTTP and 
 HTTP/1.x handling code. I'll make a PR once all tests have passed, so we 
 can talk about the changes.
>>> 
>>> Looking forward to seeing it ;)
>> 
>> 
>> 
>>> 
 
 The goals are pretty simple:
 
 1. have mod_http focus on HTTP semantics, header checks, ap_die 
 conversions,
  not chunking and CRLFs.
 2. have a mod_http1 that installs TRANSCODE filters when the connection is 
 h1.
  those do the chunking and the header serialization.
 3. Have interim responses that send META buckets, not DATA to the 
 transcode filters.
  No content filter will be confused by those.
 4. Pass FOOTERS in/out as META buckets in a similar way.
 
 The main idea is to introduce a META "HEADERS" bucket that is used for
 final/interim responses and footers as well. This will safely pass through
 all filters that do not know about it. This is similar to the ERROR bucket
 type we already have.
>>> 
>>> There are some assumptions in httpd (and possibly third-party modules)
>>> that meta buckets have a no length, so if "HEADERS" buckets have one
>>> you might need to embed it in the bucket->data struct..
>> 
>> Yes, it's 0 length like ERROR and has a struct as that holds:
>> - int status, opt 0 
>> - const char *reason, opt NULL
>> - apr_table_t *headers
>> - apr_table_t *notes
>> 
>> on responses, shallow copied from the request_rec, etc.
>> 
>> The nice thing is that the HTTP/1.x transcode out filter can
>> now check in one place that
>> - there is indeed a HEADERS before data buckets
>> - there is only one HEADERS with status >= 200
>> 
>> And "above" the transcode layer, a data bucket is content, period.
> 
> And trailers can be implemented this way too, right?

Yes, exactly. On h2 always, on h1 when the response is chunked. Otherwise the 
footers HEADERS bucket will we a NOP in h1 transcoding.

> Roy

Re: http and http/1.x separation

[Roy T. Fielding]

> On Jan 24, 2022, at 6:53 AM, Stefan Eissing  wrote:
> 
> 
> 
>> Am 24.01.2022 um 15:40 schrieb Yann Ylavic > >:
>> 
>> On Mon, Jan 24, 2022 at 3:28 PM Stefan Eissing > > wrote:
>>> 
>>> FYI: I am busy hacking away at the separation between our HTTP and HTTP/1.x 
>>> handling code. I'll make a PR once all tests have passed, so we can talk 
>>> about the changes.
>> 
>> Looking forward to seeing it ;)
> 
> 
> 
>> 
>>> 
>>> The goals are pretty simple:
>>> 
>>> 1. have mod_http focus on HTTP semantics, header checks, ap_die conversions,
>>>  not chunking and CRLFs.
>>> 2. have a mod_http1 that installs TRANSCODE filters when the connection is 
>>> h1.
>>>  those do the chunking and the header serialization.
>>> 3. Have interim responses that send META buckets, not DATA to the transcode 
>>> filters.
>>>  No content filter will be confused by those.
>>> 4. Pass FOOTERS in/out as META buckets in a similar way.
>>> 
>>> The main idea is to introduce a META "HEADERS" bucket that is used for
>>> final/interim responses and footers as well. This will safely pass through
>>> all filters that do not know about it. This is similar to the ERROR bucket
>>> type we already have.
>> 
>> There are some assumptions in httpd (and possibly third-party modules)
>> that meta buckets have a no length, so if "HEADERS" buckets have one
>> you might need to embed it in the bucket->data struct..
> 
> Yes, it's 0 length like ERROR and has a struct as that holds:
> - int status, opt 0 
> - const char *reason, opt NULL
> - apr_table_t *headers
> - apr_table_t *notes
> 
> on responses, shallow copied from the request_rec, etc.
> 
> The nice thing is that the HTTP/1.x transcode out filter can
> now check in one place that
> - there is indeed a HEADERS before data buckets
> - there is only one HEADERS with status >= 200
> 
> And "above" the transcode layer, a data bucket is content, period.

And trailers can be implemented this way too, right?

Roy

Re: http and http/1.x separation

[Stefan Eissing]

> Am 24.01.2022 um 15:40 schrieb Yann Ylavic :
> 
> On Mon, Jan 24, 2022 at 3:28 PM Stefan Eissing  wrote:
>> 
>> FYI: I am busy hacking away at the separation between our HTTP and HTTP/1.x 
>> handling code. I'll make a PR once all tests have passed, so we can talk 
>> about the changes.
> 
> Looking forward to seeing it ;)



> 
>> 
>> The goals are pretty simple:
>> 
>> 1. have mod_http focus on HTTP semantics, header checks, ap_die conversions,
>>   not chunking and CRLFs.
>> 2. have a mod_http1 that installs TRANSCODE filters when the connection is 
>> h1.
>>   those do the chunking and the header serialization.
>> 3. Have interim responses that send META buckets, not DATA to the transcode 
>> filters.
>>   No content filter will be confused by those.
>> 4. Pass FOOTERS in/out as META buckets in a similar way.
>> 
>> The main idea is to introduce a META "HEADERS" bucket that is used for
>> final/interim responses and footers as well. This will safely pass through
>> all filters that do not know about it. This is similar to the ERROR bucket
>> type we already have.
> 
> There are some assumptions in httpd (and possibly third-party modules)
> that meta buckets have a no length, so if "HEADERS" buckets have one
> you might need to embed it in the bucket->data struct..

Yes, it's 0 length like ERROR and has a struct as that holds:
- int status, opt 0 
- const char *reason, opt NULL
- apr_table_t *headers
- apr_table_t *notes

on responses, shallow copied from the request_rec, etc.

The nice thing is that the HTTP/1.x transcode out filter can
now check in one place that
- there is indeed a HEADERS before data buckets
- there is only one HEADERS with status >= 200

And "above" the transcode layer, a data bucket is content, period.

> 
> 
> Cheers;
> Yann.

Re: http and http/1.x separation

[Yann Ylavic]

On Mon, Jan 24, 2022 at 3:28 PM Stefan Eissing  wrote:
>
> FYI: I am busy hacking away at the separation between our HTTP and HTTP/1.x 
> handling code. I'll make a PR once all tests have passed, so we can talk 
> about the changes.

Looking forward to seeing it ;)

>
> The goals are pretty simple:
>
> 1. have mod_http focus on HTTP semantics, header checks, ap_die conversions,
>not chunking and CRLFs.
> 2. have a mod_http1 that installs TRANSCODE filters when the connection is h1.
>those do the chunking and the header serialization.
> 3. Have interim responses that send META buckets, not DATA to the transcode 
> filters.
>No content filter will be confused by those.
> 4. Pass FOOTERS in/out as META buckets in a similar way.
>
> The main idea is to introduce a META "HEADERS" bucket that is used for
> final/interim responses and footers as well. This will safely pass through
> all filters that do not know about it. This is similar to the ERROR bucket
> type we already have.

There are some assumptions in httpd (and possibly third-party modules)
that meta buckets have a no length, so if "HEADERS" buckets have one
you might need to embed it in the bucket->data struct..


Cheers;
Yann.

Re: http and http/1.x separation

[Graham Leggett]

On 24 Jan 2022, at 16:28, Stefan Eissing  wrote:

> The main idea is to introduce a META "HEADERS" bucket that is used for
> final/interim responses and footers as well. This will safely pass through
> all filters that do not know about it. This is similar to the ERROR bucket 
> type we already have.

Big +1.

We don’t use metadata buckets enough, and life is made significantly easier 
when used.

Regards,
Graham
—

http and http/1.x separation

[Stefan Eissing]

FYI: I am busy hacking away at the separation between our HTTP and HTTP/1.x 
handling code. I'll make a PR once all tests have passed, so we can talk about 
the changes. 

The goals are pretty simple:

1. have mod_http focus on HTTP semantics, header checks, ap_die conversions,
   not chunking and CRLFs.
2. have a mod_http1 that installs TRANSCODE filters when the connection is h1.
   those do the chunking and the header serialization.
3. Have interim responses that send META buckets, not DATA to the transcode 
filters.
   No content filter will be confused by those.
4. Pass FOOTERS in/out as META buckets in a similar way.

The main idea is to introduce a META "HEADERS" bucket that is used for
final/interim responses and footers as well. This will safely pass through
all filters that do not know about it. This is similar to the ERROR bucket 
type we already have.

It will eliminate the remaining code duplications in mod_http2, make
it leaner and use the standard HTTP filters just like for HTTP/1.x requests.
Also, h2 will no longer need to parse interim responses back into header tables
or deal with chunked encodings.

Kind Regards,
Stefan