Hi community
I am starting a DISCUSS for "PIP-259: Make the config
httpMaxRequestHeaderSize of the pulsar web server configurable".
### Motivation
We have two ways to manage pulsar's resources:
- By client API (Can manage some resources, such as `create topic`, `create
subscriber`, and so on)
- By admin API (Can manage all the resources)
The `client API` has no limit on the request length. And the `admin API`
has a limit on the request length(such as HTTP request line and HTTP
request headers), this restriction is done by the built-in web container
Jetty.
Almost resources can be created by two APIs, but can only be modified and
deleted by `admin API`. This causes us to be unable to modify or delete
resources created by `client API` with too long a name because it exceeds
Jetty's default HTTP request URI length limit.
### Goal
#### 1. For web servers
Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration
(involves two servers: the web server in pulsar and the web server in
pulsar-proxy)
#### 2.For the internal client in pulsar-proxy
Provide a way to modify Jetty-client's `httpClientRequestBufferSize`
configuration.
Since the pulsar-proxy handles HTTP requests like this: `pulsar-admin.sh`
-> `proxy web server` -> `(highlight) internal client in proxy` -> `pulsar
web server`.
When the internal client forwards a request, it forwards the request header
and the request body, and all the data passes through a buffer( we call it
Buf ), like this:
- Receive a request
- Put the request line and request headers input to the Buf.
- <strong>(highlight)</strong>Flush the Buf ( If the data in the request
line and request header exceeds the length of the buf, an error is reported
)
- Put the request body input to the Buf.
- Flush the Buf if it is full.
So we need a config to set the `buff size` of the Buf:
`pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the
internal client`.
### API Changes
#### ServiceConfiguration.java
```java
@FieldContext(
category = CATEGORY_HTTP,
doc = """
The maximum size in bytes of the request header.
Larger headers will allow for more and/or larger cookies
plus larger form content encoded in a URL.
However, larger headers consume more memory and can make a
server more vulnerable to denial of service
attacks.
"""
)
private int httpMaxRequestHeaderSize = 8 * 1024;
```
#### ProxyConfiguration.java
```java
@FieldContext(
minValue = 1,
category = CATEGORY_HTTP,
doc = """
The maximum size in bytes of the request header.
Larger headers will allow for more and/or larger cookies
plus larger form content encoded in a URL.
However, larger headers consume more memory and can make a
server more vulnerable to denial of service
attacks.
"""
)
private int httpMaxRequestHeaderSize = 8 * 1024;
@FieldContext(
minValue = 1,
category = CATEGORY_HTTP,
doc = """
the size of the buffer used to write requests to Broker.
if "httpMaxRequestHeaderSize" is large than
"httpClientRequestBufferSize", will set
"httpClientRequestBufferSize" to the value of
"httpMaxRequestHeaderSize"
"""
)
private int httpClientRequestBufferSize = httpMaxRequestHeaderSize;
```
### Anything else?
This change should cherry-pick into the previous branches ( includes
`2.9~2.11` )
If the user uses the features `RETRY Topic` or `DLQ`, it is possible that
pulsar will automatically create some topics with names that are too long
and cannot be managed, the [scenario has been discussed in the email](
https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) before
Thanks
Yubiao Feng