Repository: spark
Updated Branches:
refs/heads/master c34c42234 -> ab61ddb34
[SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP
requestHeaderSize
## What changes were proposed in this pull request?
Introducing spark.ui.requestHeaderSize for configuring Jetty's HTTP
requestHeaderSize.
This way long authorization field does not lead to HTTP 413.
## How was this patch tested?
Manually with curl (which version must be at least 7.55).
With the original default value (8k limit):
```bash
# Starting history server with default requestHeaderSize
$ ./sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# Creating huge header
$ echo -n "X-Custom-Header: " > cookie
$ printf 'A%.0s' {1..9500} >> cookie
# HTTP GET with huge header fails with 431
$ curl -H cookie http://458apiros-MBP.lan:18080/
<h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre>
# The log contains the error
$ tail -1
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
18/11/19 21:24:28 WARN HttpParser: Header is too large 8193>8192
```
After:
```bash
# Creating the history properties file with the increased requestHeaderSize
$ echo spark.ui.requestHeaderSize=10000 > history.properties
# Starting Spark History Server with the settings
$ ./sbin/start-history-server.sh --properties-file history.properties
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/attilapiros/github/spark/logs/spark-attilapiros-org.apache.spark.deploy.history.HistoryServer-1-apiros-MBP.lan.out
# HTTP GET with huge header gives back HTML5 (I have added here only just a
part of the response)
$ curl -H cookie http://458apiros-MBP.lan:18080/
<!DOCTYPE html><html>
<head>...
<link rel="shortcut icon"
href="/static/spark-logo-77x50px-hd.png"></link>
<title>History Server</title>
</head>
<body>
...
```
Closes #23090 from attilapiros/JettyHeaderSize.
Authored-by: âattilapirosâ <piros.attila.zs...@gmail.com>
Signed-off-by: Imran Rashid <iras...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab61ddb3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ab61ddb3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ab61ddb3
Branch: refs/heads/master
Commit: ab61ddb34d58ab5701191c8fd3a24a62f6ebf37b
Parents: c34c422
Author: âattilapirosâ <piros.attila.zs...@gmail.com>
Authored: Tue Nov 20 08:56:22 2018 -0600
Committer: Imran Rashid <iras...@cloudera.com>
Committed: Tue Nov 20 08:56:22 2018 -0600
----------------------------------------------------------------------
.../scala/org/apache/spark/internal/config/package.scala | 6 ++++++
core/src/main/scala/org/apache/spark/ui/JettyUtils.scala | 6 ++++--
docs/configuration.md | 8 ++++++++
3 files changed, 18 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ab61ddb3/core/src/main/scala/org/apache/spark/internal/config/package.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index ab2b872..9cc48f6 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -570,6 +570,12 @@ package object config {
.stringConf
.createOptional
+ private[spark] val UI_REQUEST_HEADER_SIZE =
+ ConfigBuilder("spark.ui.requestHeaderSize")
+ .doc("Value for HTTP request header size in bytes.")
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefaultString("8k")
+
private[spark] val EXTRA_LISTENERS = ConfigBuilder("spark.extraListeners")
.doc("Class names of listeners to add to SparkContext during
initialization.")
.stringConf
http://git-wip-us.apache.org/repos/asf/spark/blob/ab61ddb3/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
index 52a9551..316af9b 100644
--- a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
@@ -356,13 +356,15 @@ private[spark] object JettyUtils extends Logging {
(connector, connector.getLocalPort())
}
+ val httpConfig = new HttpConfiguration()
+ httpConfig.setRequestHeaderSize(conf.get(UI_REQUEST_HEADER_SIZE).toInt)
// If SSL is configured, create the secure connector first.
val securePort = sslOptions.createJettySslContextFactory().map { factory
=>
val securePort = sslOptions.port.getOrElse(if (port > 0)
Utils.userPort(port, 400) else 0)
val secureServerName = if (serverName.nonEmpty) s"$serverName (HTTPS)"
else serverName
val connectionFactories =
AbstractConnectionFactory.getFactories(factory,
- new HttpConnectionFactory())
+ new HttpConnectionFactory(httpConfig))
def sslConnect(currentPort: Int): (ServerConnector, Int) = {
newConnector(connectionFactories, currentPort)
@@ -377,7 +379,7 @@ private[spark] object JettyUtils extends Logging {
// Bind the HTTP port.
def httpConnect(currentPort: Int): (ServerConnector, Int) = {
- newConnector(Array(new HttpConnectionFactory()), currentPort)
+ newConnector(Array(new HttpConnectionFactory(httpConfig)), currentPort)
}
val (httpConnector, httpPort) =
Utils.startServiceOnPort[ServerConnector](port, httpConnect,
http://git-wip-us.apache.org/repos/asf/spark/blob/ab61ddb3/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 2915fb5..04210d8 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -973,6 +973,14 @@ Apart from these, the following properties are also
available, and may be useful
<br /><code>spark.com.test.filter1.param.name2=bar</code>
</td>
</tr>
+<tr>
+ <td><code>spark.ui.requestHeaderSize</code></td>
+ <td>8k</td>
+ <td>
+ The maximum allowed size for a HTTP request header, in bytes unless
otherwise specified.
+ This setting applies for the Spark History Server too.
+ <td>
+</tr>
</table>
### Compression and Serialization
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
