[
https://issues.apache.org/jira/browse/SPARK-49149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yichuan Huang updated SPARK-49149:
----------------------------------
Description:
Spark provides two configs to alternate the log url on the live UI and history
server, `{{{}spark.ui.custom.executor.log.url{}}}` and
`{{{}spark.history.custom.executor.log.url{}}}`. The configs support path
variables which will be replaced at runtime, but currently this only works on
Yarn. Running on k8s with the path variable doesn't work. Here's an example
{code:java}
./bin/spark-shell --conf
spark.ui.custom.executor.log.url="URL_PREFIX?appId=APP_ID&execId=EXECUTOR_ID"{code}
The log column doesn't show up in the Spark UI, and Spark driver printing
{code:java}
24/08/07 17:23:45 INFO ExecutorLogUrlHandler: Fail to renew executor log urls:
some of required attributes are missing in app's event log.. Required:
Set(APP_ID, EXECUTOR_ID) / available: Set(). Falling back to s how app's
original log urls.{code}
the empty attribute `{{{}Set(){}}}` in the above log segment indicates the
feature not supported in k8s environment, thus creating this ticket to add the
support
was:
Spark provides two configs to alternate the log url on the live UI and history
server, `spark.ui.custom.executor.log.url` and
`spark.history.custom.executor.log.url`. The configs support path variables
which will be replaced at runtime, but currently this only works on Yarn.
Running on k8s with the path variable doesn't work. Here's an example
```
./bin/spark-shell --conf
spark.ui.custom.executor.log.url="URL_PREFIX?appId=\{{APP_ID}}&execId=\{{EXECUTOR_ID}}"
```
The log column doesn't show up in the Spark UI, and Spark driver printing
```
24/08/07 17:23:45 INFO ExecutorLogUrlHandler: Fail to renew executor log urls:
some of required attributes are missing in app's event log.. Required:
Set(APP_ID, EXECUTOR_ID) / available: Set(). Falling back to s how app's
original log urls.
```
the empty attribute `Set()` in the above log segment indicates the feature not
supported in k8s environment, thus creating this ticket to add the support
> Support customized log url for Spark UI and History server in Kubernetes
> environment
> ------------------------------------------------------------------------------------
>
> Key: SPARK-49149
> URL: https://issues.apache.org/jira/browse/SPARK-49149
> Project: Spark
> Issue Type: Improvement
> Components: UI
> Affects Versions: 3.4.0, 3.4.3
> Reporter: Yichuan Huang
> Priority: Major
> Fix For: 4.0.0, 3.5.3
>
>
> Spark provides two configs to alternate the log url on the live UI and
> history server, `{{{}spark.ui.custom.executor.log.url{}}}` and
> `{{{}spark.history.custom.executor.log.url{}}}`. The configs support path
> variables which will be replaced at runtime, but currently this only works on
> Yarn. Running on k8s with the path variable doesn't work. Here's an example
> {code:java}
> ./bin/spark-shell --conf
> spark.ui.custom.executor.log.url="URL_PREFIX?appId=APP_ID&execId=EXECUTOR_ID"{code}
> The log column doesn't show up in the Spark UI, and Spark driver printing
> {code:java}
> 24/08/07 17:23:45 INFO ExecutorLogUrlHandler: Fail to renew executor log
> urls: some of required attributes are missing in app's event log.. Required:
> Set(APP_ID, EXECUTOR_ID) / available: Set(). Falling back to s how app's
> original log urls.{code}
> the empty attribute `{{{}Set(){}}}` in the above log segment indicates the
> feature not supported in k8s environment, thus creating this ticket to add
> the support
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
