[
https://issues.apache.org/jira/browse/SPARK-52327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gaurav Waghmare updated SPARK-52327:
------------------------------------
Description:
Currently, spark history server runs with one base directory immediate
subdirectories of which correspond to event logs for each application.
There are usecases for eg., multi tenancy where for the purpose of logical
separation, the event logs could be stored in separate directories at a tenant
level. To achieve this, instead of providing the path of the base directory, a
glob for the tenant directories could be provided and used in a separate
history provider similar to `org.apache.spark.deploy.history.FsHistoryProvider`.
was:
Currently, spark history server runs with one base directory immediate
subdirectories of which correspond to event logs for each application.
There are usecases for eg., multi tenancy where for the purpose of logical
separation, the event logs could be stored in separate directories at a tenant
level. To achieve this, instead of providing the path of the base directory, a
glob for the tenant directories could be provided and used in a separate
history provider.
> Glob based history provider
> ---------------------------
>
> Key: SPARK-52327
> URL: https://issues.apache.org/jira/browse/SPARK-52327
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Gaurav Waghmare
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, spark history server runs with one base directory immediate
> subdirectories of which correspond to event logs for each application.
> There are usecases for eg., multi tenancy where for the purpose of logical
> separation, the event logs could be stored in separate directories at a
> tenant level. To achieve this, instead of providing the path of the base
> directory, a glob for the tenant directories could be provided and used in a
> separate history provider similar to
> `org.apache.spark.deploy.history.FsHistoryProvider`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
