[
https://issues.apache.org/jira/browse/SPARK-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140043#comment-14140043
]
Patrick Wendell commented on SPARK-3270:
----------------------------------------
Hey There,
For the particular use case here - long-lived components that want to co-exist
with executors and share the executor lifecycle - applications can enact this
using static objects. I.e. you lazily initialize some service when accessed
from within tasks. Then when the executor terminate the object goes away.
Are there specific things preventing that approach in this case?
The reason I ask is that adding a generic service discovery mechanism to Spark,
while useful, is a fairly large new interface, and I'd guess its something
where we'd want to look at a few specific different applications to find the
common set of needs to make sure we are adding the right API's.
> Spark API for Application Extensions
> ------------------------------------
>
> Key: SPARK-3270
> URL: https://issues.apache.org/jira/browse/SPARK-3270
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Reporter: Michal Malohlava
>
> Any application should be able to enrich spark infrastructure by services
> which are not available by default.
> Hence, to support such application extensions (aka "extesions"/"plugins")
> Spark platform should provide:
> - an API to register an extension
> - an API to register a "service" (meaning provided functionality)
> - well-defined points in Spark infrastructure which can be enriched/hooked
> by extension
> - a way of deploying extension (for example, simply putting the extension
> on classpath and using Java service interface)
> - a way to access extension from application
> Overall proposal is available here:
> https://docs.google.com/document/d/1dHF9zi7GzFbYnbV2PwaOQ2eLPoTeiN9IogUe4PAOtrQ/edit?usp=sharing
> Note: In this context, I do not mean reinventing OSGi (or another plugin
> platform) but it can serve as a good starting point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
