Anoopam Sharma created SPARK-60925:
--------------------------------------
Summary: Executors tab is empty in the Spark UI when spark-core
runs on a non-filesystem classpath (OSGi/shaded), because REST API package
scanning fails
Key: SPARK-60925
URL: https://issues.apache.org/jira/browse/SPARK-60925
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.1.1, 4.2.0, 5.0.0
Reporter: Anoopam Sharma
Description:
h3. Problem
The Spark status REST API registers its JAX-RS resources via Jersey *package
scanning*:
ApiRootResource.scala (context "/api"):
holder.setInitParameter(ServerProperties.PROVIDER_PACKAGES,
"org.apache.spark.status.api.v1")
PrometheusResource.scala (context "/metrics"):
holder.setInitParameter(ServerProperties.PROVIDER_PACKAGES,
"org.apache.spark.status.api.v1")
Jersey's PackageNamesScanner enumerates the package by walking the classpath as
file/jar URLs. When spark-core is loaded from a classpath that is not
filesystem-enumerable, the scan finds no resources and the REST endpoints
(e.g. /api/v1/applications, the Executors/Environment UI tabs backed by them)
return 404 / fail to initialize. Observed failure:
org.glassfish.jersey.server.internal.scanning.ResourceFinderException:
The URI scheme bundleresource of the URI ... is not supported.
Package scanning deployment is not supported for such URIs.
This reproduces under OSGi (resources exposed as bundleresource:// URLs) and can
also affect shaded/relocated or JPMS-modular deployments.
h3. User-visible symptom
The Executors tab in the Spark Web UI does not populate. The tab is rendered by
executorspage.js, which fetches /api/v1/applications/<appId>/allexecutors.
Because
the /api/v1 Jersey servlet fails to register its resources (package scanning
throws
ResourceFinderException on bundleresource:// URLs), that REST call returns no
data
and the tab appears empty. The Environment tab and other /api/v1-backed views
are
affected the same way.
h3. Root cause
Package scanning depends on an enumerable filesystem classpath, which is an
environment assumption that does not hold for all valid deployments.
h3. Fix
Register the REST resources explicitly by class name
(ServerProperties.PROVIDER_CLASSNAMES) instead of scanning the package:
/api -> ApiRootResource, JacksonMessageWriter
/metrics -> PrometheusResource, JacksonMessageWriter
These are the only root resource classes and the only @Provider in the
org.apache.spark.status.api.v1 package; every other resource
(OneApplicationResource, StagesResource, ApplicationListResource, ...) is
reached
through JAX-RS sub-resource locators (methods returning Class[...]), which do
not
require registration/scanning. Explicit registration is environment-independent
and is the approach Jersey recommends over package scanning.
h3. Tests
All HistoryServerSuite tests (run via LevelDBBackendHistoryServerSuite /
RocksDBBackendHistoryServerSuite) pass; these exercise the live /api/v1 REST
endpoints. No functional change for the normal (filesystem classpath)
deployment.
Affected files:
- core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
- core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
