[
https://issues.apache.org/jira/browse/SPARK-21697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123208#comment-16123208
]
Steve Loughran commented on SPARK-21697:
----------------------------------------
What would a test to replicate look like?
# Create MiniDFS cluster (better: find a test with one already spun up @ class
level)
# copy JAR to it (issue: can't rely on the local test suite being in a JAR due
to SBT & IDEs doing things differently/faster than maven)
# add JAR to CP/create a CP which *only* has the JAR in
# Load something from the CP which triggers download.
If you can assume that some common library (junit.jar?) is always in a JAR then
the JAR could be uploaded by: locating its URL, translate to local path & then
use FileSystm.copyFromLocalFile() to upload.
Or: create/find a UDF JAR, copy to MiniDFSCluster, start spark SQL with the
HDFS URL. This would verify the desired codepath & be best to make sure its
gone away
> NPE & ExceptionInInitializerError trying to load UTF from HDFS
> --------------------------------------------------------------
>
> Key: SPARK-21697
> URL: https://issues.apache.org/jira/browse/SPARK-21697
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.1
> Environment: Spark Client mode, Hadoop 2.6.0
> Reporter: Steve Loughran
> Priority: Minor
>
> Reported on [the
> PR|https://github.com/apache/spark/pull/17342#issuecomment-321438157] for
> SPARK-12868: trying to load a UDF of HDFS is triggering an
> {{ExceptionInInitializerError}}, caused by an NPE which should only happen if
> the commons-logging {{LOG}} log is null.
> Hypothesis: the commons logging scan for {{commons-logging.properties}} is
> happening in the classpath with the HDFS JAR; this is triggering a D/L of the
> JAR, which needs to force in commons-logging, and, as that's not inited yet,
> NPEs
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
